Custom firmware build for R7800

[pege63]

Not at all ;-) I understand your worry: some unknown guy, newbie in this forum, puts his firmware and declares that it is better then stock FW. I'd wait when others try to flash to avoid brick of own router ;-)

P.S.
Well, to say true, I expected replies whille only from "mine" users (myopenrouter forum), who suggested me to use snbforums because myopenrouter is not so friendly to me and them.

Voxel.

Well its faster than stock FW for sure, flowing a little faster and better

 

[bodean]

May have to check this out. Running stock .20 on my R7800 now. Ill have to see how difficult the install is over stock firmware. I assume Hardware Acceleration works fine with your builds, and no issues with 2.4/5.0Ghz wifi?

 

[pege63]

Easy as stock

 

[sfx2000]

Nice project...

@Voxel - have you considered putting your project up on GitHub?

 

[bodean]

Easy as stock

Thanks, the key benefit i see is using the processor to its fullest (as I do not run separate apps on my router, or use the USB functions. My NAS is connected via ethernet and uses SMB protocol)

 

[sfx2000]

Compiled with optimization for Cortex A15 CPU (IPQ8065 is Krait, so its architecture is practically the same as Cortex A15, the same set of extended instructions vs generic ARMv7-a)

@Voxel - Qualcomm's Krait is more like an enhanced Cortex-A9 - one can build against Cortex-A15 as a build-target, but also consider that A15 has things that Krait doesn't have in HW...

Depending on your toolchain - Linaro, if I recall, has Krait specifically as a target for -mtune.. but if you're using GCC, then consider...

-mtune=cortex-a9 or -mcpu=cortex-a9

and then add...

-mfpu=neon-vfpv4 (which brings in vfp4 and neon support obviously)​

(If -mtune covers the target name, then one doesn't need to do the same with -mcpu/-mfpu, again, depends on your toolchain)

Anyways - this will give a bit better performance than building against Cortex-A15

Been there - done that with Android a few years back when Krait was the hot thing....

 

[sfx2000]

fq_codel should be a part of Netgear’s “intellectual” QoS. I do not like any QoS and do not use it, and did not investigate it in deep, but there are several references in GPL of stock firmware (and thus in my modification) to fq_codel, e.g. top config:

It's not Netgear there - it's Qualcomm, and that QoS package works really well - I don't advise to use FQ_CODEL on top of that, as they work differently...

It's one or the other - FQ_CODEL does a nice job in a generic method, but Qualcomm's package is more specific...

 

[Voxel]

May have to check this out. Running stock .20 on my R7800 now. Ill have to see how difficult the install is over stock firmware. I assume Hardware Acceleration works fine with your builds, and no issues with 2.4/5.0Ghz wifi?

My aim is to improve/extend/unlock the functionality of stock firmware keeping its advantages, including hardware acceleration. So I keep all Quallcom's (and Quantenna's for R7500) drivers from the stock firmware. No issues with 2.4/5.0GHz, otherwise I'd not publish my results.

Voxel.

 

[Voxel]

sfx2000:

Thanks for you hints and suggestions.

Regarding GitHub. I have several reasons to do not do this right now. I can explain "why" in private conversation if you are interested.

@Voxel - Qualcomm's Krait is more like an enhanced Cortex-A9 - one can build against Cortex-A15 as a build-target, but also consider that A15 has things that Krait doesn't have in HW...

Depending on your toolchain - Linaro, if I recall, has Krait specifically as a target for -mtune.. but if you're using GCC, then consider...

-mtune=cortex-a9 or -mcpu=cortex-a9

and then add...

-mfpu=neon-vfpv4 (which brings in vfp4 and neon support obviously)​

(If -mtune covers the target name, then one doesn't need to do the same with -mcpu/-mfpu, again, depends on your toolchain)

Anyways - this will give a bit better performance than building against Cortex-A15

Been there - done that with Android a few years back when Krait was the hot thing....

I use the following options for compilation:

-mcpu=cortex-a15 -mfpu=neon-vfpv4 -mtune=cortex-a15

and I avoid “-march=armv7-a” (used by Netgear) which conflicts to “-mcpu=cortex-a15”. Thanks for your suggestion regarding “cortex-a9”, but I have doubts, I did some more or less in deep investigations (starting with R7500) and own benchmarks . IMO anyway Krait is more close to Cortex-A15, but not to Cortex-A9. Here is a table for example:

https://en.wikipedia.org/wiki/Comparison_of_ARMv7-A_cores

Cortex-A15 is more advanced than Krait, but Krait is more advanced than Cortex-A9. At least I know that LEDE project people agreed with my arguments and they use the same my options for R7800.

Well, not so difficult to compare, e.g. performing full OpenSSL tests compiling my FW with option you suggested. I’ll try to do it in background and publish results.

Toolchain: I am not so free in my selection, because I have to use the compiler compatible with already precompiled by Netgear binaries (e.g. kernel objects, and some executables and shared objects). Also, kernel in Netgear GPL is 3.4.103 plus some modifications taken by Netgear (IMO) from here:


https://source.codeaurora.org/quic/qsdk/oss/kernel/linux-msm/log/?h=banana


and it means that latest compilers (>=5.x) cannot be used. Just significant compilation problems when I tried to compile the kernel by these compilers. And I cannot change the sources of kernel too much, because of compatibility with already precompiled kernel objects etc. Still playing with this (toolchain), but I suppose that my recent selection of 4.8.5 with all OpenWRT and Debian patches (up to 2016) is optimal. I used Linaro 4.8-2015 some time in my releases, but I do not see improvements in speed (vs 4.8.5).

Regards,
Voxel.

 

[Voxel]

sfx2000:

Regarding Krait and Cortex-15. I just found the patch (googled):

https://review.cyanogenmod.org/#/c/58445/

ARM: Use -mcpu=cortex-a15 when targeting MSM Krait CPUs Enable compiler optimizations specific to the Cortex-A15 processor when targeting MSM Krait CPUs. This is necessary take advantage of the UDIV/SDIV instructions supported by these processors. To accomplish this, we need to remove the -march=armv7-a ISA restriction from the compiler options because 'cortex-a15' is a superset of 'armv7-a'. Change-Id: I6215aecc11fb4f77c971de7b84f68649ef234357 Signed-off-by: Stepan Moskovchenko <stepanm@codeaurora.org>​

I.e. from the same codeaurora.

Voxel.

 

[sfx2000]

I use the following options for compilation:

-mcpu=cortex-a15 -mfpu=neon-vfpv4 -mtune=cortex-a15

and I avoid “-march=armv7-a” (used by Netgear) which conflicts to “-mcpu=cortex-a15”. Thanks for your suggestion regarding “cortex-a9”, but I have doubts, I did some more or less in deep investigations (starting with R7500) and own benchmarks . IMO anyway Krait is more close to Cortex-A15, but not to Cortex-A9. Here is a table for example:

https://en.wikipedia.org/wiki/Comparison_of_ARMv7-A_cores

Cortex-A15 is more advanced than Krait, but Krait is more advanced than Cortex-A9. At least I know that LEDE project people agreed with my arguments and they use the same my options for R7800.

Well, not so difficult to compare, e.g. performing full OpenSSL tests compiling my FW with option you suggested. I’ll try to do it in background and publish results.

Netgear is being conservative, as they need to support many different SoC's, and armv7-a will run decently enough on any Cortex-A profile..

Try it both ways - since A15 is working... might as well leave it

(little tip - take a look at what Google has done against the storm target, it's Krait 300 (which is pretty similar to what you have...)

 

[sfx2000]

It's not Netgear there - it's Qualcomm, and that QoS package works really well - I don't advise to use FQ_CODEL on top of that, as they work differently...

QCA's streamboost QoS works hand in hand with the HW flow acceleration - it's fast, it's free, and it's done

And it works really well..

Fun read below

https://www.netdev01.org/docs/IPQ806x-Hardware-acceleration_v2.pdf

 

[sfx2000]

You got me interested to see if some thing changed recently with Qualcomm, Krait, and GCC...

The kernel is still treating Krait (and Scorpion for that matter) as generic ARMV7 (similar to Marvell's P4J and regular Cortex-A)

So... with some google fu, I found Paul Chambers' drop over on GitHub...

https://github.com/paul-chambers/netgear-r7800/blob/master/.config

CONFIG_TARGET_BOARD="ipq806x"
CONFIG_TARGET_ARCH_PACKAGES="ipq806x"
CONFIG_DEFAULT_TARGET_OPTIMIZATION="-Os -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp"

So he's going with mfpu=vfp3-d16, and additionally the mfloat-abi=softfp -- My best guess at the moment is to avoid breaking binary packages included with the build...

Which goes back to my previous experience with Snapdragon on Android with JellyBean and KitKat...

CONFIG_TARGET_BOARD="ipq806x"
CONFIG_TARGET_ARCH_PACKAGES="ipq806x"
CONFIG_DEFAULT_TARGET_OPTIMIZATION="-Os -pipe -march=armv7-a -mtune=cortex-a9 -mfpu=neon-vfpv4"
CONFIG_CPU_TYPE="cortex-a9+neon-vfpv4"

Which should get one the best bang for the buck, but I'd be a bit wary of any binaries included with the build from 3rd parties...

And just to double check - I reached out to a contact - and he agreed - that should be optimal on a base build (e.g. OpenWRT, LEDE, or similar)

 

[sfx2000]

ARM: Use -mcpu=cortex-a15 when targeting MSM Krait CPUs Enable compiler optimizations specific to the Cortex-A15 processor when targeting MSM Krait CPUs. This is necessary take advantage of the UDIV/SDIV instructions supported by these processors. To accomplish this, we need to remove the -march=armv7-a ISA restriction from the compiler options because 'cortex-a15' is a superset of 'armv7-a'.

UDIV/SDIV is Thumb32, and that context switch is expensive time and resource-wise in the Cortex-A profile - which is why it's generally not used in ARM's IP blocks... these are used in the Cortex-M and Cortex-R profiles, mostly because those little guys are used in environments where Thumb is preferred (smaller RAM/Flash favors the higher code density of Thumb, but there's the tradeoff of performance)

Review here...

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0489e/CIHJICGF.html

Which goes back to my assumption above - Netgear has to support not only Qualcomm's Krait, but also the Broadcom Cortex-A9, which in the 4708/4709, doesn't have vfp or neon, and the kernel there explicitly disables thumb by default, for both performance and security purposes.

Most likely, the third party object BLOB's are built for generic ARMv7-A, and not tailored towards any specific chip...

So one way to look at Krait - it's a really fast A9 with VFP4 and Neon, and leave it at that..

 

[sfx2000]

Regarding GitHub. I have several reasons to do not do this right now. I can explain "why" in private conversation if you are interested.

PM on it's way...

Just as a heads up - GPL compliance is a touchy subject - so there had better be a very good reason...

 

[Voxel]

Netgear is being conservative, as they need to support many different SoC's, and armv7-a will run decently enough on any Cortex-A profile..

Try it both ways - since A15 is working... might as well leave it

(little tip - take a look at what Google has done against the storm target, it's Krait 300 (which is pretty similar to what you have...)

OK, thank you for your tip. It's nice to meet an expert. I am interested in any hints resulting in improvements of my FW builds. In general I already googled a lot of sites half year ago, I know expert's opinions regarding use of cortex-a9 option for Krait 300. From the other hand, if you would compare https://source.codeaurora.org/quic/qsdk/oss/kernel/linux-msm/log/?h=banana with GPL codes of Netgear you could conclude that some (not the latest, unforunately) version of this Qualcomm related development was used by Netgear. And patch mentioned above (Cortex-A15) for concrete MSM Krait (==IPQ806x) is accepted by man from this codeaurora.

Voxel.

 

[Voxel]

You got me interested to see if some thing changed recently with Qualcomm, Krait, and GCC...

The kernel is still treating Krait (and Scorpion for that matter) as generic ARMV7 (similar to Marvell's P4J and regular Cortex-A)

So... with some google fu, I found Paul Chambers' drop over on GitHub...

https://github.com/paul-chambers/netgear-r7800/blob/master/.config

CONFIG_TARGET_BOARD="ipq806x"
CONFIG_TARGET_ARCH_PACKAGES="ipq806x"
CONFIG_DEFAULT_TARGET_OPTIMIZATION="-Os -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp"

So he's going with mfpu=vfp3-d16, and additionally the mfloat-abi=softfp -- My best guess at the moment is to avoid breaking binary packages included with the build...

Which goes back to my previous experience with Snapdragon on Android with JellyBean and KitKat...

CONFIG_TARGET_BOARD="ipq806x"
CONFIG_TARGET_ARCH_PACKAGES="ipq806x"
CONFIG_DEFAULT_TARGET_OPTIMIZATION="-Os -pipe -march=armv7-a -mtune=cortex-a9 -mfpu=neon-vfpv4"
CONFIG_CPU_TYPE="cortex-a9+neon-vfpv4"

Which should get one the best bang for the buck, but I'd be a bit wary of any binaries included with the build from 3rd parties...

And just to double check - I reached out to a contact - and he agreed - that should be optimal on a base build (e.g. OpenWRT, LEDE, or similar)

Well. I know this Paul Chambers' drop. IMO it is just publishing Netgear's GPL in GitHub, but not own development. Part of config you quoted equals to config from Netgear's GPL. I.e. -Os -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp are option used by Netgear in stock FW. And it is why I wrote that IPQ806x is not used in full power in stock. So I used my own more optimal options. They do not break third-party binaries workability (e.g. Ookla speed tester or Qualcomm kernel objects) - checked in my firmware. The only thing that I cannot still use full hard float point support. Tried with R7500, there were several my previous "HF" versions (vs current "SF") , where these third party failed (precompiled with option "soft"). So now I have to use softfp, compatible with both: hard and soft. And softfp still uses FPU.

Voxel.

 

[RogerSC]

I've flashed this firmware now, and it is working well so far. It'll be up overnight, and I'll test with it more tomorrow. But so far, the performance is looking better than the stock firmware.

Great start, thanks!

 

[Voxel]

Which goes back to my assumption above - Netgear has to support not only Qualcomm's Krait, but also the Broadcom Cortex-A9, which in the 4708/4709, doesn't have vfp or neon, and the kernel there explicitly disables thumb by default, for both performance and security purposes.


Most likely, the third party object BLOB's are built for generic ARMv7-A, and not tailored towards any specific chip...

IMO Netgear is just too conservative, what is normal in principle for big multiplatform development company. To keep portability and compatibility. But (again IMO) when they did some initial mistakes starting with R7500 with new for them IPQ8064, and selected wrong options, for example the same “-mfpu=vfpv3-d16” (what is not compatible with Broadcom’s chips, i.e. specific for concrete CPU, we cannot talk about compatibility with common third-pary), they repeatedly continued this bug with R7500v2 and newest R7800. BTW, they corrected this for R9000 based on OpenWRT BB (started to use “-mfpu=neon-vfpv4” and “-mfloat-abi=hard”), so I think that these options are just a bug done on the stage of initial analyses and development, and continued due to conservativism from version to version. No additional efforts, no additional time for development is needed, just works somehow.

I.e. IMO it is just a bug, but not special action to keep compatibility with third-party and portability with Broadcom.


Some kind of proof: init scripts "powerctl" in R7500 stock FW and R7800 are identical and allows to set “turbo mode” frequency of CPU to 1400, what is correct for R7500/R7500v2, but not for R7800 (should be 1725, otherwise you get "turbo" slower than normal). The same conservative re-use of what works somehow already (bug).

Voxel.

 

[Voxel]

sfx2000:

Probably you are right:

sfx2000:
Well, not so difficult to compare, e.g. performing full OpenSSL tests compiling my FW with option you suggested. I’ll try to do it in background and publish results.

I tried not full FW, but only OpenSSL lib and util. Results are:

With “Cortex-A15” options:

compiler: arm-openwrt-linux-uclibcgnueabi-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/target-arm_uClibc-0.9.33.2_eabi/usr/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/target-arm_uClibc-0.9.33.2_eabi/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/toolchain-arm_gcc-4.8.5_uClibc-0.9.33.2_eabi/usr/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/toolchain-arm_gcc-4.8.5_uClibc-0.9.33.2_eabi/include -DNDEBUG -DTERMIOS -O3 -pipe -mcpu=cortex-a15 -mfpu=neon-vfpv4 -mtune=cortex-a15 -mfloat-abi=softfp -fhonour-copts -fpic -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              10128.63k    36545.39k    93454.51k   153856.34k   189498.22k
sha1             16474.96k    53733.59k   135326.65k   217987.34k   264574.29k
des cbc          26221.11k    27420.80k    27933.27k    28034.05k    28207.79k
des ede3         10142.87k    10337.56k    10384.90k    10445.48k    10455.72k
aes-128 cbc      72408.69k    82055.94k    86703.27k    87590.23k    87941.12k
aes-192 cbc      60969.95k    69711.85k    72732.07k    73784.32k    73160.02k
aes-256 cbc      55063.74k    61723.03k    64029.53k    64774.83k    64208.90k
sha256           23738.07k    57545.79k   105103.10k   133155.84k   144332.12k
sha512            8178.97k    32457.71k    48657.92k    67710.98k    76425.90k
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.007161s 0.000150s    139.6   6688.2
                  sign    verify    sign/s verify/s
dsa 2048 bits 0.001657s 0.001707s    603.6    585.7

With “Cortex-A9” options:

compiler: arm-openwrt-linux-uclibcgnueabi-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/target-arm_uClibc-0.9.33.2_eabi/usr/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/target-arm_uClibc-0.9.33.2_eabi/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/toolchain-arm_gcc-4.8.5_uClibc-0.9.33.2_eabi/usr/include -I/home/voxel/Netgear/R7800-V1.0.2.21SF_gpl_src/staging_dir/toolchain-arm_gcc-4.8.5_uClibc-0.9.33.2_eabi/include -DNDEBUG -DTERMIOS -O3 -pipe -mcpu=cortex-a9 -mfpu=neon-vfpv4 -mtune=cortex-a9 -mfloat-abi=softfp -fhonour-copts -fpic -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              10022.17k    41657.64k   102296.15k   164775.64k   198860.80k
sha1             13419.31k    43169.64k    97664.74k   210976.90k   265323.28k
des cbc          27856.06k    29065.88k    29455.36k    29668.26k    29955.41k
des ede3         10784.32k    10995.03k    11063.30k    11085.14k    11131.80k
aes-128 cbc      76241.89k    84184.15k    87586.90k    74547.20k    89189.00k
aes-192 cbc      64065.86k    71197.72k    73326.34k    74055.75k    74907.65k
aes-256 cbc      57941.90k    62713.00k    64122.88k    65218.83k    65892.17k
sha256           22869.00k    56828.15k   104521.22k   132966.91k   144001.71k
sha512            8096.11k    32368.51k    48547.67k    67888.46k    76414.98k
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.007003s 0.000149s    142.8   6697.3
                  sign    verify    sign/s verify/s
dsa 2048 bits 0.001627s 0.001647s    614.5    607.3

(We should compare mainly tests where no assembler acceleration is used e.g. to drop sha256 results from comparison, assembler is assembler, i.e. results will be the same)

OK, thanks. Will try to use “Cortex-A9” for next build.

Voxel.