Solved Please add BBR support for 5.04axhnd devices

[ttc0419]

5.04 devices uses 4.19 kernel which has support for BBR. Please consider add this kernel module. It can greatly improve bandwidth for server applications. The firmware still only supports reno and cubic:

admin@GT-AX6000-91F8:/tmp/home/root# cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic

 

[Crimliar]

Okay, so I'm a noob at this: If BBR/BBR2 is not on chip, then it would need to be added such that it's interventionist (disabling HW acceleration). Current interventionist QoS (such a Cake) only really works below a certain data rate threshold due to the processor cost - if the data rate is too high, the CPU bottlenecks throughput!
So who would benefit from BBR? Is there a benefit from using BBR rather than, or alongside Cake for example on slower connections?
*Sorry just trying to figure out if we're being attracted to shiny things that hurt us!

 

[ttc0419]

Okay, so I'm a noob at this: If BBR/BBR2 is not on chip, then it would need to be added such that it's interventionist (disabling HW acceleration). Current interventionist QoS (such a Cake) only really works below a certain data rate threshold due to the processor cost - if the data rate is too high, the CPU bottlenecks throughput!
So who would benefit from BBR? Is there a benefit from using BBR rather than, or alongside Cake for example on slower connections?
*Sorry just trying to figure out if we're being attracted to shiny things that hurt us!

Sorry, but I didn't quite follow you.

Are you saying BBR requires adjustments to the kernel code and disable the HW acceleration? Why is that? If so, then the current default TCP congestion control "cubic" is running "on chip"? Even openwrt has a package for the module. If it does not require any source code changes, I don't mind to disable HW acceleration since the upload speed limited by ISP will never saturate the processing power anyway.

And like I mentioned in the description, I'm running an application server on my router with a lot of TCP connections. Home internet is very congested during night, I can see a huge increase in bandwidth when using BBR on my raspberry pi.

 

[sfx2000]

And like I mentioned in the description, I'm running an application server on my router with a lot of TCP connections. Home internet is very congested during night, I can see a huge increase in bandwidth when using BBR on my raspberry pi.

Run your app server on the PI - more resources, and better availability of SW in any case (AsusWRT/Entware is great, but debian has a lot more to offer).

Adding BBR/BBR2 - it could be done, but it could break a lot of other things, including QoS schemes as BBR/BBR2 is dependent on the fq shaper to do it's thing.

Anything that plays around with the internal network stack is a bit of a risk, as performance has been defined around the stock kernel and network configuration.

I wouldn't recommend to add it to the distributed builds for AsusWRT (and forks thereof) - but one is always welcome to sync up on git, fork it, and build their own with whatever options one sees fit.

 

[ttc0419]

How about

Adding BBR/BBR2 - it could be done, but it could break a lot of other things, including QoS schemes as BBR/BBR2 is dependent on the fq shaper to do it's thing.

If I understand correctly, merlin is using CAKE for QoS. If so, How about just disabling QoS settings if you are using BBR? Changing qdisc on NIC should be trivial.

 

[sfx2000]

How about

If I understand correctly, merlin is using CAKE for QoS. If so, How about just disabling QoS settings if you are using BBR? Changing qdisc on NIC should be trivial.

BBR isn't going to be helpful - you mention you have a lot of apps running on the router, so it might help there, but the rest of your routed traffic may suffer as a result.

Rasbian (PiOS) has BBR built into their kernel, so as I mentioned earlier, might want to run your apps there for a while to even see if there is a benefit to running bbr over cubic.

There are reasons not to use bbr, see below

When to use and not use BBR | APNIC Blog

Guest Post: The difference between the bottleneck buffer size and bandwidth-delay product typically dictates when BBR performs well, study finds.

blog.apnic.net

 

[ttc0419]

BBR isn't going to be helpful - you mention you have a lot of apps running on the router, so it might help there, but the rest of your routed traffic may suffer as a result.

Thanks for the link. But what I mentioned was I have an single application with many TCP connections (nobody will run many servers on a router) and packet loss is fairly common for home internet upload link, especially during night. The cubic is overreacting and causes the upload speed to grind to halt. And I'm aware of the downsides of BBR, but in my use case, it helped a lot, and I did not notice any difference for router client traffic.

 

[sfx2000]

I have an single application with many TCP connections

What is the application, and how many TCP connections does it consume?

CUBIC and BBR are similar in performance, and in any case, running your app on the Pi and running BBR/sch_fq on it, you'll be in better shape.

BBR is not recommended for routers and switches, it has main benefit to end-points.

BTW - BBR requires sch_fq, and that is global... which can break flow caching and QoS as I previously mentioned.

As I mentioned earlier, you can always sync up to the git, and build your own - kconfig should have the kmod there, and as I mentioned, you should add the scheduler as well.

 

[sfx2000]

packet loss is fairly common for home internet upload link, especially during night. The cubic is overreacting and causes the upload speed to grind to halt. And I'm aware of the downsides of BBR, but in my use case, it helped a lot, and I did not notice any difference for router client traffic.

How do you know you're seeing high packet loss?

And that CUBIC is breaking things...

BBR starts to degrade quite heavily once you hit the limit, and it starts to fall apart, and that's where CUBIC and even RENO will out perform it.

It does sound like you're better off with SQM/AQM, and tuning ECN parameters, and this you can do on the WAN interface.

 

[ttc0419]

BTW - BBR requires sch_fq, and that is global... which can break flow caching and QoS as I previously mentioned.

Just for the record, for linux 4.13+, BBR can use TCP internal pacing, which does not require fq. See https://git.kernel.org/pub/scm/linu.../?id=218af599fa635b107cfe10acf3249c4dfe5e4123 for more info.

And I think you misunderstood my meaning. I'm not trying to make BBR as the default TCP congestion control algorithm, but adding it as an option. It should be as easy as turning on the flag in kernel config. It will not break anything by default.

 

[RMerlin]

It should be as easy as turning on the flag in kernel config.

Not necessarily, no.

Some Kernel settings will make changes to internal structures, which will then break compatibility. For instance, enabling IMQ when it wasn`t available by default was making changes to the internal skbuf struct, which would cause a crash at boot time as all the pre-compiled networking components were compiled specifically for the original skbuf structure.

That`s why I try to avoid as much as possible making changes to kernel settings, unless I were absolutely sure they wouldn`t cause issues.

 

[ttc0419]

Not necessarily, no.

Some Kernel settings will make changes to internal structures, which will then break compatibility. For instance, enabling IMQ when it wasn`t available by default was making changes to the internal skbuf struct, which would cause a crash at boot time as all the pre-compiled networking components were compiled specifically for the original skbuf structure.

That`s why I try to avoid as much as possible making changes to kernel settings, unless I were absolutely sure they wouldn`t cause issues.

Thanks for the reply. So, will the firmware crash with bbr enabled? I'm trying compiling from source, if it'll definitely break things, I'll just give up.

 

[RMerlin]

o, will the firmware crash with bbr enabled?

I have no idea. Someone would need to review what happens specifically when that setting gets enabled. Otherwise, the only way to know would be to try. If it crashes at boot, or if the system log gets "bad magic value" errors when loading precompiled modules, then it`s not compatible with these modules.

 

[ttc0419]

For those who are interested in BBR, I just built the module, uploaded to /jffs and load it using insmod on my GT-AX6000. It works without any issues. Here is the kernel module I compiled. It should work with any BCM4912 routers.

 

[sfx2000]

I have no idea. Someone would need to review what happens specifically when that setting gets enabled. Otherwise, the only way to know would be to try. If it crashes at boot, or if the system log gets "bad magic value" errors when loading precompiled modules, then it`s not compatible with these modules.

I did a private build on OpenWRT and did some testing against one of the ath10k targets (IPQ4019) - enabling the kmod-tcp-bbr along with ensuring that sch_fq is enabled - no real impact one way or the other, but I'm not enabling any hardware flow offloading, just support for cake sqm.

Same benefit was seen by just enabling BBR/sch_fq on a linux endpoint (desktop ubuntu 22.04LTS) - BBR only needs one end to use it.

I suggest not adding BBR to AsusWRT, as it would then need to be supported over time...

 

[ttc0419]

For those who wants detailed instructions, you can refer to my GitHub repo.

 

[sfx2000]

For those who wants detailed instructions, you can refer to my GitHub repo.

You're only doing half of the enablement - you also have to add the scheduler...

net.ipv4.tcp_congestion_control=bbr
net.core.default_qdisc=fq

BTW - even though it says IPV4, this also applies to IPV6

 

[ttc0419]

You're only doing half of the enablement - you also have to add the scheduler...

Please refer to my previous comments, for 4.13+, bbr does not require fq as the qdisc.

Just for the record, for linux 4.13+, BBR can use TCP internal pacing, which does not require fq. See https://git.kernel.org/pub/scm/linu.../?id=218af599fa635b107cfe10acf3249c4dfe5e4123 for more info.

And I think you misunderstood my meaning. I'm not trying to make BBR as the default TCP congestion control algorithm, but adding it as an option. It should be as easy as turning on the flag in kernel config. It will not break anything by default.

 

[sfx2000]

Please refer to my previous comments, for 4.13+, bbr does not require fq as the disc.

It can run with internal pacing as a fallback, but that fallback is not as efficient as fq

IIRC the fq scheduler is already present in the asuswrt/broadcom BSP kernel, so might as well use it.

 

[ttc0419]

IIRC the fq scheduler is already present in the asuswrt/broadcom BSP kernel, so might as well use it.

No sure about what do you mean by asuswrt/broadcom BSP kernel. But the 388.1 kernel does not have fq built-in:

admin@GT-AX6000-91F8:/tmp/home/root# echo fq > /proc/sys/net/core/default_qdisc
echo: write error: No such file or directory