Solved Bandwidth Limiter QoS

[DJones]

So I was wondering if their is a way to set the queuing discipline for Bandwidth Limiter to FQ_Codel?

The problem I’m having is I have a lot of devices on a slow internet connection and my bufferbloat score rates a F. I want the ability to hard limit devices that eat bandwidth like a smart tv, or cell phones; things like playing Amazon or Netflix which will consume everything are a problem.

-The problem with Cake is well it makes the bufferbloat score A; I can’t hard limit devices.

-FlexQoS doesn’t fix the bufferbloat issue, and some devices using things like Amazon or Netflix doesn’t seem to respect the download limit.

I’m also running a small server so I want to prioritize that above everything, unrestricted it and let it consume what it wants. I figure if the router is running a better queuing discipline then the poor QoS setup asus uses, then I might have a better time with managing bufferbloat as well. When I bandwidth limit my score ends up as a C, so I know if I’m not peaking at my max speeds bufferbloat isn’t as bad.

Don’t have a ton of experience with this stuff, but I set up my server with bbr and cake to improve latency on that endpoint, but it doesn’t help upstream in the router which handles all the other devices.

I’ve done some poking around in ssh and noticed the router only supports the congestion avoidance Cubic or Reno, but have seen FlexQoS use queuing discipline FQ_Codel so might have options for that?

Using a GT-AX11000 latest firmware.

Any advice would be appreciated

 

[dave14305]

So I was wondering if their is a way to set the queuing discipline for Bandwidth Limiter to FQ_Codel?

fq_codel is already used on Merlin for Bandwidth Limiter mode.

 

[DJones]

fq_codel is already used on Merlin for Bandwidth Limiter mode.

Excellent! That’s good to know. If you perhaps know is their anything additional I might try that could improve the bufferbloat rating further? Not sure how optimized tcp is in merlin or if their is anything that you’d recommend needs changing. Since cake says A I’m assuming it can be improved or cake is just a better queuing discipline, unsure if cake is compatible with bandwidth limiter.

My connection Speedtest from webbrowser Speedtest.net is 10.89 down, 0.90 up (routers speed test says it’s a bit higher) on a adsl2+ connection that my ISP says the plan is 11 down 1 up, but rural copper lines suck so. I set wan packet overhead to ATM 40 as it’s bridged to a modem.

 

[dave14305]

With your slow upload bandwidth, your fq_codel target value needs to be higher than the default 5ms. At 900 kbps, target should be around 15.5ms and interval around 110ms on upload. Otherwise, fq_codel will drop too many upload packets. CAKE calculates this automatically, and so will FlexQoS. But those have no device level bandwidth limiters.

See what Cake sets as target and interval when you configure it, and if my math is in the ballpark.

What values are you using in the QoS bandwidth Mbs? You need to be slightly below the actual speedtest measurements, and it’s really hard to split up 900 kbps and be usable, let alone shave off an extra 5-10% as usually recommended.

 

[DJones]

With your slow upload bandwidth, your fq_codel target value needs to be higher than the default 5ms. At 900 kbps, target should be around 15.5ms and interval around 110ms on upload. Otherwise, fq_codel will drop too many upload packets. CAKE calculates this automatically, and so will FlexQoS. But those have no device level bandwidth limiters.

See what Cake sets as target and interval when you configure it, and if my math is in the ballpark.

What values are you using in the QoS bandwidth Mbs? You need to be slightly below the actual speedtest measurements, and it’s really hard to split up 900 kbps and be usable, let alone shave off an extra 5-10% as usually recommended.

So far I’ve limited my devices mostly to around 3Mbps down and 0.3Mbps up, couple are 5Mbps, but are rarely used. My server on the other hand is unregulated and allowed full speed. I’ll attempt to use cake to verify when I get home from work.

edit: Ran a Speedtest remotely from cli from my server which is unrestricted using bandwidth limiter. Will check cake later.

(Speedtest.net)

Latency: 8.65 ms (1.13 ms jitter)

Download: 10.31 Mbps (data used: 6.3 MB)

Upload: 0.86 Mbps (data used: 394.6 kB)
Packet Loss: 0.0%


Bufferbloat test (wifi on device bandwidth limited)

Unloaded
25 ms

Download Active​

+49 ms

Upload Active​

+645 ms

grade f

Bufferbloat test (server wired unrestricted)

Unloaded
19 ms

Download Active​

+49 ms

Upload Active​

+783 ms

grade f

 

[dave14305]

When you can, run:

tc -s qdisc show dev eth0

for both Cake and fq_codel.

 

[DJones]

When you can, run:

tc -s qdisc show dev eth0

for both Cake and fq_codel.

Should I kick all devices off the network except testing device when I test?

 

[dave14305]

Should I kick all devices off the network except testing device when I test?

Not necessary. Just want to see the target and drop counters after your tests.

 

[DJones]

Not necessary. Just want to see the target and drop counters after your tests.

(FQ_Codel)

qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 592717783 bytes 4132131 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

(Cake)

qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 25550924 bytes 201570 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0



(SpeedTest.net-Cake)

14.96ms ping
8.86 down
0.65 up

(Bufferbloat Test - Cake)

LATENCY
Unloaded
51 ms

Download Active
+9 ms

Upload Active
+10 ms

Grade A

(Router manually set QoS Speed)

Upload 0.90
Download 10.82

 

[dave14305]

Ok, your WAN interface must be different on that model.

tc -s qdisc ls

Let’s have it all.

 

[DJones]

Ok, your WAN interface must be different on that model.

tc -s qdisc ls

Let’s have it all.

here's the log for both fq_codel and cake

(FQ_Codel)qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[dave14305]

here's the log for both fq_codel and cake

(FQ_Codel)qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

OK I see CAKE actually set 19.7ms target versus fq_codel default 5.0ms. So I would expect many more drops using fq_codel for upload traffic. But the fq_codel stats don't seem to show many drops, so I'm guessing you didn't run a speedtest before the fq_codel stats were captured.

Unfortunately, the only way to tune the fq_codel target using bandwidth limiter is to modify the /tmp/qos script generated by the firmware (by using a /jffs/scripts/qos-start script).

If you want to post the contents of /tmp/qos when bandwidth limiter is setup, I can try to come up with the qos-start script required to adjust your target and interval.

 

[DJones]

OK I see CAKE actually set 19.7ms target versus fq_codel default 5.0ms. So I would expect many more drops using fq_codel for upload traffic. But the fq_codel stats don't seem to show many drops, so I'm guessing you didn't run a speedtest before the fq_codel stats were captured.

Unfortunately, the only way to tune the fq_codel target using bandwidth limiter is to modify the /tmp/qos script generated by the firmware (by using a /jffs/scripts/qos-start script).

If you want to post the contents of /tmp/qos when bandwidth limiter is setup, I can try to come up with the qos-start script required to adjust your target and interval.

oh okay. Since I have no idea what I'd need to write into the script if you'd like to give it a shot it would be highly appreciated.


Here's the FQ_Codel logs after running a speedtest, wasn't immediately done after bandwidth limiter was turned back on so it might have some extra drops in the log, but might give you a idea.

(FQ_Codel)qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

#!/bin/sh
SCH="fq_codel quantum 300 limit 1000 noecn"
WAN=ppp0
tc qdisc del dev $WAN root 2>/dev/null
tc qdisc del dev $WAN ingress 2>/dev/null
tc qdisc del dev br0 root 2>/dev/null
tc qdisc del dev br0 ingress 2>/dev/null

TQAU="tc qdisc add dev $WAN"
TCAU="tc class add dev $WAN"
TFAU="tc filter add dev $WAN"
TQA="tc qdisc add dev br0"
TCA="tc class add dev br0"
TFA="tc filter add dev br0"

start()
{
$TQA root handle 1: htb
$TCA parent 1: classid 1:1 htb rate 10240000kbit

 

[dave14305]

The script is truncated, missing the good stuff.

 

[DJones]

The script is truncated, missing the good stuff.

Sorry nano truncated it. Fixed

#!/bin/shSCH="fq_codel quantum 300 limit 1000 noecn"WAN=ppp0tc qdisc del d - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[dave14305]

Your drops on ppp0 (WAN) weren’t as bad as I expected, and I see it your actual allocated bandwidth per device is generally 307.2kbit. At that bandwidth, a full packet might take up to 45ms to send. Very extreme.

As a quick experiment, you can try these commands:

/tmp/qos stop
sed -Ei '/TQAU parent 2:[3-9][0-9] .*SCH/ s/$/ target 45ms interval 140ms/g' /tmp/qos
/tmp/qos start

Run another test and see if it changes for the better. To undo the changes above, just run:

service restart_qos

 

[DJones]

Your drops on ppp0 (WAN) weren’t as bad as I expected, and I see it your actual allocated bandwidth per device is generally 307.2kbit. At that bandwidth, a full packet might take up to 45ms to send. Very extreme.

As a quick experiment, you can try these commands:

/tmp/qos stop
sed -Ei '/TQAU parent 2:[3-9][0-9] .*SCH/ s/$/ target 45ms interval 140ms/g' /tmp/qos
/tmp/qos start

Run another test and see if it changes for the better. To undo the changes above, just run:

service restart_qos

Didn’t seem to improve the results of the bufferbloat grade. Still Grade F unfortunately according to https://www.waveform.com/tools/bufferbloat

Edit: Nvm I’m a idiot and didn’t read. Used service restart_QoS which purged the tmp file. re ran it still seems Grade F though..

 

[dave14305]

You might also consider trying FlexQoS again. But give your server IP a rule to treat all its traffic as Gaming (or another category you prioritize at the top, but won’t have much other traffic) and then set maximum bandwidth limits on all the remaining lower priority categories (75-90%) if you want to leave some slack for the server.

With so little bandwidth to work with, you should probably avoid restricting other devices from using it when no one else is using it either.

Without getting too detailed, is this server sending data out to the internet through your 1Mbps upload link (internet users downloading data from your server)? Or is it just a private server for your local network?

 

[DJones]

You might also consider trying FlexQoS again. But give your server IP a rule to treat all its traffic as Gaming (or another category you prioritize at the top, but won’t have much other traffic) and then set maximum bandwidth limits on all the remaining lower priority categories (75-90%) if you want to leave some slack for the server.

With so little bandwidth to work with, you should probably avoid restricting other devices from using it when no one else is using it either.

Without getting too detailed, is this server sending data out to the internet through your 1Mbps upload link (internet users downloading data from your server)? Or is it just a private server for your local network?

Well it does act as a NAS as well locally on the network, but it’s using the upload mostly externally. It’s running a game server usually has 2-6 people on it in the afternoon.

 

[DJones]

Played around with the target changed it to 75ms and that seems to fix my grade to a A. So perhapd was just a little low, since this is only a tmp file do I just put this as script under /jffs/scripts/qos-start so it does this on boot?

nano /jffs/scripts/qos-start

#!/bin/sh

sed -Ei '/TQAU parent 2:[3-9][0-9] .*SCH/ s/$/ target 95ms interval 190ms/g' /tmp/qos

Save file

chmod a+rx /jffs/scripts/qos-start

Reboot