What's new

Limited but effective QoS for Asus routers with CTF/hardware acceleration

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

ml70

Regular Contributor
It was a major disappointment that Asus' routers, for example my AC66, cannot have QoS on without dropping CTF, the hardware acceleration, and thereby cutting router performance drastically. However i've found a way to have limited yet meaningful QoS-like solution while keeping CTF on.

It's based on a 5 band prio qdisc with TOS classification done at iptables. Could be done with the default 3 band prio qdisc too, but 5 gives finer control in a home network environment.

Bands are 0-4, smaller number means higher priority. Large downloads, including Asus' own Download Master, are put to band 4, and in band 0 we find interactive traffic, dns and icmp.

Unfortunately there aren't much ways to measure its performance except looking at the tc stats, so the improvement is mostly anecdotal, but i think i do see a large difference. Before, with link fully saturated with downloads or both down & up, i couldn't even reload websites anymore. Now, i surf with ease.

This is a proof of concept and a skeleton to start improving on, not a whole solution which is more about iptables in general. It's expected you know your way around linux shell and iptables already. Tc you don't have to touch, except for statistics.

Dns and icmp are handled at both prerouting and postrouting because i'm unsure if CTF affects them and the stats don't quite match, if you know better please comment. Also, tc shows stats stuck at zero for band 0 for unknown reasons, something to do with CTF?

Note that it's your own responsibility to make sure traffic in upper bands never saturates the connection, and thus starves the lower bands entirely. You've been warned...

Edit: this is for a connection which uses PPPOE to connect to isp, if you're using direct cable, just replace ppp0 with eth0.

Experimented with a TBF qdisc on eth0 while this on ppp0, but never got the performance quite right. Recommend just a plain pfifo qdisc at eth0, like "tc qdisc replace dev eth0 pfifo ; ifconfig eth0 txqueuelen NN", where NN is 1/10 seconds worth of 1500 byte packets, for example for a 10 Mbit connection, NN=800.

Code:
# Queue 1000 packets per band
ifconfig ppp0 set txqueuelen 1000

tc qdisc replace dev ppp0 root prio bands 5 priomap 4 2 1 3 3 4 4 4 0 0 0 0 2 2 2 2

# comment out if you don't want your existing table flushed
iptables -t mangle -F

# tos 16 minimize delay interactive -> band 0
# dns
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A OUTPUT -p udp --dport 53 -j RETURN
# telnet ssh
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j RETURN
# icmp
iptables -t mangle -A PREROUTING -p icmp -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p icmp -j RETURN

# tos 4 maximize reliability -> band 1
# preferred mac addresses
iptables -t mangle -A PREROUTING -m mac --mac-source 00:11:22:33:44:55 -j TOS --set-tos 4
iptables -t mangle -A PREROUTING -m mac --mac-source 00:11:22:33:44:55 -j RETURN

# tos 2 minimize monetary cost -> band 2
# everyone in our internal network
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j TOS --set-tos 2

# tos 8 maximize throughput bulk -> band 3
# preferred bulk traffic
# all transfers from internal network larger than 1 MB are put into preferred bulk category
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8

# tos 0 normal -> band 4
# slowest bulk traffic here, will (hopefully) include default traffic from router itself; download master

### POSTROUTING
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A POSTROUTING -p icmp -j TOS --set-tos 16

# check results with
# tc -s -d qdisc
# tc -s -d class show dev ppp0
# iptables -t mangle -nvL --line-numbers
 
Last edited:
I have 3 large torrent downloads going on, absolutely saturating the connection, even 2 would be enough for that, but i'm purposely overloading it for testing.

Meanwhile kids are watching youtube without any stuttering or buffering (their tablets are on band 1), and rest of family is surfing speedily (on band2). It's almost better than Asus' full QoS :) The router is so snappy it makes me suspect the full QoS with 5 sfq and 5 htb and no CTF is too heavy to process for AC66's single core.

The *average* downlink utilization is 100% of my rated bandwidth :D
 
I did exactly the same and it doasn't work for me. I have Asus RT-AC68U with the newest Merlin.

Code:
admin@Media:/tmp/mnt/Media/jffs# tc -s -d qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 3060275226 bytes 6928442 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 1990639353 bytes 1604475 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 7884114865 bytes 5949546 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc prio 8002: dev ppp0 root refcnt 2 bands 5 priomap  4 2 1 3 3 4 4 4 0 0 0 0 2 2 2 2
 Sent 12408824 bytes 53624 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 


admin@Media:/tmp/mnt/Media/jffs# tc -s -d class show dev ppp0
class prio 8002:1 parent 8002: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 8002:2 parent 8002: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 8002:3 parent 8002: 
 Sent 64734 bytes 1396 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 8002:4 parent 8002: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 8002:5 parent 8002: 
 Sent 12382217 bytes 52952 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 


admin@Media:/tmp/mnt/Media/jffs# iptables -t mangle -nvL --line-numbers
Chain PREROUTING (policy ACCEPT 57745 packets, 67M bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1      101  6820 TOS        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:22:23 TOS set 0x10/0xff
2      101  6820 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:22:23
3      851 64492 TOS        icmp --  *      *       0.0.0.0/0            0.0.0.0/0            TOS set 0x10/0xff
4      851 64492 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
5        2    64 TOS        all  --  *      *       0.0.0.0/0            0.0.0.0/0            MAC XX:XX:XX:XX:XX:XX TOS set 0x04/0xff
6        2    64 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            MAC XX:XX:XX:XX:XX:XX
7     5402 5442K TOS        all  --  *      *       192.168.2.0/24       0.0.0.0/0            TOS set 0x02/0xff
8        0     0 TOS        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:22:23 TOS set 0x10/0xff

Chain INPUT (policy ACCEPT 54682 packets, 67M bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 3060 packets, 175K bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 30593 packets, 6441K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1      330 21754 TOS        udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 TOS set 0x10/0xff
2      330 21754 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
3        0     0 TOS        udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 TOS set 0x10/0xff

Chain POSTROUTING (policy ACCEPT 33660 packets, 6618K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1       72  251K TOS        tcp  --  *      *       192.168.2.0/24       0.0.0.0/0            connbytes 1048576 connbytes mode bytes connbytes direction both TOS set 0x08/0xff
2        0     0 TOS        udp  --  *      *       192.168.2.0/24       0.0.0.0/0            connbytes 1048576 connbytes mode bytes connbytes direction both TOS set 0x08/0xff
3      956 79229 TOS        icmp --  *      *       0.0.0.0/0            0.0.0.0/0            TOS set 0x10/0xff
admin@Media:/tmp/mnt/Media/jffs#

When I use transsmision on router without speed limit (and only 6 peers) I have ping over 1000 (from XX.XX.XX.XX.XX.XX and the same from other PCs). My internet connection s about 2,6 mbits.

Code:
ifconfig ppp0 set txqueuelen 1000
doasn't worked for me, I used:

Code:
ifconfig ppp0 txqueuelen 200

Code:
ppp0       Link encap:Point-to-Point Protocol  
           inet addr:XX.4.252.133  P-t-P:XX.1.4.202  Mask:255.255.255.255
           UP POINTOPOINT RUNNING MULTICAST  MTU:1492  Metric:1
           RX packets:4907659 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2936883 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:200 
           RX bytes:1766341626 (1.6 GiB)  TX bytes:544450590 (519.2 MiB)
 
Last edited:
It's a mystery why your TOS rules don't work, all traffic seems to be still with TOS 0 and thus goes to band 4.

In order to gain more insight you could match tos rules in nat table postrouting chain. The counts should verify that the tos values are not getting assigned. Like "iptables -t nat -I POSTROUTING 1 -m tos --tos 0x10" for each tos value.

If they're not assigned...try without the /0xff ? Or is that only added by AC68 iptables display? Need to find some way for the tos markings to stick.

Btw why doesn't the "ifconfig ppp0 set txqueuelen 1000" work? I don't think you should be out of memory on AC68 (each of the 5 band consumes 1000 packets of memory).
 
Last edited:
Hello.

This information about trafficc management trough firewal is amazing!
I am using this router ASUS NT-R66U in comunity of 30 persons.

Here is normally online 15-20 users, mix of windows, linux and android users.
But the total internet connection is very low.
We should have: 1MB upload and 8MB download.
Really i see in traffic usage max: 100 KB/s upload and 500 KB/s download.

Becouse of big local network usage from windows services, it is imposible to do not use Hardaware NAT acceleration.
The local network just start colapsing on long response of all traffic.
So i cannost use QoS system at any point.

Now i am using your setup, which work perfectly with few users.
It actually slow down downloading at some point.
But still when there is more users and more people downloading, the web browsing take me a long time (sometimes timeout).
If i start download, it download with no problem one file at 40-100 kB/s, or two files at at 40-100 kB/s EACH.

So downloading is restricted somehow, but web-browsing take long time even if i dont download.
So i need MORE restrictions to people that use downloading.

Please can you help me ? Its even posible ?

THANK YOU!

----------------------------------------------
PRIORITIES THAT MAYBE CAN HELP:

  • 1. dns, dhcp, and important services.
  • 2. local network.
  • 3. ssh (except big downloads, until 10MB )
  • 4. all small net usage, like until 100-500 kb ???
  • 5. web browsing until 2MB - most important is web users feelings!
  • 6. all net usage until 10MB - google maps, photos, ..
  • 7. all net usage until 50MB - video or music streaming, small files
  • 8. all net usage until 100MB - big files or videos


----------------------------------------------
HERE IS MY SETUP:

I use standart ethernet WAN port to acces internet.
For local users WIFI 2Ghz and 5Ghz, plus ethernet ports for LAN network users (with extensions like switch and wifi bridge).
All users obtain IP adress directly from ASUS router.
And most of users connect trought EThernet ports, not wifi.

IFCONFIG:

Code:
admin@RT-N66U-D7F8:/tmp/home/root# ifconfig
br0  Link encap:Ethernet  HWaddr 08:60:6E:21:D7:F8
  inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:16476862 errors:0 dropped:0 overruns:0 frame:0
  TX packets:11461580 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:0
  RX bytes:1581343125 (1.4 GiB)  TX bytes:2286997652 (2.1 GiB)
eth0  Link encap:Ethernet  HWaddr 08:60:6E:21:D7:F8
  inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:580748107 errors:0 dropped:0 overruns:0 frame:0
  TX packets:570541820 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:300
  RX bytes:1001445332 (955.0 MiB)  TX bytes:2816054946 (2.6 GiB)
  Interrupt:4 Base address:0x2000
eth1  Link encap:Ethernet  HWaddr 08:60:6E:21:D7:F8
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:18327689 errors:3 dropped:0 overruns:0 frame:177812263
  TX packets:30980002 errors:2309 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:2198513463 (2.0 GiB)  TX bytes:3930941888 (3.6 GiB)
  Interrupt:3 Base address:0x8000
eth2  Link encap:Ethernet  HWaddr 08:60:6E:21:D7:FC
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:4177287 errors:0 dropped:0 overruns:0 frame:240729
  TX packets:7572764 errors:3355 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:1223388752 (1.1 GiB)  TX bytes:1486997662 (1.3 GiB)
  Interrupt:5 Base address:0x8000
lo  Link encap:Local Loopback
  inet addr:127.0.0.1  Mask:255.0.0.0
  UP LOOPBACK RUNNING MULTICAST  MTU:16436  Metric:1
  RX packets:2812 errors:0 dropped:0 overruns:0 frame:0
  TX packets:2812 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:0
  RX bytes:346768 (338.6 KiB)  TX bytes:346768 (338.6 KiB)
vlan1  Link encap:Ethernet  HWaddr 08:60:6E:21:D7:F8
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:250120592 errors:0 dropped:0 overruns:0 frame:0
  TX packets:302622343 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:0
  RX bytes:414158400 (394.9 MiB)  TX bytes:3706856336 (3.4 GiB)
admin@RT-N66U-D7F8:/tmp/home/root#



FIREWALL SCRIPT:

Code:
admin@RT-N66U-D7F8:/tmp/home/root# cat /jffs/scripts/firewall-start
#!/bin/sh

logger "firewall"
logger "firewall" "START applying firewall-start rules"

#Queue 1000 packets per band ==> ifconfig eth0 txqueuelen NN", where NN is 1/10 seconds worth of 1500 byte packets, for example for a 10 Mbit connection, NN=800.

ifconfig eth0 txqueuelen 300

#tc qdisc replace dev eth0 root pfifo
tc qdisc replace dev eth0 root prio bands 5 priomap 4 2 1 3 3 4 4 4 0 0 0 0 2 2 2 2

# comment out if you don't want your existing table flushed
iptables -t mangle -F

# tos 16 minimize delay interactive -> band 0

# dns
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A OUTPUT -p udp --dport 53 -j RETURN

# telnet ssh
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j RETURN

# icmp
iptables -t mangle -A PREROUTING -p icmp -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p icmp -j RETURN

# tos 4 maximize reliability -> band 1
# preferred mac addresses - DONT WANT, ALL USERS ARE SAME PRIORITY
#iptables -t mangle -A PREROUTING -m mac --mac-source 00:11:22:33:44:55 -j TOS --set-tos 4
#iptables -t mangle -A PREROUTING -m mac --mac-source 00:11:22:33:44:55 -j RETURN

# tos 2 minimize monetary cost -> band 2
# everyone in our internal network
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j TOS --set-tos 2


# tos 8 maximize throughput bulk -> band 3
# preferred bulk traffic
# all transfers from internal network larger than 1 MB are put into preferred bulk category
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8

# tos 0 normal -> band 4
# slowest bulk traffic here, will (hopefully) include default traffic from router itself; download master


### POSTROUTING
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A POSTROUTING -p icmp -j TOS --set-tos 16


logger "firewall"
logger "firewall" "SHOW pfifo limits:"

tc -s -d qdisc | logger

logger "firewall"
logger "firewall" "SHOW FIREWALL RULES:"

iptables -t mangle -nvL --line-numbers | logger

logger "firewall"
logger "firewall" "FINISHED"
admin@RT-N66U-D7F8:/tmp/home/root#


IPTABLES CHECK:

Code:
admin@RT-N66U-D7F8:/tmp/home/root#
admin@RT-N66U-D7F8:/tmp/home/root# iptables -t mangle -nvL --line-numbers
Chain PREROUTING (policy ACCEPT 24M packets, 2906M bytes)
num  pkts bytes target  prot opt in  out  source  destination
1  1238 87268 TOS  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpts:22:23 TOS set 0x10
2  1238 87268 RETURN  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpts:22:23
3  308K  33M TOS  icmp --  *  *  0.0.0.0/0  0.0.0.0/0  TOS set 0x10
4  308K  33M RETURN  icmp --  *  *  0.0.0.0/0  0.0.0.0/0
5  16M 1469M TOS  all  --  *  *  192.168.1.0/24  0.0.0.0/0  TOS set 0x02
6  0  0 TOS  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpts:22:23 TOS set 0x10

Chain INPUT (policy ACCEPT 5148K packets, 754M bytes)
num  pkts bytes target  prot opt in  out  source  destination

Chain FORWARD (policy ACCEPT 18M packets, 2128M bytes)
num  pkts bytes target  prot opt in  out  source  destination

Chain OUTPUT (policy ACCEPT 4805K packets, 886M bytes)
num  pkts bytes target  prot opt in  out  source  destination
1  755K  49M TOS  udp  --  *  *  0.0.0.0/0  0.0.0.0/0  udp dpt:53 TOS set 0x10
2  755K  49M RETURN  udp  --  *  *  0.0.0.0/0  0.0.0.0/0  udp dpt:53
3  0  0 TOS  udp  --  *  *  0.0.0.0/0  0.0.0.0/0  udp dpt:53 TOS set 0x10

Chain POSTROUTING (policy ACCEPT 23M packets, 3008M bytes)
num  pkts bytes target  prot opt in  out  source  destination
1  13309  534K TOS  tcp  --  *  *  192.168.1.0/24  0.0.0.0/0  connbytes 1048576:4294967295 bytes direction both TOS set 0x08
2  17056  26M TOS  udp  --  *  *  192.168.1.0/24  0.0.0.0/0  connbytes 1048576:4294967295 bytes direction both TOS set 0x08
3  366K  43M TOS  icmp --  *  *  0.0.0.0/0  0.0.0.0/0  TOS set 0x10
admin@RT-N66U-D7F8:/tmp/home/root#
admin@RT-N66U-D7F8:/tmp/home/root#

SOME RANDOM USAGE:

traffic-monitor.png


THANK YOU!
Petr.
 
Last edited:
I dont think its right to put DNS and ICMP in band 0. DNS doesnt require an immediate response while ICMP is usually given low priority (pings as example).

NTP, FPS games and such should be given band 0 because response time is important for them but they take very little bandwidth. VOIP stuff dont really need fast response but do require some bandwidth so i would place them below games.

Other games can can go into band 1 or 2 while streaming goes below gaming, followed by http and than downloads and other stuff.

If it is possible try to reserve a bit of bandwidth in download for uploads and vice versa.

Its good to be able to perform QoS with hardware acceleration. I'd like to know what throughputs you get from it.
 
I dont think its right to put DNS and ICMP in band 0. DNS doesnt require an immediate response while ICMP is usually given low priority (pings as example).

NTP, FPS games and such should be given band 0 because response time is important for them but they take very little bandwidth. VOIP stuff dont really need fast response but do require some bandwidth so i would place them below games.

Other games can can go into band 1 or 2 while streaming goes below gaming, followed by http and than downloads and other stuff.

If it is possible try to reserve a bit of bandwidth in download for uploads and vice versa.

Its good to be able to perform QoS with hardware acceleration. I'd like to know what throughputs you get from it.

OK, maybe my priority ideas are wrong. Thank You.

But I must say, that i dont know how to improve the setup to limit more the users that download and block the connection.
I need to limit all downloads and sure uploads in very short time and amount of data, to improve web browsing of many users.
I dont think there is anyone playing games, or using VOIP.
But sometimes, when somebody download, web browsing is very very slow, one page can take 10-20 seconds to load.
This setup i have posted, is almost the same as first example.
I hope ml70, or anyone with experience will give the answer :)
If You need more information, ask for direct command or file, i will repost.
Thank You.
 
simple. Http browsing, than other traffic, than torrent/p2p/downloads. a lot of people forget to include 'others' Im not sure that you can use bandwidth control using priority which is what you're using unless you use a non consumer router. However using priorities will affect uploads as well as downloads at the same time.

If you can have different que sizes for different priorities than give priority 0 a smaller que because it needs lower latency. Certain things like downloads, torrents and such can have a big que. http too. Remember that it applies for uploads too so if you're a streamer than you will need to reduce the que size for your stream priority for lower latency. Bigger ques increase latency but theres a bigger buffer.
 
Last edited:
This is an interesting application I haven't seen for a while! Sad to see it received little traction when first appeared last year.

I would suggest automate with some sort of scripts to read user preferences and generate the iptables rules. Focus on average number of rules a packet has to traverse through iptables...for better efficiency.

Also have better integration with overall firewall setup on routers. Can ditch the existing Asus iptables setup as it looks cluttered and possibly inefficient.
 
I tried few changes:
- add WEB priority with data less than 5MB to band 1
- add WEB priority with data more than 5MB to band 2
- add small data priority with data less than 512kB data to band 2
- add all other data rule to band 0

I am really amater in this, hope it makes sense :)
If you feel some critics or improves, please tell me.

Code:
admin@RT-N66U-D7F8:/tmp/home/root# cat  /jffs/firewall-start
#!/bin/sh

logger "firewall"
logger "firewall" "START applying firewall-start rules"

#Queue 1000 packets per band ==> ifconfig eth0 txqueuelen NN", where NN is 1/10 seconds worth of 1500 byte packets, for example for a 10 Mbit connection, NN=800.

ifconfig eth0 txqueuelen 300

#tc qdisc add dev eth0 root pfifo
tc qdisc replace dev eth0 root prio bands 5 priomap 4 2 1 3 3 4 4 4 0 0 0 0 2 2 2 2

# comment out if you don't want your existing table flushed
iptables -t mangle -F



# tos 16 minimize delay interactive -> BAND 0
#-----------------------------------------
# dns
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A OUTPUT -p udp --dport 53 -j RETURN
#-----------------------------------------
# telnet ssh (less than 5mb)
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j RETURN
#-----------------------------------------
# icmp
iptables -t mangle -A PREROUTING -p icmp -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p icmp -j RETURN




# tos 4 maximize reliability -> BAND 1
#-----------------------------------------
#web prefered, less than 5MB:
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 4
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes -j RETURN
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 4
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes -j RETURN




# tos 2 minimize monetary cost -> BAND 2
#-----------------------------------------
# everyone in our internal network
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j TOS --set-tos 2
#-----------------------------------------
#web prefered, more than 5MB:
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 2
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes -j RETURN
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 2
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes -j RETURN
#-----------------------------------------
#very small traffic 512kB
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 0:524288 --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 2
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 0:524288 --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 2




# tos 8 maximize throughput bulk -> BAND 3
#-----------------------------------------
# preferred bulk traffic
# all transfers from internal network larger than 1 MB are put into preferred bulk category
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8



# tos 0 normal -> BAND 4
#-----------------------------------------
# slowest bulk traffic here, will (hopefully) include default traffic from router itself; download master
iptables -t mangle -A POSTROUTING -p udp -j TOS --set-tos 0
iptables -t mangle -A POSTROUTING -p tcp -j TOS --set-tos 0




### POSTROUTING
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A POSTROUTING -p icmp -j TOS --set-tos 16



logger "firewall"
logger "firewall" "SHOW pfifo limits:"

tc -s -d qdisc | logger

logger "firewall"
logger "firewall" "SHOW FIREWALL RULES:"

iptables -t mangle -nvL --line-numbers | logger

logger "firewall"
logger "firewall" "FINISHED"
admin@RT-N66U-D7F8:/tmp/home/root#
 
I tried few changes:
- add WEB priority with data less than 5MB to band 1
- add WEB priority with data more than 5MB to band 2
- add small data priority with data less than 512kB data to band 2
- add all other data rule to band 0

I am really amater in this, hope it makes sense :)
If you feel some critics or improves, please tell me.

Code:
admin@RT-N66U-D7F8:/tmp/home/root# cat  /jffs/firewall-start
#!/bin/sh

logger "firewall"
logger "firewall" "START applying firewall-start rules"

#Queue 1000 packets per band ==> ifconfig eth0 txqueuelen NN", where NN is 1/10 seconds worth of 1500 byte packets, for example for a 10 Mbit connection, NN=800.

ifconfig eth0 txqueuelen 300

#tc qdisc add dev eth0 root pfifo
tc qdisc replace dev eth0 root prio bands 5 priomap 4 2 1 3 3 4 4 4 0 0 0 0 2 2 2 2

# comment out if you don't want your existing table flushed
iptables -t mangle -F



# tos 16 minimize delay interactive -> BAND 0
#-----------------------------------------
# dns
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A OUTPUT -p udp --dport 53 -j RETURN
#-----------------------------------------
# telnet ssh (less than 5mb)
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j RETURN
#-----------------------------------------
# icmp
iptables -t mangle -A PREROUTING -p icmp -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p icmp -j RETURN




# tos 4 maximize reliability -> BAND 1
#-----------------------------------------
#web prefered, less than 5MB:
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 4
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes -j RETURN
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 4
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 0:5242880 --connbytes-dir both --connbytes-mode bytes -j RETURN




# tos 2 minimize monetary cost -> BAND 2
#-----------------------------------------
# everyone in our internal network
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j TOS --set-tos 2
#-----------------------------------------
#web prefered, more than 5MB:
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 2
iptables -t mangle -A PREROUTING -p tcp --dport 80 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes -j RETURN
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes  -j TOS --set-tos 2
iptables -t mangle -A PREROUTING -p tcp --dport 443 -m connbytes --connbytes 5242880: --connbytes-dir both --connbytes-mode bytes -j RETURN
#-----------------------------------------
#very small traffic 512kB
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 0:524288 --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 2
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 0:524288 --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 2




# tos 8 maximize throughput bulk -> BAND 3
#-----------------------------------------
# preferred bulk traffic
# all transfers from internal network larger than 1 MB are put into preferred bulk category
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p tcp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -p udp -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j TOS --set-tos 8



# tos 0 normal -> BAND 4
#-----------------------------------------
# slowest bulk traffic here, will (hopefully) include default traffic from router itself; download master
iptables -t mangle -A POSTROUTING -p udp -j TOS --set-tos 0
iptables -t mangle -A POSTROUTING -p tcp -j TOS --set-tos 0




### POSTROUTING
iptables -t mangle -A OUTPUT -p udp --dport 53 -j TOS --set-tos 16
iptables -t mangle -A PREROUTING -p tcp --dport 22:23 -j TOS --set-tos 16
iptables -t mangle -A POSTROUTING -p icmp -j TOS --set-tos 16



logger "firewall"
logger "firewall" "SHOW pfifo limits:"

tc -s -d qdisc | logger

logger "firewall"
logger "firewall" "SHOW FIREWALL RULES:"

iptables -t mangle -nvL --line-numbers | logger

logger "firewall"
logger "firewall" "FINISHED"
admin@RT-N66U-D7F8:/tmp/home/root#

What tests do you have to ensure it's indeed working as expected, especially under heavy traffic? And what will be CPU utilisation under such situations?
 
What tests do you have to ensure it's indeed working as expected, especially under heavy traffic? And what will be CPU utilisation under such situations?

Yes, thats the problem. I really dont know how to get some statisctics.

I only test it with browsing and downloadig myself. And I ask people about their personal experience.
I was running the "first" original setup for one month. Now i run new setup and ask people if they feel improvement :)

With 25 devices connected is CPU 0-5% and RAM 25-30%.

If anyone can explain me how to check or improve my setup, i will be happy. Thats the reason why i write here.
 
@Sonic See if this helps. Not a turn-key solution so you still have legwork to do.
 
Some necromancy on this old thread, the original solution is missing the tc traffic classifier part it'd need to function properly, I might've had some remains of tc filters created by Asus stock qos solution running which made it work, without it'll only work for bands 1-3. Sorry.

But to have a nicely working QoS all that is needed is the stock pfifo_fast qdisc with its 3 bands (1,2,3). What you need to know is that the kernel matches internal Linux priorities onto the bands as follows:
Code:
Binary Decimal   Meaning                       Linux priority   Band
-----------------------------------------------------------------------------
1000   8         Minimize delay (md)           6                0
0000   0         Normal Service                0                1
0010   2         Maximize reliability (mr)     0                1
0001   1         Minimize monetary cost (mmc)  1                2
0100   4         Maximize throughput (mt)      2                2

From this follows, that for a minimal QoS solution all you need to do is to mark packets with Minimize-Delay (for band 0) or Minimize-Cost (band 2).
Code:
iptables -t mangle -A PREROUTING -i br0 -p tcp -m multiport --dports 80,443 -j TOS --set-tos Minimize-Delay

This above is all you need to avoid complaints from family members that their web-thing doesn't work while you're maximizing your down/upload.
Others things to consider for example are -p icmp, -p udp 53,123 (dns and time), -p tcp 465,587,993 (gmail ports for email software and handphones).

You can replace builtin pfifo_fast with a 3 band prio for verifying, because pfifo_fast does not allow its bands to be viewed by tc -s class ls dev ppp0 (priomap here is same as in pfifo_fast).
Code:
tc qdisc replace dev ppp0 root prio bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
To remove, use 'tc qdisc delete dev ppp0 root', the default pfifo_fast qdisc will be recreated automatically.

If you have a separate vm/pc doing any downloading or other heavy use, add this last
Code:
iptables -t mangle -A PREROUTING -i br0 -s 192.168.1.200 -j TOS --set-tos Minimize-Cost
and all traffic from 192.168.1.200 will go to band 2, except tcp:80,443 traffic which will also be caught by the previous rule for band 0.

That is the preferred solution, if you run a torrent client on your main pc it's more difficult to write rules for all the connections it makes, because only incoming will use the chosen incoming port.
To catch outgoing connections for TOS classification you'd have to separate the port ranges by use, for example in Windows have other programs use the ephemeral range only
Code:
netsh int ipv4 set dynamicport tcp start=49152 num=16384
netsh int ipv4 set dynamicport udp start=49152 num=16384
and then limit the torrent client to use ports below 49152, or the other way round.

It seems that running the classful qdisc's (tbf etc) is what bogs an old Asus router down, using a multiband classless qdisc (pfifo_fast, prio) is the only QoS solution if you want to achieve at least 100/100 mbps performance.
 
Either I've misunderstood something very badly, or then I've actually found out the truth about Asus' TOS implementation:

This is the generic Linux TOS/Priomap
Code:
TOS     Bits  Means                    Linux Priority    Band
------------------------------------------------------------
0x0     0     Normal Service           0 Best Effort     1
0x2     1     Minimize Monetary Cost   1 Filler          2
0x4     2     Maximize Reliability     0 Best Effort     1
0x6     3     mmc+mr                   0 Best Effort     1
0x8     4     Maximize Throughput      2 Bulk            2
0xa     5     mmc+mt                   2 Bulk            2
0xc     6     mr+mt                    2 Bulk            2
0xe     7     mmc+mr+mt                2 Bulk            2
0x10    8     Minimize Delay           6 Interactive     0
0x12    9     mmc+md                   6 Interactive     0
0x14    10    mr+md                    6 Interactive     0
0x16    11    mmc+mr+md                6 Interactive     0
0x18    12    mt+md                    4 Int. Bulk       1
0x1a    13    mmc+mt+md                4 Int. Bulk       1
0x1c    14    mr+mt+md                 4 Int. Bulk       1
0x1e    15    mmc+mr+mt+md             4 Int. Bulk       1
which, as I understand it, means that should you number the bands sequentially 0..15, then TOS 0x10 should go to band 8 (same as its Bit order).

But testing this theory out, TOS 0x10 goes to band 6 instead. It seems the TOS codes map to their Linux Priority equivalents, tos 0x0,0x4 -> band 0, 0x2 -> band 1, 0x8 -> band 2, 0x10 -> band 6.

With this information we can finally plan a properly working 4 band priomap with one extra band , the last one #4, signifying N/A.
Code:
# TOS 0x10 Minimize-Delay       Linux Priority 6 -> Band 0
# TOS 0x00 Normal-Service       Linux Priority 0 -> Band 1
# TOS 0x08 Maximize-Throughput  Linux Priority 2 -> Band 2
# TOS 0x02 Minimize-Cost        Linux Priority 1 -> Band 3
#
tc qdisc replace dev ppp0 root prio bands 5 priomap 1 3 2 4 4 4 0 4 4 4 4 4 4 4 4 4
Lo and behold, it finally truly works this time.
Code:
$ tc -s -d class ls dev ppp0
class prio 8026:1 parent 8026:
 Sent 4540102 bytes 59537 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 8026:2 parent 8026:
 Sent 5451 bytes 81 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 8026:3 parent 8026:
 Sent 81753 bytes 242 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 8026:4 parent 8026:
 Sent 6047667 bytes 10238 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 8026:5 parent 8026:
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

No tc filters are needed
Code:
$ tc -s -d filter ls dev ppp0

$

Usage example in mangle table:
Code:
# Band 0 for everything important
-A PREROUTING -i br0 -p icmp -j TOS --set-tos Minimize-Delay
-A PREROUTING -i br0 -p icmp -j RETURN
-A PREROUTING -i br0 -p udp -m multiport --dports 53,123 -j TOS --set-tos Minimize-Delay
-A PREROUTING -i br0 -p udp -m multiport --dports 53,123 -j RETURN
-A PREROUTING -i br0 -p tcp -m multiport --dports 20:23,43,80,110,143,443,465,587,993 -j TOS --set-tos Minimize-Delay
-A PREROUTING -i br0 -p tcp -m multiport --dports 20:23,43,80,110,143,443,465,587,993 -j RETURN
# Band 1 is the default for all traffic, an average pc here
-A PREROUTING -i br0 -s 192.168.1.10 -j TOS --set-tos Normal-Service
-A PREROUTING -i br0 -s 192.168.1.10 -j RETURN
# Band 2 for a not so important pc
-A PREROUTING -i br0 -s 192.168.1.200 -j TOS --set-tos Maximize-Throughput
-A PREROUTING -i br0 -s 192.168.1.200 -j RETURN
# Band 3 for the pc downloading Linux ISO's
-A PREROUTING -i br0 -s 192.168.1.252 -j TOS --set-tos Minimize-Cost
-A PREROUTING -i br0 -s 192.168.1.252 -j RETURN
PRE or POSTrouting now that's the question, please do set me straight on this?
POST would capture traffic from router, but comparing packet amounts PRE vs POST some traffic just disappears between PRE and POST, is this Broadcom's CTF in action? And if relying on POST, would these packets then skip TOS classification?
Should the rules be divided into 2 chains, PRE for lan, OUTPUT or POST (which?) for router?
 
Similar threads

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top