FlexQoS FlexQoS 1.2.5 - Flexible QoS Enhancement Script for Adaptive QoS

[dave14305]

Shouldn't speedtest traffic categorized under File Transferring? I was doing a speedtest and File Transfer category is not showing.

Where were you running the Speedtest from? Router or client? What did the tracked connections show for that device during the test?

 

[Jerry Loh]

Where were you running the Speedtest from? Router or client? What did the tracked connections show for that device during the test?

Client side ya, both windows app and web. I have no idea what the tracked connections shows.
This is the newest debug, anything wrong? how do I know QOS is properly working?

FlexQoS v1.2.5 released 2021-06-13Debug date : 2021-08-21 03:44:35+0800 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[dave14305]

Client side ya, both windows app and web. I have no idea what the tracked connections shows.

What do you mean? It’s directly beneath all the graphs.

 

[Jerry Loh]

What do you mean? It’s directly beneath all the graphs.

I think it's this. Then its correct.

When i ran speedtest with "Internet Speed" tab, it is showing in Upload section only no matter download or upload testing is carried out. Not sure is my download is bugged or too many tracked connections.

 

[dave14305]

I think it's this. Then its correct.
View attachment 35872

When i ran speedtest with "Internet Speed" tab, it is showing in Upload section only no matter download or upload testing is carried out. Not sure is my download is bugged or too many tracked connections.
View attachment 35873

Router speedtest will only show in Uploads. But client speedtest should appear in both if it’s working correctly.

 

[dave14305]

Today, my family and I were streaming a program and I was disappointed to see it was being classified as Web Surfing, despite my custom rule for all traffic from the streaming device to be classified as Streaming.

After some analysis of the Tracked Connections from the device, I determined that the stream was happening over IPv6, so it eluded my IPv4-based iptables rule for the device. Sad.

After reviewing my old notes and posts from the FreshJR thread I was reminded of the difficulty creating iptables rules for local IPv6 addresses because in most cases, the addresses are random and change frequently for privacy. Hence, any iptables rule in FlexQoS that includes a local or remote IPv4 address isn’t applied to IPv6 traffic.

So I’m on a bit of a quest to solve this puzzle. Since we cannot rely on a stable, predictable IPv6 address, we need to figure out what we have to work with. MAC address is the obvious missing link, but not available in the POSTROUTING table.

I found a clever post at OpenWRT that has put me on an interesting track using ipsets. I’m only experimenting with manual commands at this point, but this is the approach I’m pursuing (it’s a work-in-progress so not necessarily efficient or fully thought through yet):

• We know the local IPv4 address for a device. Derive the MAC address of the IPv4 address.
• Create a hash:mac ipset to hold the MAC address for up to 1 day (my lease duration). ipset create firestick-mac hash:mac timeout 86400
• Create an iptables rule to capture the MAC address when the known IPv4 address initiates a new connection anywhere. iptables -t mangle -I PREROUTING -m conntrack --ctstate NEW -s 192.168.50.7 -j SET --add-set firestick-mac src --exist
• Create a hash:ip ipset for IPv6 address for the device to hold the temporary IPv6 addresses as they change, for up to 1 hour. ipset create firestick hash:ip family inet6 timeout 3600
• Create an ip6tables rule to capture the local IPv6 address when the MAC address initiates a connection. ip6tables -t mangle -I PREROUTING -m conntrack --ctstate NEW -m set --match-set firestick-mac src -j SET --add-set firestick src --exist
• As the device initiates connections, the firestick ipset should always contain the most recent IPv6 addresses it has been assigned. Now we can use that firestick ipset in an ip6tables rule to capture my streaming traffic.
• Create ip6tables rules to mimic the IPv4 rule I setup in the FlexQoS GUI. ip6tables -t mangle -A FlexQoS -o br0 -m set --match-set firestick dst -p tcp -j MARK --set-xmark 0x8004ffff/0xc03fffff ip6tables -t mangle -A FlexQoS -o eth0 -m set --match-set firestick src -p tcp -j MARK --set-xmark 0x4004ffff/0xc03fffff

It remains to be seen if this is workable for every situation involving local IPs. But so far it seems to be capturing the idle phone home traffic that the Amazon Firestick is doing while I type this.

# ip6tables -t mangle -nvL FlexQoS
Chain FlexQoS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport dports 500,4500 multiport sports 500,4500 MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport sports 500,4500 multiport dports 500,4500 MARK xset 0x4006ffff/0xc03fffff
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport sports 3478:3481 mark match 0x80000000/0xc03fffff MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport dports 3478:3481 mark match 0x40000000/0xc03fffff MARK xset 0x4006ffff/0xc03fffff
   40  4060 MARK       tcp      *      br0     ::/0                 ::/0                 match-set firestick dst MARK xset 0x8004ffff/0xc03fffff
  149 12608 MARK       tcp      *      eth0    ::/0                 ::/0                 match-set firestick src MARK xset 0x4004ffff/0xc03fffff

There’s no reward for keeping this half-baked idea to myself, so I post it for anyone to comment on or test themselves if they follow the logic. FreshJR always said he had some ideas how to solve this, but never elaborated.

 

[Tomo]

Hello guys, after a long time i decided to upgrade my Ac86u from 384.17 to the newest 386.3.2, i reset every thing , formatted the JFFS partition and then enabled it but my question is, where is the amtm menù ? i wanted install the new FlexQos script , maybe i need of a usb stick ?

 

[slidermike]

Yes you want a drive for a swap file.
Once you set the router to mount the usb drive in the gui, reboot so it takes effect.
Then from the command line you type "amtm" without the brackets.
The menu should be pretty self explanatory.
I would create the swap drive first in the amtm window.

 

[Tomo]

oh ok, because with the old FreshJr script there was not need of the usb stick/swap file , when i read that amtm was included in the firmware i thought it was ready to use.
Maybe i go for manual install, would be ok

edit: ok manual install is ok, i have configured all options, add gaming rule and all run perfect, great work Dave !! donation incoming soon

 

[slidermike]

Glad you got the outcome you were looking for.

For clarity, amtm is a shell or menu that allows you to just click a few buttons to seamlessly download/install any of the various scripts supported.
amtm is in the firmware now but that does not mean the applications come installed and ready to run.
You still have to hit the command line and from amtm, install and configure the applications you like.
Your not having to download amtm now is all.

Some (not all) of the apps require a swap file. An external usb drive is the preferred method.

 

[dave14305]

oh ok, because with the old FreshJr script there was not need of the usb stick/swap file , when i read that amtm was included in the firmware i thought it was ready to use.
Maybe i go for manual install, would be ok

edit: ok manual install is ok, i have configured all options, add gaming rule and all run perfect, great work Dave !! donation incoming soon

Grazie mille! For the record, you do not need a USB stick or a swap file for FlexQoS.

 

[Kingp1n]

Today, my family and I were streaming a program and I was disappointed to see it was being classified as Web Surfing, despite my custom rule for all traffic from the streaming device to be classified as Streaming.

After some analysis of the Tracked Connections from the device, I determined that the stream was happening over IPv6, so it eluded my IPv4-based iptables rule for the device. Sad.

After reviewing my old notes and posts from the FreshJR thread I was reminded of the difficulty creating iptables rules for local IPv6 addresses because in most cases, the addresses are random and change frequently for privacy. Hence, any iptables rule in FlexQoS that includes a local or remote IPv4 address isn’t applied to IPv6 traffic.

So I’m on a bit of a quest to solve this puzzle. Since we cannot rely on a stable, predictable IPv6 address, we need to figure out what we have to work with. MAC address is the obvious missing link, but not available in the POSTROUTING table.

I found a clever post at OpenWRT that has put me on an interesting track using ipsets. I’m only experimenting with manual commands at this point, but this is the approach I’m pursuing (it’s a work-in-progress so not necessarily efficient or fully thought through yet):

• We know the local IPv4 address for a device. Derive the MAC address of the IPv4 address.
• Create a hash:mac ipset to hold the MAC address for up to 1 day (my lease duration). ipset create firestick-mac hash:mac timeout 86400
• Create an iptables rule to capture the MAC address when the known IPv4 address initiates a new connection anywhere. iptables -t mangle -I PREROUTING -m conntrack --ctstate NEW -s 192.168.50.7 -j SET --add-set firestick-mac src --exist
• Create a hash:ip ipset for IPv6 address for the device to hold the temporary IPv6 addresses as they change, for up to 1 hour. ipset create firestick hash:ip family inet6 timeout 3600
• Create an ip6tables rule to capture the local IPv6 address when the MAC address initiates a connection. ip6tables -t mangle -I PREROUTING -m conntrack --ctstate NEW -m set --match-set firestick-mac src -j SET --add-set firestick src --exist
• As the device initiates connections, the firestick ipset should always contain the most recent IPv6 addresses it has been assigned. Now we can use that firestick ipset in an ip6tables rule to capture my streaming traffic.
• Create ip6tables rules to mimic the IPv4 rule I setup in the FlexQoS GUI. ip6tables -t mangle -A FlexQoS -o br0 -m set --match-set firestick dst -p tcp -j MARK --set-xmark 0x8004ffff/0xc03fffff ip6tables -t mangle -A FlexQoS -o eth0 -m set --match-set firestick src -p tcp -j MARK --set-xmark 0x4004ffff/0xc03fffff

It remains to be seen if this is workable for every situation involving local IPs. But so far it seems to be capturing the idle phone home traffic that the Amazon Firestick is doing while I type this.

# ip6tables -t mangle -nvL FlexQoS
Chain FlexQoS (1 references)
pkts bytes target     prot opt in     out     source               destination       
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport dports 500,4500 multiport sports 500,4500 MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport sports 500,4500 multiport dports 500,4500 MARK xset 0x4006ffff/0xc03fffff
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport sports 3478:3481 mark match 0x80000000/0xc03fffff MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport dports 3478:3481 mark match 0x40000000/0xc03fffff MARK xset 0x4006ffff/0xc03fffff
   40  4060 MARK       tcp      *      br0     ::/0                 ::/0                 match-set firestick dst MARK xset 0x8004ffff/0xc03fffff
  149 12608 MARK       tcp      *      eth0    ::/0                 ::/0                 match-set firestick src MARK xset 0x4004ffff/0xc03fffff

There’s no reward for keeping this half-baked idea to myself, so I post it for anyone to comment on or test themselves if they follow the logic. FreshJR always said he had some ideas how to solve this, but never elaborated.

Yes I remember the headches with the FreshJR script while running IPV6.

I eventually just disabled IPV6 since the gaming rules would also not apply. Guess it's the same for streaming.

I see your currently testing the possibility of workaround with IPV6....will you include this in a future beta Flex script test or it has to de done manually at this time?

Thanks!

 

[dave14305]

I see your currently testing the possibility of workaround with IPV6....will you include this in a future beta Flex script test or it has to de done manually at this time?

Eventually it will find its way to the develop branch once I think through everything. I imagine the impact is on a small number of users since IPv6 isn’t in widespread use.

 

[raion969]

Just for sayin i also use ipv6

Also with the flexqos didnt work tge post before i also had the problems at the beginning when i dont enter the right wan paket overhead, after this all worked normal because i saw that you didnt put any in it?

 

[raion969]

is it good to set all the other devices in the bandwith priority tap to verly low when i want to have no ping fluctating and so on while gaming ?

 

[sentinelvdx]

Hi @dave14305 , can you please remind me how to mark all traffic from an IP as "Streaming" or whatever I want?
I want to mark my TV's IP as Streaming no matter what traffic is, and a couple of devices mor the same way.

Thanks!!!

 

[dave14305]

Hi @dave14305 , can you please remind me how to mark all traffic from an IP as "Streaming" or whatever I want?
I want to mark my TV's IP as Streaming no matter what traffic is, and a couple of devices mor the same way.

Thanks!!!

Create a rule with the device’s Local IP and the Class as Streaming.

 

[sentinelvdx]

Create a rule with the device’s Local IP and the Class as Streaming.

With no Mark at all? Like this?

Update: Thanks, didn't thought about adding them without marking

 

[dave14305]

Today, my family and I were streaming a program and I was disappointed to see it was being classified as Web Surfing, despite my custom rule for all traffic from the streaming device to be classified as Streaming.

After some analysis of the Tracked Connections from the device, I determined that the stream was happening over IPv6, so it eluded my IPv4-based iptables rule for the device. Sad.

After reviewing my old notes and posts from the FreshJR thread I was reminded of the difficulty creating iptables rules for local IPv6 addresses because in most cases, the addresses are random and change frequently for privacy. Hence, any iptables rule in FlexQoS that includes a local or remote IPv4 address isn’t applied to IPv6 traffic.

So I’m on a bit of a quest to solve this puzzle. Since we cannot rely on a stable, predictable IPv6 address, we need to figure out what we have to work with. MAC address is the obvious missing link, but not available in the POSTROUTING table.

I found a clever post at OpenWRT that has put me on an interesting track using ipsets. I’m only experimenting with manual commands at this point, but this is the approach I’m pursuing (it’s a work-in-progress so not necessarily efficient or fully thought through yet):

• We know the local IPv4 address for a device. Derive the MAC address of the IPv4 address.
• Create a hash:mac ipset to hold the MAC address for up to 1 day (my lease duration). ipset create firestick-mac hash:mac timeout 86400
• Create an iptables rule to capture the MAC address when the known IPv4 address initiates a new connection anywhere. iptables -t mangle -I PREROUTING -m conntrack --ctstate NEW -s 192.168.50.7 -j SET --add-set firestick-mac src --exist
• Create a hash:ip ipset for IPv6 address for the device to hold the temporary IPv6 addresses as they change, for up to 1 hour. ipset create firestick hash:ip family inet6 timeout 3600
• Create an ip6tables rule to capture the local IPv6 address when the MAC address initiates a connection. ip6tables -t mangle -I PREROUTING -m conntrack --ctstate NEW -m set --match-set firestick-mac src -j SET --add-set firestick src --exist
• As the device initiates connections, the firestick ipset should always contain the most recent IPv6 addresses it has been assigned. Now we can use that firestick ipset in an ip6tables rule to capture my streaming traffic.
• Create ip6tables rules to mimic the IPv4 rule I setup in the FlexQoS GUI. ip6tables -t mangle -A FlexQoS -o br0 -m set --match-set firestick dst -p tcp -j MARK --set-xmark 0x8004ffff/0xc03fffff ip6tables -t mangle -A FlexQoS -o eth0 -m set --match-set firestick src -p tcp -j MARK --set-xmark 0x4004ffff/0xc03fffff

It remains to be seen if this is workable for every situation involving local IPs. But so far it seems to be capturing the idle phone home traffic that the Amazon Firestick is doing while I type this.

# ip6tables -t mangle -nvL FlexQoS
Chain FlexQoS (1 references)
pkts bytes target     prot opt in     out     source               destination        
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport dports 500,4500 multiport sports 500,4500 MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport sports 500,4500 multiport dports 500,4500 MARK xset 0x4006ffff/0xc03fffff
    0     0 MARK       udp      *      br0     ::/0                 ::/0                 multiport sports 3478:3481 mark match 0x80000000/0xc03fffff MARK xset 0x8006ffff/0xc03fffff
    0     0 MARK       udp      *      eth0    ::/0                 ::/0                 multiport dports 3478:3481 mark match 0x40000000/0xc03fffff MARK xset 0x4006ffff/0xc03fffff
   40  4060 MARK       tcp      *      br0     ::/0                 ::/0                 match-set firestick dst MARK xset 0x8004ffff/0xc03fffff
  149 12608 MARK       tcp      *      eth0    ::/0                 ::/0                 match-set firestick src MARK xset 0x4004ffff/0xc03fffff

There’s no reward for keeping this half-baked idea to myself, so I post it for anyone to comment on or test themselves if they follow the logic. FreshJR always said he had some ideas how to solve this, but never elaborated.

There's a 1.2.6 beta on the develop channel to see if this works for others using IPv6. It seems to work for me in my initial tests.

If you don't use IPv6, the only other change in this version is a change to classify outbound DNS, DoT and NTP as Net Control. Previously, these bypassed QoS completely.

In case you forgot how, you can switch to the develop branch using flexqos develop

This only works with Local IPs in rules. If an IPv4 Remote IP is also present in the rule, the IPv6 rule support won't work.

I haven't coded in weeks, so this beta could be rough around the edges.

 

[Kingp1n]

@dave14305

I would like to try the beta. I had a couple of questions.

Off course I will 1st enabled IPV6 on the router.

Do I need to do anything on any of my devices specifically for my gaming devices once they get an IPV6 IP?

Do I go to my devices and see what IP (IPV6) they're using to add to current the gaming rule or even for devices that are streaming?

Thanks.