[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

Oops i meant this command?
nvram get wan0_ifname

That was debugging users did when they noticed iptable rules were created on the wrong interface. Do not worry about that command

 

[skeal]

Hello im see this in system log:


Jun 10 12:13:55 kernel: br0: received packet on vlan1 with own address as source address
Jun 10 12:13:55 kernel: br0: received packet on vlan1 with own address as source address

I have the alternative script because i receve net by vlan and i use a swith tl-link sg105E i have config vlan´s in the tp

http://whatsyourrouter.com/mybb/showthread.php?tid=304

 

[FreshJR]

Pushed an update to the ALTERNATE_WAN_INTERFACE version of the script.

The update was to fix the bug reports regarding improper interface detection.

The alternate version can be found here.

https://www.snbforums.com/threads/r...-and-inner-workings.36836/page-76#post-409166

--

v5 release uses a different firewall-start entry.

As a result, you will have to fully uninstall any previous version of the script, and fresh install this one.

Eg.

/jffs/scripts/FreshJR_QOS -uninstall
--upload new script--
/jffs/scripts/FreshJR_QOS -install

Once again, this is only intended for users with non-eth0 interfaces!

 

[Atomic7431]

Pushed an update to the ALTERNATE_WAN_INTERFACE version of the script.
The update was to fix the bug reports regarding improper interface detection.

The alternate version can be found here.

https://www.snbforums.com/threads/r...-and-inner-workings.36836/page-76#post-409166

--

v5 release uses a different firewall-start entry.

As a result, you will have to fully uninstall any previous version of the script, and fresh install this one.

Eg.

/jffs/scripts/FreshJR_QOS -uninstall
--upload new script--
/jffs/scripts/FreshJR_QOS -install

Once again, this is only intended for users with non-eth0 interfaces!

Thanks for this i have it installed now, will report any oddities in the logs.

 

[el pescador]

Thanks Fresh.
Will do this as i have a vlan fibre connection.

Fresh do you know anything about this?

https://www.isoc.org/inet97/proceedings/F3/F3_1.HTM

 

[FreshJR]

Thanks Fresh.
Will do this as i have a vlan fibre connection.

Fresh do you know anything about this?

https://www.isoc.org/inet97/proceedings/F3/F3_1.HTM

That article is a little dated and shouldn’t apply.

TCP transfers initially have slow start and the server keeps ramping up speeds until it detects packet loss.

Once it hits packet loss, speed would ramp down as a function of packet loss until equilibrium is reached.

Article stated that concurrent tcp connections would have high possibility ramp up/down in sync.

With that said, the findings in the article is are longer valid.

Now a days we also have ECN information instead of only limiting speeds as a function of packet less. CoDel on QOS takes this a step further and does intelligent queuing based both ECN and packet age. This changes the dynamic entirely.


---

If you are concerned with UDP traffic you should first understand that it cannot really be rate limited / QOS’d. Only TCP can be flow controlled.

In my head sticking all UDP traffic into NetControl makes perfect sense. This would cause QOS traffic rate limiting techniques to only apply to TCP traffic.

Ultimatey, I decided not to mess with what isn’t broken.

I decided that for us UDP traffic priority isn’t a problem. Most UDP should primarily be VoIP & Gaming. This UDP traffic should never be getting dropped by our QOS setup since these categories have high priority assigned in relation to bulk traffic.

As such the effects of sticking UDP into NetControl would not make a difference for most users. Instead they would lose details when viewing traffic usuage by data type via QOS statistics.

Feel free to correct me if I am wrong in my line of thinking between QOS rate limiting and UDP traffic. I believe I am correct from what I have read so far.

As always, feel free to experiment and post your findings. This thread is about inner workings and improvements of QOS. Suggestions don’t have to come from me.

 

[el pescador]

Thanks.my main concern is udp packet loss.
I know ea dont show it properly...bf1 netgraph for example...only shows if its over a second.

Was suspicious when i realised udp loss is always over zero lol

I presume that paper really only applies if i have a heavily congested network so with router capacity and bandwidth i guess that why you say its not relevant.
From our side at least.

What about congested networks en route?

Also how can i tell a udp packet size being used by say FIFA without wireshark?

Want to go armed with info before i start questioning or shaming them lol.

 

[FreshJR]

Thanks.my main concern is udp packet loss.

What about congested networks en route?

It is not likely packet loss is occuring because the network is congested en route. (Who knows)

It's most likely your ISP throwing out packets because you are exceeding your allotted speeds.
This would mean QOS is not properly limiting speeds prebottleneck.

If it's not your ISP, then it means QOS is throwing out gaming packets to stay within your alloted speeds instead of bulk traffic.
This is weird since it should target bulk traffic first.

Are we talking about PC or Console?

The following code will push to system log the details about:
{ 1 random incoming download, udp packet, destined towards device 192.168.2.100, each second }

iptables -I POSTROUTING -o br0 -t mangle -p udp -d 192.168.2.100/32 -m limit --limit 1/s -j LOG

To disable the logging run

iptables -D POSTROUTING -o br0 -t mangle -p udp -d 192.168.2.100/32 -m limit --limit 1/s -j LOG

---

When viewing the log, just focus on the mark.

Mark 0x8008 ???? will send that traffic into "Gaming"
Mark 0x8000 0000 will also send that traffic into "Gaming" if the custom gaming rule enabled for that device

 

[FreshJR]

QOS + VPN Client fix via custom rules.

Tunneled VPN traffic when the router is acting as a VPN client does not appear on the upload/download portions of QOS correctly.

You have to understand the difference between a router acting as a VPN Server or acting as a VPN Client before using this rule.

Spoiler: Difference between VPN Client & VPN Server

The router has the ability to act as a VPN Client on behalf of specific LAN devices.

This means that:

1) Local devices connect to the router as normal
2) A portion of these devices (depending on configuration) will have their entire communications forwarded to a private VPN Server **typically paid**
3) The VPN server will fetch/receive data from the destination/website that the local device wanted to communicate with
4) The VPN server will return the data, in encrypted form, to the local device.

This configuration is typically used when you dont want your ISP to snoop on the contents/destination of your traffic.
​

The router also has the ability to act as a VPN Server

This is the reverse of the above situation:

1) You are on a work/public WiFi network and you dont want your work/public network admin to spy on you.
2) Instead of paying for a private VPN Server, you can have your router host one for you, and tunnel your traffic via your local ISP
3) Your traffic will remain private from whatever network you are connected too.
4) As an added bonus, when connecting to your own hosted VPN, you can access your LAN devices just as if you never left your network.

This configuration is typically used when you want to access LAN devices or maintain privacy on a network away from your own.​

Custom Rules for when Router is acting as a VPN Client

Spoiler: VPN Client

These custom rules fix tunneled traffic from being zero-rated (whitelisted) or have traffic in the wrong section (reversed) when using QOS.

Download:
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -j MARK --set-xmark 0x80000000/0xC0000000  &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -j MARK --set-xmark 0x80000000/0xC0000000

"Incoming/download" traffic is being whitelisted/reversed on the download interface since Asus is marking "download" traffic with an "upload" mark.
The rule above switches the "upload" mark back into a "download" mark.
Since the original mark is preserved, download traffic will remain correctly identified by traffic type.

192.168.2.100/32 should be changed to match the iprange of your tunneled devices​

Upload:
iptables -D OUTPUT -t mangle -o $wan ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
iptables -A OUTPUT -t mangle -o $wan ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark_up}

The upload mark assigned to upload traffic is completely lost after entering the VPN tunnel.
As a result, we have to lump ALL upload vpn traffic into a fixed user defined category.

192.168.2.1 should be changed to the IP of your router

-p udp --dport 25000 should be changed to the protocol and port of your VPN host/provider.​

Enjoy!

Custom Rules for when Router is acting as a VPN Server

Spoiler: VPN Server

No solution here

This might not be possible to fix because download traffic in this configuration is never forwarded to the br0 interface.
Traffic has to transverse the br0 interface to have its download limited via QOS.

You can read this post of mine for a detailed explanation.
https://www.snbforums.com/threads/just-a-quick-question-about-qos.47220/

If someone has ideas how to fix this via routing changes, please go right ahead!
That area is beyond my skill level.

 

[skeal]

Great news, thanks to @Martineau for providing a VPN server, I have been able to develop another QOS fix.
Some of you ** damn pirates ** are going to be happy with this one.

As you might of noticed, the tunneled VPN traffic when the router is acting as a VPN client does not appear on the upload/download portions of QOS.

You have to understand the difference between a router acting as a VPN Server or acting as a VPN Client before attempting to use this rule.

Spoiler: Difference between VPN Client & VPN Server

The router has the ability to act as a VPN Client on behalf of specific LAN devices.

This means that:

1) Local devices connect to the router as normal
2) A portion of these devices (depending on configuration) will have their entire communications forwarded to a private VPN Server **typically paid**
3) The VPN server will fetch/receive data from the destination/website that the local device wanted to communicate with
4) The VPN server will return the data, in encrypted form, to the local device.

This configuration is typically used when you dont want your ISP to snoop on the contents/destination of your traffic.
​

The router also has the ability to act as a VPN Server

This is the reverse of the above situation:

1) You are on a work/public WiFi network and you dont want your work/public network admin to spy on you.
2) Instead of paying for a private VPN Server, you can have your router host one for you, and tunnel your traffic via your local ISP
3) Your traffic will remain private from whatever network you are connected too.
4) As an added bonus, when connecting to your own hosted VPN, you can access your LAN devices just as if you never left your network.

This configuration is typically used when you want to access LAN devices or maintain privacy on a network away from your own.​

Custom Rules for when Router is acting as a VPN Client

Spoiler: VPN Client

These custom rules fix tunneled traffic from being zero-rated (whitelisted) on QOS.

**THESE RULE ARE NOT APPLICABLE WHEN THE ROUTER IS ACTING AS A VPN SERVER**

Download:

iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -j MARK --set-xmark 0xC0000000/0x80000000

"Incoming/download" traffic is being whitelisted since Asus is marking "download" traffic with an "upload" mark.
The rule above switches the "upload" mark back into a "download" mark.
Since the original mark is preserved, download traffic will remain correctly identified by traffic type.

192.168.2.200/32 should be the iprange of your tunneled devices​

Upload:

iptables -A POSTROUTING -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

The upload mark assigned to upload traffic is completely lost after entering the VPN tunnel.
As a result, we have to lump ALL upload vpn traffic into a fixed user defined category.

192.168.2.1 should be changed to the IP of your router
-p udp --dport 25000 should be changed to the ports of your VPN host/provider.​

Enjoy!

Custom Rules for when Router is acting as a VPN Server

Spoiler: VPN Server

No solution here

This might not be possible to fix because download traffic in this configuration is never forwarded to the br0 interface.
Traffic has to transverse the br0 interface to have its download limited via QOS.

You can read this post of mine for a detailed explanation.
https://www.snbforums.com/threads/just-a-quick-question-about-qos.47220/

If someone has ideas how to fix this via routing changes, please go right ahead!
That area is beyond my skill level.

I get this when running the second iptables command at ssh prompt:

iptables v1.4.15: unknown option "--set-mark"

 

[FreshJR]

I get this when running the second iptables command at ssh prompt:

iptables v1.4.15: unknown option "--set-mark"

You cant run the second iptables command directly via SSH since the variable ${Downloads_mark} is only defined when executed from the FreshJR_QOS script itself.

For testing feel free to use this command temporarily.

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark 0x40030001

After testing, both rules should go within the upload/download portions of the script itself.

 

[skeal]

You cant run the second iptables command directly via SSH since the variable ${Downloads_mark} is only defined when executed from the FreshJR_QOS script itself.

For testing feel free to use this command temporarily.

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark 0x40030001

After testing, both rules should go within the upload/download portions of the script itself.

The above command you mentioned to test and the other lines; can i change {Downloads_mark} to {Streaming_mark} ? For instance:

iptables -A POSTROUTING -t mangle -o br0 -d 192.168.14.67/32 -j MARK --set-xmark 0xC0000000/0x80000000

 

[FreshJR]

The above command you mentioned to test and the other lines; can i change {Downloads_mark} to {Streaming_mark} ? For instance:

iptables -A POSTROUTING -t mangle -o br0 -d 192.168.14.67/32 -j MARK --set-xmark 0xC0000000/0x80000000

Come again

This command

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark 0x40030001

is equivalent to

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

I just manually expanded ${Downloads_mark} --> 0x40030001 so you would be able run in directly from SSH as you requested.

Yes you can change ${Downloads_mark} into whatever you perfer for your personal use.

All UPLOAD tunneled traffic will be counted against whatever category you define. This is your choice.

 

[waeking]

Asus Router 68U with version 384.5 firmware

Download:

iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -j MARK --set-xmark 0xC0000000/0x80000000

I am getting this error :
iptables v1.4.15: unknown option "--set-xmark"

Upload:

iptables -A POSTROUTING -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

I am getting this error :
iptables v1.4.15: unknown option "--set-mark"

Do I need to install entware's iptables?

 

[FreshJR]

Asus Router 68U with version 384.5 firmware

Download:

iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -j MARK --set-xmark 0xC0000000/0x80000000

I am getting this error :
iptables v1.4.15: unknown option "--set-xmark"

Upload:

iptables -A POSTROUTING -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

I am getting this error :
iptables v1.4.15: unknown option "--set-mark"

Do I need to install entware's iptables?

RMerlin v384.5??

Those commands should work out of the box. I am stumped!

I personally cannot upgrade past 384.4. There is no reason those commands stopped working on the v384.5 update

As for --set-mark, that has to be ran from the script itself unless you manually expand the ${Downloads_mark} variable. See @skeal's question for the same exact issue.

 

[waeking]

RMerlin v384.5??

Those commands should work out of the box. I am stumped!

I personally cannot upgrade past 384.4. There is no reason those commands stopped working on the v384.5 update

As for --set-mark, that has to be ran from the script itself. See @skeal same exact issue.

Sorry I deleted this post and never hit the post reply button, so I thought..... It should not have been posted.
I mistyped and didnt include the "-j MARK"

However This solution works awsome. The graphs in the RMerlin don't represent what is actually going on. but the actual qos is working splendidly!

 

[FreshJR]

However This solution works awsome. The graphs in the RMerlin don't represent what is actually going on. but the actual qos is working splendidly!

RMerlin's charts are always a true representation of the QOS system.
Are both oth the upload & download charts are not showing activity?

If so, then I would say the rules are not working at all in your case.

Did you set up the variables matching your configuration.

 

[skeal]

Come again

This command

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark 0x40030001

is equivalent to

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

I just manually expanded ${Downloads_mark} --> 0x40030001 so you would be able run in directly from SSH as you requested.

Yes you can change ${Downloads_mark} into whatever you perfer for your personal use.

All UPLOAD tunneled traffic will be counted against the category you define.

Can I change this:

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Downloads_mark}

To:

iptables -A OUTPUT -t mangle -o eth0 ! -s 192.168.2.1 -p udp --dport 25000 -j MARK --set-mark ${Streaming_mark}

or is this:

0x40030001

not right for streaming.

 

[waeking]

Then I am sorry to say the rules are not working in your case.
RMerlin's charts are always a true representation of the QOS system.

Both the upload & download charts are not showing activity?
Were you sure to set up the variables to match your configuration.

All is still showing as upload

In my opinion the graphs are nice to look at, but the functionality is what counts. So far seems to be functioning just fine. will test more on Monday. I will be out of the office till then. Thanks!

 

[FreshJR]

All is still showing as upload

In my opinion the graphs are nice to look at, but the functionality is what counts.

When I did not have the rules applied, both upload && download traffic did NOT show activity via QOS statistics.

RMerlin's graphs are always correct.

We can continue this discussion via PM. I think your configuration isn't working.