1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

Discussion in 'Asuswrt-Merlin' started by FreshJR, Jan 12, 2017.

  1. Therion87

    Therion87 Occasional Visitor

    Joined:
    Oct 13, 2015
    Messages:
    48
    Would moving VoIP further down be fine if you do not have dedicated VoIP? I take this statement to mean that enabling WiFi Calling on your cellphone does NOT fall into the VoIP category. What would be some real scenarios that would fall into that category for an average home user? Would this be things like FaceTime and Skype?

     
  2. Please support SNBForums! Just click on this link before you buy something from Amazon and we'll get a small commission on anything you buy. Thanks!
  3. NGRhodes

    NGRhodes Regular Contributor

    Joined:
    Oct 3, 2014
    Messages:
    82
    Location:
    West Yorkshire, UK.
    Hi,

    It may be worth noting that there in release/src/asustools/squashfs-root/usr/sbin/ there is a svqos2 which is now utilising hfsc for the AC86U 3.0 kernel; the code looks pretty identical apart from this change, probably something that needs to be tested.

    Also, I was looking at how we could achieve per-host isolation and stumbled on this (https://serverfault.com/questions/7...dwidth-per-ip-using-htb-a-cidr-range-in-linux), which also turns out how Gargoyle does it:

    Code:
    tc qdisc add dev eth0 root handle 1: htb default 12
    tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit ceil 100mbit
    # Here, you want a fair qdisc
    tc qdisc add dev eth0 parent 1:1 handle 101: sfq perturb 10
    # Put a range in the filter
    tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 192.168.1.0/24 flowid 1:1
    # Create 254 bucket, each ip src will be attached to one bucket
    tc filter add dev eth0 parent 1:1 protocol ip handle 10 flow hash keys nfct-src divisor 254
    The bottom line is the cool bit.
    fq_codel should do a good enough job (within each [per-ip] class), but I think it may be possible to then create sub classes in each bucket based on Adaptive QoS, might remove/reduce the need to rely on additional per application rules.
    Over the winter months I am going to try and create a proof of concept script, then the step after this would be creating something more generic that correctly considers total ip addresses available (in-use or available) and setting suitable rates and incorporating into the FreshJR script.
     
    Gitsum, Vexira and FreshJR like this.
  4. JohnSmith

    JohnSmith Occasional Visitor

    Joined:
    May 2, 2018
    Messages:
    33
    Location:
    Canada
    Yes I am sure, as I tried both dport & sport , as well as TCP and UDP on both upload rules for testing purpose, and in all 8 tests (4 tests per 2 rules), Upload traffic still got classified into "Web Surfing".

    When I ran "netstat -na" on the Window machine at a command prompt while the download was happening, it showed TCP
    On the ASUS router, when I used the Network Tools, for "netstat-nat" and by "Source IP" it shows TCP

    Are there any logging commands that I can run to determine where these Upload packets are being classified into?
     
  5. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    915
    I was planning to do something like this earlier to fix the device priority but I stopped.

    Asus already dynamically modifies the TC structure as devices enter & leave the network.

    Code:
    1:1 - root HTB
     1:10 - NetControl
        10:1 -  NetControl HTB
        10:2 - device1
        10:3 - device2
        10:4 - device3
     1:11 - VoOP
       11:1 - VoIP  HTB
        11:2 - device1
        11:3 - device2
        11:4 - device3
     1:12
     1:13
     1:14
     1:15
     1:16
    
    I was planning to intercept Asus's TC commands for the sub structure and fix the rate/ciels/ & priority for each device to my liking.
    I find that the TC parameters Asus is using to dynamically implement the per device priority in the substructure leads to a terrible end result (this is my opinion).

    Calculating and changing rate/priority per networked device via the existing structure wasn't an issue at all.
    I was going to piggyback off RMerlins fakeTC to intercept the commands and run some calculations to make it work. (In doing that, I did find that the QOS commands issued from the bwdpi engine issues are time sensitive before it throws errors and wipes the entire TC structure and restarts, I did find a way to work around that).

    Everything was going good but I ran into an issue.

    If my iPhone was device2. Once it left the network, the next device that connected to the network would will replace my iPhone as device2 instead of being assigned device4.

    They are filting devices via an ID assigned within the mark and not filtering by device IP. I will pull up a picture of the mark bitwise structure I created and post it here so it can be clear.

    EDIT: [​IMG]

    Since I couldn't figure out which mark corresponds with which device (as the correlation is random and tracked internally somewhere), I stopped trying at this point to fix it.


    ---

    I see that you are trying to do it in the reverse order and that will work even better. (I was trying to maintain the existing structure with different parameters, but do perfer it in the reverse order as you mentioned).

    The reason I perfer first by device and they by traffic, is that even if traffic identification is poor. Each device will always have its own guaranteed bandwidth and will use it like having a dedicated connection even if identification completely fails. The traffic prioritization that would be occuring in the subclass would be nice on per machine traffic performance but blips in performance wouldn't make or break QOS quality.

    A word of advice is you will HAVE to use a different set of class IDs for each perIP class and also for the all the perTraffic class within each perIP class so Asus dynamic modification do not interfere with your structure.

    (We could block all of Asus dynamic modifcations via fakeTC but you would then also have to cross your fingers and hope Asus doesn't try to wipe the new structure).

    My approach would be to use fakeTC to block the Asus structure entirely and then create our own. The issue is that the bwdpi executable that issues the TC commands would definitely not be happy.

    If someone can comment out the QOS check, I would be happy to join you in creating this new QOS structure that dynamically adjusts based on devices present on the network.

    Both Wifi-Calling and FaceTime were unidentified. I created special rules for both those cases. Try making some calls and see if the VoIP category grows. I believe Skype was correctly identified during the time I last tested it, but that was a few years ago.

    @JohnSmith

    Why do you keep using so many filtering elements?

    Eg.
    Code:
    iptables -D POSTROUTING -t mangle -o $wan -s X.X.X.X/32 -m mark --mark 0x801400c2/0xc03fffff -p tcp --dport 49152:65535 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -s X.X.X.X/32 -m mark --mark 0x801400c2/0xc03fffff -p tcp --dport 49152:65535 -j MARK --set-mark ${Downloads_mark_up}
    
    Could be condensed into this

    Code:
    iptables -D POSTROUTING -t mangle -o $wan -m mark --mark 0x801400c2/0xc03fffff -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -m mark --mark 0x801400c2/0xc03fffff -j MARK --set-mark ${Downloads_mark_up}
    
    And has higher chance of the match not being void by a mismatch with the source IP, transfer protocol, or ports used.

    Go ahead and use this to log packets.

    Code:
    iptables -D POSTROUTING -t mangle -o $wan -m mark --mark 0x801400c2/0xc03fffff -m limit --limit 1/s -j LOG &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -m mark --mark 0x801400c2/0xc03fffff -m limit --limit 1/s -j LOG
    
     
    Last edited: Nov 1, 2018
  6. Sinner

    Sinner Senior Member

    Joined:
    Sep 30, 2017
    Messages:
    289
    Location:
    Canada
    The facetime discord teamspeak ventrillo and alot of online games being unidentified is why i find myself using voip for my unidentified at the top and manually moving anything out i need to. Others for me is unused
     
  7. JohnSmith

    JohnSmith Occasional Visitor

    Joined:
    May 2, 2018
    Messages:
    33
    Location:
    Canada
    Thanks for the log code FreshJR! So I used it to see if on the upload, the "Lets Encrypt" traffic would get logged to show why it is being classified under the "Web Surfing" and not the "File Transferring", and no logs ever showed up in the system log. Is this not where the "LOG" command puts this information if it matches your code, or is it somewhere else?


    So I have gone back to my rule that puts this ONE computers "Lets Encrypt" traffic uploading under "File Transferring" (unlike your code above that puts ALL computers "Lets Encrypt" traffic under File Transferring, which I didn't want to happen yet unless necessary in the long run), and it works

    Code:
    iptables -D POSTROUTING -t mangle -o $wan -s X.X.X.X/32 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -s X.X.X.X/32 -j MARK --set-mark ${Downloads_mark_up}
    I was just hoping to tie it down further to not only an IP address, but to its specific ports, in case other applications get classified under the "Lets Encrypt" classification, and I would then consider that traffic to be in "Web Surfing" (or higher, depending on what it is or does).
     
  8. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    915
    Is there significant upload bandwidth being identified as "Lets Encrypt" shown in the "App Analysis" tab after you click on a specific device via the WebUI?

    I messed up the log rule. 0x8 marks are download, 0x4 marks are upload. (The log rule was trying to look download traffic on upload interface, thats why nothing hit)

    This is same reason why your original upload rule was not hitting at all. The mark prefix was not 0x4.
    I will make the results of -appDB clearer in the next release.

    Try these

    Code:
    iptables -D POSTROUTING -t mangle -o $wan -m mark --mark 0x401400c2/0xc03fffff -m limit --limit 1/s -j LOG &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -m mark --mark 0x401400c2/0xc03fffff -m limit --limit 1/s -j LOG
    
     
    Last edited: Nov 2, 2018
  9. JohnSmith

    JohnSmith Occasional Visitor

    Joined:
    May 2, 2018
    Messages:
    33
    Location:
    Canada
    Thanks FreshJR, that was exactly it (I missed that as well when looking at the "iptables -vL -t mangle", argh, it's always the smallest change), change "0x8" to "0x4" in uploads, (also used "dport" in the upload rule, as "sport" was used in the download rule) and classification is now working properly for that one computer......

    Code:
    iptables -D POSTROUTING -t mangle -o $wan -s X.X.X.X./32 -m mark --mark 0x401400c2/0xc03fffff -p tcp --dport 563 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
    iptables -A POSTROUTING -t mangle -o $wan -s X.X.X.X/32 -m mark --mark 0x401400c2/0xc03fffff -p tcp --dport 563 -j MARK --set-mark ${Downloads_mark_up}
     
  10. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    1,393
    Location:
    Australia
  11. WAN-Noob

    WAN-Noob New Around Here

    Joined:
    Nov 7, 2018
    Messages:
    2
    @FreshJR
    Hi, I am using this script on ASuS RT-AC68U with merlin's fw and it's working great. I get good enough bandwidth when my family is watching netflix. So thanks for doing all the hard work.
    I'm very noob in regards to iptables and tc and was wondering if this script could be modified to use on custom linux router? Could you give any pointers if possible?
    I use ppp to connect to internet on a debian router.
     
  12. Hallux

    Hallux Occasional Visitor

    Joined:
    Aug 29, 2014
    Messages:
    12
    Vexira likes this.
  13. Kingp1n

    Kingp1n Regular Contributor

    Joined:
    Feb 27, 2018
    Messages:
    88
    @FreshJR, I see per comments above you may be working on an updated script? Once released, wld you recommend updating to it?
     
  14. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    915
    no, the update is just text updates for the rarely used AppDB command to make the output a little more clear.
    It is so minor I have been holding off on pushing it unless I get more questions about it.

    Yes and no. The deep packet inspection system that categorizes traffic by type that TC filters on will only work on the Asus router it was licensed for.

    You can still use TC on the linux box to implement QOS by allocating bandwidth & priority per networked device instead of allocating bandwidth & priority per traffic type.

    (Alloting good limits and priority per device will give equivalent performance to limiting by traffic type. It is just a different approach. This setup will even perform better than using DPI since bandwidth will always be allocated as you expect it to compared to the situations where DPI is performing poorly, which this script tries to mitigate).
     
    Last edited: Nov 8, 2018
    Kingp1n likes this.
  15. WAN-Noob

    WAN-Noob New Around Here

    Joined:
    Nov 7, 2018
    Messages:
    2
    Thanks for clarifying that, I will try to find more resources on tc and learn which is the best way to implement for a small home.
    PS: Thanks again for your awesome script for Asus.
     
    FadgewackeR likes this.
  16. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    1,393
    Location:
    Australia
    @FreshJR is it rate an pps that mean traffic is being accounted for by the particular category or just rate, ive been trying to see if Instagram video chat is classed as voip, only seen the rate move but not pps.
     
  17. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    915
    Both should be showing up.

    You can try using AppAnalysis in the WebUI and then look it up with the AppDB command within the script. This method should have less guessing and even works when the network is busy.

    Barebones you need to perform rate limiting and then each user should have either fq-codel or cake as their queuing discipline.

    A step head is an HTB on the root where you filter each user into their own class that will have a minimum + maximum bandwidth & priority per user.

    Should be many examples online as this is common.
     
    Last edited: Nov 9, 2018
  18. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    1,393
    Location:
    Australia
    @FreshJR thank you ill look into it
     
Please support SNBForums! Just click on this link before you buy something from Amazon and we'll get a small commission on anything you buy. Thanks!