Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

Discussion in 'Asuswrt-Merlin' started by FreshJR, Jan 12, 2017.

  1. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    Lol - a huge spike in Net Control. Clearly ASUS is misclassifying that traffic. Probably because it's google.
     
  2. Please support SNBForums! Just click on this link before you buy something from Amazon and we'll get a small commission on anything you buy. Thanks!
  3. medwatt

    medwatt Regular Contributor

    Joined:
    Jun 13, 2017
    Messages:
    54
  4. brummygit

    brummygit Regular Contributor

    Joined:
    Jan 19, 2013
    Messages:
    164
    My RT-AC68U on 382.3 Alpha 1 sees this as Web Traffic using signature file 2.050. What signature version are you running?

    @deddc23efb I'm not yet using the new FreshJR script
     
  5. medwatt

    medwatt Regular Contributor

    Joined:
    Jun 13, 2017
    Messages:
    54
    Mine is 1.200 (I'm on firmware 380.69) and it gets identified as Net Control Packets.
     
  6. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    I poked around a little bit. The 0x8014 classification is an indication of the application class. The 8 is direction and the actual category is 0x14 == 20 decimal. You can grep through the db files in /tmp/bwdpi/*.db to get a glimpse of what ASUS is doing. That classification is for Network protocols.

    If you're willing to risk actual network protocols, you could delete that rule and write a new rule that mapped 0x8014 to the web treatment (it looks like you have web set to second priority? 0x11? There are three level of Network protocol classification (18,19,20) that all get mapped to 1:10 treatment.

    I think this is truly a problem with the Trend Micro dpi.

    <doh - 8013 not 8014 in all of this.>
    /dedd
     
    Last edited: Jan 13, 2018
  7. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    That's very interesting. My signature file is 2.050 as well.
    The mapping info I was looking at is in the /tmp/bwdpi directory. I'm completely guessing at the number that are thinking about (hex 14 == 20 == things in the db files) . Maybe those files aren't updated by a signature file update and the problem has been fixed in 382.3?

    I disabled my FreshJr script and tried that download with after a clean reboot and it still goes to Net Control. I suspect this is a bug that ASUS has fixed and it requires an upgrade to something after 382.1_2 - sounds like 382.3 has the fix. Thanks brummygit!

    /dedd
     
  8. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    @medwatt On the previous versions we noticed that ASUS was missing a rule for "mark 0x80130000".

    We noticed this because a lot of HTTPS traffic was bypassing QOS entirely. As such we found the missing rule for that classification and created it as shown below, in previous versions of the script.

    Code:
    ${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic    (rule missing from ASUS default rule set)
    
    Fast forward to today, On ASUS's updated firmware this rule is not missing anymore. You can see it present here.

    Code:
    filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:10
      mark 0x80130000 0xc03f0000 (success 9471)
    
    Since we saw it was not missing on current firmware, I have removed my original, now duplicate, rule. As you pointed out, ASUS has decided to classify 0x8013 traffic as "Net Control", while I decided it was more appropriate to stick that traffic into "Web Surfing".

    I didn't notice that ASUS decided to choose Net Control when the introduced the previously missing rule. I still think Web Surfing is way more appropriate. The old rule is coming back from as was setup on previous version of the script.

    The "Web Surfing" rule is coming back. Thanks for pointing it out.

    @Nodiaque I bet you downloaded the file from pastebin and uploaded it to the router. The below is in caps so it grabs other peoples attention who did the same thing.

    THE PASTEBIN DOWNLOAD ADDS INCOMPATIBLE "NEW LINE" CHARACTERS. THIS MAKES THE SCRIPT NOT EXECUTE IN LINUX/SHELL.

    Read the first post on how to convert the New Line characters using Notepad++. On previous versions, you probably downloaded the supplied ".txt" file on the forum. That one was directly uploaded from my computer so it had correct new line characters.
     
    Last edited: Jan 13, 2018
    Vexira likes this.
  9. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    If brummygit is correct, then ASUS has done something about this in 382.3 and classifying true network protocol traffic as web might be undesirable (or whatever ASUS is shoving in that class in 382.3) Something to watch out for.

    People can fix their files using dos2unix in the ASUS cli as well.
     
  10. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    Maybe, maybe not.

    In 382.3 maybe @brummygit received a definitions update that specifically identifies that specific HTTPS traffic stream as websurfing.

    We would have to test other HTTPS traffic streams to see if they don't fall into Net Control.

    @brummygit can you post how you 0x8013 rule is setup from TC filter show dev br0?

    Can you run a dslreports (https) version speedtest

    Asus just might be identifying google drive HTTPS traffic. We should confirm.

    While it goes into net control, I don't think it's actually syn and awks getting caught by 0x8013. It didn't create any issues before.

    edit:

    • on 382.2 signature version 2.050, the google docs document is "Web Surfing" and https dsl reports speedtest is "Net Control". Confirms my suspicions.
    @medwatt The updated version of the script is below.

    v382.2 version of FreshJR_QOS (typo present, use v382.3)
    https://pastebin.com/pA10JSXT

    *now put unidentified https traffic into "Web Surfing" instead of "Net Control" like old versions of the script.
     
    Last edited: Jan 18, 2018
    Vexira likes this.
  11. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    Oddly enough on my system the Google TLS traffic is collecting against 0x8014

    filter parent 1: protocol all pref 12 u32 fh 803::800 order 2048 key ht 803 bkt 0 flowid 1:10
    mark 0x80090000 0xc03f0000 (success 299)
    --
    filter parent 1: protocol all pref 21 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:10
    mark 0x80120000 0xc03f0000 (success 58)
    --
    filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:10
    mark 0x80130000 0xc03f0000 (success 33897)
    --
    filter parent 1: protocol all pref 23 u32 fh 804::800 order 2048 key ht 804 bkt 0 flowid 1:10
    mark 0x80140000 0xc03f0000 (success 249435)


    htpps://www.dslreports.com/speedtest goes to web surfing for me R382.1_2 .
     
    Vexira likes this.
  12. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    That's weird. I still decided to revert it. I ran for over a year with 0x8013 going into web surfing instead of net control.
    On previous versions of ASUS's firmware that rule was missing entirely, LOL.
     
  13. medwatt

    medwatt Regular Contributor

    Joined:
    Jun 13, 2017
    Messages:
    54
    If you can recall, most of my unidentified traffic when downloading was collected against 0x8014 in earlier firmwares. They are now identified against 0x8013 in 382.2
     
  14. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    Are you sure?

    This line is straight from v1.92 in the first post.

    Code:
    ${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic  (rule missing from ASUS default rule set)
    
    It was clearly 0x8013 that was missing and giving issues.
    While on the other hand 0x8014 is actual net control that should not be touched.

    If https traffic is now collected against 8013 and going into net control, then it will go into web downloads now. That is the same behavior I am currently experiencing and the same behavior I experienced in the past.
     
    Vexira likes this.
  15. medwatt

    medwatt Regular Contributor

    Joined:
    Jun 13, 2017
    Messages:
    54
    I did the changes myself. You can check the private conversation I had with you in the past for a pastebin file. Compare it with the one I posted today and you'll see.
     
    Vexira likes this.
  16. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
    Yeah I think that was why I was confused earlier too. I'm definitely seeing it go to 0x8014 on 382.1_2. I just disabled FreshJR_QOS and restarted to make sure things were clean and that's what I get. I haven't tried upgrading to anything newer.

    I installed the same rule as you are recommending FreshJR, except for 0x8014 and it works fine (as expected.) Not sure what else is getting pushed down to Web rather than Net. This is the problem with ASUS's approach here.

    /dedd
     
  17. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    Can you post a few links that get pushed to 8014. This is odd/
     
    Vexira likes this.
  18. deddc23efb

    deddc23efb Regular Contributor

    Joined:
    Nov 28, 2017
    Messages:
    66
    Location:
    Canada
  19. FreshJR

    FreshJR Very Senior Member

    Joined:
    Oct 8, 2016
    Messages:
    530
    Currently https traffic from google drive is hitting 0x800d & https version of dslreports is hitting 0x8013.

    This grep filter works great to compare success hits upon fresh QOS restart with low traffic.

    Code:
    tc filter show dev br0 | grep "mark" | grep -v "success 0"
    
    Also I made a typo in script v382.2. Ninja edit incoming. (Use v382.3)
     
    Last edited: Jan 18, 2018
    Vexira and Gitsum like this.
  20. brummygit

    brummygit Regular Contributor

    Joined:
    Jan 19, 2013
    Messages:
    164
    Code:
    filter parent 1: protocol all pref 22 u32
    filter parent 1: protocol all pref 22 u32 fh 802: ht divisor 1
    filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt                                                                                                              0 flowid 1:10
      mark 0x80130000 0xc03f0000 (success 1930057)
    I can't run a speed test at the moment as my network is very busy
     
    Vexira likes this.
  21. h3lmut

    h3lmut Occasional Visitor

    Joined:
    Oct 20, 2013
    Messages:
    24
    dsl reports with https is in "net control" category with this script

    with ac56u 382.2 Beta 3 (released today)

    signature version: 2.050 Updated : 2018/01/06 01:00
     
    Last edited: Jan 17, 2018
Please support SNBForums! Just click on this link before you buy something from Amazon and we'll get a small commission on anything you buy. Thanks!