x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware (1-Nov-2020)

  • ATTENTION! You'll notice a Prefix dropdown when you create a thread. If your post applies to one of the topics listed, please use that Prefix for your post. When browsing the thread list you can use the Prefix to filter the view.
  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

Xentrk

Part of the Furniture
"ipset destroy TEMP" issued the "ipset v7.6: Set cannot be destroyed: it is in use by a kernel component" message

all the other commands you mentioned responded with correct results
I'll go ahead and make the small change I proposed. Sounds like there was a hiccup of some kind. I'll take a look to see if I need to add some error trapping.
 

Xentrk

Part of the Furniture
Hi Xentrk,

first let me thank you as usual for your replies.

I have condensed all my answers in a single post.

Policy Rules (strict) has always been active for me. My first question is how do you analyse dnsmasq.log? I know how to follow it from Diversion (or use the tail command you gave) but is there a way to analyse it retrospectively? I see you use a grep command below, is there some sort of editor I can use? Or is the grep the best way to query dnsmasq.log?

I fully understand what you were doing for HBO and this is the same I was trying for Netflix but, for some reason, it wasn't adding the entries to the IPSET (I guess this was due to the funny looking entries in the dnsmasq config file as, once cleansed, it started working again).



I watched Netflix before and after. I realised the entries were being added when watching it but, for some reason, the counter was stuck at 6 entries, until I did all the cleansing. I will monitor if this happens again. I am very new to Unix scripts etc so it may well be that I was doing something wrong. And I understand why I don't need both the dnsmasq and the ASNUM.

The above is very useful, thanks! I assume I will get the new autoscan once you release it in GitHub.

All the rest (Amazon and Disney+) are working well so I am not planning to change anything there.

Thanks again!
autoscan.sh has been updated and posted to GitHub. Please see the announcement on this post for usage instructions.

Either the ASN and dnsmasq methods I have posted on GitHub for Netflix will work. You don't need both as it is redundant.

The method comes to down several factors. For example, the ASN for Amazon US will work with both Amazon Prime traffic and Disney. For some, it may create an issue. So using the dnsmasq method is just a way to get very specific where as the ASN method may cast too wide of a net. Or, Content Delivery Netwok may prevent ASN method from working.

Here is what I ran into with HBOMAX. When I use the ASN Lookup Tool available in x3mRouting, the HBO site belongs to ASN 14618 and 16509. But I already route all AWS traffic to my private IP yet HBO was flagging an error.

Code:
asn hbomax.com

-----------------------------
| ASN lookup for hbomax.com |
-----------------------------

- Resolving "hbomax.com"... 6 IP addresses found:

  52.205.19.251 +PTR ec2-52-205-19-251.compute-1.amazonaws.com
                +ASN 14618 (AMAZON-AES, US)
                +ORG Amazon.com, Inc.
                +NET 52.200.0.0/13 (AT-88-Z)
                +ABU [email protected]
                +GEO Ashburn, Virginia (US)

35.167.130.181 +PTR ec2-35-167-130-181.us-west-2.compute.amazonaws.com
                +ASN 16509 (AMAZON-02, US)
                +ORG Amazon.com, Inc.
                +NET 35.160.0.0/13 (AMAZO-ZPDX9)
                +ABU [email protected]
                +GEO Portland, Oregon (US)

    52.24.41.24 +PTR ec2-52-24-41-24.us-west-2.compute.amazonaws.com
                +ASN 16509 (AMAZON-02, US)
                +ORG Amazon.com, Inc.
                +NET 52.24.0.0/14 (AT-88-Z)
                +ABU [email protected]
                +GEO Portland, Oregon (US)
<snip>

So, that is what lead to the updates to autoscan.sh. I needed to see the FQDN for the reply records to see what was going on. It was then that I saw domains owned by content delivery network. To do the mining, I forced all traffic to use VPN Client 1 (my private IP) and selected all of the options on the HBOMAX website and clicked around on the streaming app. From that exercise, the script told me the top level domain names I needed to use for the dnsmasq method. The reason I also added the FQDN for the reply records is my main router is a pfSense appliance and it has a feature to route by FQDN. It did not work with just the query records. I also had to include the reply records. But I also found it gave me a better idea of what is going on. Now, sometimes we get lucky and find that only one term is need to query dnsmasq to find the records e.g. "hbo". But sometimes, there may be other records the streaming app generates that don't contain the term. In that case, the getdomainnames.sh script or the follow the log file option in diversion can help shed light on other domain names being queried.
 

CannaLucente

Occasional Visitor
Hi Xentrk,

thanks for the explanation.
Interestingly for Netflix the ASN wasn't working at all for me, I was only able to see a single movie :-D the 2906 created 146 entries in the IPSET while the dnsmasq has created 800+ as of now.
Just run the update in amtm and got the new autoscan.sh, really helpful to see the new FQDN information.

Cheers.
 

Xentrk

Part of the Furniture
Hi Xentrk,

thanks for the explanation.
Interestingly for Netflix the ASN wasn't working at all for me, I was only able to see a single movie :-D the 2906 created 146 entries in the IPSET while the dnsmasq has created 800+ as of now.
Just run the update in amtm and got the new autoscan.sh, really helpful to see the new FQDN information.

Cheers.
You reminded me that I ran into a similar issue a few months back and added the dnsmasq method for netflix to fix the issue and kept the AS2906. But I didn't have to make a similar change on my pfSense appliance. Things do change though and it may require another analysis to see what is going on. Adding amazonaws.com or amazonvideo.aws may be required. I'll take another look at it.
 

Xentrk

Part of the Furniture
Following are the Netflix domains I mined using a Fire TV and Web Browser yesterday.

Code:
dnsmasq=netflix.com,netflix.net,nflxext.com,nflximg.com,nflxso.net,nflxvideo.net

The netflix.net only appeared if I accessed NF from a browser. As a result, I'll do some more analysis and stream using iOS, Roku and Android Box to make sure I have collected all of the domains.

I'll update the README and summarize the domains for the services I use in a new section so it's not buried in the text and solicit input from others to help us with all of our selective routing needs. Stay tuned.
 

CannaLucente

Occasional Visitor
I haven't done testing as detailed as yours but entering the same 6 domains using the dnsmasq method, my IPSET site has currently 1225 entries (vs the 146 I get from asnum 2906). I don't get the netflix unblocker/proxy error any longer... I only access via Fire TV or dedicated app on the TV.

The error on firetv only comes up if you block 8.8.8.8 on the router as otherwise it's hardcoded in the netflix app...

Thanks for the updates! Let me know if there's any testing I can help you with...
 

Xentrk

Part of the Furniture
I haven't done testing as detailed as yours but entering the same 6 domains using the dnsmasq method, my IPSET site has currently 1225 entries (vs the 146 I get from asnum 2906). I don't get the netflix unblocker/proxy error any longer... I only access via Fire TV or dedicated app on the TV.

The error on firetv only comes up if you block 8.8.8.8 on the router as otherwise it's hardcoded in the netflix app...

Thanks for the updates! Let me know if there's any testing I can help you with...
Thanks for the confirmation.

The ipset size difference is because the ASN method uses CIDR format (e.g. 23.246.0.0/18) which represents a range of IPv4 addresses whereas the dnsmasq method collect individual IPv4 addresses assigned to the domain (there can be more than one).

I force all LAN clients to use the DNS of the router via the DNSFilter option rather than explicitly blocking 8.8.8.8
 

MarcoPolo

Occasional Visitor
I have just discovered and installed (directly version 2) this wonderful tool!
Thank you for creating this and maintaining it :)

If I can allow myself a feedback:

- After the initial installation the firewall-start file was no longer executable, I had to do a "chmod".

- In the documentation there seems to be some small mistakes:
In the examples given, sometimes "ipset_name=" is missing as in "x3mRouting AMAZON autoscan=amazon" and "x3mRouting WIMIPCOM ip=104.27.198.90,104.27.199.90".

- I had a quota problem with " " during my initial tests. I didn't immediately understand why the IPSETs were empty when I used "x3mRouting ipset_name=XXXX asnum=xxxxx,yyyyy,zzzzzz":
"curl -fsL --retry 3 --connect-timeout 3 "https://api.hackertarget.com/aslookup/?q=xxxx" --> "API count exceeded - Increase Quota with Membership"
How to modify the code of x3mRouting.sh to be able to use the Hackertarget APIs with a membership?

On the same subject, I read in the documentation "The IPv4 addresses are downloaded from ipinfo.io. ipinfo.io may require whitelisting if you use an ad-blocker program. If x3mRouting is unable to download the IP addresses from ipinfo.io, it will attempt to download using the aslookup tool on api.hackertarget.com/aslookup/" but I can't find any reference in the code to ipinfo.io: i have the feeling that the unique and default method is with the Hackertarget APIs (which has quotas, so).
Did I miss something?

Thank you
 
Last edited:

Xentrk

Part of the Furniture
I have just discovered and installed (directly version 2) this wonderful tool!
Thank you for creating this and maintaining it :)

If I can allow myself a feedback:

- After the initial installation the firewall-start file was no longer executable, I had to do a "chmod".

- In the documentation there seems to be some small mistakes:
In the examples given, sometimes "ipset_name=" is missing as in "x3mRouting AMAZON autoscan=amazon" and "x3mRouting WIMIPCOM ip=104.27.198.90,104.27.199.90".

- I had a quota problem with " " during my initial tests. I didn't immediately understand why the IPSETs were empty when I used "x3mRouting ipset_name=XXXX asnum=xxxxx,yyyyy,zzzzzz":
"curl -fsL --retry 3 --connect-timeout 3 "https://api.hackertarget.com/aslookup/?q=xxxx" --> "API count exceeded - Increase Quota with Membership"
How to modify the code of x3mRouting.sh to be able to use the Hackertarget APIs with a membership?

On the same subject, I read in the documentation "The IPv4 addresses are downloaded from ipinfo.io. ipinfo.io may require whitelisting if you use an ad-blocker program. If x3mRouting is unable to download the IP addresses from ipinfo.io, it will attempt to download using the aslookup tool on api.hackertarget.com/aslookup/" but I can't find any reference in the code to ipinfo.io: i have the feeling that the unique and default method is with the Hackertarget APIs (which has quotas, so).
Did I miss something?

Thank you
Thanks for the comments on the documentation. I updated the README. ipinfo.io is no longer used so I removed the reference. ipinfo.io was limiting my downloads during the development when I was hitting it hard. So I switched to hackerytarget.com. I don't trap the error you describe but will have to do so now based on your feedback or look at other options. If you are using ASN method to get IPv4 addresses belonging to Amazon, you can change to the aws_region method instead as the source is a json file hosted by AWS. I'll research the amount of downloads allowed. x3mRouting does change permission on firewall-start if x3mRouting creates it. But not if the file already exists when x3mRouting inserts a line of code. I'll do some analysis on it. Thank you.
 

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top