Selective Routing for Netflix

[Xentrk]

@Xentrk,

I wanted to report that the dvd.netflix .com site now loads properly from Safari, Chrome and Firefox and the updated ipset directive is also loaded automatically loaded in/jffs/configs/dnsmasq.conf.add.

As I mentioned earlier, the only issue that remains is that my DVD Netflix still cannot access this site - not sure why that is. But when I add 207.45.72.0/22 to the Policy Rules, the app works well.

Thanks again for your hard work and the prompt update of the script!

This line of code in the HTML page source references an IP address that the IPSET list is not capturing.

var _vsvrname = _csma_trimmer('website.prod.dvdco.netflix.com_s(207.45.72.90:443)

It is not a valid domain name. The sister script IPSET_Netflix.sh uses AS numbers to load the IPSET list rather than the IPSET directive in dnsmasq.conf.add. I can add the AS number 394406 to the code and it will then add 207.45.72.0/22 to the IPSET list.

For now, you have a work around by adding 207.45.72.0/22 to the OpenVPN client GUI. An alternative is to add 207.45.72.0/22 to the ipset list manually using the command below followed by the ipset save command to save the ipset list to the backup so the list contents will survive a reboot.

ipset add x3mRouting_NETFLIX_DNSMASQ 207.45.72.0/22
ipset save x3mRouting_NETFLIX_DNSMASQ > /opt/tmp/x3mRouting_NETFLIX_DNSMASQ

 

[Marin]

@Xentrk - Thank you very much!

 

[st3v3n]

With the various outages Wednesday, including FB, then Apple/iCloud (and others) over the week, it would seem whomever is inducing these episodes/problems, they're sending a clear enough message to those who follow this kind of thing, that they can do it openly and without consequences. Our iPads were not happy throughout Thursday, even though we don't use icloud. Apple usually doesn't say much until their issues are resolved, but this time it was a massive problem. Anything that touched parts of Apple's infrastructure went a little wonky; synth notes in various apps wouldn't shut off, Sound Cloud demanded users set up new accounts and suddenly old accounts were gone. Other app accounts were also blocked, and/or requested the owner sett up a new account. The app update function was the flakiest, until almost midnight central north American time. Video streaming continued, but paused or reloaded at various places. It will be interesting to see how Apple's new video streaming effort will fare. Good luck to all.

 

[vission]

@Xentrk
I think you've done a great job with your script, I'm still working on getting it to work properly against amazon prime, but that is just because I haven't had enough time to get back to it. I wanted to ask do you think what you have done could possibly be baked into the merlin code so that we could just configure this through the UI without copying scripts and editing lines of code.

Thanks!

 

[Xentrk]

@Xentrk
I think you've done a great job with your script, I'm still working on getting it to work properly against amazon prime, but that is just because I haven't had enough time to get back to it. I wanted to ask do you think what you have done could possibly be baked into the merlin code so that we could just configure this through the UI without copying scripts and editing lines of code.

Thanks!

The IPSET_Netflix.sh script in the repository will route both Amazon Prime and Netflix. Amazon Prime routing was not done intentionally when I first wrote the script. But because Netflix hosts on the Amazon AWS server farm in US, I included all of the AWS server farm addresses in US, which includes Amazon Prime. This past year, Amazon Prime also started blocking known VPN servers. IPSET_Netflix_Domains.sh uses a different technique and only includes the Amazon servers Netflix hosts on.

I do have a solution that @Martineau shared with me that is more integrated with the firmware GUI and "user friendly". See the screen shot in the post. This past weekend, I finished the coding and have it working for routing IPSET lists thru the VPN interfaces (I think some additional changes would be required to route an IPSET list thru the WAN interface though). I have to "beautify" the code do a QA check before posting to GitHub. I also need to write an installer. Unfortunately, I had to leave town on Sunday and work will be delayed until I return in approximately two weeks.

The only part that requires coding by the user is creating the IPSET list. To help simplify the process, @Martineau came up with script based solution of passing parms to the script for IPSET lists specified inside of dnsmasq to make it more user friendly for users. I made some mods so the user can also specify manually created lists or an AS number. Amazon is unique because the source of the IPv4 addresses is json file.

Following is an example what I came up with so far.

sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_AMAZON_ipset.sh

sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_MANUAL_ipset.sh BBC
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_MANUAL_ipset.sh BBC_IPLAYER
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_MANUAL_ipset.sh CBS
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_MANUAL_ipset.sh SLINGTV_LIST

sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_ASN_ipset.sh HULU AS23286
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_ASN_ipset.sh NETFLIX AS2906
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_ASN_ipset.sh SLINGTV AS35873

sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_DNSMASQ_ipset.sh CBS_WEB cbs.com,www.cbs.com,cbsnews.com,www.cbsnews.com,cbssports.com,www.cbssports.com
sh /jffs/scripts/Asuswrt-Merlin-Selective-Routing/load_DNSMASQ_ipset.sh BBC_WEB bbc.com,bbci.co.uk,bbc.co.uk

After trying all of the different firmware, pfSense really shines when it comes to selective routing. It can all be done in the Web GUI using the package pfBlockerNG to create the lists followed by defining the routing rules in the Firewall-LAN screen. The changes described above will definitely help make selective routing more user friendly for Asuswrt-Merlin Firmware users.

 

[Adamm]

@Xentrk - Thank you very much!

Policy routing is broken on the AX88U is most likely why you were having issues, I detailed the problem here.

 

[Marin]

Policy routing is broken on the AX88U is most likely why you were having issues, I detailed the problem here.

Yep, just read it. Thanks much @Adamm!

 

[Xentrk]

Issue with AX88U may have to do with a change in firmware handling of marks. Some interesting info on use of marks in the thread

https://www.snbforums.com/threads/selective-routing-with-asuswrt-merlin.9311/page-23#post-241676

Definitely not my area of expertise. A search on “0x” and “fwmark” in the AsusWRT Merlin repo returns many matches.

 

[sone]

Thank you again for this amazing script as it's been working great for me. I do have one weird problem.. since using it every once in a while duckduckgo.com searches go through the WAN instead of the VPN. You can tell by typing What's My IP in DDG search and sometimes it'll show the WAN IP, but if you click on a link that provides IP check from DDG.. it'll show the VPN IP. I can only guess that DDG is using Amazon Web Services/IP's that Netflix uses. Is there any way to circumvent this so DDG searches always through the VPN while keeping the script running to route Netflix traffic through WAN?

thanks

 

[Xentrk]

Thank you again for this amazing script as it's been working great for me. I do have one weird problem.. since using it every once in a while duckduckgo.com searches go through the WAN instead of the VPN. You can tell by typing What's My IP in DDG search and sometimes it'll show the WAN IP, but if you click on a link that provides IP check from DDG.. it'll show the VPN IP. I can only guess that DDG is using Amazon Web Services/IP's that Netflix uses. Is there any way to circumvent this so DDG searches always through the VPN while keeping the script running to route Netflix traffic through WAN?

thanks

Hello @sone, glad the script is helping you. There are two scripts in the repository that one can select from, IPSET_Netflix.sh and IPSET_Netflix_Domains.sh. Which one are you using? Also, what is your router model and firmware version?

Depending on the script, here is how to analyze:

First, do an nslookup on duckduck.go. Here are my results:

Non-authoritative answer:
Name:    duckduckgo.com
Addresses:  50.18.200.106
          54.241.2.241

Then see if one of the duckduckgo.com IPv4 addresses somehow inserted itself in the IPSET list. Depending on the script you used, the commands are below, where x.x.x.x is the IPv4 address returned by the nslookup above.

IPSET_Netflix.sh

ipset -L x3mRouting_NETFLIX | grep x.x.x.x
ipset -L x3mRouting_AMAZONAWS | grep x.x.x.x

IPSET_Netflix_Domains.sh

ipset -L x3mRouting_NETFLIX_DNSMASQ | grep x.x.x.x

If you do find that the IPv4 address somehow inserted itself in the IPSET list, you can delete it using the command below by specifying the IPSET name and the IPv4 address:

ipset del IPSET_NAME x.x.x.x

But we also need to determine how duckduckgo.com IPv4 address got inserted into the IPSET list. It also appears that whatismyip.com is also going thru the WAN. So, repeat the steps for whatismyip.com.

However, you could be experiencing a browser caching issue. Web browsers like to cache content. I use IP detection web sites in my development and testing of selective routing scripts. It is not uncommon for me to change routing to another VPN location, then go to an IP detection web site that I had used while connected to the prior VPN server and get a false reading because of browser caching. I normally use Firefox. To overcome the caching issue, I open up a new session in another browser like Chrome or Edge. Then, validate the IP address using the IP detection web site.

 

[Xentrk]

@sone, I had another idea of what could be the issue.

The script IPSET_Netflix.sh downloads all of the IPv4 addresses for the Amazon US region. I found the following address in the list:

54.254.0.0/16

When doing the nslookup on duckduckgo.com, I see

54.254.135.186

So, duckduckgo.com may be matching here. Later on, I can do a CIDR lookup to see if the /16 subnet would match the duckduckgo.com site. Perhaps duckduckgo.com is hosting on Amazon AWS servers which is why you are experiencing the issue you report.

 

[Xentrk]

@sone, I did a web search and confirmed that duckduckgo.com is hosted on Amazon AWS servers. Also confirmed the IPv4 address range using the site https://www.ipaddressguide.com/cidr. That explains the results you are getting since the IPSET_Netflix.sh scripts pulls all IPv4 addresses for Amazon AWS.

You can try removing the entry from the IPSET list.

ipset del x3mRouting_NETFLIX 54.254.0.0/16

Then, test to make sure everything works. If it works, we would need to modify the script to stop the periodic refreshes of the list. I have been doing testing the past year and found the list to be static and does not require frequent refreshes.

Similarly, you may want to test with the IPSET_Netflix_Domains.sh script to see if that resolves the issue as it should only be collecting the IPv4 addresses of the Amazon AWS servers Netflix uses.

 

[sone]

Thank you so much @Xentrk your information is so helpful. DDG is currently not going through the WAN but as soon as it does it again I'll will use your steps to determine if its in the routing set and delete. The browsing cache makes sense so I'll take that into consideration as well.

oh just saw your recent posts as well.. yes I'm using your IPSET_Netflix.sh not the domain one. Good catch! I'll check out the domain version also to see if that helps. Thanks again!

 

[Xentrk]

After more testing and dusting off my old AC86U, I can confirm this issue is unique to the AX88U. @RMerlin

Steps to reproduce;

Get current IP for comparison (http://whatismyip.host/my-ip-address-details )

ip rule add fwmark 0x7000/0x7000 table 254 prio 9990

ip rule add fwmark 0x1000/0x1000 table 111 prio 9991

iptables -A PREROUTING -t mangle -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

Get new IP for comparison (http://whatismyip.host/my-ip-address-details )

Just reread your post and noticed you are using 0x7000/0x7000 for the WAN iface. Following are the fwmark/bitmask combinations I am currently using in my Selective Routing scripts after consulting with @Martineau .

    FWMARK_WAN="0x8000/0x8000"
    FWMARK_OVPNC1="0x1000/0x1000"
    FWMARK_OVPNC2="0x2000/0x2000"
    FWMARK_OVPNC3="0x4000/0x4000"
    FWMARK_OVPNC4="0x7000/0x7000"
    FWMARK_OVPNC5="0x3000/0x3000"

The priority numbers of the ifaces are inverted when compared to the OpenVPN Client number. That was also a recent change. You can see the code on the repo. I appreciate if you can try it as I need to know if there is an issue with the use of fwmark/bitmask for routers using the HND platform.

 

[Adamm]

Just reread your post and noticed you are using 0x7000/0x7000 for the WAN iface. Following are the fwmark/bitmask combinations I am currently using in my Selective Routing scripts after consulting with @Martineau .

    FWMARK_WAN="0x8000/0x8000"
    FWMARK_OVPNC1="0x1000/0x1000"
    FWMARK_OVPNC2="0x2000/0x2000"
    FWMARK_OVPNC3="0x4000/0x4000"
    FWMARK_OVPNC4="0x7000/0x7000"
    FWMARK_OVPNC5="0x3000/0x3000"

The priority numbers of the ifaces are inverted when compared to the OpenVPN Client number. That was also a recent change. You can see the code on the repo. I appreciate if you can try it as I need to know if there is an issue with the use of fwmark/bitmask for routers using the HND platform.

Looks like that did the trick, not sure exactly why the previous method only worked on my AC86U and not on my AX88U but guess it works now

 

[Xentrk]

Looks like that did the trick, not sure exactly why the previous method only worked on my AC86U and not on my AX88U but guess it works now

Whoot! That is great news. I had been digging around off an on the past week or two trying to figure out what the issue could me. I was looking at the vpnflix.sh script your wrote on the misc repository on GitHub this morning and the light when off. I am glad it is working. I have another selective routing project planned soon and wanted to make sure it was compatible with HND routers.

 

[Adamm]

Whoot! That is great news. I had been digging around off an on the past week or two trying to figure out what the issue could me. I was looking at the vpnflix.sh script your wrote on the misc repository on GitHub this morning and the light when off. I am glad it is working. I have another selective routing project planned soon and wanted to make sure it was compatible with HND routers.

So now that my ISP has finally enabled IPv6 I noticed a pretty big flaw with selective routing. Because traffic priorities IPv6, the new rules are essentially ignored (most VPN providers don't support IPv6 for security reasons).

Not quite sure what the "official" way to go about forcing the specific traffic to use IPv4, rather then disabling/deprioritizing IPv6 all together.

 

[Martineau]

Whoot! That is great news. I had been digging around off an on the past week or two trying to figure out what the issue could me. I was looking at the vpnflix.sh script your wrote on the misc repository on GitHub this morning and the light when off. I am glad it is working. I have another selective routing project planned soon and wanted to make sure it was compatible with HND routers.

So RT-AX88U users could potentially now be limited to only three VPN Clients?

 

[Martineau]

So now that my ISP has finally enabled IPv6 I noticed a pretty big flaw with selective routing.

Not quite sure what the "official" way to go about forcing the specific traffic to use IPv4, rather then disabling/deprioritizing IPv6 all together.

Currently @RMerlin's firmware doesn't support IPv6 VPNs, so if you have an IPv6 WAN ISP, you have to disable IPv6 on the VPN ISP's tunnel.

 

[Xentrk]

So RT-AX88U users could potentially now be limited to only three VPN Clients?

Why is that? Curious minds want to know. Conflict with use of fwmarks/bitmasks vs. firmware?