x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware

  • ATTENTION! You'll notice a Prefix dropdown when you create a thread. If your post applies to one of the topics listed, please use that Prefix for your post. When browsing the thread list you can use the Prefix to filter the view.
  • ATTENTION! As of November 1, 2020, you will not be able to reply to threads 6 months after the thread is opened. Threads will not be locked, so posts may still be edited by their authors.

Xentrk

Part of the Furniture


x3mRouting Version 2.4.0
now available! 29 September, 2020

Migrating from Version 1.0
Version 2.0.0 Update Process - update instructions
Version 2.0.0 Changes - see what's changed!

x3mRouting includes selective routing features for LAN Clients, OpenVPN Clients and OpenVPN Servers.

If coming from the first generation of x3mRouting, please read the updated instructions to become familiar with the new features and usage instructions. Refer to the Version 2.0.0 Changes section for a description of the changes and the Version 2.0.0 Update Process section for the update instructions.

Project Development

I used Amazon Prime, BBC, CBS All Access, Hulu, Netflix and Sling streaming media services in developing the project and include them in the usage examples on the x3mRouting GitHub repository.

Introduction

The features of x3mRouting include three selective routing methods to select from:

1. x3mRouting for LAN Clients

An alternative approach to automate and easily assign LAN clients to a WAN or OpenVPN Client interface using a list of DHCP Static Lease clients to assign interfaces. This method eliminates the need to enter the LAN Client information and IP addresses in the OpenVPN Client Screen.

Code:
#########################################################
# Assign the interface for each LAN client by entering  #
# the appropriate interface number in the first column  #
# 0 = WAN                                               #
# 1 = OVPNC1                                            #
# 2 = OVPNC2                                            #
# 3 = OVPNC3                                            #
# 4 = OVPNC4                                            #
# 5 = OVPNC5                                            #
#########################################################
0 192.168.1.150 SamsungTV
1 192.168.1.151 Samsung-Phone
2 192.168.1.152 Asus-Laptop
2 192.168.1.153 iPad
1 192.168.1.154 Lenovo-Laptop
2. OpenVPN Client Screen, OpenVPN Event and x3mRouting Script

Policy_Routing_Screen.PNG

Provides the ability to create IPSET lists using x3mRouting script and selectively route the IPSET lists by entering the IPSET name in a modified OpenVPN Client Screen.

3. OpenVPN Event and x3mRouting Script

Provides the ability to
  • Create and selectively route IPSET lists to the VPN Client interface for all traffic or specific devices.
  • Bypass a VPN Client interface for all traffic or specific devices.
  • Route VPN Server 1, 2 or both to one of the VPN Clients.
  • Selectively route VPN Server 1, 2 or both to the same routing rules as an IPSET.
4. getdomainnames.sh and autoscan.sh Scripts

These two scripts can be used to analyze the domain names being used for a website or streaming service.

The getdomainnames.sh script will create a uniquely sorted list of domain names gathered from dnsmasq.log that you collect by accessing a website or streaming service. Use the script to help determine the domain names used by a website or streaming service.

The autoscan.sh script is used to search for one or more terms in dnsmasq.log. It will return the top level domain names if there is a match.

Detailed descriptions and usage examples of each method are available on the x3mRouting GitHub repository.

Requirements
  1. An Asus router with Asuswrt-Merlin firmware installed.
  2. A USB drive with entware installed. Entware can be installed using amtm - the SNBForum Asuswrt-Merlin Terminal Menu
  3. Policy Rules (Strict) or Policy Rules enabled on the OpenVPN Client screen.
Installation

You can install x3mRouting from an SSH command line or using amtm.

To install from the SSH command line, copy and paste the command below into an SSH session:

Code:
sh -c "$(curl -sL https://raw.githubusercontent.com/Xentrk/x3mRouting/master/Install_x3mRouting.sh)"
To access the installation menu, type the command x3mMenu on a command line. The x3mRouting programs are installed in the /jffs/scripts/x3mRouting directory.

InstallationMenu.PNG


Acknowledgements

Please see the GitHub repository for all of the Acknowledgements
 
Last edited:

Xentrk

Part of the Furniture
x3mRouting Usage Notes

Required parameters are listed inside the braces: { }
Optional parameters are listed inside of the brackets [ ]
Valid parameter values are listed in parenthesis ( )

Create IPSET List with Routing Rules:
Code:
x3mRouting {src iface} (ALL|1|2|3|4|5)
           {dst iface} (0|1|2|3|4|5)
           ** src/dst NOTES Start **
             Valid SRC and DST Combinations
             1) VPN Client Routing
                - Use this SRC and DST combination to route all IPSET list traffic to a VPN Client:
                  ALL 1, ALL 2, ALL 3, ALL 4, ALL 5
             2) VPN Bypass Routing
                - Use this SRC and DST combination to bypass the VPN Client for an IPSET list and
                  route to the WAN interface:
                  1 0, 2 0, 3 0, 4 0, 5 0
           ** src/dst NOTES End **
           {ipset_name}
           ['autoscan='keyword1[,keyword2]...] Scans for keywords and creates IPSET list using
                                               the dnsmasq method
           ['asnum='asnum[,asnum]...] ASN method
           ['aws_region='US[,EU]...]  Amazon method
           ['dnsmasq='domain[,domain]...] dnsmasq method
           ['ip='ip[,ip][,cidr]...] Equivalent to manual method
           ['src='src_ip]
           ['src_range='from_ip-to_ip]
           ['dir='save_restore_location] if 'dir' not specified, defaults to /opt/tmp
           ['del']
Create IPSET List with no Routing Rules:
Code:
x3mRouting {ipset_name=}
           ['autoscan='keyword1[,keyword2]...] Scans for keywords and creates IPSET list using
                                               the dnsmasq method
           ['asnum='asnum[,asnum]...] ASN method
           ['aws_region='US[,EU]...]  Amazon method
           ['dnsmasq='domain[,domain]...] dnsmasq method
           ['ip='ip[,ip][,cidr]...] Equivalent to manual method
           ['dir='save_restore_location] if 'dir' not specified, defaults to /opt/tmp
           ['del']
VPN Server to VPN Client Routing:
Code:
x3mRouting {'server='1|2|both} {'client='1|2|3|4|5} ['del']
VPN Server to existing LAN routing rules for one or more IPSET list
Code:
x3mRouting {'server='1|2|both} {'ipset_name='IPSET[,IPSET]...} ['del']
 
Last edited:

Jack Yaz

Part of the Furniture
It looks like a lot of files are needed - are they loose in /jffs/scripts or do you create your own directory?

YazFi users - I don't anticipate any compatibility issues, but if you run into anything drop me a line and I'll work on it ASAP.
 

thelonelycoder

Part of the Furniture
It looks like a lot of files are needed - are they loose in /jffs/scripts or do you create your own directory?
Everything in /jffs/ goes into the /x3mRouting folder. The main x3mRouting file is /opt/bin/x3mRouting.
 

JFlynnXYZ

New Around Here
Hey guys, I seem to be having some issues with my setup using this. I have a feeling I'm making a dumb mistake but have tried two days to get this to work with no luck. I have my vpn setup and am trying to get access to Netflix by using this routing method. I'm having no luck with it.

I've tried the both the Netflix methods (originally doing it using netflix-vpn-bypass)
* load_ASN_ipset.sh NETFLIX AS2906
* load_DNSMASQ_ipset.sh NETFLIX amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

I've attached a log of the connection I made with dnsmasq --log-async --log-queries.

Any help is much appreciated :)
 

Attachments

Xentrk

Part of the Furniture
Hey guys, I seem to be having some issues with my setup using this. I have a feeling I'm making a dumb mistake but have tried two days to get this to work with no luck. I have my vpn setup and am trying to get access to Netflix by using this routing method. I'm having no luck with it.

I've tried the both the Netflix methods (originally doing it using netflix-vpn-bypass)
* load_ASN_ipset.sh NETFLIX AS2906
* load_DNSMASQ_ipset.sh NETFLIX amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

I've attached a log of the connection I made with dnsmasq --log-async --log-queries.

Any help is much appreciated :)
The entries in the dnsmasq.log file indicate that the IP addresses of the Netflix domain names are getting added to the IPSET list NETFLIX. You can verify this using the command:

Code:
ipset -L NETFLIX
or the command
Code:
liststats
You are using Method 2 of the scripts. With this method, the IPSET is created but not the routing. For the routing to work, you have to add the IPSET list name to the OpenVPN Client Screen. The screen only allows routing through the VPN interface and does not have the capability to route thru the WAN interface. For my use case, I have a VPN provider that provides me with a Private VPN Address to get around their VPN blocks. But most people use shared VPN servers that Netflix blocks. In this case, you need to route Netflix to the WAN interface using Method 3. With Method 3, the syntax is similar to Method 2. The difference is the name of the script and adding the interface name to the list of parameters you pass to the script.

With Netflix, there are two options that will work. You can use the ASN or the DNSMASQ methods:

ASN
Code:
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX AS2906
DNSMASQ
Code:
sh load_DNSMASQ_ipset_iface.sh 0 NETFLIX amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
To check that the routing rules are in affect, type the command
Code:
iptables -nvL PREROUTING -t mangle --line
Result
Code:
hain PREROUTING (policy ACCEPT 5808K packets, 6404M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        1    60 MARK       all  --  tun13  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2     661K  863M MARK       all  --  tun15  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3        1    60 MARK       all  --  tun14  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
4    76880   70M MARK       all  --  tun12  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
5    2030K 2737M MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
6        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
7        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK set 0x1000
8    1067K   60M MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set HULU_WEB dst MARK set 0x1000
9    33488 6945K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON dst MARK set 0x1000
10    129K 9898K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set MOVETV dst MARK set 0x3000
11   27284 5635K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set CBS_WEB dst MARK set 0x3000
12       0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set BBC_WEB dst MARK set 0x4000
Also, make sure Policy Rules (Strict) or Policy Rules are enabled on the OpenVPN Client Screen.
 
Last edited:

JFlynnXYZ

New Around Here
The entries in the dnsmasq.log file indicate that the IP addresses of the Netflix domain names are getting added to the IPSET list NETFLIX. You can verify this using the command:

Code:
ipset -L NETFLIX
or the command
Code:
liststats
You are using Method 2 of the scripts. With this method, the IPSET is created but not the routing. For the routing to work, you have to add the IPSET list name to the OpenVPN Client Screen. The screen only allows routing through the VPN interface and does not have the capability to route thru the WAN interface. For my use case, I have a VPN provider that provides me with a Private VPN Address to get around their VPN blocks. But most people use shared VPN servers that Netflix blocks. In this case, you need to route Netflix to the WAN interface using Method 3. With Method 3, the syntax is similar to Method 2. The difference is the name of the script and adding the interface name to the list of parameters you pass to the script.

With Netflix, there are two options that will work. You can use the ASN or the DNSMASQ methods:

ASN
Code:
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX AS2906
DNSMASQ
Code:
sh load_DNSMASQ_ipset_iface.sh 0 NETFLIX amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
To check that the routing rules are in affect, type the command
Code:
iptables -nvL PREROUTING -t mangle --line
Result
Code:
hain PREROUTING (policy ACCEPT 5808K packets, 6404M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        1    60 MARK       all  --  tun13  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2     661K  863M MARK       all  --  tun15  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3        1    60 MARK       all  --  tun14  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
4    76880   70M MARK       all  --  tun12  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
5    2030K 2737M MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
6        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
7        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK set 0x1000
8    1067K   60M MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set HULU_WEB dst MARK set 0x1000
9    33488 6945K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON dst MARK set 0x1000
10    129K 9898K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set MOVETV dst MARK set 0x3000
11   27284 5635K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set CBS_WEB dst MARK set 0x3000
12       0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set BBC_WEB dst MARK set 0x4000
Also, make sure Policy Rules (Strict) or Policy Rules are enabled on the OpenVPN Client Screen.
Perfect! I misunderstood some of the meaning of the scripts (very new to the whole router game) and simply ran the option 3 DNSMASQ code as you described and now it works perfectly!

Do you have some kind of tip-jar or someway I can say thanks? :) Excellent work on the scripts, from all the people involved!

Also, I fixed a bug (just a naming error with one of the functions) and did a pull request on GitHub https://github.com/Xentrk/x3mRouting/pull/4
 

Xentrk

Part of the Furniture
Perfect! I misunderstood some of the meaning of the scripts (very new to the whole router game) and simply ran the option 3 DNSMASQ code as you described and now it works perfectly!

Do you have some kind of tip-jar or someway I can say thanks? :) Excellent work on the scripts, from all the people involved!

Also, I fixed a bug (just a naming error with one of the functions) and did a pull request on GitHub https://github.com/Xentrk/x3mRouting/pull/4
I merged the update. Thanks for letting me know about the typo.

No tip jar at the moment. But I appreciate the offer. A lot of work and effort went into the project. Thanks for the feedback and grateful you got it working.
 

Torson

Regular Contributor
Hi, that's exactly what I was looking for ...

My initial use case is just like the one detailed in post #6 - using a shared VPN provider, one device is routed through the VPN client (Policy Rules are enabled) to all destinations. Using method 3 I ran:
Code:
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX AS2906
Netflix comes up with the usual VPN related error. A check if the routing rules are in effect yields:
Code:
Chain PREROUTING (policy ACCEPT 97647 packets, 122M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     2018 1942K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2    11860 1111K MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3     488K   95M BWDPI_FILTER  udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0
4     5209  350K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK or 0x8000
Does QoS get in the way of selective re-routing or any other issue?

___________________Edit____later _____in_____the_____day___________

I turned QoS off, and the routing rules look like this:
Code:
Chain PREROUTING (policy ACCEPT 751 packets, 90157 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        7   968 MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3    64714 3794K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK or 0x8000
Still no dice... Netflix complains about a VPN or Proxy. Is the TV too smart?

***Running 384.13 Stable on RT-AC66U_B AiMesh mode with a similar model node + amtm, diversion & skynet latest***
 
Last edited:

Xentrk

Part of the Furniture
Hi, that's exactly what I was looking for ...

My initial use case is just like the one detailed in post #6 - using a shared VPN provider, one device is routed through the VPN client (Policy Rules are enabled) to all destinations. Using method 3 I ran:
Code:
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX AS2906
Netflix comes up with the usual VPN related error. A check if the routing rules are in effect yields:
Code:
Chain PREROUTING (policy ACCEPT 97647 packets, 122M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     2018 1942K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2    11860 1111K MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3     488K   95M BWDPI_FILTER  udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0
4     5209  350K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK or 0x8000
Does QoS get in the way of selective re-routing or any other issue?

___________________Edit____later _____in_____the_____day___________

I turned QoS off, and the routing rules look like this:
Code:
Chain PREROUTING (policy ACCEPT 751 packets, 90157 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        7   968 MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3    64714 3794K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX dst MARK or 0x8000
Still no dice... Netflix complains about a VPN or Proxy. Is the TV too smart?

***Running 384.13 Stable on RT-AC66U_B AiMesh mode with a similar model node + amtm, diversion & skynet latest***
The routing iptables rules look okay to me. I see packets are traversing the iptables chain. Check the IPSET list has values using the command liststats or ipset -L NETFLIX.

Try the DNSMASQ method and/or trying combining the ASN with the DNSMASQ method, which means running both scripts.

One forum member had to combine the ASN and DNSMASQ method in the first iteration of x3mRouting called netflix-vpn-bypass project. I suspect it has to do with Content Delivery Networks (CDN) caching content to be geographically closer to the streaming location.

DNSMASQ Method for Netflix
Code:
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 NETFLIX netflix.com,nflxext.com,nflximg.net,nflxso.net
If you still have issues, try adding amazonaws.com to the list of domains. Netflix hosts on Amazon servers. Similarly, you can try the ASN method for Amazon:

Code:
sh /jffs/scripts/x3mRouting/load_AMAZON_ipset_iface.sh 0 AMAZON-US US
I think one of the testers in EU ended up specifying the GLOBAL region. US works for me.
 
Last edited:

Torson

Regular Contributor
Thank you for the reply... I wish it was that simple :)
Actually I had to run both, DNSMASQ and ASN methods for Netflix (amazonws.com is in the list of domains.)

As for ipset -L NETFLIX:
Code:
ipset -L NETFLIX
Name: NETFLIX
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 8324
References: 1
Number of entries: 152
Members:
...followed by 152 IP ranges
and liststats:
Code:
AMAZON - 43
NETFLIX - 152
SkySports - 1
Skynet-Blacklist - 154594
Skynet-BlockedRanges - 1688
Skynet-IOT - 0
Skynet-Master - 2
Skynet-Whitelist - 5067
For Amazon and Netflix it's a SmartTV IP that I have in the rules for routing section.

I have Amazon GLOBAL with 43 ranges.
Please note that also SkySports does not route to VPN client 2:
Code:
 sh load_DNSMASQ_ipset_iface.sh 2 SkySports www.skysports.com
For SkySports I use a PC, which works just fine if I add it to the rules for routing in the router GUI for VPN client 2 but does not selectively route through interface 2 on the router.

Is there anything else that I'm missing in the big picture?
 
Last edited:

Xentrk

Part of the Furniture
Thank you for the reply... I wish it was that simple :)
Actually I had to run both, DNSMASQ and ASN methods for Netflix (amazonws.com is in the list of domains.)

As for ipset -L NETFLIX:
Code:
ipset -L NETFLIX
Name: NETFLIX
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 8324
References: 1
Number of entries: 152
Members:
...followed by 152 IP ranges
and liststats:
Code:
AMAZON - 43
NETFLIX - 152
SkySports - 1
Skynet-Blacklist - 154594
Skynet-BlockedRanges - 1688
Skynet-IOT - 0
Skynet-Master - 2
Skynet-Whitelist - 5067
For Amazon and Netflix it's a SmartTV IP that I have in the rules for routing section.

I have Amazon GLOBAL with 43 ranges.
Please note that also SkySports does not route to VPN client 2:
Code:
 sh load_DNSMASQ_ipset_iface.sh 2 SkySports www.skysports.com
For SkySports I use a PC, which works just fine if I add it to the rules for routing in the router GUI for VPN client 2 but does not selectively route through interface 2 on the router.

Is there anything else that I'm missing in the big picture?
Is Netflix working for you now?

I have Netflix working on FireTV, Roku, Nvidia Shield. I also tested using the iOS streaming media apps and streaming from a browser. My current set up is:

Code:
sh /jffs/scripts/x3mRouting/load_AMAZON_ipset_iface.sh 1 AMAZON_US US

sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX AS2906

sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 3 BBC_WEB bbc.co.uk,bbc.com,bbc.gscontxt.net,bbci.co.uk,bbctvapps.co.uk,ssl-bbcsmarttv.2cnt.net,llnwd.net
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 5 CBS_WEB cbs.com,cbsnews.com,cbssports.com,cbsaavideo.com,omtrdc.net,akamaihd.net,irdeto.com,cbsi.com,cbsig.net
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 1 HULU_WEB hulu.com,hulustream.com,akamaihd.net
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 5 MOVETV movetv.com
If you still have issues with Netflix, the next thing we can do is check the Routing Policy Database (RPDB) priorities. Please type the command ip rule at the command line and post results.

For skysports.com, you can drop the "www" and just use skysports.com. Also, there are other domains you need to include.

For example,
Code:
sh load_DNSMASQ_ipset_iface.sh 0 SKY sky.com autoscan
Will get additional domains:

Code:
ipset=/akadns.net/edgekey.net/edgesuite.net/epgsky.com/sky.com/SKY from 'a674.hsar.cdn.sky.com.edgesuite.net/adm.sky.com/assets.sky.com/assets.sky.com-secure.edgekey.net/awk.epgsky.com' etc
Similarly, you can try the ASN method for sky.com using AS5607

Code:
# nslookup skysports.com
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain

Name:      skysports.com
Address 1: 90.216.129.62

# whob 90.216.129.62
IP: 90.216.129.62
Origin-AS: 5607
Prefix: 90.192.0.0/11
AS-Path: 18106 2914 5607
AS-Org-Name: BSKYB-BROADBAND-AS
Org-Name: Sky UK Limited
Net-Name: BSKYB-BROADBAND
Cache-Date: 1564643813
Latitude: 51.508530
Longitude: -0.125740
City: London
Region: England
Country: United Kingdom
Country-Code: GB
Or, mine dnsmasq to get the top level domain names using the method described on the netflix-vpn-bypass project. The other technique I've used is to go to the website, right click on the page and select "View Source Code" form the menu. I then search on ".com" and ".net" or "href" to see what domains the site is using. For example, skysports.com website also has references to sky.com, skybet.com, akamai.net and 365dm.com.
 
Last edited:

Xentrk

Part of the Furniture
@Torson

Here are some sky.com domain names posted by @Martineau on the post which may be of some help. I took the raw list and processed it using the script getdomainnames.sh.

Code:
analytics.faw.sky.com
awk.epgsky.com
config.ott.sky.com
config.skyq-b.interactive.sky.com
desktop.client.ott.sky.com
entity.search.sky.com
images.metadata.sky.com
init.sky.com
metrics.sky.com
p.sky.com
qgo-desktop-sgw.skyanywhere.com
static.skyq-b.interactive.sky.com
uk.imageservice.sky.com
Alternatively, you can try the ASN method: https://bgp.he.net/AS5607
 
Last edited:

Torson

Regular Contributor
@Xentrk, thank you for the patience and guidance. I made some progress - I now have Netflix and Amazon running smoothly - tested briefly several movies/tv shows some I previously watched, some new; they all play well, no surprises there.
It's all a bit convoluted, but that's how it seems to work North of the border where Rogers is the cable king and I'm spending some time here.
As for Skysports, it starts playing the clips then it comes up with the dreaded geolocation thing... Actually I picked up a lot of .com and .net entries from the pages' source code. However, something is still missing - I'll keep digging.
Also, I whitelisted some sites that were blocked by Diversion, but eventually I disabled it altogether while testing.

Here is my current setup:
Code:
#!/bin/sh
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX-812 AS812
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX-14618 AS14618
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX-394406 AS394406
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 NETFLIX-2906 AS2906
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 AMAZON-812 AS812
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 0 AMAZON-16509 AS16509
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 2 SkySports e0.365dm.com,e1.365dm.com,e2.365dm.com,fresh8.co,annex.fresh8.co,cdn-gs.fresh8.co,player.ooyala.com,static.chartbeat.com,assets.adobedtm.com,img.skysports.com,video.skysports.com,cdn.livefyre.com,app-cdn.spot.im,cdn.teads.tv,services.babator.com,eu-services.babator.com,widgets.oddschecker.com,pagead2.googlesyndication.com,metrics-api.librato.com,cloud-static.storage.googleapis.com,analytics.ooyala.com,l.ooyala.com,a248.e.akamai.net,ads.pubmatic.com,bskyb.demdex.net,config.seedtag.com,cs.seedtag.com,csync.smartadserver.com,dynamic-cdn.spot.im,fonts.googleapis.com,imasdk.googleapis.com,player.h-cdn.com,player2.h-cdn.com,recirculation.spot.im,secure-assets.rubiconproject.com,sync.richaudience.com,us-u.openx.net
Code:
/tmp/mnt/asus/conf# iptables -nvL PREROUTING -t mangle --line
Chain PREROUTING (policy ACCEPT 18537 packets, 7329K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2      409  357K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3       33  2563 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-812 dst MARK or 0x8000
4       84 21377 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-14618 dst MARK or 0x8000
5        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-394406 dst MARK or 0x8000
6        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-2906 dst MARK or 0x8000
7       32  2522 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON-812 dst MARK or 0x8000
8       84 10551 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON-16509 dst MARK or 0x8000
9       42  6706 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set SkySports dst MARK or 0x2000
I tried to add an ASN Skysports rule, but didn't make any difference.
The following part looks like an over sized hammer hitting a small nail, but the router is happy and everything else works like a charm (AiMesh included.)

Code:
 liststats
AMAZON-16509 - 3063
AMAZON-812 - 574
NETFLIX-14618 - 188
NETFLIX-2906 - 152
NETFLIX-394406 - 2
NETFLIX-812 - 574
SkySports - 152
SkySports1 - 0
Skynet-Blacklist - 152854
Skynet-BlockedRanges - 1638
Skynet-IOT - 0
Skynet-Master - 2
Skynet-Whitelist - 5015
..and finally
Code:
 ip rule
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
10101:  from 192.168.1.238 lookup ovpnc1
32766:  from all lookup main
32767:  from all lookup default
...now it would be nice to have the ability to concatenate the ASN statements based on ipset.

Thanks again, and let me know if you have any ideas for Skysports and/or comments on the working stuff.
 
Last edited:

Xentrk

Part of the Furniture
Thanks again, and let me know if you have any ideas for Skysports and/or comments on the working
stuff.
The only ideas on Skysports and sky is to try AS5607 , the domain names I posted above and/or mining dnsmasq.log for the domain names gathered from surfing the website. I did some web searching and found the site below may be useful to find the top level domain names for skysports.com and sky.com.

https://www.yougetsignal.com/tools/web-sites-on-web-server/


upload_2019-8-3_15-27-19.png

upload_2019-8-3_15-28-42.png


You can use the domain names listed above in the dnsmasq method. Try it first without the amazon.com domain.

For Netflix, I don't see any packets traversing the chain for these two IPSET lists:

Code:
5        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-394406 dst MARK or 0x8000
6        0     0 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-2906 dst MARK or 0x8000
That is one of the reasons for having separate routing rule per IPSET list as it helps in analysis and finding out if packets are traversing the routing rules. Routing rules are processed in the order listed. The traffic may be matching first on chains 3 and 4 and not reaching chains 5 and 6.

Code:
3       33  2563 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-812 dst MARK or 0x8000
4       84 21377 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX-14618 dst MARK or 0x8000
You can eliminate one of these lists are they are duplicates and the routing rules are the same:

AMAZON-812
NETFLIX-812

I don't understand why AS812 is needed unless one is living in Canada.

The AMAZON-16509 and NETFLIX-14618 list should be included in the Amazon method using the US region:

Code:
sh /jffs/scripts/x3mRouting/load_AMAZON_ipset_iface.sh 2 AMAZON-US US
The other setting I recommend is to put the Router's IP address in the OpenVPN Client Screen and route it to the WAN iface. For my use case, I find the entry for the router is required in the OpenVPN Client 1 Screen when using more than one OpenVPN Client instance and have multiple rules routing traffic through more than one VPN interface and other traffic through the WAN interface.
 

Xentrk

Part of the Furniture
Attention: x3mRouting for LAN Clients Method 1 users

The program x3mRouting_client_config.sh that generates the file
/jffs/configs/x3mRouting_client_rules (example below) has been updated for Asuswrt-Merlin 384.13.

/jffs/configs/x3mRouting_client_rules
Code:
#########################################################
# Assign the interface for each LAN client by entering  #
# the appropriate interface number in the first column  #
# 0 = WAN                                               #
# 1 = OVPNC1                                            #
# 2 = OVPNC2                                            #
# 3 = OVPNC3                                            #
# 4 = OVPNC4                                            #
# 5 = OVPNC5                                            #
#########################################################
0 192.168.1.150 SamsungTV
1 192.168.1.151 Samsung-Phone
2 192.168.1.152 Asus-Laptop
2 192.168.1.153 iPad
1 192.168.1.154 Lenovo-Laptop
To update, type x3mRouting at the command line and select option 5 from the menu to update:

[5] = Check for updates to existing x3mRouting installation
 

CMc

Occasional Visitor
Coming from the old script, is there a method (or actions) you recommend to 'upgrade' to your new script, to avoid conflicts etc? (or import existing settings).
 

Xentrk

Part of the Furniture
Coming from the old script, is there a method (or actions) you recommend to 'upgrade' to your new script, to avoid conflicts etc? (or import existing settings).
You shouldn't have to take any actions. You can always backup the /jffs partition or the /jffs/scripts/x3mRouting directory as an extra safety precaution. I sometimes make a backup directory of an existing installation during testing of changes so I can easily recover if need be.

Code:
mkdir /jffs/scripts/x3mRouting-bkup
cp /jffs/scripts/x3mRouting/*.* /jffs/scrits/x3mRouting-bkup
The script mentioned above is probably a script that is executed one time during the initial set-up and not run again, unless one adds more clients to static DHCP lease reservations and wants to create an updated list containing the newly added clients. If there is an issue, I could give you a command to download a prior version of the script on GitHub. This script does make a backup of /jffs/configs /x3mRouting_client_rules if one exists. So, if you did add new clients and need to generate a new file, you can copy/paste the contents from the backup file to the new file. Then, you only have to enter the interface assignments for the new clients.
 

Torson

Regular Contributor
Getting close to a fully functional system - all components work, but not together:

1. Netflix and Amazon work properly when the router's IP address is added to the OVPN Client 1 (along with the TV's IP) and routed to the WAN interface. However, Skysports is not working.

Code:
Chain PREROUTING (policy ACCEPT 3834 packets, 1143K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      208  122K MARK       all  --  tun12  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2     2461 1290K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
4    41417 2447K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set ROGERS_812 dst MARK or 0x8000
5     4461 2250K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX_14618 dst MARK or 0x8000
6    52088   74M MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON_16509 dst MARK or 0x8000
7     3589  980K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set SSp_rev dst MARK or 0x2000
8    16158 2161K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set SkySports_masq dst MARK or 0x2000
Code:
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
10001:  from 192.168.1.1 lookup main
10101:  from 192.168.1.238 lookup ovpnc1
32766:  from all lookup main
32767:  from all lookup default
Code:
AMAZON_16509 - 3064
NETFLIX_14618 - 188
ROGERS_812 - 574
SSp_rev - 2
SkySports_masq - 232
2. Skysports works when removing the router's WAN interface from the OVPN Client 1. Netflix and Amazon do not work anymore.
That's the only difference:
Code:
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
10101:  from 192.168.1.238 lookup ovpnc1
32766:  from all lookup main
32767:  from all lookup default
Which would be the middle way where all 3 work together?
 

Xentrk

Part of the Furniture
Getting close to a fully functional system - all components work, but not together:

1. Netflix and Amazon work properly when the router's IP address is added to the OVPN Client 1 (along with the TV's IP) and routed to the WAN interface. However, Skysports is not working.

Code:
Chain PREROUTING (policy ACCEPT 3834 packets, 1143K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      208  122K MARK       all  --  tun12  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2     2461 1290K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3        0     0 MARK       all  --  tun21  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
4    41417 2447K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set ROGERS_812 dst MARK or 0x8000
5     4461 2250K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set NETFLIX_14618 dst MARK or 0x8000
6    52088   74M MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set AMAZON_16509 dst MARK or 0x8000
7     3589  980K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set SSp_rev dst MARK or 0x2000
8    16158 2161K MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set SkySports_masq dst MARK or 0x2000
Code:
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
10001:  from 192.168.1.1 lookup main
10101:  from 192.168.1.238 lookup ovpnc1
32766:  from all lookup main
32767:  from all lookup default
Code:
AMAZON_16509 - 3064
NETFLIX_14618 - 188
ROGERS_812 - 574
SSp_rev - 2
SkySports_masq - 232
2. Skysports works when removing the router's WAN interface from the OVPN Client 1. Netflix and Amazon do not work anymore.
That's the only difference:
Code:
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
10101:  from 192.168.1.238 lookup ovpnc1
32766:  from all lookup main
32767:  from all lookup default
Which would be the middle way where all 3 work together?
Good news is you are closer now. But the results are a unexpected so I don't have an easy answer at the moment :confused:

Two things are going on with selective routing.

The first item is to create the IPSET lists for the traffic we want to route. Go back to the first setting in your post. Then, add AS2906 for Netflix as that one has been the go to ASN for Netflix for several years now for myself and others.
Code:
sh load_ASN_ipset_iface.sh 0 NETFLIX-2906 AS2906
Test and see what happens.

How are you creating the IPSET lists for Sky? Did you try including AS5607? I wonder if the IPSET lists for sky.com still need some additional entries. If so, then the proper matching is not occurring which may create issues. So if you can add AS5607, that will help eliminate that as a concern.

Traffic is matched according to the RPDB priorities. The lower number has a higher priority.
  • Rogers, Netflix and Amazon match prio 9990 and route to the WAN.
  • Sky matches prio 9994 and routes to ovpnc2.
  • 192.168.1.238 matches prio 10101 and routes to ovpnc1.
You can manually change the RPDB database. If you still have issues after you have entered the routers IP address to the OVPNC1 screen and route it to the WAN iface, remove the priority rule:
Code:
ip rule del prio 10001
Then, add it back as the first priority above the fwmark/bitmask rules and see what happens:
Code:
ip rule add from 192.168.1.1 table main prio 9000
 

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top