1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Selective Routing for Netflix

Discussion in 'VPN' started by Xentrk, Dec 9, 2017.

  1. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    This project has been superseded by the x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware project.

    The purpose of this project is to selectively route Netflix traffic to the WAN interface, or one of the five OpenVPN clients, on Asuswrt-Merlin firmware, using the method espoused by @Martineau on snbforums.com.

    The routing of Netflix traffic is a frequent topic on this and other forums. Since January 2016, Netflix blocks VPN providers. This project was originally developed to bypass the OpenVPN client for Netflix traffic and route it to the WAN interface. The scripts now provide the ability to route Netflix traffic to an OpenVPN Client if desired. If you want a VPN provider who can circumvent the Netflix VPN ban, see my blog post Why I use Torguard as my VPN Provider to learn more.

    There are two selective routing scripts used in this project. Each one uses a different method to collect the IPv4 addresses required for selective routing. Both scripts use the features of IPSET to collect IPv4 addresses in IPSET lists and match against the IPSET lists. Users can select the script that works best for their environment.

    IPSET_Netflix.sh collects the IPv4 addresses used by Netflix from https://ipinfo.io using the Autonomous System Number (ASN) assigned to Netflix. Amazon AWS supplies the list of IPv4 addresses in the json file at https://ip-ranges.amazonaws.com/ip-ranges.json

    Only the Amazon AWS US Regions are extracted from ip-ranges.json. As a result, the script will also route all Amazon AWS traffic bound for the US, including Amazon Prime traffic, to the WAN interface.

    IPSET_Netflix_Domains.sh uses the IPSET feature built into dnsmasq to dynamically generate the IPv4 address used by Netflix and Amazon AWS dynamically. This approach can be useful when your ISP is using the Netflix Open Connect Network.

    Project documentation and install instructions are available on github.com. Earlier versions of the scripts posted throughout this thread may no longer work.
     
    Last edited: Jul 28, 2019
    sugar0, amplatfus, buddyp and 2 others like this.
  2. Adamm

    Adamm Part of the Furniture

    Joined:
    Mar 26, 2013
    Messages:
    2,204
    I suggest you have your script create a file "/jffs/shared-*-whitelist" with * being a name of your choosing. Skynet and AB will detect this and automatically whitelist any domains listed.
     
  3. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    Thanks for the tip! I need some time to dwell on a name for awhile first. hmmm..

    Edit: Update made in post 1 above by borrowing some of TLC's ABS Code!
     
    Last edited: Dec 9, 2017
  4. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    On my pfSense appliance, the pfBlockerNG package allows me to create IPv4 lists for Selective Routing using AS numbers and JSON file formats. So, for the Amazon AWS AS numbers, I am using

    https://ip-ranges.amazonaws.com/ip-ranges.json

    as the source rather than ipinfo.io.

    Need to research how to parse out json file formats. Would be nice to use the json file format if possible. I should probably look at BBCan177's pfBlockerNG code for clues!
     
    Last edited: Dec 9, 2017
  5. steef84

    steef84 Occasional Visitor

    Joined:
    Jan 4, 2015
    Messages:
    24
    Cool script. Using v 1.2 on AC86U.
    However netflix still complaining im using a unblocker. Using this with dutch vpn server. Ipset V6. Policy Rules enabled in VPN client. NO IPv6. Are there specific ASN for Dutch users? Or do I miss a certain clue :) ?
    No errors on running the script, however it complains multiple times that "Element cannot be added to the set: it's already added" after rerunning script :) But thats no real error
     
  6. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    Thanks for testing. I was wondering how this would work for those in Europe.

    The Element cannot be added to the set: it's already added message happens if you run the script multiple times. The IPSET list has already been created. When you run it again, it tries to load the IP addresses to the IPSET list. But the IP addresses already are added.
    You can ignore it. The IPSET lists don't survive a router reboot and the script is intended to be invoked at start-up in nat-start for example e.g.
    #!/bin/sh
    sh /jffs/scripts/IPSET_Netflix.sh


    ipset -L NETFLIX command will list the contents of the IPSET list NETFLIX

    There may be a flaw in the iptables lines. My fiber connection to my house is down and I am piggy backing off my neighbor until it is fixed. So I am limited in my testing.

    Try changing these lines
    Code:
    iptables -t mangle -D PREROUTING -m set --match-set NETFLIX dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -m set --match-set NETFLIX dst -j MARK --set-mark 0x7000/0x7000
    
    to

    Code:
    iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
    
    Once my internet connection is working again, I plan on fixing this and posting and update the orginal post if it works.

    When I look at Netflix domains at https://bgp.he.net/search?search[search]=netflix&commit=Search, I see USA and UK IP addresses. Most of the IPv6 addresses are in the UK. I wonder if all European Netflix uses the UK servers? I can update the script to include IPv6 addresses to see if that helps as well. Or, create one version that uses USA IP addresses for the USA users and another that uses the UK IP addresses for EU users. Try the iptables change first though and let me know what happens.

    Some VPN providers provide Private IP addresses in USA that allow streaming of Netflix. I route my NF traffic to Private IP address rather than WAN. If you want to get USA Netflix, which I heard has more content (https://www.finder.com/netflix-usa-vs-world-content), I can create another version of the script to do this. I would recommend my VPN provider as I know it works and you can cancel after 30 days if you don't like it or want to use another provider. Some providers may not openly advertise this feature and you may have to contact their customer support to see if they offer it.

    But lets try the iptables change first. I would like to have this work for European users . Thanks for helping with the testing.

    Do you have dnsmasq enabled? If so, we can use dnsmasq.log file to determine the domain names your Netflix is using. We can then do an nslookup on the domain names to determine the IP address it is using and see if your streaming device is using USA, UK or both US and UK NF servers.
     
    Last edited: Dec 14, 2017
  7. Raphie

    Raphie Senior Member

    Joined:
    Sep 19, 2017
    Messages:
    215
    What exactly does this script allow you to do? Does it allow me to connect to different regions without spoofing via VPN?
     
  8. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    The script collects the IPv4 addresses of a company. In this case, Netflix and Amazon AWS, creates IPSET lists using the IP addresses, creates a fwmark for the WAN interface, marks the packets containing the IP addresses with the WAN fmwark 0x7000/0x7000 and routes the traffic to the WAN bypassing the VPN. It can be modified to route to one of the OpenVPN tunnel clients with creating an IPSET list for OpenVPN Client 1 = OVPNC1, associated fwmark 0x1000 and iptables commands. Example:

    Code:
    ipset create OVPNC1 hash:net family inet hashsize 1024 maxelem 65536
    
    # WAN
    ip rule del fwmark 0x7000
    ip rule add fwmark 0x7000 table 254 prio 9990
    
    #VPN Client 1
    ip rule del fwmark 0x1000
    ip rule add fwmark 0x1000 table 111 prio 9991
    
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 dst,dst -j MARK --set-mark 0x1000/0x1000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 dst,dst -j MARK --set-mark 0x1000/0x1000
    
    Do you want to test?

    There are recent reports of this not working for AC68U owners though. :(
     
  9. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    My internet is still down. They usually come out to my house the same day. I hope they arrive before I have to leave at Noon today. It is a top priority for me to test updated iptables once it is back online.
     
  10. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    The iptable command changes:
    Code:
    iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
    worked in my test. It bypassed my VPN tunnel and was routed to the WAN interface. I went to my web browser and typed netflix.com to make sure. I then got the Thai version. Updated to beta version 1.3. Please retest and provide feedback.
     
    Last edited: Dec 14, 2017
  11. steef84

    steef84 Occasional Visitor

    Joined:
    Jan 4, 2015
    Messages:
    24
    Thanks for you long answer and clear answer. It makes me understand it a bit more!
    Tonight i tried your 1.3 on my router. Netflix still complains :) ... Dont know what it its. Tried to use dutch and a american vpn client, but proxy still detected by NF.
    Im also wondering what will happen when using UK ASN. It may be the cause of using Europe settings. Setting the different IP tables didnt work for me at least.

    Now looking how to enabled dnsmasq log :)

    EDIT: and also i did ask my vpn provider about supplying private ip
     
    Last edited: Dec 15, 2017
  12. st3v3n

    st3v3n Very Senior Member

    Joined:
    Feb 24, 2016
    Messages:
    501
    Location:
    Central US
    Xentrx, Have been following this thread with much interest. Can confirm it's still possible to locate a VPN who will assign you a private IP for a fee, to you want watch US Netflix or Amz. Affording the IP; and how long the vpn can manage to perform their VPN wizardry are the questions, subject to how well their resident wizards manage their resources. There's not many of these companies left who can and very few who are willing to talk unless you pay to become a customer. It took me two solid two weeks of inquiries before I decided to lay the plastic down but so far, it's the cats meow.
     
  13. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    Thanks for testing. Do you have these settings?

    1. Static IP addresses are set up all clients in DHCP tab.
    2. Redirect Internet Traffic = Policy Rules or Policy Rules(Strict) for the VPN Client
    3. IP addresses of clients that need to use the VPN tunnel are listed in the OpenVPN Client page in the Rules for routing client traffic through the tunnel section.

    Can you please post output of these commands:

    ip rule

    iptables -nvL PREROUTING -t mangle --line

    I will make another script to focus on sending traffic to the UK servers. Do you have IPv6 enabled?

    Regarding dnsmasq, I use AB-Solution and it handles the creation of it for you which makes the set up easy. Great thing AB-Solution.
     
  14. steef84

    steef84 Occasional Visitor

    Joined:
    Jan 4, 2015
    Messages:
    24
    I've answered the asked questions in blue. Thanks for the help so far! Willing to get this script working for EU users also :):)

     
    Last edited: Dec 17, 2017
    Ro berto likes this.
  15. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    Thanks for the information. That will help! I will start working on new script with the changes to see if it makes a difference for you. I can try it on my end to see if it routes me to the EU library.
     
    Ro berto and steef84 like this.
  16. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    I see some of the domains belong to Akamai. I did a nslookup on their domain names against two sites I use. I got referred to Akamai. But could not find a match against the IPv4 address. To handle this, I do an nslookup on the the Akamai domain names you supplied and also included the AS numbers assigned to Akamai - As35944 and AS20940. I am not sure how doing do may impact your other streaming media sources though. I suggest first commenting out the sections that perform the nslookup for the Akamai AS Numbers, then uncomment and retest if it still does not work. If none of this works for you, I may want you to include a pic of your OpenVPN Web Gui screen just to make sure there is not some setting there that is causing the issue. If that does not work, I can try my original suggestion of filtering just for the EU IPv4 addresses.
    Code:
    #!/bin/sh
    set -xo
    ####################################################################################################
    #
    # Author: Xentrk
    # Dec-20-2017 Beta V 1.4 Added lookup for four CDN Akamai Technology domains.
    #
    #####################################################################################################
    # Script Description:
    #
    # The purpose of this script is for selective routing of Netflix traffic using
    # Autonomous System Numbers (ASNs). ASNs are assigned to entities such as Internet
    # Service Providers and other large organizations that control blocks of IP addresses.
    #
    # Netflix and other services that use Amazon AWS servers are blocking VPN's.
    #
    # This script will
    #   1. Create shared whitelist entry for ipinfo.io in /jffs/shared-SelectiveRouting-whitelist for use by AB-Solution and Skynet.
    #      Otherwise, ipinfo.io may be blocked and the script will not work.
    #     2. Obtain the IPv4 addresses used by Netflix and Amazon AWS USA from ipinfo.io.
    #      IPv6 addresses are excluded in this version.
    #   3. Create the IPSET list NETFLIX
    #   4. Add the IPv4 address to the IPSET list NETFLIX
    #   5. Route IPv4 addresses in IPSET list NETFLIX to WAN interface.
    #
    # Note 1: IPSET syntax differs between version 6 and 4.5
    #             Syntax for ipset v6
    #                ipset create WAN0 list:set
    #                ipset add WAN0 setlist (e.g. SPEEDTEST)
    #             for routers running ipset v4.5 (ipset -V)
    #                ipset -N WAN0 setlist (e.g. SPEEDTEST)
    #
    # Note 2: In the event one needs to use IPv6 in the future, the syntax is: ipset -N NETFLIX-v6 hash:net family ipv6
    #
    # Note 3: Troubleshooting
    #
    #             You can use these sites for AS validation and troubleshooting to lookup ASNs:
    #
    #               https://bgp.he.net/AS16509 (Click on the prefixes tab to view IP addresses)
    #               http://ipinfo.io/AS2906
    #        
    # Note 4: Required OpenVPN Client Settings
    #
    #         - Redirect Internet Traffic = Policy Rules or Policy Rules (Strict)
    #         - Others?
    #
    #######################################################################
    logger -t "($(basename $0))" $$ Starting IPSET_Netflix_2.sh..." $0${*:+ $*}."
    
    # check shared-SelectiveRouting-whitelist so ipinfo.io is not blocked
    # by AB-Solution and Skynet
    
    echo -e "\n checking shared-SelectiveRouting-whitelist"
    if [ -s "/jffs/shared-SelectiveRouting-whitelist" ];then
      echo " file found, no further checks"
    else
      echo "generating missing file"
      echo "ipinfo.io" > /jffs/shared-SelectiveRouting-whitelist
    fi
    
    ipset create NETFLIX hash:net family inet hashsize 1024 maxelem 65536
    ipset create AMAZONAWS hash:net family inet hashsize 1024 maxelem 65536
    ipset create AKAMAI hash:net family inet hashsize 1024 maxelem 65536
    
    # Pull all IPv4s listed for Netflix USA - AS2906
    netsv4=`curl http://ipinfo.io/AS2906 2>/dev/null | grep -E "a href.*2906\/" | grep -v ":" |sed 's/^.*\">//; s/<.*//; /^\s*$/d'`
    for net in $netsv4
    do
      ipset add NETFLIX $net
    done
    unset netsv4
    
    # Pull all IPv4s listed for Amazon AWS - AS16509
    netsv4=`curl http://ipinfo.io/AS16509 2>/dev/null | grep -E "a href.*16509\/" | grep -v ":" | sed 's/^.*\">//; s/<.*//; /^\s*$/d'`
    for net in $netsv4
    do
      ipset add AMAZONAWS $net
    done
    unset netsv4
    
    for domain_name in cdn-0.nflximg.com occ-0-2433-2430.1.nflxso.net occ-1-2433-2430.1.nflxso.net occ-2-2433-2430.1.nflxso.net
        do
          echo "domain name:" $domain_name
          for ip in $(nslookup $domain_name | awk '/^Name:/,0{if (/^Addr/)print $3}' | grep -v ":"); do
            echo "ip address is:" $ip
            ipset add AKAMAI $ip
          done
        done
    unset ip
    unset domain_name
    
    # Pull all IPv4s listed for Akamai - 35994
    netsv4=`curl http://ipinfo.io/AS35994 2>/dev/null | grep -E "a href.*35994\/" | grep -v ":" |sed 's/^.*\">//; s/<.*//; /^\s*$/d'`
    for net in $netsv4
    do
      ipset add AKAMAI $net
    done
    unset netsv4
    
    # Pull all IPv4s listed for Akamai - 20940
    netsv4=`curl http://ipinfo.io/AS20940 2>/dev/null | grep -E "a href.*20940\/" | grep -v ":" |sed 's/^.*\">//; s/<.*//; /^\s*$/d'`
    for net in $netsv4
    do
      ipset add AKAMAI $net
    done
    unset netsv4
    
    ###########################################################
    #Create table to contain items added automatically by wan #
    ###########################################################
    ip rule del prio 9990
    ip rule add from 0/0 fwmark 0x7000 table main prio 9990
    
    iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set NETFLIX dst -j MARK --set-mark 0x7000/0x7000
    
    iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set AMAZONAWS dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set AMAZONAWS dst -j MARK --set-mark 0x7000/0x7000
    
    iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set AKAMAI dst -j MARK --set-mark 0x7000/0x7000
    iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set AKAMAI dst -j MARK --set-mark 0x7000/0x7000
    
    logger -t "($(basename $0))" $$ Ending IPSET_Netflix_2.sh..." $0${*:+ $*}."
     
  17. MrFahrenheit

    MrFahrenheit New Around Here

    Joined:
    Dec 25, 2017
    Messages:
    4
    hi and Merry Christmas, first of all thank you for all the effort you've put into advising and creating this script.
    I've got a bit of an issue when I run the script and was hoping someone could help, whenever I run it I get the message:

    Code:
    iptables: No chain/target/match by that name.
    I don't get any other errors prior to that, answers to previously asked questions here if that helps:

    1. Static IP addresses are set up all clients in DHCP tab - yes for required clients i.e. media server and printer
    2. Redirect Internet Traffic = Policy Rules or Policy Rules(Strict) for the VPN Client - Policy Rules
    3. IP addresses of clients that need to use the VPN tunnel are listed in the OpenVPN Client page in the Rules for routing client traffic through the tunnel section. Yes

    Can you please post output of these commands:

    ip rule

    Code:
    0:    from all lookup local
    9990:    from all fwmark 0x7000 lookup main
    32766:    from all lookup main
    32767:    from all lookup default
    iptables -nvL PREROUTING -t mangle --line

    Code:
    Chain PREROUTING (policy ACCEPT 42537 packets, 5726K bytes)
    num   pkts bytes target     prot opt in     out     source               destination
    1        0     0 MARK       all  --  !eth0  *       0.0.0.0/0            192.168.0.7          MARK or 0x8000
    TIA
     
  18. MrFahrenheit

    MrFahrenheit New Around Here

    Joined:
    Dec 25, 2017
    Messages:
    4
    Managed to resolve my issue, needed to add the following to the start of the script:

    Code:
    modprobe xt_set.ko
    Unfortunately I'm able to test because my VPN provider is playing up but will feedback once it's sorted

    UPDATE: back online and seems to be working perfectly. Just tried with Amazon Prime Video and it bypasses the VPN. Thanks again! :)
     
    Last edited: Dec 25, 2017
  19. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,420
    Location:
    The Land of Smiles
    Thanks for the feedback. What router model and firmware are you using? Did you use the version posted in the OP at the top?

    I did not intend for Amazon Prime to bypass VPN. :eek: Perhaps the Amazon AWS reference is doing that. I will have to test that out.
     
  20. MrFahrenheit

    MrFahrenheit New Around Here

    Joined:
    Dec 25, 2017
    Messages:
    4
    Asus RT-AC5300 on Merlin WRT 380.69
    The victory was shortlived - the AWS rules do seem to be allowing Prime Video playback but only on my desktop, when I used the app on my Shield TV I get blocked for using a VPN.

    Netflix doesn't seem to be working either but I'm guessing that's because your script targets US servers/IPs? Any suggestions welcome :)