Hi Xentrk. dnsmasq.conf and logfile are both blank after that. My dnsmasq.conf always revert back to its native state.
https://pastebin.com/raw/a8vnKxjt
Selective Routing with Asuswrt-Merlin
- Thread starter janosek
- Start date
https://pastebin.com/raw/a8vnKxjt
Xentrk
Part of the Furniture
Good grief. dnsmasq logging is preventing you from finalizing your objective. If I had an Apple TV, I could try on my end. Installing AB-Solution should fix your dnsmasq logging issues.
My vpn provider sometimes works on BBC iPlayer. Most of the time it doesn't. I would have to pay the extra $$ for a private IP for it to work all of the time. For about 5 minutes this morning. I was able to navigate around the iPlayer on Kodi and only picked up six more domain names. Three of these I already had in the list. The three new domaines are at the end of the list.
a1089.d.akamai.net
a1104.w10.akamai.net
a2.w10.akamai.net
account-origin-live.bbc.net.uk
account.bbc.com
b1rbsov.bidi.live.bbc.co.uk
bbcdotcom.2cnt.net
bbciplayer.metafaq.com
bn1305.storage.live.com
bootstrapcdn.jdorfman.netdna-cdn.com
cd-megavolt.90fe2324ce3eb149.xhst.bbci.co.uk
component.iplayer.api.bbc.co.uk
e3891.dscf.akamaiedge.net
e3891.f.akamaiedge.net
e8218.dscb1.akamaiedge.net
emp.bbc.co.uk
emp.bbci.co.uk
fig.bbc.co.uk
fig.bbc.net.uk
gn.symcd.com
ibl.api.bbc.co.uk
ichef.bbc.co.uk
ichef.bbci.co.uk
iplayer-web.files.bbci.co.uk
iplayerhelp.external.bbc.co.uk
ipv4only.arpa
live-ibl-componen-3y285w56k7w5-887784694.eu-west-1.elb.amazonaws.com
live-matc-componen-14ucw7bt4o3x5-61844696.eu-west-1.elb.amazonaws.com
live-noti-componen-9nj5c6fwh1nl-1633728249.eu-west-1.elb.amazonaws.com
live-tvip-componen-poadok30hype-1266449070.eu-west-1.elb.amazonaws.com
login.live.com
maxcdn.bootstrapcdn.com
mm.bidi.bbc.co.uk
mobile.pipe.aria.microsoft.com
music.files.bbci.co.uk
mvt.api.bbc.com
mybbc-analytics.files.bbci.co.uk
mybbc.files.bbci.co.uk
nav.files.bbci.co.uk
navpromo.90fe2324ce3eb149.xhst.bbci.co.uk
navpromo.api.bbci.co.uk
ocsp.comodoca.com
ocsp.usertrust.com
open-live.bbc.net.uk
open.live.bbc.co.uk
polling.bbc.co.uk
preferences.notifications.api.bbc.co.uk
r.bbci.co.uk
s.w.org
sa-live.com
sa.bbc.co.uk
search.bbc.co.uk
search.bbc.net.uk
search.files.bbci.co.uk
session-origin-live.bbc.net.uk
session.bbc.co.uk
session.bbc.com
ssl.bbc.co.uk
ssl.bbc.net.uk
static.bbc.co.uk
static.bbci.co.uk
stats.bbc.co.uk
uf2f.com
vod-dash-uk-live.akamaized.net
vod-dash-uk-live.bbcfmt.hs.llnwd.net
vod-thumb-uk-live.akamaized.net
vod-thumb-uk-live.bbcfmt.hs.llnwd.net
www-bbc-com.bbc.net.uk
www.bbc.co.uk
www.bbc.com
www.bbc.net.uk
a.files.bbci.co.uk
ve-hls-uk-live.akamaized.net
vs-hls-uk-live.akamaized.net
a1104.w10.akamai.net
a2.w10.akamai.net
account-origin-live.bbc.net.uk
account.bbc.com
b1rbsov.bidi.live.bbc.co.uk
bbcdotcom.2cnt.net
bbciplayer.metafaq.com
bn1305.storage.live.com
bootstrapcdn.jdorfman.netdna-cdn.com
cd-megavolt.90fe2324ce3eb149.xhst.bbci.co.uk
component.iplayer.api.bbc.co.uk
e3891.dscf.akamaiedge.net
e3891.f.akamaiedge.net
e8218.dscb1.akamaiedge.net
emp.bbc.co.uk
emp.bbci.co.uk
fig.bbc.co.uk
fig.bbc.net.uk
gn.symcd.com
ibl.api.bbc.co.uk
ichef.bbc.co.uk
ichef.bbci.co.uk
iplayer-web.files.bbci.co.uk
iplayerhelp.external.bbc.co.uk
ipv4only.arpa
live-ibl-componen-3y285w56k7w5-887784694.eu-west-1.elb.amazonaws.com
live-matc-componen-14ucw7bt4o3x5-61844696.eu-west-1.elb.amazonaws.com
live-noti-componen-9nj5c6fwh1nl-1633728249.eu-west-1.elb.amazonaws.com
live-tvip-componen-poadok30hype-1266449070.eu-west-1.elb.amazonaws.com
login.live.com
maxcdn.bootstrapcdn.com
mm.bidi.bbc.co.uk
mobile.pipe.aria.microsoft.com
music.files.bbci.co.uk
mvt.api.bbc.com
mybbc-analytics.files.bbci.co.uk
mybbc.files.bbci.co.uk
nav.files.bbci.co.uk
navpromo.90fe2324ce3eb149.xhst.bbci.co.uk
navpromo.api.bbci.co.uk
ocsp.comodoca.com
ocsp.usertrust.com
open-live.bbc.net.uk
open.live.bbc.co.uk
polling.bbc.co.uk
preferences.notifications.api.bbc.co.uk
r.bbci.co.uk
s.w.org
sa-live.com
sa.bbc.co.uk
search.bbc.co.uk
search.bbc.net.uk
search.files.bbci.co.uk
session-origin-live.bbc.net.uk
session.bbc.co.uk
session.bbc.com
ssl.bbc.co.uk
ssl.bbc.net.uk
static.bbc.co.uk
static.bbci.co.uk
stats.bbc.co.uk
uf2f.com
vod-dash-uk-live.akamaized.net
vod-dash-uk-live.bbcfmt.hs.llnwd.net
vod-thumb-uk-live.akamaized.net
vod-thumb-uk-live.bbcfmt.hs.llnwd.net
www-bbc-com.bbc.net.uk
www.bbc.co.uk
www.bbc.com
www.bbc.net.uk
a.files.bbci.co.uk
ve-hls-uk-live.akamaized.net
vs-hls-uk-live.akamaized.net
You can try it on your end to see if the three new additions make a difference.
Eureka! It works.
I had not realised that dnsmasq.conf can only be edited with the dnsmasq.conf.add file. This is why my logging wasn't working. My firmware kept replacing all the extra config things you asked me to put in dnsmasq.conf taking it back to default each time.
Anyway, as I suspected, the AppleTV makes a few calls of its own which iPlyaer uses to establish client location. One of them is to an itunes domains. Also there were a few bbc related domains that were specific to the AppleTV. It could be one or both techniques they are using to establish geolocation.
Here are some more domains:
I would like to somehow clean up my list and make some of the a753.w10.akamai type domains more general so that when those server numbers change, I am still covered. Any ideas how to figure out the appropriate ranges? Is there a way to get an output of the ip's from that Ipset VPN Routing script of yours?
Thanks so much for the help.
I had not realised that dnsmasq.conf can only be edited with the dnsmasq.conf.add file. This is why my logging wasn't working. My firmware kept replacing all the extra config things you asked me to put in dnsmasq.conf taking it back to default each time.
Anyway, as I suspected, the AppleTV makes a few calls of its own which iPlyaer uses to establish client location. One of them is to an itunes domains. Also there were a few bbc related domains that were specific to the AppleTV. It could be one or both techniques they are using to establish geolocation.
Here are some more domains:
Code:
a.files.bbci.co.uk
a753.w10.akamai.net
a936.w16.akamai.net
appletv.iplayer.api.bbc.co.uk
bbc01.sitestat.com
bbcfmt-ic-5896f100-0b3f9b-vodhlsuklive.s.loris.llnwd.net
e673.e9.akamaiedge.net
guzzoni.apple.com
ichef-bbci.bbc.net.uk
itunes.apple.com.edgekey.net
sylvan.apple.com
vod-hls-uk-live.bbcfmt.hs.llnwd.net
xp.itunes-apple.com.akadns.netI would like to somehow clean up my list and make some of the a753.w10.akamai type domains more general so that when those server numbers change, I am still covered. Any ideas how to figure out the appropriate ranges? Is there a way to get an output of the ip's from that Ipset VPN Routing script of yours?
Thanks so much for the help.
Just pinging you in case you stopped following this thread...
Xentrk
Part of the Furniture
I am happy this is finally working for you. Sounds like your iptables issues are resolved as well? Did you find out what the issue was and what did you do to fix it?
Regarding the akamai domains, see Note 2 on this thread for the two options you can use to mine the IP addresses and subnets of the domain names:
https://www.snbforums.com/threads/selective-routing-with-asuswrt-merlin.9311/page-29#post-349474
One method requires installation of entware package which I don't think you have installed.
Go to https://www.ultratools.com/ to lookup ASN.
There is also the nslookup method here, which we used previously.
Code:
for domain_name in $(awk '{ print $1 }' /jffs/scripts/BBCdns)
do
echo "domain name:" $domain_name
for ip in $(nslookup $DNS | awk '/^Name:/,0{if (/^Addr/)print $3}'); do
echo "ip address is:" $ip
ipset add IPLAYER $ip
done
doneYou can see which one gives you the best results.
Once you get it working, you can try and remove some of the domain names to see if they are required or not. Some of the apple domain names may be generic to the apple tv device and may not impact iPlayer. A little testing is required to determine this and it helps when others are not on the network in case you need to reboot.
Last edited:
Yeah I wanted to give it a few days to check it persists before explaining it. I didn't do much differently from what was written here by Xentrk and some others. The only thing different was specifically using dnsmasq.conf.add to add lines to dnsmasq.conf, as otherwise the changes don't persist. I guess thats probably quite elementary to most people here.
I can give you the gist of what I did though:
1. Enable jffs partition and dns filtering like in the wiki. (Also give your streaming device a manual static ip)
2. Start your TCP protocol VPN Client 1 and set DNS to Exclusive and Policy Rules (Strict). Enter streaming box fixed ip and then 0.0.0.0 so for example. "192.168.1.100 0.0.0.0 VPN". This is temporary, just so that you can grab all the domains you need. Check that you can surf and play iPlayer on your streaming box.
3. Open winSCP or the like and create a folder - /tmp/mnt/logs/
4. Go to /jffs/configs and create dnsmasq.conf.add file
5.Write in it
domain-needed
log-facility=/tmp/mnt/logs/dnsmasq.log
log-async=5
log-queries
6. Save it.
7. Open Putty and enter "cd /tmp/mnt/logs/" then "service restart_dnsmasq" then "tail -f dnsmasq.log > logfile"
8. Open iPlayer on your streaming device and watch some things, go to different pages of the app, load up some live tv, some archived shows, etc.
9. Go back to Putty and press Ctrl - C to stop the tail.
10. Go back to winSCP /tmp/mnt/logs/ and hit Refresh, (Ctrl R). You should have a file called logfile now.
11. Go to /jffs/scripts/
12. Create a blank file called BBCdns.
13. Create a blank file called getdomainnames.sh and write in the following code but in place of 192.168.1.100 type in your streaming box fixed IP.
https://pastebin.com/raw/9KfiVG26
14. Make sure you have correct permissions (I think 0777) and make the script executable by entering command "chmod a+rx /jffs/scripts/getdomainnames.sh"
15. Type command "sh getdomainnames.sh logfile" hit enter to run the script. This filters out the dnsmasq tail log stuff that is relevant to your streaming box and creates list of domains that were accessed and writes them to BBCdns file.
16. Inspect domains trawled. Add any others you have seen around (Xentrk has a long list). Remove ones that are clearly not related, although be careful as some appear unrelated, but it breaks without these.
17. Create a file called OVPNC1 and write in it the fixed IP of your streaming box. Save.
18. Create another script called "IPSET_VPN_Routing.sh"
19. Enter this code:
https://pastebin.com/raw/VigxYAaN
20. Make executable and ensure correct permissions.
21. Remove Policy Rule for your router from your VPN Client GUI page, but leave Redirect Internet Traffic on Policy Rules (strict) - Not sure if that is necessary, but it works for me.
22. Run script "sh IPSET_VPN_Routing.sh"
23. Check that your streaming box works and that BBC iPlayer works at the same time as Netflix or some US based streaming service also runs. If it doesn't put your hands in the air and scream. (Throw your streaming box out the window.) If it does, go to step 24.
24. Rejoice.
25. Figure out if you need this script to run on WAN/ NAT/ VPN start, in which case you will have to ask someone in more of the know how to ensure that, or just experiment with the naming of the script or both.
26. Tell me what you did in step 25.
27. I have heard it said that you should probably stop the dnsmasq logging as it might burn out a partition? If true then rename dnsmasq.conf.add to something like dnsmasq.conf.donotadd
28. Monitor over a few days to weeks and if it stops working, recheck the domains and add them to the BBCdns file.
Done.
Last edited:
Xentrk
Part of the Furniture
Good summary. For #25, put a call to the vpn routing script inside nat-start.
Now that you have dnsmasq logging working, best practice is to log output to a USB drive. You can use free formatting software such as "Mini Tool Partition" to partition and format a USB drive in linux format. I normally use EXT2 for smaller thumb drives. I have a 2GB drive with four partitions of 500MB each. This is used by entware, absolution, skynet firewall addition and the nvram backup and restore utility.
Last edited:
I think nat-start won't do it for me at least, since the script seems to run only for a couple of hours before I have to restart it again. Meanwhile there has been no restarts.
Any ideas of what could be causing the stop so often? Maybe the VPN is restarting?
Also, do you know how I can get this to work with UDP vpn configs? I tried changing the TCP part of your IPSET_VPN_Routing.sh script to UDP, but it doesn't work. UDP have much better performance than TCP protocol (apparently).
So does Rappy’s guide will solve the error caused with iptables? I’m experiencing the same thing. Trying to route Hulu and Netflix on WAN and everything else over VPN but the iptable error came across and I haven’t found a fix for it yet. I’m running 382 beta 1 on AC86U
Sent from my iPhone using Tapatalk
Sent from my iPhone using Tapatalk
Thanks for the steps... much appreciated! Seems like, https://pastebin.com/raw/pBydGJmu, is not available. Can you repost this?
No, it doesn't solve the errors. I still get this iptables: No chain/target/match by that name. after many of the lines. But for some reason the script still fundamentally works. No idea why.
Having said that, the script currently seems to be broken again. I think the IP ranges have shifted again. I will try again with the dnsmasq.conf.add later on to see what is being accessed today. Eventually I am sure the full domain ranges will be logged.
Edit: Got it working again, but it's erratic. No extra domains were being accessed. The script seems to work more reliably if I use PUTTY to start it instead of SCP. And the "up" and "route-up" command in the Custom Configuration GUI both seem to break not only the script but my entire connection. The "openvpn-event" script in jffs/scripts/ doesn't seem to do anything either.
The script is a bit pointless as it is right now as I have to restart it everytime I watch something, which I could have done before with the Client On/Off toggle.
Last edited:
Yeah, sorry must have set an expiration time for it. Will edit this into the original instructions:
https://pastebin.com/raw/9KfiVG26
Yup, the entire of my script ran just fine, except the error message appeared when it issued
Code:
iptables -t mangle -A PREROUTING -m set --match-set $IPSET dst -j MARK --set-mark $FW_MARKNo idea if it is because of the Merlin build 382 beta 1. My AC68U that was on Tomato ran the exact script just fine. I even compare the iptable version of both and they are the same.
Sent from my iPhone using Tapatalk
Xentrk
Part of the Furniture
The tuning part can take some time. For my routing, It would work most of the time. But sometimes, it would default back to another Openvpn tunnel. If I rebooted the Roku, that would fix it sometimes. I did some more mining and editing of the domain names before it was consistent. I would focus on using the domain names until you finish tuning it. Then, try the conversion to ip addresses.
For example, try removing domain names that don't have the iplayer or uk reference in them to see if that solves the problem with getting other streaming to work after you have finished watching iplayer:
Code:
guzzoni.apple.com
itunes.apple.com.edgekey.net
sylvan.apple.comOr, try to see if the above domains are logged when navigating or watching other streaming media on the Apple TV. If you see them, then these are probably candidates for removal from selective routing and may fix your issue.
nat-start works for me. If you have to bounce a VPN client or do something other changes in the gui, then you may probably will need to rerun the script.
Thanks @Rappy and @Xentrk but it still doesn't work.
I'm trying the simple use case to work where I only want to route clients through VPN. In my OVPNC1 file, I have added a few devices that should route through VPN, but they are still going through WAN.
I have configured VPN to use TCP and also removed all Policy Rules and left the Redirect Internet Traffic on Policy Rules (strict) on.
After execution, same issue of routing table not found
I'm trying the simple use case to work where I only want to route clients through VPN. In my OVPNC1 file, I have added a few devices that should route through VPN, but they are still going through WAN.
I have configured VPN to use TCP and also removed all Policy Rules and left the Redirect Internet Traffic on Policy Rules (strict) on.
Code:
#!/bin/sh
#logger -t "($(basename $0))" $$ Starting IPSET_VPN_Routing.sh..." $0${*:+ $*}."
# Uncomment the line below for debugging
set -xo
ipset create LAN_GW hash:net family inet hashsize 1024 maxelem 65536
ipset create OVPNC1 hash:net family inet hashsize 1024 maxelem 65536
#ipset create OVPNC2 hash:net family inet hashsize 1024 maxelem 65536
# extract LAN ip addresses
ipset add LAN_GW $(nvram get lan_ipaddr)
# extract OVPNC1 ip addresses
for ip in $(awk '{ print $1 }' /jffs/scripts/OVPNC1)
do
ipset add OVPNC1 $ip
done
# extract OVPNC2 ip addresses
#for ip in $(awk '{ print $1 }' /jffs/scripts/OVPNC2)
# do
# ipset add OVPNC2 $ip
# done
# WAN
ip rule del fwmark 0x7000
ip rule add fwmark 0x7000 table 254 prio 9990
#VPN Client 1
ip rule del fwmark 0x1000
ip rule add fwmark 0x1000 table 111 prio 9991
ip route flush cache
###########################################################
#Create table to contain items added automatically by wan #
###########################################################
# WAN
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
# VPN Client 1
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
# Route SlingTV Domain Names to VPN Client 2
#for DNS in $(awk '{ print $1 }' /jffs/scripts/AmazonPrimeDNS)
# do
# iptables -t mangle -D PREROUTING -i br0 -d $DNS -j MARK --set-mark 0x7000
# iptables -t mangle -A PREROUTING -i br0 -d $DNS -j MARK --set-mark 0x7000
# done
#logger -t "($(basename $0))" $$ Ending IPSET_VPN_Routing.sh..." $0${*:+ $*}
Code:
admin@RT-AC68U:/jffs/scripts# sh IPSET_VPN_Routing.sh
errexit off
noglob off
ignoreeof off
interactive off
monitor off
noexec off
stdin off
xtrace on
verbose off
noclobber off
allexport off
notify off
nounset off
vi off
pipefail off
+ ipset create LAN_GW hash:net family inet hashsize 1024 maxelem 65536
+ ipset create OVPNC1 hash:net family inet hashsize 1024 maxelem 65536
+ nvram get lan_ipaddr
+ ipset add LAN_GW 192.168.1.1
+ awk { print $1 } /jffs/scripts/OVPNC1
+ ipset add OVPNC1 192.168.1.106
+ ipset add OVPNC1 192.168.1.104
+ ipset add OVPNC1 192.168.1.116
+ ip rule del fwmark 0x7000
RTNETLINK answers: No such file or directory
+ ip rule add fwmark 0x7000 table 254 prio 9990
+ ip rule del fwmark 0x1000
RTNETLINK answers: No such file or directory
+ ip rule add fwmark 0x1000 table 111 prio 9991
+ ip route flush cache
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables: No chain/target/match by that name.
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables: No chain/target/match by that name.
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables: No chain/target/match by that name.
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables: No chain/target/match by that name.Shouldn't this:
Route SlingTV Domain Names to VPN Client 2
go to mark 0x1000?
It seems like you are overcomplicating things for now with two clients. Try to get it to work with just one first.
Also, I still get those exact same iptable errors, but the script still works. As in I can watch iPlayer and then go to HBOgo and go to netflix and only iPlayer is playing through the VPN.
Try completing the script though and see if it works like mine does. As I keep saying mine shows errors at the
WAN
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
# VPN Client 1
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
# VPN Client 1
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
Similar threads
Similar threads
Similar threads
-
-
OpenVPN / site to site with special security/routing constraints
- Started by MarkusI
- Replies: 7
-
Bug Report: nvram chilli_enable=1 when guest network is removed causes no routing to occur
- Started by Acru
- Replies: 2
-
[Help] Routing VPN server through VPN client (external VPN Provider)
- Started by GuardYaGrill
- Replies: 14
-
-
-
-
Offloading ssl/tls for https (the router handling the https, routing http through)
- Started by JCompagner
- Replies: 7
-
-
Latest threads
-
-
Yet another question about VLAN w/ RT-AX88U Pro in AP mode.
- Started by mbze430
- Replies: 0
-
-
Question about (normal) guest network with ASUS ROG Rapture GT-AX6000
- Started by HHawk
- Replies: 2
-
Sign Up For SNBForums Daily Digest
Get an update of what's new every day delivered to your mailbox. Sign up here!