Selective Routing with Asuswrt-Merlin

RMerlin · Feb 28, 2016

I'm trying to make sense out of how asuswrt currently uses marks, and it's not that simple. I see numerous locations where the firmware uses a mask when marking a packet for CTF bypass (--set-mark 0x01/0x7), and this mask makes little sense to me. I would expect the mask to need to be 0x01/0x7F or something like that to avoid trashing any of the QoS/BW limiter bits. So right now, I'm not sure if I should also update this 0x7 masks all over the place to also mask whichever bit I allocate to the NAT loopback (or if I am misunderstanding what the mask does - iptables's man page is lacking in details on how these masks work).

RMerlin · Feb 28, 2016

I think I found a place where they better explain how they work when using --set-mark:

If a mask is specified then only those bits set in the mask are modified.

So, set-mark 0x01/0x7 means that QoS would get wiped (since bits 2 and 3 are set to 0 with a 0x01 mark), but the NAT loopback would be untouched (they are outside of that mask).

RMerlin · Feb 28, 2016

ASAT said:

Here's what that looks like.

Code:

0000000001111111 = 0x007F = 127    ASUS QoS and bandwidth limiter
0000000010000000 = 0x0080 = 128    OK
0000000100000000 = 0x0100 = 256    OK
0000001000000000 = 0x0200 = 512    OK
0000010000000000 = 0x0400 = 1024   OK
0000100000000000 = 0x0800 = 2048   Merlin NAT Loopback
0001000000000000 = 0x1000 = 4096   OK
0010000000000000 = 0x2000 = 8192   OK
0100000000000000 = 0x4000 = 16384  OK
1000000000000000 = 0x8000 = 32768  Merlin NAT Loopback

CORRECTED

I see references to using the 0x100 bit in qos.c, so that bit is probably reserved as well (tho the code specifically setting that bit is currently disabled)

Code:

                /* Beginning of the Rule */
                /*
                        if transferred != NULL, class_num must bt 0x1~0x6, not 0x101~0x106
                        0x1~0x6         : keep tracing this connection.
                        0x101~0x106     : connection will be considered as marked connection, won't detect again.
                */

john9527 · Feb 28, 2016

RMerlin said:
I think I found a place where they better explain how they work when using --set-mark:

So, set-mark 0x01/0x7 means that QoS would get wiped (since bits 2 and 3 are set to 0 with a 0x01 mark), but the NAT loopback would be untouched (they are outside of that mask).

So, without a mask specified, all bits are modified? That would mean anyone currently using a '--set-mark 1' (without a mask) would not only potentially invalidate some QoS, but also the NAT loopback.

RMerlin · Feb 28, 2016

The beauty of open source code: if the documentation is lacking, go for the source

This is the connmark code, from kernel sources:

Code:

case XT_CONNMARK_SET:
                newmark = (ct->mark & ~info->ctmask) ^ info->ctmark;
                if (ct->mark != newmark) {
                        ct->mark = newmark;
                        nf_conntrack_event_cache(IPCT_MARK, ct);
                }
                break;

ct->mark is the current packet's mark. The info struct contains the new mark and mask values.

So according to this, the mask determines which bits could potentially be cleared, but not those who could be set. Three examples I've computed to better understand these masks:

Code:

ct = 10
mask = f0
mark 1

(10 & 0f) or 1
0 or 1
resulting mark: 1
[code]

[code]
ct = 10
mask = 0f
mark 1

(10 & f0) or 1
10 or 1
resulting mark: 11

So if I were to set the new 0x8000 loopback bit, on something with a QoS mark of 0x14:

Code:

ct=14
mask = 8000
mark = 8000

(14 & 7fff) or 8000
14 or 8000

resulting mark: 8014

RMerlin · Feb 28, 2016

john9527 said:
So, without a mask specified, all bits are modified? That would mean anyone currently using a '--set-mark 1' (without a mask) would not only potentially invalidate some QoS, but also the NAT loopback.

That's correct (see the three examples I just computed based on the actual kernel source code).

I think I read somewhere that if not provided, a mask of 0xffffffff was assumed, which would match what you describe.

So when in a context where numerous sources can modify various bits, it's probably best to always use masks, and ensure you know who touches which bits. Probably why Asus added that 0x7 mask a few years ago (initially they weren't using any mask).

I'll flip the NAT loopback bit to 0x8000, and use a mask of 0x8000 when setting it. We'll see if it might not also resolves a few random issues related to the NAT loopback and/or QoS.

Hemanshu · Mar 23, 2016

Guys I have a question for which I've looked for an answer for a while.. I live in the US and use PIA as my VPN provider.

I use transmission via entware on my RT-AC66U. Hence currently I had EVERYTHINg going through my VPN client.

Now since I also use netflix on various devices, netflix has stopped working as it detects a VPN (on any/all devices).

How do I let netflix bypass the VPN while keeping all else going through the VPN? Is there a way to enable that? I tried using policy rules to let certain devices to bypass VPN completely, but I suspect by using policy rules I've somehow stopped transmission from going through VPN as well?

MRammeloo · Apr 6, 2016

Hi,

This topic covers my needs pretty well, and though there are a lot of really helpful posts, I cannot get ‘it’ to work.

What is my situation? I have:
- A Synology 413J NAS, with a fixed LAN IP 192.168.1.40, behind…:
- An Asus RT-N66U router, with Merlin 378.55 firmware
- An active AirVPN account

What is my desired end-state?
1. Send all WAN – NAS traffic through a (Air-) VPN tunnel, while at the same time…
2. Accept non-VPN traffic on specific ports, for example ‘FTP-ports’.
3. Block all 1. traffic if the VPN connection goes down, while at the same time still allow 2. traffic. If this is somehow impossible, blocking 1. traffic has priority.

What have I done so far?
1. Before ‘trying this VPN stuff’, I used to have port forwarding rules om my router via the GUI: 21 and 55536-55543. This worked OK, I was able to reach my server from a WAN.
2. I used the AirVPN config generator, and installed it in the Merlin GUI.
3. I changed 'policy routing' and 'start with WAN'. See this screenshot for all settings: http://puu.sh/o8roe/71b567df93.png

4. I created (with notepad++) a nat-start.sh file, uploaded in jffs/scripts/, chmod-ded it to 777, rebooted. Contents of this file:

Code:

#!/bin/sh

touch /tmp/000wanstarted

TAG_MARK=8
ip rule  del  fwmark  $TAG_MARK  2>  /dev/null > /dev/null
ip rule  add  fwmark  $TAG_MARK  table  main  prio 1100
ip route  flush  cache

iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range xxx.xxx.xxx.xxx -p tcp --dport 21 -j MARK --set-mark $TAG_MARK
iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range xxx.xxx.xxx.xxx -p udp --dport 21 -j MARK --set-mark $TAG_MARK
iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range xxx.xxx.xxx.xxx -p tcp --dport 55536:55543 -j MARK --set-mark $TAG_MARK

(Thanks to Martineau, post #423)

What are the results?
1. All WAN-NAS traffic seems to go through the tunnel just fine. I haven’t found a way to really check this, but while downloading a file (both nzb and torrent), I can see the numbers on the VPN Status screen go up. Besides, I have a test torrent, that (correctly) displays my VPN IP as the tracker IP. So, this part of my desired end-state seems to be covered.
2. I cannot reach my NAS through FTP, using my public (i.e. non-VPN) IP
3. I doubt if the ‘kill switch’ really works. If I disable the VPN client, the torrent client immediately continues sending & receiving traffic, using my public IP. However, I can imagine that by disabling the VPN client, I also disable the policy routing rules that goes along with it. If so, is there another way to test the kill switch/policy routing? If not, something else goes wrong…
4. No 000wanstarted is created

What have I done to fix items 2. and 3.?
1. I removed the port forwarding rules in my GUI, unfortunately to no avail…
2. Nothing else… Although I don’t consider myself a complete PC/network newbie, I must admit that I am well out of my comfort/knowledge zone with this stuff. I am not afraid to try stuff, but as I don’t (really) understand what I am doing, ‘trial and error’ is not the way ahead I guess… Furthermore, #423 had some testing commands, but the output makes no sense to me at all.

Some additional questions:
1. Do I need port forwarding to the FTP server on my NAS or not? I guess I do, because with or without VPN, the router needs to know where the server is...
2. In #423, Martinau suggests naming the file VPN_PortSelect.sh. Howevver, if you do so, how does it get started during boot?

As always, I am sure the problem is something really simple, I hope you can give me some pointers in the right direction!

Kind regards,

Martijn

peraburek · Apr 7, 2016

try this for if VPN1 client is down this should kill all traffic

Code:

iptables -I FORWARD ! -o tun11 -s 192.168.1.40 -j DROP

the same goes for VPN2 client, only difference is interface id

Code:

iptables -I FORWARD ! -o tun12 -s 192.168.1.40 -j DROP

regarding access to FTP outside of VPN tunnel, try adjusting this code

Code:

#!/bin/sh
sleep 10
for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
echo 0 > $i
done

# Delete table 100 and flush any existing rules if they exist.

ip route flush table 100
ip route del default table 100
ip rule del fwmark 1 table 100
ip route flush cache
#iptables -t mangle -F PREROUTING
# This line breaks connection from LAN to VPN LAN address via ddns.

# Copy all non-default and non-VPN related routes from the main table into table 100.
# Then configure table 100 to route all traffic out the WAN gateway and assign it mark "1"
#
# NOTE: Here I assume the OpenVPN tunnel is named "tun11".

ip route show table main | grep -Ev ^default | grep -Ev tun11 \
| while read ROUTE ; do
ip route add table 100 $ROUTE
done
ip route add default table 100 via $(nvram get wan_gateway)
ip rule add fwmark 1 table 100
ip route flush cache

# Define the routing policies for the traffic. The rules will be applied in the order that they
# are listed. In the end, packets with MARK set to "0" will pass through the VPN. If MARK is set
# to "1" it will bypass the VPN.

iptables -t mangle -A PREROUTING -i br0 -p tcp --sport 1234 --match iprange --src-range 192.168.1.40 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -i br0 -p tcp --match multiport --sports 1234:5678 --match iprange --src-range 192.168.1.41 -j MARK --set-mark 1

MRammeloo · Apr 7, 2016

@peraburek : thanks, I will try this later. This is all supposed to go into one file? And it replaces ALL the code that was in my original file? What name would you suggest for this file?

Furthermore, I assume the first line at the bottom is for port 21, and the second is for ports 55536:55543, correct? And I assume the target IP in the second line should be .40 as well, correct?

And what about port forwarding in the router?

How do I actually find out the name of the tunnel (i.e. tun11)?

Sorry for all the noob questions...

Martijn

Martineau · Apr 7, 2016

MRammeloo said:
Hi,

4. I created (with notepad++) a nat-start.sh file, uploaded in jffs/scripts/, chmod-ded it to 777, rebooted. Contents of this file:

As per the Wiki, the recognised customisable script files names are shown here

Code:

https://github.com/RMerl/asuswrt-merlin/wiki/User-scripts

so the file should be called nat-start not nat-start.sh

I recommend using different script names to the ones described in the Wiki so that nat-start calls my truly custom code say VPN_Routing.sh so admin/maintenance is so much easier as the script execution is modularised so I can then call say my VPN_Routing.sh explicitly from any other script (init-start,wan-start etc.) or even via cron etc.

I don't use AirVPN, but I have a Synology DS110J and a DS416 hosting services (not FTP) that are defined in the Port Forwarding rules, while I have concurrent VPN1/2 clients connected outbound.

Incoming requests via the WAN I/P (my VPN ISP does not allow unsolicited incoming VPN requests) to the ports on either NAS work fine.

MRammeloo · Apr 7, 2016

Martineau said:
As per the Wiki, the recognised customisable script files names are shown here

Code:

https://github.com/RMerl/asuswrt-merlin/wiki/User-scripts

so the file should be called nat-start not nat-start.sh

I recommend using different script names to the ones described in the Wiki so that nat-start calls my truly custom code say VPN_Routing.sh so admin/maintenance is so much easier as the script execution is modularised so I can then call say my VPN_Routing.sh explicitly from any other script (init-start,wan-start etc.) or even via cron etc.

I don't use AirVPN, but I have a Synology DS110J and a DS416 hosting services (not FTP) that are defined in the Port Forwarding rules, while I have concurrent VPN1/2 clients connected outbound.

Incoming requests via the WAN I/P (my VPN ISP does not allow unsolicited incoming VPN requests) to the ports on either NAS work fine.

Martineau,

1. I created a new file, called nat-start. Still no luck.
2. Then I started entering all lines one by one in putty. Some observations:
a. The line 'touch /tmp/test' immediatly yielded results. This never happened before, leading me to believe that my script doesn't run at all, and it never did.
b. Entering the first iptables line, produced an error, saying something like ip xxx.xxx etc. is not valid. I always assumed all the xx-es were meant as some kind of mask, this seems to be wrong. After feeling totally stupid, I changed the xx-es to the IP of my server 192.168.1.40 and the error in putty was gone.
3. However, after entering all the lines in putty, I still cannot reach my FTP server from a WAN device...

My script is now:

#!/bin/sh

touch /tmp/test1618

TAG_MARK=8
ip rule del fwmark $TAG_MARK 2> /dev/null > /dev/null
ip rule add fwmark $TAG_MARK table main prio 1100
ip route flush cache

iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range 192.168.1.40 -p tcp --dport 21 -j MARK --set-mark $TAG_MARK
iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range 192.168.1.40 -p udp --dport 21 -j MARK --set-mark $TAG_MARK
iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range 192.168.1.40 -p tcp --dport 55536:55543 -j MARK --set-mark $TAG_MARK

Conclusions:
- my script does not run at boot time, not even after waiting for 10 minutes
- manually entering the script in putty gives no errors, but still has no use, therefore (?) something in the script itself is not right as well

(Edit: 'Enable JFFS custom scripts and configs' IS set to 'yes' in the GUI.)

(Edit2: the sys log says 'Jan 1 01:00:12 custom script: Running /jffs/scripts/nat-start'. So, the script appears to be started, but I cannot find any evidence that it actually does anything)

Any more ideas?

Kind regards,

Martijn

RMerlin · Apr 7, 2016

Make sure your script is properly encoded for UNIX.

Code:

dos2unix /jffs/scripts/nat-start

MRammeloo · Apr 8, 2016

RMerlin said:
Make sure your script is properly encoded for UNIX.

Code:

dos2unix /jffs/scripts/nat-start

Hi RMerlin,

Too bad:
-sh: dos2unix: not found

Edit:
However... Notepad++ hast the option to convret the EOL to UNIX, and that did the trick. The 'touch' line created a file in tmp. Conclusion: for the first time, the script actually executed.

Edit 2: unfortunately, I still cannot reach my FTP server on my public IP from a WAN IP.

Is there some lof file that might be useful posting here?

Martijn
(To state the obvious, I'm working from a Win 10 machine...)

Martineau · Apr 8, 2016

MRammeloo said:
Too bad:
-sh: dos2unix: not found

Does this remove the CR characters in lieu of the missing utility

Code:

sed -i 's/\r//g' /jffs/scripts/nat-start

[/QUOTE]

MRammeloo · Apr 8, 2016

Martineau said:
Does this remove the CR characters in lieu of the missing utility

Code:

sed -i 's/\r//g' /jffs/scripts/nat-start

[/QUOTE]
Martineau,

I have solved the EOL issue, thanks to a very handy feature in notepad++. Because of the 'touch-line', I can confirm that the script is actually being executed.

Still no joy regarding reaching the server though...

Martijn

Lirilin · Aug 2, 2016

Hello! trying to config Policy Routing on AC88U (380.61 Beta1) but faced some problems.
with TUN interface everything works well, but my VPN-provider only offer TUN-TCP, and it's speed just 10mbps
with TAN-UDP while redirect all traffic i can get near 50mbps but when i turn on Policy rules they just do not work, all traffic just go to WAN
and i do not see any rules when checked via ssh: ip rule

RMerlin · Aug 2, 2016

Policy-based routing only works with TUN.

jaydee99 · Oct 21, 2016

Hi,

Could anyone please tell me why i still can't access 5000 and 5001 ports on my synology with this nat-start script :

Code:

Synology_NAS="192.168.0.2"

ip rule add fwmark 1 table  111 prio 30001
ip rule add fwmark 2 table  112 prio 30002
ip rule add fwmark 3 table main prio 30003
ip route flush cache

iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range $Synology_NAS -p tcp --dport 5000:5001 -j MARK --set-mark 3

Thanx !

Martineau · Oct 22, 2016

jaydee99 said:
Hi,

Could anyone please tell me why i still can't access 5000 and 5001 ports on my synology with this nat-start script :

Code:

Synology_NAS="192.168.0.2" ip rule add fwmark 1 table 111 prio 30001 ip rule add fwmark 2 table 112 prio 30002 ip rule add fwmark 3 table main prio 30003 ip route flush cache iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range $Synology_NAS -p tcp --dport 5000:5001 -j MARK --set-mark 3

Thanx !

That's the problem with old posts, ASUS,@RMerlin,@john9527 are actively tweaking their proprietary modificatons, so old posts have to be reviewed for compatibility ;-)

I would check your RPDB table to ensure that the order of the RPDB fwmark rules is correct, and that the appropriate -t mangle PREROUTING chain rule is being triggerred.

Both @john9527/@RMerlin suggest new best practices when it comes to applying custom fwmarks, namely stay way from fwmarks such as 1,2,3 etc. and possibly preserve any existing fwmarks when you apply your custom bit setting by including a mask.

RMerlin creates RPDB prio rules starting at 100x (which seems unnecessarily restrictive?), so if you want to force selective fwmark to the WAN then you will need to set your fwmark rule prio to 999 or less

i.e.

Code:

ip rule add fwmark 0x7000 table main prio 999

iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range $Synology_NAS -p tcp --dport 5000:5001 -j MARK --set-mark 0x7000/0x7000

I currently create the RPDB table as follows using a custom version of RMerlins /vpnrouting.sh script.
NOTE: Given there are 5 VPN clients I have chosen fwmarks 0x1000 - 0x5000 for the clients and 0x7000 for the WAN although @john9527 hints that in his future firmware he will apparently use 0x3000 for his own purposes :-(

Code:

admin@RT-AC68U:/tmp/home/root# ip rule

0:    from all lookup local
10000:    from all fwmark 0x7000 lookup main
10001:    from all fwmark 0x1000 lookup NewYork
10002:    from all fwmark 0x2000 lookup HongKong
10005:    from all fwmark 0x5000 lookup Maidenhead
10101:    from 172.0.0.1 lookup NewYork
10102:    from 168.0.0.1 to 173.230.240.197 lookup NewYork
10103:    from 10.88.8.114 lookup NewYork
10301:    from 172.0.0.2 lookup HongKong
10901:    from 172.0.0.5 lookup Maidenhead
32766:    from all lookup main
32767:    from all lookup default

Effectively real I/P address 10.88.8.114 is designated (via the VPN Policy routing GUI) to use VPN1 (NewYork)

I have just issued the following:

Code:

admin@RT-AC68U:/tmp/home/root# iptables -t mangle -A PREROUTING -i br0 -m iprange --src-range 10.88.8.114 -p tcp -j MARK --set-mark 0x7000/0x7000

admin@RT-AC68U:/tmp/home/root# iptables -nvL --line -t mangle

Chain PREROUTING (policy ACCEPT 29 packets, 3269 bytes)
num  pkts bytes target  prot opt in  out  source  destination
1  108  7595 MARK  all  --  tun15  *  0.0.0.0/0  0.0.0.0/0  MARK xset 0x1/0x7
2  544  184K MARK  all  --  tun12  *  0.0.0.0/0  0.0.0.0/0  MARK xset 0x1/0x7
3 1822  852K MARK  all  --  tun11  *  0.0.0.0/0  0.0.0.0/0  MARK xset 0x1/0x7
4    0     0 MARK  all  --  tun21  *  0.0.0.0/0  0.0.0.0/0  MARK xset 0x1/0x7
5   12  608  MARK  all  --  !vlan2 *  0.0.0.0/0  xxx.xxx.xxx.xxx  MARK or 0x8000
6  333 24787 MARK  tcp  --  br0    *  0.0.0.0/0  0.0.0.0/0  source IP range 10.88.8.114-10.88.8.114 MARK or 0x7000

and now 10.88.8.114 is no longer is routed via VPN1 (NewYork) but instead is routed via the WAN.
NOTE: Removing the RPDB entry

Code:

10103:    from 10.88.8.114 lookup NewYork

would have the same effect!

Thread starter	Title	Forum	Replies	Date
E	Multi source selective port forward?	Asuswrt-Merlin	1	Dec 28, 2023
M	OpenVPN / site to site with special security/routing constraints	Asuswrt-Merlin	7	Apr 8, 2024
	Bug Report: nvram chilli_enable=1 when guest network is removed causes no routing to occur	Asuswrt-Merlin	2	Jan 9, 2024
G	[Help] Routing VPN server through VPN client (external VPN Provider)	Asuswrt-Merlin	14	Dec 27, 2023
P	IPv6 routing table flags	Asuswrt-Merlin	2	Dec 21, 2023
F	Help with routing 2 VLAN's to the same Ethernet Port	Asuswrt-Merlin	36	Nov 1, 2023
	RT-AC86U VPN Director Not Routing Through WAN	Asuswrt-Merlin	1	Oct 16, 2023
	Offloading ssl/tls for https (the router handling the https, routing http through)	Asuswrt-Merlin	7	Oct 9, 2023
A	Policy-based port routing to bypass NordVPN client	Asuswrt-Merlin	10	Sep 12, 2023
A	whats the command to enable routing to my secondary wan?	Asuswrt-Merlin	3	Sep 8, 2023

Selective Routing with Asuswrt-Merlin

Asuswrt-Merlin dev

Asuswrt-Merlin dev

Asuswrt-Merlin dev

Part of the Furniture

Asuswrt-Merlin dev

Asuswrt-Merlin dev

New Around Here

New Around Here

Senior Member

New Around Here

Part of the Furniture

New Around Here

Asuswrt-Merlin dev

New Around Here

Part of the Furniture

New Around Here

New Around Here

Asuswrt-Merlin dev

Occasional Visitor

Part of the Furniture

Similar threads

Similar threads

Sign Up For SNBForums Daily Digest