Can't get SNAT/MASQ to work.

[k.alle]

I have been battling this for some time now and vacuumed the internet for examples but it does not simply work for me.

Some data:
Router: RT-AX88U
Firmware Version: Merlin 386.7_2
Router IP: 192.168.1.1
Server local IP: 192.168.1.23


I'm sure there is something trivial that I have missed or what I'm trying to do is impossible.
There is LAN, WAN, TUN11 & TUN12 + Guest network configured.
I "only" need to sort out the TUN11 <-> LAN bit.


Any help I can get is highly appreciated since I have very little knowledge with iptables etc.



OK, I'll try to explain my issue as good as I can.

What does work:
Accessing the server from outside the local network works perfectly.

These are the current rules that allows outside connections:

iptables -I FORWARD -i tun11 -p tcp -d 192.168.1.23 --dport 80 -j ACCEPT
iptables -t nat -I PREROUTING -i tun11 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.23

If I put my computer on it's own VPN client on the computer itself I gain access, but I guess I am leaving the LAN in that scenario.

What does not work:
Connections are being dropped when accessing my server through the EXT_IP from a device within the LAN.
From TCPDUMP I see that the local IP is not masqueraded so I believe the server is replying to the local IP instead of the router that has NAT:ed the package.


What I have tried:

iptables -t nat -I POSTROUTING -i tun11 -p tcp --dst 192.168.1.23 --dport 80 -j SNAT --to-source 192.168.1.1
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -d 192.168.1.0/24 -p tcp --dport 80 -j MASQUERADE

And different variants of the above and also tried setting --to-source to external IP.
But no matter what I do the local IP is always present when looking at TCPDUMP and connection is refused.



Here's the rest of the pertinent IP tables (I think).
I see in the POSTROUTING that tun11 already is MASQ'd but I'm not sure I'm reading all this correctly.

iptables -t nat -vL -n

Chain PREROUTING (policy ACCEPT 15374 packets, 1374K bytes)
 pkts bytes target     prot opt in     out     source               destination
   42  1800 DNSVPN1    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
26385 1711K DNSVPN1    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
   42  1800 DNSVPN2    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
 4877  309K DNSVPN2    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
  922 50440 DNAT       tcp  --  tun11  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:192.168.1.23
25824 1195K GAME_VSERVER  all  --  *      *       0.0.0.0/0            EXTERNAL IP
25824 1195K VSERVER    all  --  *      *       0.0.0.0/0            EXTERNAL IP
 4982  315K DNSFILTER  udp  --  br+    *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
   20   820 DNSFILTER  tcp  --  br+    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53

Chain INPUT (policy ACCEPT 1886 packets, 269K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1092 packets, 134K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 3341 packets, 213K bytes)
 pkts bytes target     prot opt in     out     source               destination
 145K   15M MASQUERADE  all  --  *      tun11   0.0.0.0/0            0.0.0.0/0
24727 3832K MASQUERADE  all  --  *      tun12   0.0.0.0/0            0.0.0.0/0
 4375 1719K MASQUERADE  all  --  *      br101  !EXTERNAL IP       0.0.0.0/0
19605 4714K MASQUERADE  all  --  *      br0     192.168.1.0/24       192.168.1.0/24

iptables -vL -n

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2723  197K INPUT_PING  icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
  12M   11G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
 1172 68920 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
2294K  447M PTCSRVWAN  all  --  !br0   *       0.0.0.0/0            0.0.0.0/0
1793K  663M PTCSRVLAN  all  --  br0    *       0.0.0.0/0            0.0.0.0/0
    2    80 DROP       tcp  --  !lo    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5152
1793K  663M ACCEPT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0            state NEW
2189K  440M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 dpt:68
  335 13500 INPUT_ICMP  icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     udp  --  br2    *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    2   656 ACCEPT     udp  --  br2    *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     udp  --  br2    *       0.0.0.0/0            0.0.0.0/0            udp dpt:68
  790  130K DROP       all  --  br2    *       0.0.0.0/0            0.0.0.0/0
 104K 7179K OVPNSI     all  --  *      *       0.0.0.0/0            0.0.0.0/0
 104K 7179K OVPNCI     all  --  *      *       0.0.0.0/0            0.0.0.0/0
48405 2024K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 3569  335K ACCEPT     tcp  --  tun11  *       0.0.0.0/0            192.168.1.23         tcp dpt:80
    0     0 DROP       all  --  br0    vlan1   0.0.0.0/0            0.0.0.0/0
3162K 3354M ACCEPT     all  --  tun11  br0     0.0.0.0/0            0.0.0.0/0
2831K 1045M ACCEPT     all  --  br0    tun11   0.0.0.0/0            0.0.0.0/0
6248K 6757M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
51808 7299K OVPNSF     all  --  *      *       0.0.0.0/0            0.0.0.0/0
11934  752K WGNPControls  all  --  br2    *       0.0.0.0/0            0.0.0.0/0
   15   960 ACCEPT     all  --  br2    br101   0.0.0.0/0            0.0.0.0/0
    0     0 other2wan  all  --  !br0   br101   0.0.0.0/0            0.0.0.0/0
  217 15565 ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0
 7652  338K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
43924 6945K NSFW       all  --  *      *       0.0.0.0/0            0.0.0.0/0
31927 6189K ACCEPT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0
11997  755K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate DNAT
    0     0 DNSFILTER_DOT  tcp  --  br+    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:853
    0     0 OVPNCF     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Thanks in advance!

 

[eibgrad]

Several points here.

First, when port forwarding over the OpenVPN client, you don't need any FORWARD rules. The FORWARD chain already contains a rule that automatically forwards any traffic that entered the router via a DNAT rule.

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination      
...
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate DNAT
…

Second, you can't specify an input network interface on a POSTROUTING rule since that information is *lost* at that point. You have to specify the source IP/network instead, which is always known.

Invalid:

iptables -t nat -I POSTROUTING -i tun11 -p tcp --dst 192.168.1.23 --dport 80 -j SNAT --to-source 192.168.1.1

Valid:

iptables -t nat -I POSTROUTING -s 10.8.0.0/24 -p tcp --dst 192.168.1.23 --dport 80 -j SNAT --to-source 192.168.1.1

The invalid rules should have generated an error. The fact you didn't notice tells me you're not testing these rules via SSH first. Not a good idea, since you're otherwise flying blind if you just stick them in the event scripts (e.g., nat-start and/or firewall-start).

It sounds like your objective here is to have NAT loopback working. But that's already automatically enabled by the router. There's no need to add your own.

Also, what's w/ this bridge "br101"? That seems to have popped up out of nowhere. I get the impression there's more going on here than you've explicitly stated, like perhaps your own VLANs and bridges configuration via the CLI?? It may be irrelevant, but I'd still like to now ALL the details, just to be sure.

 

[k.alle]

Several points here.

First, when port forwarding over the OpenVPN client, you don't need any FORWARD rules. The FORWARD chain already contains a rule that automatically forwards any traffic that entered the router via a DNAT rule.

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination      
...
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate DNAT
…

Second, you can't specify an input network interface on a POSTROUTING rule since that information is *lost* at that point. You have to specify the source IP/network instead, which is always known.

Invalid:

iptables -t nat -I POSTROUTING -i tun11 -p tcp --dst 192.168.1.23 --dport 80 -j SNAT --to-source 192.168.1.1

Valid:

iptables -t nat -I POSTROUTING -s 10.8.0.0/24 -p tcp --dst 192.168.1.23 --dport 80 -j SNAT --to-source 192.168.1.1

The invalid rules should have generated an error. The fact you didn't notice tells me you're not testing these rules via SSH first. Not a good idea, since you're otherwise flying blind if you just stick them in the event scripts (e.g., nat-start and/or firewall-start).

It sounds like your objective here is to have NAT loopback working. But that's already automatically enabled by the router. There's no need to add your own.

Also, what's w/ this bridge "br101"? That seems to have popped up out of nowhere. I get the impression there's more going on here than you've explicitly stated, like perhaps your own VLANs and bridges configuration via the CLI?? It may be irrelevant, but I'd still like to now ALL the details, just to be sure.

Thank you for the swift response, very kind!

• FORWARD rule

You are correct, I removed it and it still works.
Good catch!
It must be a remnant from when I was trying to get the port forward over TUN11 working.

• POSTROUNTING rule

Right again!
My mistake, copied from a "work in progess" document.
I believe I used '-o tun11' (or just omitted it), just tried that again it rendered no fault message.

I do test them through SSH first, for the same reasons you point out.
Don't want to lock myself out in case of some bad command.

• NAT loopback

I think so, I have seen 'NAT hairpin' mentioned as well. Is that the same thing?

If it is activated by default then I don't know what I'm doing wrong.
The local IP still shows up in TCPDUMP with dropped connection.
As soon as I try from outside my LAN it works perfectly.

tcpdump -i br0 port 80 -n

21:57:41.714776 IP 192.168.1.26.61996 > EXT_IP.80: Flags [S], seq 3089832091, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:57:41.714893 IP EXT_IP.80 > 192.168.1.26.61996: Flags [R.], seq 0, ack 3089832092, win 0, length 0

• br101

I think that showed up after setting up my Guest network, maybe?
The only things I have been doing through the CLI is the previous mentioned forwarding rules since the GUI can't handle conditional port forwarding over VPN connections.

As per my first post, my network consists of:

• LAN
• WAN
• VPN Client 1 (tun11)
• VPN Client 2 (tun12)
• Guest network

There is also 2,4GHz & 5GHz Wifi SSID's.
I can also add that I have DNS filtering activated, routing all Port 53 traffic to my PiHole instance.
Ah, almost forgot. I have IPTV on one of the LAN-ports as well.

 

[eibgrad]

NAT loopback and NAT hairpin are different names for the same thing.

I have seen problems w/ port forwarding and/or NAT loopback when CTF (NAT acceleration) is enabled. It tends to break things. That's why enabling AiProtection, for example, *silently* disables it. You might want to explicitly disable it and see if it makes a difference.

Set LAN>Switch Control>NAT acceleration to Disable.

 

[k.alle]

I see..
Will disabling NAT acceleration lower throughput?

It would be nice to test this theory out.
However I don't have that option available to me under Switch Control.

The QoS and AiProtection options is forcing me to accept some terms from a 3rd party service.
Not to keen on that if it can be avoided.

Is there a shell command that could be used instead?

 

[eibgrad]

I don't know why that router wouldn't have it. My RT-AC68U certainly does.

Yes, disabling it can reduce throughput at the top end. Most modern consumer-grade routers are going to top out around 300-400Mbps w/o CTF. They simply are underpowered given what some ISPs are now offering (gigabit and beyond). But the manufacturers don't want to deal w/ the expense of redesigning their hardware. They'd rather "stretch the soup" w/ these kinds of hacks. But they tend to break stuff since they bypass the firewall in certain situations in order to do their magic.

All that said, it's only going to matter if your ISP is actually capable of delivering enough bandwidth for it to make a difference! IOW, if you're only getting 100Mbps from your ISP, the feature is pointless. You might as well disable it and avoid all the nasty side-effects. But since the manufacturer doesn't KNOW what any given customer is going to have in terms of available bandwidth, they enable it by default, then disable it when certain features are KNOWN to be incompatible.

P.S. I'm NOT an expert on NAT acceleration, and specifically CTF. I know just enough to despise it.

 

[eibgrad]

P.S. You might want to dump nvram to see if there's a relevant variable.

nvram show | grep ctf

In my case, it's apparently ctf_nonat_force (set to 0 since I *always* disable it in the GUI).

 

[eibgrad]

The QoS and AiProtection options is forcing me to accept some terms from a 3rd party service.
Not to keen on that if it can be avoided.

You could always make a backup of JFFS and the settings before doing so, then restore both afterwards.

For all I know, CTF may not be the issue. It's just that I've been burned enough times in the past to be rightfully suspicious.

 

[k.alle]

I don't know why that router wouldn't have it. My RT-AC68U certainly does.

Yes, disabling it can reduce throughput at the top end. Most modern consumer-grade routers are going to top out around 300-400Mbps w/o CTF. They simply are underpowered given what some ISPs are now offering (gigabit and beyond). But the manufacturers don't want to deal w/ the expense of redesigning their hardware. They'd rather "stretch the soup" w/ these kinds of hacks. But they tend to break stuff since they bypass the firewall in certain situations in order to do their magic.

All that said, it's only going to matter if your ISP is actually capable of delivering enough bandwidth for it to make a difference! IOW, if you're only getting 100Mbps from your ISP, the feature is pointless. You might as well disable it and avoid all the nasty side-effects. But since the manufacturer doesn't KNOW what any given customer is going to have in terms of available bandwidth, they enable it by default, then disable it when certain features are KNOWN to be incompatible.

P.S. I'm NOT an expert on NAT acceleration, and specifically CTF. I know just enough to despise it.

Oh, I see.
Some good information right there. Thanks!

I have a symmetric 1Gbps internet connection and I can fully saturate it with the AX88U.
So it would hurt to miss out on that.

Nevertheless, it would be nice to see if the loopback can start to work with NAT accl disabled in order to narrow it down.

 

[k.alle]

P.S. You might want to dump nvram to see if there's a relevant variable.

nvram show | grep ctf

In my case, it's apparently ctf_nonat_force (set to 0 since I *always* disable it in the GUI).

Don't know what this tells me.
Maybe our models are too different?

RT-AX88U:/tmp/home/root# nvram show | grep ctf
size: 78837 bytes (52235 left)

 

[k.alle]

You could always make a backup of JFFS and the settings before doing so, then restore both afterwards.

For all I know, CTF may not be the issue. It's just that I've been burned enough times in the past to be rightfully suspicious.

Yes, might do that as a last resort.


But we have no other way to force SNAT/MASQ with HW accel on?
Using iptables, ip route and such?

 

[eibgrad]

Yes, might do that as a last resort.


But we have no other way to force SNAT/MASQ with HW accel on?
Using iptables, ip route and such?

As I said, the necessary rule is already there in the firewall. There's nothing missing or that needs to be added. The issue is whether NAT acceleration (CTF) is causing it to be bypassed. Adding your own rule is redundant and would presumably have the same results.

 

[k.alle]

OK, got it.

Stupid question incoming.
Could I move the (currently failed by router) SNAT/MASQ downstream to the server?
It's a linux machine with iptables.

E.g. some iptable rule on the server that rewrites the incoming IP to router IP before answering?

[EDIT]
In pseudo code:
1. If DestIp=ExternalIP on Port=80 then SourceIP=RouterIP
2. ...
3. Profit?

 

[eibgrad]

You lost me.

The point of NAT loopback is to be able to reference your *public* IP from inside your own LAN when accessing the server. The idea being that if you're using a portable device and routinely accessing that server both locally and remotely, you don't have to reconfigure the portable device back and forth between a private and public IP as you change environments. IOW, it's purely a convenience, NOT a necessity.

So I fail to see the relevance of *moving* anything from the router to the server, or any other device. NAT loopback is *only* relevant to the router given it is the one doing the port forwarding.

Let's assume, for the sake of argument, the problem is CTF. Are you going to disable it just so you can have access to NAT loopback? Probably NOT, esp. when you can simply reference the server by its local IP address when physically connected to the LAN.

This whole discussion only matters if in fact CTF is NOT the problem, and some other misconfiguration is causing the problem. But I fail to see what that might be at this point.

BTW, when using NAT loopback, are you referencing a DDNS name? Are you sure it's correct (i.e., matches your current public IP)? Are you even sure the server is reachable by your local IP (so far, that's just been an assumption)?

 

[k.alle]

Yes of course it is a convenience.
And for some services hosted on the server it is a huge pain in the back having to deal with self signed certificates (which some services refuse due to security) for LAN access when I have a perfectly working LetsEncrypt cert on the public facing side.

If the problem is CTF I will have to reconsider my current hardware and try to find something that is capable of both NAT accel & NAT loopback.

The idea was just for "science" to see if offloading the rewrite to the server itself would improve the situation as a hail mary, but as I said stupid question.


Yes I am using the DDNS name.
Yes I am sure it is correct, if I use the browser on my phone and disable the WiFi (leaving the LAN) I immediately can access the server.
Yes, the server is reachable on local IP as well.

The issue is clearly visible using TCPDUMP, the local IP of the machine making the request is never rewritten when targeting the DDNS IP.

 

[eibgrad]

Yes of course it is a convenience.
And for some services hosted on the server it is a huge pain in the back having to deal with self signed certificates (which some services refuse due to security) for LAN access when I have a perfectly working LetsEncrypt cert on the public facing side.

But how is NAT loopback going to solve that problem? When you NAT that traffic, what the server *sees* is the LAN ip of the router, NOT the public IP. IOW, from the server's perspective, it *is* a local access, NOT a public one. So whatever issues you have w/ local access will remain.

The issue is clearly visible using TCPDUMP, the local IP of the machine making the request is never rewritten when targeting the DDNS IP.

This is why I'm suspicious. CTF is known to bypass the firewall for certain things. In most cases, it's a black box, so we tend to learn of these issues through direct experience, NOT because it's inevitable. It varies from implementation to implementation. On FT (FreshTomato), for example, they'll mark certain packets in the mangle table for the purposes of bypassing CTF for known problematic situations.

 

[k.alle]

But how is NAT loopback going to solve that problem? When you NAT that traffic, what the server *sees* is the LAN ip of the router, NOT the public IP. IOW, from the server's perspective, it *is* a local access, NOT a public one. So whatever issues you have w/ local access will remain.

Not sure.
If loopback was working, would it not be possible to SNAT the external IP on the incoming LAN request?
That way the server would see the external IP and return it to the router thus not trip up on the cert.
I am not at all familiar how all this works in practice.

However, there are numerous guides on internet for (i presume) other routers that says "just enable SNAT/MASQ (loopback/hairpin) and the problem is solved".
I can not get to that stage unfortunately.

This is why I'm suspicious. CTF is known to bypass the firewall for certain things. In most cases, it's a black box, so we tend to learn of these issues through direct experience, NOT because it's inevitable. It varies from implementation to implementation. On FT (FreshTomato), for example, they'll mark certain packets in the mangle table for the purposes of bypassing CTF for known problematic situations.

That could well be true.
I will try that out later if I can't figure something else out.

 

[eibgrad]

Not sure.
If loopback was working, would it not be possible to SNAT the external IP on the incoming LAN request?
That way the server would see the external IP and return it to the router thus not trip up on the cert.
I am not at all familiar how all this works in practice.

However, there are numerous guides on internet for (i presume) other routers that says "just enable SNAT/MASQ (loopback/hairpin) and the problem is solved".
I can not get to that stage unfortunately.

Well I don't know what you've been reading. Those sites may have not been considering your ultimate objective here, which has only recently come to light. Namely, you're trying to get around the cert issue wrt LAN access. That's NOT the intent of NAT loopback. Granted, you share a common "first step", which is being able to reference the server by its public IP on the WAN while on the LAN, but the second step may indeed require your own NAT loopback rule w/ the public IP. But I don't know if that will work. It will certainly return the traffic back to the router (since it's the default gateway), but I don't know if when it undoes the DNAT whether it will return the packet to the LAN client, or attempt to return to the public IP. I'm just not sure how tightly bound the DNAT is to the SNAT under that scenario. I *think* it will work as intended, but until I see the results w/ my own eyes, I'm NOT willing to guarantee it (as soon as I assume something will work as I expected, half the time it doesn't because I end up having overlooked something).

 

[ColinTaylor]

I don't know why that router wouldn't have it. My RT-AC68U certainly does.

The HND routers don't have CTF they use flow cache instead which is completely different. That's why there's no option for it in the GUI.

 

[ColinTaylor]

I "only" need to sort out the TUN11 <-> LAN bit.

I think I'm missing the point of this statement. Everything that follows seems to be about using NAT loopback on the WAN DDNS address to access an internal server. So I fail to see the relevance of tun11 in this scenario.

You don't describe how you have configured tun11 or any related policy routing rules so perhaps they have a bearing. But ignoring that for the moment, to get NAT loopback to work you would need to create a standard port forwarding rule for port 80 under "WAN - Virtual Server / Port Forwarding" which as far as I can see you haven't done.