Static routes not working as expected in Asuswrt-Merlin?

[metamul]

Yesterday we replaced a Linksys WRT54GL running DD-WRT v24 sp2 with an Asus RT-AC66U running Asuswrt-Merlin 3.0.0.4.354.29 Beta 1. The transition went fairly smoothly--with one important exception.

We run an OpenVPN server inside our network, in "routing" (i.e., not "bridged") mode. The problem, in a nutshell, is this: the VPN clients can ping the VPN server and *some* of the server's peers/clients, but not all. After much poking around we believe the problem is that static routes work differently in Asuswrt-Merlin than they did in DD-WRT.

Here's the "not a nutshell" version. The internal network runs on 192.168.10.x; the VPN network runs on 10.8.0.x. There's a static route defined on the router that directs 10 traffic to the VPN server on the 192 network. If someone inside the 192 network pings someone in the 10 network things work as expected: packets are sent to the router, which sends them to the VPN server, which sends them to the client. But if someone in the 10 network pings someone on the 192 network it's hit or miss. After much poking around we narrowed it down to this: if the someone they ping happens to have a local static route for 10 they (the client on 10) get a reply; if not, the packet appears to go to the router (i.e., the internal gateway), where it's either dropped or forwarded on to the external gateway. To put that a slightly different way: requests from 192 to 10 and replies from 10 to 192 seem to work, and requests from 10 to 192 seem to work, but replies from 192 to 10 don't. (Via the router, that is.)

Ultimately we believe the problem is in the iptables rules. Are we absolutely positive? Not yet. But while we continue to gnaw away at the problem we thought we'd bring our thinking here to people who know more about these things than we do.

There's quite a bit of stuff we can supply in terms of logs, dumps, etc. but given our current theory we'll start with just two simple things: the routing table and the iptables rules from the router. Please let us know if you spot anything odd...

(Just to be clear: we haven't tweaked "under the hood" in either case. We're just providing an "under the hood" dump of the configuration.)

Thanks!

===

# netstat -nr

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
99.99.99.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.8.0.0 192.168.10.10 255.255.255.0 UG 0 0 0 br0
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
99.99.99.0 0.0.0.0 255.255.254.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 99.99.99.1 0.0.0.0 UG 0 0 0 eth0

===

# iptables --list

Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere state NEW
ACCEPT all -- anywhere anywhere state NEW
ACCEPT udp -- anywhere anywhere udp spt:bootps dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:1723
ACCEPT gre -- anywhere anywhere
DROP all -- anywhere anywhere

Chain FORWARD (policy DROP)
target prot opt source destination
all -- anywhere anywhere account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
all -- anywhere anywhere account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere
DROP icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate DNAT
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain FUPNP (0 references)
target prot opt source destination

Chain PControls (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere

Chain logaccept (0 references)
target prot opt source destination
LOG all -- anywhere anywhere state NEW LOG level warning tcp-sequence tcp-options ip-options prefix `ACCEPT '
ACCEPT all -- anywhere anywhere

Chain logdrop (0 references)
target prot opt source destination
LOG all -- anywhere anywhere state NEW LOG level warning tcp-sequence tcp-options ip-options prefix `DROP'
DROP all -- anywhere anywhere

 

[RMerlin]

If it seems to be hit-or-miss between devices one thing I would double check are your netmasks.

 

[coxhaus]

If you’re static for the 10 network does not point to a router or network then you will need a static for every host otherwise your main router will send the packets to the default gateway out the WAN port for the hosts without statics.

 

[metamul]

It's "hit or miss" because some hosts have the static route defined locally. We never did that ourselves; my guess is that DD-WRT was sending (and some hosts were accepting) ICMP REDIRECT packets. That would make sense; the gateway for 10 (the VPN server) is on the same network.

We definitely don't need to manually add a static routes to each host to make this work. Again: it was working Monday morning before we swapped the router.

At this point the theory is that either there's a missing FORWARD rule or that the priority of the existing rules is wrong. Here's a closer look at the current FORWARD chain. The DROP for INVALID comes before the ACCEPT for br0. Is that right or wrong? We're trying to figure it out.

In the meantime, one thing we're going to do is plug the old router into a machine so that we can dump its iptables rules for comparison.

===

# iptables -nvL FORWARD

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1173K 1388M all -- eth0 br0 0.0.0.0/0 0.0.0.0/0 account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
565K 90M all -- br0 eth0 0.0.0.0/0 0.0.0.0/0 account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
1704K 1475M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 DROP all -- !br0 eth0 0.0.0.0/0 0.0.0.0/0
10946 571K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
6 880 ACCEPT all -- br0 br0 0.0.0.0/0 0.0.0.0/0
0 0 DROP icmp -- eth0 * 0.0.0.0/0 0.0.0.0/0
7809 754K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate DNAT
25782 1816K ACCEPT all -- br0 * 0.0.0.0/0 0.0.0.0/0

===

 

[metamul]

Here are side-by-side comparisons of the rules in DD-WRT and Asus. (The first line of each stanza is the command-line. It includes the router's host name so you can tell which stanza is which.) I've included all three chains (FORWARD, INPUT, OUTPUT) for completeness, but our focus is really on the FORWARD chain.

No surprise, but the two are completely different. DD-WRT has a default ACCEPT policy with a final DROP rule; Asus has a default DROP policy. DD-WRT has a single DROP rule at the end; Asus interleaves ACCEPT and DROP rules. DD-WRT has no rules for INVALID; Asus does. And so on.

I've been finding indications that INVALID is tricky, especially in FORWARD. See, for example, http://serverfault.com/questions/440156/packets-marked-invalid-in-forward-rule. (Inconclusive, certainly, but not the only such example.)

===

root@DD-WRT:~# iptables -nvL FORWARD
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 47 -- * vlan1 192.168.10.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * vlan1 192.168.10.0/24 0.0.0.0/0 tcp dpt:1723
0 0 ACCEPT 0 -- br0 br0 0.0.0.0/0 0.0.0.0/0
0 0 TCPMSS tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
0 0 lan2wan 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 0.0.0.0/0 192.168.10.10 udp dpt:1194
0 0 TRIGGER 0 -- vlan1 br0 0.0.0.0/0 0.0.0.0/0 TRIGGER type:in match:0 relate:0
0 0 trigger_out 0 -- br0 * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- br0 * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0

admin@RT-AC66U:/tmp/home/root# iptables -nvL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1214K 1426M all -- eth0 br0 0.0.0.0/0 0.0.0.0/0 account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
593K 96M all -- br0 eth0 0.0.0.0/0 0.0.0.0/0 account: network/netmask: 192.168.10.0/255.255.255.0 name: lan
1771K 1518M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 DROP all -- !br0 eth0 0.0.0.0/0 0.0.0.0/0
459 30566 ACCEPT all -- br0 * 0.0.0.0/0 0.0.0.0/0
11312 591K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
7 950 ACCEPT all -- br0 br0 0.0.0.0/0 0.0.0.0/0
0 0 DROP icmp -- eth0 * 0.0.0.0/0 0.0.0.0/0
8163 780K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate DNAT
26555 1861K ACCEPT all -- br0 * 0.0.0.0/0 0.0.0.0/0

root@DD-WRT:~# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
508 39119 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1723
0 0 ACCEPT 47 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP udp -- vlan1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:520
0 0 DROP udp -- br0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:520
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:520
0 0 DROP icmp -- vlan1 * 0.0.0.0/0 0.0.0.0/0
0 0 DROP 2 -- * * 0.0.0.0/0 0.0.0.0/0
1 71 ACCEPT 0 -- lo * 0.0.0.0/0 0.0.0.0/0 state NEW
866 57845 logaccept 0 -- br0 * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0

admin@RT-AC66U:/tmp/home/root# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
350 17696 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
397K 36M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
133 23204 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 state NEW
118K 8340K ACCEPT all -- br0 * 0.0.0.0/0 0.0.0.0/0 state NEW
624 219K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1723
0 0 ACCEPT 47 -- * * 0.0.0.0/0 0.0.0.0/0
398 38973 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

root@DD-WRT:~# iptables -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 1368 packets, 127K bytes)
pkts bytes target prot opt in out source destination

admin@RT-AC66U:/tmp/home/root# iptables -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 479K packets, 64M bytes)
pkts bytes target prot opt in out source destination

 

[metamul]

I've been finding indications that INVALID is tricky, especially in FORWARD. See, for example, http://serverfault.com/questions/440156/packets-marked-invalid-in-forward-rule. (Inconclusive, certainly, but not the only such example.)

Well, I think I can prove the point now. I deleted that rule (the rule that DROPs INVALID on the FORWARD chain) and everything started working.

Any chance of making that change permanent in Asuswrt-Merlin?

 

[RMerlin]

Well, I think I can prove the point now. I deleted that rule (the rule that DROPs INVALID on the FORWARD chain) and everything started working.

Any chance of making that change permanent in Asuswrt-Merlin?

I will have to see what are the security implications.

In the mean time, you can easily drop that rule through a user script.

 

[metamul]

I will have to see what are the security implications.

I'll see if I can track down any interesting (and authoritative) info on the topic.

In the mean time, you can easily drop that rule through a user script.

Yes, that's what we've done. For anyone else who might be interested, it looks like this:

===

#!/bin/sh

iptables -D FORWARD -m state --state INVALID -j DROP

===

That's in /jffs/scripts/firewall-start.

 

[metamul]

Well, I withdraw my suggestion for a change. I can find hints about INVALID, but nothing definitive. I don't know enough about the issue myself, and neither does anyone else on this end, so there's no basis for suggesting the change other than "it works for us!". That's not good enough.

For the record (in case Google sends others to this thread some day) here are some of the hints we found. Note that our case is in some ways different. For example, we're specifically talking about INVALID packets going through the FORWARD chain, specifically in the context of a static route defined in the router. Some of the references below concern more general configs.

By the way, one thing I/we saw: in many of the more advanced iptables configs DROPs for INVALID in FORWARD were limited to proto tcp.

===

"Some packets are randomly detected as "INVALID" (and then droped) althought ip_conntrack state is ESTABLISHED. That packets are legitimate traffic and not attacks."

http://forum.mandriva.com/en/viewtopic.php?t=42953

===

"Seems like most firewalling documentation suggests DROP'ing
all --state INVALID packets. eg.,

iptables -A INPUT -m state --state INVALID -j DROP

so I hook that up with a -j LOG to see what it catches.
About half of that is HTTP which otherwise seem legitimate.
What am I missing ? I don't want to miss legitimate hits
but i don't want security holes, either."

...

"I've found state INVALID to be not useful, seems to catch the wrong stuff."

http://us.generation-nt.com/answer/iptables-state-invalid-http-help-197480341.html

===

"I set up some iptables rules so it logs and drops the packets that are INVALID (--state INVALID). Reading the logs how can I understand why the packet was considered invalid?"

...

"For debugging specific INVALID packets, looking at them in Wireshark from a dump might work... I have seen packets with SACK fields (actually the normal sequence numbers are changed by the firewall and the ones in SACK option isn't, resulting in SACK values being invalid) that was broken by a firewall being dropped as invalid..."

http://unix.stackexchange.com/quest...packet-was-considered-invalid-by-the-iptables

===

"I am seeing a significant number of matches for the following iptables rule:

iptables -A INPUT -m state --state INVALID -j LOG --log-prefix INPUT-BAD-PACKET:
iptables -A INPUT -m state --state INVALID -j DROP

Before it is suggested that perhaps the server is under attack, I am fairly well certain it is not. A great summary of the problem I am seeing is here: http://www.webmasterworld.com/forum40/1642.htm."

http://mailman.nginx.org/pipermail/nginx/2008-May/005055.html

===

 

[RMerlin]

Now that you mention these references, I do remember reading something about the INVALID condition occasionally catching innocent packets.

I suspect also this is one of these things that will depend on which specific kernel version you are using. If it's a Netfilter bug, it might have been fixed at some point (or be a regression from some point?).

I think being able to monitor exactly what packets are getting hit by the INVALID rule would be a good place to start, especially if you can easily reproduce the scenario with a specific device over an OpenVPN tunnel. Maybe the INVALID rule is doing what it really should, and there IS something wrong with the packet it received. It could indicate an actual bug in OpenVPN, for example.

One recommendation I could make for your particular case: perhaps have the INVALID check only done on packets that aren't sent to, or from the tunnel interface?

 

[metamul]

I think being able to monitor exactly what packets are getting hit by the INVALID rule would be a good place to start, especially if you can easily reproduce the scenario with a specific device over an OpenVPN tunnel. Maybe the INVALID rule is doing what it really should, and there IS something wrong with the packet it received. It could indicate an actual bug in OpenVPN, for example.

One recommendation I could make for your particular case: perhaps have the INVALID check only done on packets that aren't sent to, or from the tunnel interface?

Yes, I thought about adding a LOG before the DROP to see exactly what was being dropped and why. But the lack of clarity around INVALID gives me pause. By this point I've seen a lot of evidence that people who know a lot more about networking than I do get stumped by this. If they get stumped I wouldn't say the prospects for me figuring it out are very good! ("INVALID" is a bit of a misnomer. It's really more like "none of the above". That's part of the problem: it's a catch-all for what doesn't fit anywhere else--for whatever reason.)

There certainly seems to be a lot of mystery around this topic. For example, if I'm reading things correctly (i.e. correctly labeling the "before" and "after") DD-WRT used to DROP INVALID on FORWARD but no longer does. I couldn't find any discussion about why or when the change happened. Furthermore, even when they used to DROP they did it *after* the ACCEPT br0 -> br0 line; Asus has it the other way around, DROPing INVALID before ACCEPTING br0 -> br0.

I've thought about posting at netfilter and/or DD-WRT to see if I could learn a bit more. If I learn anything interesting I'll let you know.

 

[metamul]

Merlin, the lightbulb went off. I think I understand exactly what's happening. I'll use a ping request to map it out--specifically, a ping request from an OpenVPN client to a peer of the OpenVPN server.

The client sends the ping through the tunnel. That means it's sent as an encrypted UDP packet to the router, which then forwards the packet to the server. The server then routes the packet to its peer. The peer receives the ping request and replies--but it doesn't know how to route the reply (the reply is going to 10 while the peer is on 192) so it sends the packet to the gateway. From the perspective of the gateway this is a ping reply to a ping request it never saw. A reply without a request isn't ESTABLISHED. It obviously isn't NEW or RELATED either--so it's INVALID by default. But in fact the router knows what to do with the packet because there's a static route defined for 10 with another gateway, etc.

If the OpenVPN server is running on the peer's gateway (as it can with Asus' firmware) then everything is fine because the initial routing of the ping request from the server to the peer happens inside the firewall. That renders the connection ESTABLISHED. If, on the other hand, the OpenVPN server isn't running on the peer's gateway there will be a problem with the current iptables rules; our setup, which includes defining a static route in the gateway, is completely standard, completely by the book--and it doesn't work.

In a sense the problem here isn't unique to OpenVPN; it could happen whenever traffic is "asymmetric"--that is, whenever the router sees traffic that came in a different door and has to be sent back to that door. But probably few things are OpenVPN-like, so that may be a moot point.

Given all of this your earlier suggestion was along the right lines. I think we can now explain *why* it was along the right lines. And we could, if you like, take it a step further by saying that such a rule should be added to the FORWARD chain whenever a static route is defined.

 

[zack4805]

updates?

Any updates on this?
I am having a similar problem. What's weird is that I can run pings on either side of the vpn tunnel (which is not on the router) to computers on the two subnets but I can't establish Remote Desktop or vnc sessions.
By using the iptables command that metamul gave the sessions start working, I am just worried about any security implications that this has.

Thanks,
Zack

 

[cmoskowitz]

I'm having this same problem.

Are there any updates here?

 

[cmoskowitz]

Sorry to be a nudge but I can't figure out my issue here and it is similar to what other are seeing. My network, 192.168.10.0/24, from open vpn, I can ping 90% of my hosts and connect to them externally VPN'd in, but about 10% I can't and there are no specific rules not allowing them. Does anyone have a recommendation?

 

[RMerlin]

A workaround was already posted in post #8.

 

[cmoskowitz]

Thanks Merlin, I don't have any scripts setup, was afraid to wear out the ram. I 'll look into implementing this.

 

[RMerlin]

Thanks Merlin, I don't have any scripts setup, was afraid to wear out the ram. I 'll look into implementing this.

Writing is what wears off the flash, not reading. A script will only require writing 2-3 times (depending on how much troubleshooting you have to do while setting it up) and will not have any real effect on your flash durability.

 

[cmoskowitz]

ok, thank you. I think I"m retarded, I enabled it but the /jffs/ directory is empty. Can I download them somewhere?

 

[RMerlin]

ok, thank you. I think I"m retarded, I enabled it but the /jffs/ directory is empty. Can I download them somewhere?

You must reboot after enabling JFFS. Once the partition is mounted, you will see the configs and scripts directory automatically created there.