PSA - RT-AC68U with CTF/NAT Acceleration may allow IPv6 UDP traffic through firewall

[bengalih]

This is a PSA post based on some issues I just worked through. I wanted to share it here in case anyone else came across something similar as well as a warning for all of you on older routers like the AC68U who are also using IPv6 (and have public IPs for any clients). I don't know how wide-spread this issue is, and I don't know if there are certain things about my configuration that exacerbated this issue, but it could be considered a fairly severe security problem for some.

TL/DR: If using NAT Acceleration (CTE) (LAN > Switch Control) with IPv6, some UDP traffic may get through your firewall even without allowing it via a firewall rule.

I found this out serendipitously while I was trying to setup a Minecraft server for my son. I ran into some strangeness with that task and my IPv6/AAAA setup that had me dig a bit deeper and do some experimentation (basically I wanted to see if Minecraft would work over IPv6, just for the heck of it). After I had it working, I realized that the IPv6 traffic was reaching my internal server (internal as in on the LAN side of the router, but ofc with a public IPv6 address) without me having ever allowed any ports. I got very nervous because I had thought that the router was providing IPv6 firewall services and yet the remote client was able to connect without me opening any ports.

I setup a quick web server on my box and attempted to reach it externall via the IPv6 address and I was unable to. I allowed the port through the IPv6 firewall and I could reach it. Here everything looked like it behaved as it should. I then started Wireshark on my system and launched the Minecraft client on my phone while on cellular (the only remote testing I can currently do), and sure enough I saw UDP packets getting through on the standard UDP 19132 port. So I launched netcat on my phone and started sending some UDP pings to 19132 and went onto some other online scanning sites and had them ping the same port...none of those came through to my IPv6 address!

So, basically I the firewall was seemingly behaving as it should for most standard tests, but somehow the Minecraft client was getting UDP packets through to the same port that was blocked to the other commands. I of course audited all sorts of things, like UPnP and port forwards, etc (things that I knew I already had turned off, but made sure). I even popped into the IRC channel and Merlin confirmed my ip6tables appeared to look good...nothing should be getting through!

I worked on the issue again this morning and took a bunch of captures both on my (remote) Android client, my server and on the router itself. What I saw was interesting: If I captured on all interfaces on the RT-AC68U I saw some packets come in over eth0 (WAN), and some packets come out over eth2 (LAN). However I was seeing packets come out of the LAN and also reach my server that I never saw ingress through the WAN port. And the ones that I was seeing come in through the WAN were not the ones coming out of the LAN and hitting my server. In short, some of the UDP (what I could see come in) were being blocked, but other UDP that I could not see come in was somehow shooting out the LAN port to my server.

I had just recently fixed another issue thanks to @ColinTaylor and in that thread he had reminded me about NAT Acceleration:

NAT (or hardware) acceleration is a Broadcom chipset feature that increases the throughput of WAN to LAN traffic by bypassing the router's CPU. This allows the router to achieve near gigabit routing speeds. Without it the traffic has to be processed in software and is limited by the power of the router's CPU (hence my 300Mbps guess).

The problem with NAT acceleration (AFAIK) is it only works for TCP traffic. So in theory the router shouldn't even try to use it for UDP. But because your test showed that you could only send data after the initial connection it sounded like the router was erroneously trying to use NAT acceleration.

Because this was fairly fresh in my mind, I theorized that I wasn't seeing certain traffic in my tcpdump because it was never reaching the kernel in the same way to be processed by tcpdump or the ipv6tables rules. I went ahead and disabled NAT Acceleration and voila! All the packets were now showing coming in over eth0 AND properly being blocked by the firewall. I repeated the testing multiple times and can definitively say this was the fix.

Unfortunately, I can't leave NAT acceleration off, as it cuts my speed down to about 1/3 of what I get with it on. For now I think I will be removing IPv6 until I can upgrade the router.

Additional Notes:
- I tested this same scenario with IPv4 and did not have an issue.

- Interestingly, the packet flows that I saw were identical across dozens of tests. It was always the same packets that didn't show up coming through the WAN but making it out the other end to my server. These were all UDP packets to the same port so should have been processed in the same way by any ip6table rules. Because of the way this could be recreated, I don't believe it was due to the router being stressed and not being able to keep up with traffic. Nothing else was going on during these tests, and I suspect that the Broadcomm implementation of CTE is designed in such a way to be deterministic.

- If you would like to try these tests on your own I'm not 100% sure how you could recreate them short of what I did. I'm sure there are other UDP (and possibly TCP?!?!) flows that might be subjected to this issue, but I don't know what those could be. For my part, I used the Minecraft client on Android. This is a Bedrock client and uses UDP 19132 to connect which should be the same as any other Bedrock client (PC, Switch, XBOX, PS, etc.). So having anyone you know with Minecraft (except the Java PC edition which uses TCP) try to access your server over IPv6 will hopefully show the issue. You do not even need to be running a Minecraft server - just having Wireshark running monitoring your IPv6 is enough to see the connection requests come in. Be sure that they are connecting over IPv6. I'm not sure the client will take an IPv6 address directly, but you can force it by ensuring the FQDN you give them only resolves to an AAAA record.

- After figuring out the root cause, I googled some of my new terms and found this:

Enabling NAT Acceleration (CTF) breaks Port Forwarding? (RT-AC68U w/ 386.12_4)

Hi, I've come across an issue with port forwarding not working when IPTraffic is disabled. My intention was to enable CTF to increase throughput. EDIT1: Ahh, further research shows NAT acceleration and port forwarding are incompatible. EDIT2: Why is it that port forwarding for TCP works, but...

www.snbforums.com

It is not exactly the issue I was seeing, but my guess is they are related to some degree. This post makes me think that (luckily) this issue may indeed only be affecting UDP traffic and TCP is safe. In short though, I wouldn't fully trust the RT-AC68U (or really any model that uses NAT Acceleration) to be properly preotecting your public IPv6 devices behind it without putting it through the paces.

All of the above is simply my observations. Clearly disabling NAT Acceleration was the solution, but it might not be the only one. The reasons for why are strictly hypotheses. If you disagree with any of the above, or want to discuss, I'm all ears.

 

[Crimliar]

Okay, so my head hurts a little after reading that... Basic NAT should have no effect on IPv6, there is no Network Address Translation to do with standard IPv6! With IPv6 we have local IP addresses, and global IP addresses. Those global IP addresses will work device to device within your LAN just the same as a restricted local IP address (that can't be reached from the outside) would. Using global IP addresses within your LAN, the data never exits your LAN, that's as far as I'm aware how it's supposed to be!
So, if I look at the IPv6 DNS address provided by my router to the client devices here, it uses the Router's global IPv6 address, not a local one.

I apologise in advance if I've got the wrong end of the stick, misread something, or just plain got it wrong!

 

[bengalih]

So, your point about IPv6 not requiring NAT is mostly accurate. For pure IPv6 with global addresses, I would say it almost entirely accurate and since my situation is with global IP addresses I would tend to agree that "NAT should have no effect." The routing to the IPv6 addresses should be based on the IPv6 Firewall rules and NAT is not involved. That being said I can only say two things:

1) Regardless of the theory - the fact is that disabling NAT Accelration on the router makes the IPv6 firewall behave properly is fact. With NAT Acceleration enabled, some UDP packets from the gaming client make it through while others (to the same UDP port) do not. Furthermore, with NAT Acceleration enabled, performing a tcpdump on the WAN interface does not truly show all traffic that is coming into said interface. All of those are facts. I'm open to debate either on the soundness of my testing, or the root reason for the observed behavior, but it is all gathered empirically.

2) No one knows how the NAT Acceleration works. To quote @RMerlin (from IRC):

Only Broadcom knows how CTF works. Even Asus does not have access to its source code.

So, while I agree that NAT Acceleration sounds like it should deal only with NAT, and we've established NAT is/should not be occuring with global IPv6, it is not unreasonable to theorize that there is some unoptimal logic in the code on the chips. I'm sure the NAT Acceleration code is not going to fully bypass IPv6 as it needs to examine every incoming packet first do determine what to do. In this case, it seems clear to me that something in hardware is manipulating the packets due to the fact that we don't ever see them ingress on the WAN (only egress on the LAN) when NAT Acceleration is enabled.
Also, while I may be wrong on this, "NAT Acceleration" is just how Asus referes to the CTF Broadcomm technology. I don't believe that CTF, in general, is technically just a "NAT" thing, it is a hardware based packet offload technology and could very well be designed to deal (if somewhat improperly) with all of this.

To your other point re: using global IPv6 addresses locally. I'm not sure I understand why you bring this up apart from the fact that you misread, or I was not clear:

Correct - global IPv6 addresses can be used to communicate locally within the LAN (in the same way LLA/ULA could). I suppose it depends on the implementation but yes, they should be able to communicate internally without going out to the WAN, in the same way if you assigned public IPv4 range as your local range it shoudl theoretically (mostly) work as long as routing tables were correct.

So yes you are correct again when you state:

So, if I look at the IPv6 DNS address provided by my router to the client devices here, it uses the Router's global IPv6 address, not a local one.

However, this has nothing to do with the devices talking with each other internally. My tests were done with a remote client (Android on cellular network), so the communication is being done with a remote global IPv6 address to a global IPv6 address on my local network. Obviously I have no concerns with other global IPv6 addresses local to my LAN talking to each other without a firewall. The whole point is the firewall should be blocking traffic from non local global IPv6 addresses, which it does not fully do if NAT Acceleration is used.

Your reply however does bring up a good point that there are different ways IPv6 can be configured on the router and I should have shared mine for full disclosure:

If this is still unclear, or you still believe that something is wrong with either my configuration, testing methods, or theory, please let me know. If I can solve the problem I'd love to (short of getting a new router that can handle my throughput with CTF). If I can't solve the problem, I'd still like to properly understand the root-cause.

thanks for replying

 

[Ripshod]

Is this specific to this router alone? Or does this spill over to all Asus routers?
No-one is going to spend much time fixing firmware for an EoL router and, I'm sure others will admit, this is not a known problem. For sure if @RMerlin had see this he'd have fixed it in his firmwares, as well as discuss this issue somewhere on these forums.

 

[bengalih]

I can't say, as I only have this router to test with. I took this to the IRC first because I was trying to get some quick assistance, so Merlin is aware of my findings (I couldn't tell 100% if agreed with them or not, and I didn't want to bother him). He did ackknoweldge that my firewall rules looked right and based on them no UDP traffic should have been getting through.

I fully understand that no firmware fix is coming for this, but I still wanted to put this out there because I know there are still alot of people running this router.

As far as it affecting others...well, I would at least suggest that any router using CTF on a Broadcomm chip might be affected(?)
It's also possible other CTF implementations could suffer similarly. Still, I would recommend others who have Asus routers, especially with CTF enabled try some tests simialr to mine for their own peace of mind.

I think this has gone undiagnosed for several reasons:

1) It seems to only affect IPv6 and UDP combined. My testing methods are limited, I'm not sure how to test it in every permutation. The Minecraft client uses UDP and can work over IPv4 and IPv6, so I was able to use that to test both. In general, most traffic is still IPv4/TCP and so that gets put through the paces every day and I, like everyone else, has never seen an issue with it.

2) The problem doesn't seem to manifest under all standard testing. If you trigger a remote port scan to ping all your UDP ports on an IPv6 address, they will most likely all get blocked and you will not see any of those attempted scans reach your device. This is the behavior I saw. I both monitored with Wireshark and wrote a simply python UDP listener that just ponged a response back to any incoming packet. Under these circumstances all the traffic was blocked as you would expect it to be.
However, when met with a more robust UDP exchange, like that coming from the game client, something in the CTF was clearly intercepting it and forwarding it on to my local server.

IOW, casual testing of the port didn't seem to be enough as the firewall did block basic port testing if the firewall did not have rules enabled.

The purpose of my post was not to ask for a fix, I don't think that is possible with this model. It was as a warning to others running the same hardware and also to notify others that it is possible it could affect other models as well.

I also wasn't necessarily asking for help (primarily because I don't think there is a fix), but rather to show my finding. That said, I am happy to take advice to offer a possible fix or update my findings based on errros in my methods.

BTW, if anyone doesn't have the ability to test or would like me to use the same game client to "connect" to an IPv6 you have (again you don't need to be running the game server to see the connection attempts come through), I would be happy to do so if you DM me a hostname with an AAAA record.