bengalih
Senior Member
This is a PSA post based on some issues I just worked through. I wanted to share it here in case anyone else came across something similar as well as a warning for all of you on older routers like the AC68U who are also using IPv6 (and have public IPs for any clients). I don't know how wide-spread this issue is, and I don't know if there are certain things about my configuration that exacerbated this issue, but it could be considered a fairly severe security problem for some.
TL/DR: If using NAT Acceleration (CTE) (LAN > Switch Control) with IPv6, some UDP traffic may get through your firewall even without allowing it via a firewall rule.
I found this out serendipitously while I was trying to setup a Minecraft server for my son. I ran into some strangeness with that task and my IPv6/AAAA setup that had me dig a bit deeper and do some experimentation (basically I wanted to see if Minecraft would work over IPv6, just for the heck of it). After I had it working, I realized that the IPv6 traffic was reaching my internal server (internal as in on the LAN side of the router, but ofc with a public IPv6 address) without me having ever allowed any ports. I got very nervous because I had thought that the router was providing IPv6 firewall services and yet the remote client was able to connect without me opening any ports.
I setup a quick web server on my box and attempted to reach it externall via the IPv6 address and I was unable to. I allowed the port through the IPv6 firewall and I could reach it. Here everything looked like it behaved as it should. I then started Wireshark on my system and launched the Minecraft client on my phone while on cellular (the only remote testing I can currently do), and sure enough I saw UDP packets getting through on the standard UDP 19132 port. So I launched netcat on my phone and started sending some UDP pings to 19132 and went onto some other online scanning sites and had them ping the same port...none of those came through to my IPv6 address!
So, basically I the firewall was seemingly behaving as it should for most standard tests, but somehow the Minecraft client was getting UDP packets through to the same port that was blocked to the other commands. I of course audited all sorts of things, like UPnP and port forwards, etc (things that I knew I already had turned off, but made sure). I even popped into the IRC channel and Merlin confirmed my ip6tables appeared to look good...nothing should be getting through!
I worked on the issue again this morning and took a bunch of captures both on my (remote) Android client, my server and on the router itself. What I saw was interesting: If I captured on all interfaces on the RT-AC68U I saw some packets come in over eth0 (WAN), and some packets come out over eth2 (LAN). However I was seeing packets come out of the LAN and also reach my server that I never saw ingress through the WAN port. And the ones that I was seeing come in through the WAN were not the ones coming out of the LAN and hitting my server. In short, some of the UDP (what I could see come in) were being blocked, but other UDP that I could not see come in was somehow shooting out the LAN port to my server.
I had just recently fixed another issue thanks to @ColinTaylor and in that thread he had reminded me about NAT Acceleration:
Because this was fairly fresh in my mind, I theorized that I wasn't seeing certain traffic in my tcpdump because it was never reaching the kernel in the same way to be processed by tcpdump or the ipv6tables rules. I went ahead and disabled NAT Acceleration and voila! All the packets were now showing coming in over eth0 AND properly being blocked by the firewall. I repeated the testing multiple times and can definitively say this was the fix.
Unfortunately, I can't leave NAT acceleration off, as it cuts my speed down to about 1/3 of what I get with it on. For now I think I will be removing IPv6 until I can upgrade the router.
Additional Notes:
- I tested this same scenario with IPv4 and did not have an issue.
- Interestingly, the packet flows that I saw were identical across dozens of tests. It was always the same packets that didn't show up coming through the WAN but making it out the other end to my server. These were all UDP packets to the same port so should have been processed in the same way by any ip6table rules. Because of the way this could be recreated, I don't believe it was due to the router being stressed and not being able to keep up with traffic. Nothing else was going on during these tests, and I suspect that the Broadcomm implementation of CTE is designed in such a way to be deterministic.
- If you would like to try these tests on your own I'm not 100% sure how you could recreate them short of what I did. I'm sure there are other UDP (and possibly TCP?!?!) flows that might be subjected to this issue, but I don't know what those could be. For my part, I used the Minecraft client on Android. This is a Bedrock client and uses UDP 19132 to connect which should be the same as any other Bedrock client (PC, Switch, XBOX, PS, etc.). So having anyone you know with Minecraft (except the Java PC edition which uses TCP) try to access your server over IPv6 will hopefully show the issue. You do not even need to be running a Minecraft server - just having Wireshark running monitoring your IPv6 is enough to see the connection requests come in. Be sure that they are connecting over IPv6. I'm not sure the client will take an IPv6 address directly, but you can force it by ensuring the FQDN you give them only resolves to an AAAA record.
- After figuring out the root cause, I googled some of my new terms and found this:
www.snbforums.com
It is not exactly the issue I was seeing, but my guess is they are related to some degree. This post makes me think that (luckily) this issue may indeed only be affecting UDP traffic and TCP is safe. In short though, I wouldn't fully trust the RT-AC68U (or really any model that uses NAT Acceleration) to be properly preotecting your public IPv6 devices behind it without putting it through the paces.
All of the above is simply my observations. Clearly disabling NAT Acceleration was the solution, but it might not be the only one. The reasons for why are strictly hypotheses. If you disagree with any of the above, or want to discuss, I'm all ears.
TL/DR: If using NAT Acceleration (CTE) (LAN > Switch Control) with IPv6, some UDP traffic may get through your firewall even without allowing it via a firewall rule.
I found this out serendipitously while I was trying to setup a Minecraft server for my son. I ran into some strangeness with that task and my IPv6/AAAA setup that had me dig a bit deeper and do some experimentation (basically I wanted to see if Minecraft would work over IPv6, just for the heck of it). After I had it working, I realized that the IPv6 traffic was reaching my internal server (internal as in on the LAN side of the router, but ofc with a public IPv6 address) without me having ever allowed any ports. I got very nervous because I had thought that the router was providing IPv6 firewall services and yet the remote client was able to connect without me opening any ports.
I setup a quick web server on my box and attempted to reach it externall via the IPv6 address and I was unable to. I allowed the port through the IPv6 firewall and I could reach it. Here everything looked like it behaved as it should. I then started Wireshark on my system and launched the Minecraft client on my phone while on cellular (the only remote testing I can currently do), and sure enough I saw UDP packets getting through on the standard UDP 19132 port. So I launched netcat on my phone and started sending some UDP pings to 19132 and went onto some other online scanning sites and had them ping the same port...none of those came through to my IPv6 address!
So, basically I the firewall was seemingly behaving as it should for most standard tests, but somehow the Minecraft client was getting UDP packets through to the same port that was blocked to the other commands. I of course audited all sorts of things, like UPnP and port forwards, etc (things that I knew I already had turned off, but made sure). I even popped into the IRC channel and Merlin confirmed my ip6tables appeared to look good...nothing should be getting through!
I worked on the issue again this morning and took a bunch of captures both on my (remote) Android client, my server and on the router itself. What I saw was interesting: If I captured on all interfaces on the RT-AC68U I saw some packets come in over eth0 (WAN), and some packets come out over eth2 (LAN). However I was seeing packets come out of the LAN and also reach my server that I never saw ingress through the WAN port. And the ones that I was seeing come in through the WAN were not the ones coming out of the LAN and hitting my server. In short, some of the UDP (what I could see come in) were being blocked, but other UDP that I could not see come in was somehow shooting out the LAN port to my server.
I had just recently fixed another issue thanks to @ColinTaylor and in that thread he had reminded me about NAT Acceleration:
Because this was fairly fresh in my mind, I theorized that I wasn't seeing certain traffic in my tcpdump because it was never reaching the kernel in the same way to be processed by tcpdump or the ipv6tables rules. I went ahead and disabled NAT Acceleration and voila! All the packets were now showing coming in over eth0 AND properly being blocked by the firewall. I repeated the testing multiple times and can definitively say this was the fix.
Unfortunately, I can't leave NAT acceleration off, as it cuts my speed down to about 1/3 of what I get with it on. For now I think I will be removing IPv6 until I can upgrade the router.
Additional Notes:
- I tested this same scenario with IPv4 and did not have an issue.
- Interestingly, the packet flows that I saw were identical across dozens of tests. It was always the same packets that didn't show up coming through the WAN but making it out the other end to my server. These were all UDP packets to the same port so should have been processed in the same way by any ip6table rules. Because of the way this could be recreated, I don't believe it was due to the router being stressed and not being able to keep up with traffic. Nothing else was going on during these tests, and I suspect that the Broadcomm implementation of CTE is designed in such a way to be deterministic.
- If you would like to try these tests on your own I'm not 100% sure how you could recreate them short of what I did. I'm sure there are other UDP (and possibly TCP?!?!) flows that might be subjected to this issue, but I don't know what those could be. For my part, I used the Minecraft client on Android. This is a Bedrock client and uses UDP 19132 to connect which should be the same as any other Bedrock client (PC, Switch, XBOX, PS, etc.). So having anyone you know with Minecraft (except the Java PC edition which uses TCP) try to access your server over IPv6 will hopefully show the issue. You do not even need to be running a Minecraft server - just having Wireshark running monitoring your IPv6 is enough to see the connection requests come in. Be sure that they are connecting over IPv6. I'm not sure the client will take an IPv6 address directly, but you can force it by ensuring the FQDN you give them only resolves to an AAAA record.
- After figuring out the root cause, I googled some of my new terms and found this:
Enabling NAT Acceleration (CTF) breaks Port Forwarding? (RT-AC68U w/ 386.12_4)
Hi, I've come across an issue with port forwarding not working when IPTraffic is disabled. My intention was to enable CTF to increase throughput. EDIT1: Ahh, further research shows NAT acceleration and port forwarding are incompatible. EDIT2: Why is it that port forwarding for TCP works, but...
www.snbforums.com
All of the above is simply my observations. Clearly disabling NAT Acceleration was the solution, but it might not be the only one. The reasons for why are strictly hypotheses. If you disagree with any of the above, or want to discuss, I'm all ears.
