Strange packet loss with IPv6 going through my Asus RT-AX88U running 3004.388.4

[RamGuy]

Greetings,

I've started to have issues with my Static Global IPv6 traffic going through my Asus RT-AX88U, running AsusWRT-Merlin firmware 3004.388.4. I have static IPv4 and IPv6 from my ISP routed through their Business solution, and there haven't been any issues with the ISP for the past eight years, and this doesn't look like an ISP issue either. The ISP has already confirmed that they can't locate any fault.

I have static /56 Global IPv6 that I'm running as a single /64 on my home network using SLAAC. There is a link network between myself and the ISP where the default gateway on the ISP side responds to ICMP, making troubleshooting easier.

I can run continuous ICMPv6 for hours between my server, my desktop, my wifes desktop, my MacBook Pro, my Mac mini etc. Without any issues. The Asus RT-AX88U responds perfectly without any packet loss when I ping the routers IP within my distributed IPv6 /64 scope. But as soon as I start pinging anything outside my IPv6 /64 scope I have a ton of packet loss. First, I began by running continuous ICMPv6 towards Cloudflare and Google DNS using their IPv6 addresses. This gives a ton of packet loss, from every client I use for testing. I then tried to move things a tad bit closer in the loop and ran continuous ICMPv6 towards my default gateway (the IPv6 address of my ISP within the link network) and this also ends up with heaps of packet loss.

I have pretty much confirmed this to be my Asus RT-AX88U. As I have static IP it's easy for me to simply grab my MacBook Pro and patch myself directly into my GPON modem and run my MacBook Pro directly within the link network I have with my ISP. The MacBook Pro can run ICMPv6 towards Cloudflare DNS, Google DNS and my default gateway without any packet loss.

If I patch my MacBook Pro directly on the LAN of the Asus RT-AX88U, simply to remove all my Ubiquiti gear from the equation, it's back to having heaps of packet loss.


This is only IPv6 having packet loss. Doing all the same using IPv4 gives no packet loss at all.

I'm at a loss at this point. I have set the system log on the RT-AX88U into debug + all, but there is nothing in the system logs giving me any pointers to what is going on. And to make it even stranger, if I use the "Network Tools - Network Analysis" on the RT-AX88U, which for some reason don't let you type in IPv6 addresses, even though the tool supports IPv6.. It doesn't allow for ":", which is rather stupid when the tool supports IPv6, but I digress. So I simply told the tool to utilise IPv6 towards one.one.one.one and the router itself doesn't get any packet loss.

So for whatever reason this behaviour with a ton of IPv6 packet loss only occurs for clients routing themselves through the Asus RT-AX88U, while the Asus RT-AX88U seemingly have no problem with it's own IPv6 traffic.


Does anyone have any pointers on how I'm supposed to get a better understanding of what is going on?

 

[ColinTaylor]

I suspect the router's IPv6 firewall. In the past there were some bugs where it was blocking required packets rather than allowing them. I suggest you SSH into the router and flush all the IPv6 filter rules and set the INPUT/FORWARD/OUTPUT policy to ACCEPT. Then see if that's made a difference.

P.S. Probably also worth examining the current rules with ip6tables-save

 

[RamGuy]

I have to admit, I don't have much experience with IP tables. I have the SPI firewall disabled in the WebUI and ip6tables-save outputs this:

# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:08:35 2023
*nat
REROUTING ACCEPT [6505:1215489]
:INPUT ACCEPT [3031:251099]
:OUTPUT ACCEPT [660:72904]
OSTROUTING ACCEPT [2145:311659]
COMMIT
# Completed on Wed Oct 4 20:08:35 2023
# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:08:35 2023
*mangle
REROUTING ACCEPT [37277:8349655]
:INPUT ACCEPT [21164:3912359]
:FORWARD ACCEPT [14199:3712355]
:OUTPUT ACCEPT [19965:11869069]
OSTROUTING ACCEPT [33277:15532647]
COMMIT
# Completed on Wed Oct 4 20:08:35 2023
# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:08:35 2023
*filter
:INPUT ACCEPT [1360:553257]
:FORWARD ACCEPT [12594:3613127]
:OUTPUT ACCEPT [19965:11869069]
NSFILTER_DOT - [0:0]
:ICMP_V6 - [0:0]
:ICMP_V6_LOCAL - [0:0]
:IControls - [0:0]
:OUTPUT_DNS - [0:0]
:OUTPUT_IP - [0:0]
:OVPNCF - [0:0]
:OVPNCI - [0:0]
:OVPNSF - [0:0]
:OVPNSI - [0:0]
Controls - [0:0]
:UPNP - [0:0]
:VPNCF - [0:0]
:VPNCI - [0:0]
:WGCF - [0:0]
:WGCI - [0:0]
:WGNPControls - [0:0]
:WGSF - [0:0]
:WGSI - [0:0]
:logaccept - [0:0]
:logdrop - [0:0]
:logdrop_dns - [0:0]
:logdrop_ip - [0:0]
-A INPUT -p ipv6-nonxt -m length --length 40 -j ACCEPT
-A INPUT -i br0 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p ipv6-icmp -j ICMP_V6_LOCAL
-A INPUT -p ipv6-icmp -j ICMP_V6
-A FORWARD -j WGSF
-A FORWARD -j OVPNSF
-A FORWARD ! -i br0 -o eth0 -j DROP
-A FORWARD -i br0 -o br0 -j ACCEPT
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -p ipv6-nonxt -m length --length 40 -j ACCEPT
-A FORWARD -p ipv6-icmp -j ICMP_V6
-A FORWARD -j WGCF
-A FORWARD -j OVPNCF
-A OUTPUT -p udp -m udp --dport 53 -m u32 --u32 "0x30>>0xf&0x1=0x0" -j OUTPUT_DNS
-A OUTPUT -p tcp -m tcp --dport 53 -m u32 --u32 "0x34>>0x1a&0x3c@0x8>>0xf&0x1=0x0" -j OUTPUT_DNS
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m limit --limit 1/sec -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 129 -j ACCEPT
-A ICMP_V6 -j DROP
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 131 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 132 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 133 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 141 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 142 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 143 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 148 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 149 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 151 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 152 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 153 -j ACCEPT
-A ICMP_V6_LOCAL -j RETURN
-A OUTPUT_DNS -m string --hex-string "|10706f697579747975696f706b6a666e6603636f6d00|" --algo bm --to 65535 --icase -j logdro p_dns
-A OUTPUT_DNS -m string --hex-string "|0d72666a656a6e666a6e65666a6503636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|1131306166646d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logd rop_dns
-A OUTPUT_DNS -m string --hex-string "|0f376d667364666173646d6b676d726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_ dns
-A OUTPUT_DNS -m string --hex-string "|0d386d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0f3966646d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_ dns
-A OUTPUT_DNS -m string --hex-string "|1265666274686d6f6975796b6d6b6a6b6a677403636f6d00|" --algo bm --to 65535 --icase -j lo gdrop_dns
-A OUTPUT_DNS -m string --hex-string "|086861636b7563647403636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|076c696e77756469056633333232036e657400|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0f6c6b6a68676664736174727975696f03636f6d00|" --algo bm --to 65535 --icase -j logdrop_ dns
-A OUTPUT_DNS -m string --hex-string "|0b6d6e627663787a7a7a313203636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|077131313133333303746f7000|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|057371353230056633333232036e657400|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|077563746b6f6e6503636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0e7a786376626d6e6e666a6a66777103636f6d00|" --algo bm --to 65535 --icase -j logdrop_dn s
-A OUTPUT_DNS -m string --hex-string "|0a65756d6d6167766e627003636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0b726f75746572736173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|037777770b726f757465722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_ dns
-A OUTPUT_DNS -m string --hex-string "|0377777709617375736c6f67696e03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0d72657065617461722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|037777310b726f757465722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_ dns
-A logaccept -m state --state NEW -j LOG --log-prefix "ACCEPT " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logaccept -j ACCEPT
-A logdrop -m state --state NEW -j LOG --log-prefix "DROP " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop -j DROP
-A logdrop_dns -j LOG --log-prefix "DROP_DNS " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop_dns -j DROP
-A logdrop_ip -j LOG --log-prefix "DROP_IP " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop_ip -j DROP
COMMIT
# Completed on Wed Oct 4 20:08:35 2023

 

[RamGuy]

I did try to disable IPv6 on the RT-AX88U, turn it off for five minutes. Turn it back on, re-enable IPv6, turn it off again for five minutes, and turn it on again. Didn't make any difference.

 

[RamGuy]

I tried to enable the SPI firewall, this made ICMPv6 have no packet loss. That sounds very strange to me. Now the output looks like this:

# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:13:46 2023
*nat
REROUTING ACCEPT [662:108718]
:INPUT ACCEPT [414:32984]
:OUTPUT ACCEPT [17:1437]
OSTROUTING ACCEPT [74:9641]
COMMIT
# Completed on Wed Oct 4 20:13:46 2023
# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:13:46 2023
*mangle
REROUTING ACCEPT [4764:804386]
:INPUT ACCEPT [3315:579324]
:FORWARD ACCEPT [1283:161938]
:OUTPUT ACCEPT [3071:1725934]
OSTROUTING ACCEPT [4327:1884054]
COMMIT
# Completed on Wed Oct 4 20:13:46 2023
# Generated by ip6tables-save v1.4.15 on Wed Oct 4 20:13:46 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [3071:1725934]
NSFILTER_DOT - [0:0]
:ICMP_V6 - [0:0]
:ICMP_V6_LOCAL - [0:0]
:IControls - [0:0]
:OUTPUT_DNS - [0:0]
:OUTPUT_IP - [0:0]
:OVPNCF - [0:0]
:OVPNCI - [0:0]
:OVPNSF - [0:0]
:OVPNSI - [0:0]
Controls - [0:0]
:UPNP - [0:0]
:VPNCF - [0:0]
:VPNCI - [0:0]
:WGCF - [0:0]
:WGCI - [0:0]
:WGNPControls - [0:0]
:WGSF - [0:0]
:WGSI - [0:0]
:logaccept - [0:0]
:logdrop - [0:0]
:logdrop_dns - [0:0]
:logdrop_ip - [0:0]
-A INPUT -p ipv6-nonxt -m length --length 40 -j ACCEPT
-A INPUT -i br0 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p ipv6-icmp -j ICMP_V6_LOCAL
-A INPUT -p ipv6-icmp -j ICMP_V6
-A INPUT -j DROP
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -j WGSF
-A FORWARD -j OVPNSF
-A FORWARD -i br0 -o eth0 -j ACCEPT
-A FORWARD -i br0 -o br0 -j ACCEPT
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -p ipv6-nonxt -m length --length 40 -j ACCEPT
-A FORWARD -p ipv6-icmp -j ICMP_V6
-A FORWARD -j WGCF
-A FORWARD -j OVPNCF
-A FORWARD -j DROP
-A OUTPUT -p udp -m udp --dport 53 -m u32 --u32 "0x30>>0xf&0x1=0x0" -j OUTPUT_DNS
-A OUTPUT -p tcp -m tcp --dport 53 -m u32 --u32 "0x34>>0x1a&0x3c@0x8>>0xf&0x1=0x0" -j OUTPUT_DNS
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m limit --limit 1/sec -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 129 -j ACCEPT
-A ICMP_V6 -j DROP
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 131 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 132 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 133 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 141 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 142 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 143 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 148 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 149 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 151 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 152 -j ACCEPT
-A ICMP_V6_LOCAL -p ipv6-icmp -m icmp6 --icmpv6-type 153 -j ACCEPT
-A ICMP_V6_LOCAL -j RETURN
-A OUTPUT_DNS -m string --hex-string "|10706f697579747975696f706b6a666e6603636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0d72666a656a6e666a6e65666a6503636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|1131306166646d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0f376d667364666173646d6b676d726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0d386d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0f3966646d617361787373736171726b03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|1265666274686d6f6975796b6d6b6a6b6a677403636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|086861636b7563647403636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|076c696e77756469056633333232036e657400|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0f6c6b6a68676664736174727975696f03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0b6d6e627663787a7a7a313203636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|077131313133333303746f7000|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|057371353230056633333232036e657400|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|077563746b6f6e6503636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0e7a786376626d6e6e666a6a66777103636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0a65756d6d6167766e627003636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0b726f75746572736173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|037777770b726f757465722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0377777709617375736c6f67696e03636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|0d72657065617461722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A OUTPUT_DNS -m string --hex-string "|037777310b726f757465722d6173757303636f6d00|" --algo bm --to 65535 --icase -j logdrop_dns
-A logaccept -m state --state NEW -j LOG --log-prefix "ACCEPT " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logaccept -j ACCEPT
-A logdrop -m state --state NEW -j LOG --log-prefix "DROP " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop -j DROP
-A logdrop_dns -j LOG --log-prefix "DROP_DNS " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop_dns -j DROP
-A logdrop_ip -j LOG --log-prefix "DROP_IP " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop_ip -j DROP
COMMIT
# Completed on Wed Oct 4 20:13:46 2023

 

[RamGuy]

Upon disabling the SPI for IPv6 again, the packet loss returns.

 

[ColinTaylor]

The IPv6 firewall should always be enabled.

That said it should probably work either way. I suspect the problem is the absence of this line with the firewall disabled:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Which perhaps means that pings are being rate-limited by this command:

-A ICMP_V6 -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m limit --limit 1/sec -j ACCEPT

But that's just a guess.

 

[RamGuy]

Thank you. Everything seems to be behaving as expected now.