IPv6 Traceroute broken in 388.6 ?

jorhett · May 1, 2024

I've been chasing down an IPv6 issue with my provider. At this point they've fixed their routing, and all traffic over IPv6 appears to be working... except IPv6 traceroute

Code:

$ traceroute6 -n gitlab.jorhett.com
traceroute6 to foobar.jorhett.com (2001:1969:a109:8::8) from 2204:29c0:8121:8f8f:ac8f:e7fa:6a30:1f5f, 64 hops max, 28 byte packets
 1  2204:29c0:8121:8f8f::1  3.650 ms  1.657 ms  1.628 ms
 2  * * *
 3  * * *

All other traffic seems to be working fine

Code:

$ dig +short @2606:4700:4700::1111 foobar.jorhett.com aaaa
2001:1969:a109:8::8

$ curl -6 https://foobar.jorhett.com
<html><body>It works!</body></html>

I'm not certain that it's a local problem, but I'm struggling to prove this. Anyone got any local commands I can run to prove that the traceroute outbound packet or reply isn't being dropped by the Asus?

SomeWhereOverTheRainBow · May 1, 2024

jorhett said:
I've been chasing down an IPv6 issue with my provider. At this point they've fixed their routing, and all traffic over IPv6 appears to be working... except IPv6 traceroute

Code:

$ traceroute6 -n gitlab.jorhett.com traceroute6 to foobar.jorhett.com (2001:1969:a109:8::8) from 2204:29c0:8121:8f8f:ac8f:e7fa:6a30:1f5f, 64 hops max, 28 byte packets 1 2204:29c0:8121:8f8f::1 3.650 ms 1.657 ms 1.628 ms 2 * * * 3 * * *

All other traffic seems to be working fine

Code:

$ dig +short @2606:4700:4700::1111 foobar.jorhett.com aaaa 2001:1969:a109:8::8 $ curl -6 https://foobar.jorhett.com <html><body>It works!</body></html>

I'm not certain that it's a local problem, but I'm struggling to prove this. Anyone got any local commands I can run to prove that the traceroute outbound packet or reply isn't being dropped by the Asus?

Wouldn't be the first time anyone's reported strange behaviors with traceroute. I thinks here it is important to understand the factors that impact, and the limitations that are presented with traceroute. If all factors are addressed (e.g. potential firewall issues or misconfigurarions) , and limitations are understood, then it can be difficult otherwise to explain behavioral oddities.

Crimliar · May 1, 2024

While I get back more than you guys. What I get is distinctly odd!

Code:

Tracing route to gitlab.jorhett.com [2001:1868:a108:80::9]
over a maximum of 30 hops:

  1     2 ms     2 ms     3 ms  2a00:im:not:daft::1
  2     4 ms     4 ms     4 ms  2a00:2302::1102:203:508
  3     *        *        *     Request timed out.
  4     *       12 ms     *     2a00:2302::1102:100:3f
  5     *        *        *     Request timed out.
  6    13 ms    12 ms    12 ms  peer8-et4-0-5.telehouse.ukcore.bt.net [2a00:2380:14::c5]
  7    12 ms    14 ms    13 ms  2a00:2000:2073:2::6a
  8    14 ms    14 ms    14 ms  2a00:2000:2073:2::3d
  9   153 ms   154 ms   192 ms  2001:668:0:2:ffff:0:5995:8bde
 10   148 ms   148 ms   148 ms  2001:668:0:3:ffff:2:0:5ae
 11   158 ms   158 ms   158 ms  2607:f060:0:3::2a3:0
 12   161 ms   162 ms   160 ms  2607:f060:0:3::b:0
 13   158 ms   158 ms   157 ms  2607:f060:0:3::a:1
 14   161 ms   160 ms   160 ms  2607:f060:0:3::82:1
 15     *        *        *     Request timed out.
 16   164 ms   158 ms   158 ms  2607:f060:0:4::90
 17   160 ms   160 ms   160 ms  2607:f060:0:5::f9:1
 18   157 ms   158 ms   158 ms  2607:f060:0:5::10c:1
 19   159 ms   160 ms   160 ms  2607:f060:0:5::101:1
 20   158 ms   159 ms   158 ms  2607:f060:0:5::18:0
 21   161 ms   160 ms   162 ms  2607:f060:0:5::14:0
 22   157 ms   158 ms   158 ms  2001:1868::341
 23   159 ms   160 ms   160 ms  2001:1868:a108:80::9

Trace complete.

RMerlin · May 2, 2024

Traceroute is fine and working as intended. Not every hop will answer to ICMP packets, some will deprioritize them, etc...

learning_curve · May 2, 2024

jorhett said:
I've been chasing down an IPv6 issue with my provider. At this point they've fixed their routing, and all traffic over IPv6 appears to be working... except IPv6 traceroute

Code:

$ traceroute6 -n gitlab.jorhett.com traceroute6 to foobar.jorhett.com (2001:1969:a109:8::8) from 2204:29c0:8121:8f8f:ac8f:e7fa:6a30:1f5f, 64 hops max, 28 byte packets 1 2204:29c0:8121:8f8f::1 3.650 ms 1.657 ms 1.628 ms 2 * * * 3 * * *

All other traffic seems to be working fine

Code:

$ dig +short @2606:4700:4700::1111 foobar.jorhett.com aaaa 2001:1969:a109:8::8 $ curl -6 https://foobar.jorhett.com <html><body>It works!</body></html>

I'm not certain that it's a local problem, but I'm struggling to prove this. Anyone got any local commands I can run to prove that the traceroute outbound packet or reply isn't being dropped by the Asus?

FWIW but without posting all of your data etc online, I can traceroute6 / dig AAAA / curl -6 (i.e. all IPv6) and the exact same again (but all IPv4) to: gitlab.jorhett.com and/or jorhett.com without any problems at all, from any macOS device on my own LAN and/or from a useful traceroute online site (on which you can test; domain or IPv4 or IPv6) like this one: https://tools.keycdn.com/traceroute So it that might indicate that it is a local IPv6 config issue with your own ISP package / setup - possibly (but DO factor in post #2 from @SomeWhereOverTheRainBow first of all...)

My own ISP provides FTTH IPv4 / IPv6 Dual Stack c/w Router (that's setup as a Bridge to my own AX86U). Some time ago (i.e. not specific to any specific Merlin Firmware Release) I noticed, that in my case / with my setup, both traceroute6 AND dig AAAA enquiries (but not Curl -6 oddly...) for whatever reason, always run far better / much faster and with no timed out connections at all - IF - I use a VPN Client (on any macOS device on my own LAN) that has IPV6 as well as IPv4 remote IP addresses. No idea why this is & I've never bothered to investigate why this is, mainly because; IPv6 itself, works effortlessly and perfectly (without needing a VPN Client) on absolutely everything else if/when needed. I've only posted this, just in case, for peace of mind, you wanted to try some Non-Local traceroute6 AND dig AAAA enquiries from any of your own devices on your own LAN etc (but via VPN with / without IPv6 ) as well as, the trace route site that's already been mentioned.

jorhett · May 2, 2024

SomeWhereOverTheRainBow said:
Wouldn't be the first time anyone's reported strange behaviors with traceroute. I thinks here it is important to understand the factors that impact, and the limitations that are presented with traceroute. If all factors are addressed (e.g. potential firewall issues or misconfigurarions) , and limitations are understood, then it can be difficult otherwise to explain behavioral oddities.

Sorry, I should have been specific. I'm a network engineer for a living. I know exactly the UDP packets and ICMP replies sent back at each stop. There's nothing complex about this to me. It's not a question of any configuration other than the Asus router.

RMerlin said:
Traceroute is fine and working as intended. Not every hop will answer to ICMP packets, some will deprioritize them, etc...

As above, I know what traceroute results looked like in January (still have them in the email thread with the ISP) and further, I can traceroute from the router itself which would seem to indicate the packets being dropped are by the Asus on relay, given that all other packets are returned to my assigned /64

Code:

/tmp/home/root# traceroute6 gitlab.jorhett.com
traceroute to gitlab.jorhett.com (2001:1868:a108:80::9), 30 hops max, 16 byte packets
 1  2604:21c0:8121:1::feed:1 (2604:21c0:8121:1::feed:1)  15.149 ms  15.630 ms  15.851 ms
 2  core-he2-rtr1.sailx.co (2604:21c0:100:f1::1eaf)  16.387 ms  15.747 ms  15.876 ms
 3  t0-0-0-0-p99.bgp-he2-rtr1.sailx.co (2604:21c0:100:e::101)  16.387 ms  15.893 ms  15.768 ms
 4  10gigabitethernet5-16.core2.fmt2.he.net (2001:470:1:75e::1)  16.012 ms  23.819 ms  15.871 ms
...

learning_curve said:
I've only posted this, just in case, for peace of mind, you wanted to try some Non-Local traceroute6 AND dig AAAA enquiries from any of your own devices on your own LAN etc

Sorry, I should have included this before. I'm a network engineer, I tested everything in every direction before I opened this query.

1. I can dig from any machine to any remote DNS server. I can't actually find any TCP, UDP, or other ICMP being blocked.
2. If I open up a firewall rule I can dig (udp 53) or traceroute (udp 33434-33534) to any node from outside to the inside.
3. There's no problems of any type with IPv4 traceroute.

The ONLY thing that's not working is IPv6 traceroute from nodes on ethernet or wifi behind the Asus, and it fails to work even if I allow ANY traffic to reach the node running traceroute with ANY packet. So there is something very specific to automatic handling of return ICMP responses to outbound UDP which isn't working.

jorhett · May 2, 2024

Also my provider has shared their Cisco mesh configuration with me (engineer --> engineer) and there's absolutely nothing in their configuration which would impact specific ICMP codes. Further that other customers on the same peering node with sequentially assigned /64s have no problem.

They're completely at a loss to explain this.

ColinTaylor · May 2, 2024

jorhett said:
I can traceroute from the router itself which would seem to indicate the packets being dropped are by the Asus on relay, given that all other packets are returned to my assigned /64

Try setting Firewall > IPv6 Firewall > Enable IPv6 Firewall = No and see if that makes a difference.

SomeWhereOverTheRainBow · May 2, 2024

jorhett said:
Sorry, I should have been specific. I'm a network engineer for a living. I know exactly the UDP packets and ICMP replies sent back at each stop. There's nothing complex about this to me. It's not a question of any configuration other than the Asus router.

As above, I know what traceroute results looked like in January (still have them in the email thread with the ISP) and further, I can traceroute from the router itself which would seem to indicate the packets being dropped are by the Asus on relay, given that all other packets are returned to my assigned /64

Code:

/tmp/home/root# traceroute6 gitlab.jorhett.com traceroute to gitlab.jorhett.com (2001:1868:a108:80::9), 30 hops max, 16 byte packets 1 2604:21c0:8121:1::feed:1 (2604:21c0:8121:1::feed:1) 15.149 ms 15.630 ms 15.851 ms 2 core-he2-rtr1.sailx.co (2604:21c0:100:f1::1eaf) 16.387 ms 15.747 ms 15.876 ms 3 t0-0-0-0-p99.bgp-he2-rtr1.sailx.co (2604:21c0:100:e::101) 16.387 ms 15.893 ms 15.768 ms 4 10gigabitethernet5-16.core2.fmt2.he.net (2001:470:1:75e::1) 16.012 ms 23.819 ms 15.871 ms ...

Sorry, I should have included this before. I'm a network engineer, I tested everything in every direction before I opened this query.

1. I can dig from any machine to any remote DNS server. I can't actually find any TCP, UDP, or other ICMP being blocked.
2. If I open up a firewall rule I can dig (udp 53) or traceroute (udp 33434-33534) to any node from outside to the inside.
3. There's no problems of any type with IPv4 traceroute.

The ONLY thing that's not working is IPv6 traceroute from nodes on ethernet or wifi behind the Asus, and it fails to work even if I allow ANY traffic to reach the node running traceroute with ANY packet. So there is something very specific to automatic handling of return ICMP responses to outbound UDP which isn't working.

I don't think anyone commented or questioned your level of expertise, but thank you for sharing. I concur with what @ColinTaylor says; further, it is something I would have tried before creating this thread. Try disabling the ipv6 firewall on the Asus router to see if you get the same results using traceroute. In my original post I alluded to -(e.g. potential firewall issues or misconfigurarions), but I was not stating that to be the only possible issue. As you have seen from RMerlin, there are other possible outcomes at play here as well.

jorhett · Monday at 4:45 PM

ColinTaylor said:
Try setting Firewall > IPv6 Firewall > Enable IPv6 Firewall = No and see if that makes a difference.

Sorry for the delayed response. No change, either with 388.6 or with 388.7 which I just upgraded to.

SomeWhereOverTheRainBow said:
As you have seen from RMerlin, there are other possible outcomes at play here as well.

Huh? RMerlin's only response was a vague statement that defies observation:

Not every hop will answer to ICMP packets, some will deprioritize them, etc...

1. I have traceroute results from the WAN interface that work just fine
2. I have years of different traceroute results sent in various diagnostic messages

Yes, any random node on any given day might not be responding well -- but we're not talking about any single node. No traceroute works to any location, via any path. Further, it works fine from the router itself, and if I disconnect the Asus and put something else there it works fine from that.

I'm reporting a real problem with repeatable impact that removing the Asus produces different results. Statements like that are just waving your hands and trying to pass off total failure by referring to well-known ICMP handling priority in routing engines -- which absolutely might impact any single node, or a few nodes, but not stop it entirely. And that node behavior would be repeatable with other hardware.

Again, I am not certain that the Asus is entirely at fault. What I'm asking for here is a way to prove that. A way to show that these packets aren't being dropped by the Asus. Can we please focus on objective, repeatable, provable reality here? Let's not wave our hands and suggest it's impossible to know. Instead, let's gather data. How can we be certain that the packet drops are not being done by the Asus? Let's talk about that.

jorhett · Monday at 4:48 PM

To be specific @RMerlin, what's the best way to:

* Confirm the IPv6 firewall outbound session mappings?
* See any packets dropped because they don't match the firewall and/or don't map to outbound traffic?

SomeWhereOverTheRainBow · Monday at 5:03 PM

jorhett said:
* Confirm the IPv6 firewall outbound session mappings?

Well not trying to stir a hornets nest, but have you checked the outbound connections tracking page from the webui while trying to perform the trace route on a separate window? I am not sure if you will find what you are looking for there, I just know there is a page dedicated for outbound connection. Also, what level of logging are you using right now? Have you adjusted the syslog to detect events such as dropped packets from the firewall? If you have some log indications of dropped packets during the time you are running the traceroute, you may be able to identify whats going on. Otherwise, I am not sure what else you can do to obtain the granularity you are seeking.

Thread starter	Title	Forum	Replies	Date
	IPV6 and OpenVPN on Asus Merlin on Asus GT-AXE11000	Asuswrt-Merlin	0	Sunday at 2:52 PM
C	VPN doing a lot of unrecognized options without them even being in the config IPV6 issue	Asuswrt-Merlin	3	May 8, 2024
U	DDNS and ipv6: eth0 not getting WAN ipv6 address.	Asuswrt-Merlin	0	May 3, 2024
T	Wireguard VPN Disable IPV6	Asuswrt-Merlin	0	Apr 20, 2024
K	Have global ipv6 address but unable to access internet over ipv6	Asuswrt-Merlin	39	Apr 8, 2024
J	Help - duckdns has started returning an IPv6 IP address	Asuswrt-Merlin	1	Mar 27, 2024
M	IPv6 and Router Advertisement adding route to delegated /56 via WAN interface	Asuswrt-Merlin	0	Mar 15, 2024
A	DNS/TLS IPv6.	Asuswrt-Merlin	4	Mar 8, 2024
L	When I access the System Log --> IPv6 tab the web server crashes	Asuswrt-Merlin	0	Feb 28, 2024
V	IPv6 Passthrough failure on USB WAN	Asuswrt-Merlin	4	Feb 27, 2024

Search

Search

IPv6 Traceroute broken in 388.6 ?

jorhett

Regular Contributor

SomeWhereOverTheRainBow

Part of the Furniture

Crimliar

Very Senior Member

RMerlin

Asuswrt-Merlin dev

learning_curve

Senior Member

jorhett

Regular Contributor

jorhett

Regular Contributor

ColinTaylor

Part of the Furniture

SomeWhereOverTheRainBow

Part of the Furniture

jorhett

Regular Contributor

jorhett

Regular Contributor

SomeWhereOverTheRainBow

Part of the Furniture

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Members online