RT-AX86U Pro running 3006.102.4 dropping IPv6 ULA address

[hartacus]

I use native IPv6, my ISP provides a static /64. I also use a ULA assigned to the router via 'dhcpc-event' script, together with a corresponding range of ULAs provided to LAN devices via 'dnsmasq-postconf' script, to help Pi-hole resolve local hostnames of IPv6 clients sensibly.
After updating to asuswrt-merlin 3006.102.4, I have noticed that the IPv6 ULA assigned to the router is being periodically removed.
The 'dhcpc-event' script checks whether the ULA exists in the output of ip addr show dev

ip addr show dev br0

and if not it adds the ULA via

ip -6 addr add [ULA]/64 dev br0

The system log shows that it's being silently removed (not sure by what), and then re-added again next time the script runs - the script logs a different message when the ULA is found vs when the ULA is added, and it's intermittent.
Prior to the update, the script would log the message when the ULA was added at boot, and then subsequent messages would just be saying that the ULA was found over and over again.
Has anything changed, like flushing all IPv6 addresses when receiving DHCPv6 assignment from the ISP, or anything else that would cause the address to drop?
IPv6 connectivity is fine through all this, it's just messing with the ULA addressing and consequently breaking Pi-hole IPv6 client name resolution.

 

[dave14305]

There are a few places in the firmware where it will run ip -6 addr flush scope global dev br0. Search these files for flush in quotation marks. That would do it.

asuswrt-merlin.ng/release/src/router/rc/udhcpc.c at main · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

asuswrt-merlin.ng/release/src/router/rc/services.c at main · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

 

[dave14305]

Wondering out loud if the Pi-Hole could be configured to advertise just the ULA instead of the router? You’d have to be very careful and know what to do if it breaks everything.

Edit:

https://www.reddit.com/r/ipv6/comments/1af0r2v/using_rpi_to_advertise_ula_how_to_make_it_work

 

[hartacus]

Thanks for the suggestion. I could, perhaps, if it can deal with AiMesh, but I read somewhere it would probably need manual routing to be set up, and that's a whole lot of mess for something that worked properly before this update.

 

[hartacus]

There are a few places in the firmware where it will run ip -6 addr flush scope global dev br0. Search these files for flush in quotation marks. That would do it.

asuswrt-merlin.ng/release/src/router/rc/udhcpc.c at main · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

asuswrt-merlin.ng/release/src/router/rc/services.c at main · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

I've taken a bit of a look through these, and services.c in particular.
In version 3004, the only functions that flush IPv6 global scope addresses are stop_ipv6 and stop_ipv6_tunnel.
In the main branch, it's also in set_ip6_sdnaddr and _add_ip6_lansdnaddr. In both of those, it seems to flush global scope addresses before adding a new (?) address.
I don't know for sure if this is what's causing the issue, because there doesn't seem to be anything logged when the addresses are flushed. However, this is a change from one version to the next so it's possible. Curious about what these functions are used for, why they're triggered (if they are), and if it's possible to disable SDN or whatever triggers them (if they are).

From udhcpc.c, I think the only times the global scope addresses are flushed is when the ISP allocated prefix has changed. That's not happening in my case.

 

[hartacus]

I guess those SDN-related functions are something to do with Guest Network Pro (UI page is SDN.asp). I didn't have any guest networks set up. I've set one up now to see if it makes any difference, but it does not, the ULA still gets silently removed.

 

[dave14305]

What is the lifetime of your ISP delegated prefix (in /tmp/wan0_bound6.env)? Does it coincide in any way with the removal of the ULA?

 

[hartacus]

That is interesting. The prefix seems to have a lifetime of 600s, if I'm reading that correctly:

PREFIXES=[prefix]/56,600,600,class=000b1ca0

The dhcpc-event script is being called every 5 minutes. For the most part, every second run re-adds the ULA, implying it was removed between 5 and 10 minutes after it was added. However, that's not 100% consistent - sometimes two consecutive runs of the script will re-add the ULA, implying it was removed 5 minutes or less after being added.

Jul  2 21:50:34 admin: ULA IPv6 Address found.. Skipping
Jul  2 21:55:37 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:00:41 admin: ULA IPv6 Address found.. Skipping
Jul  2 22:05:44 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:10:47 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:15:50 admin: ULA IPv6 Address found.. Skipping
Jul  2 22:20:53 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:25:56 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:30:59 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:36:02 admin: ULA IPv6 Address found.. Skipping
Jul  2 22:41:05 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:46:08 admin: ULA IPv6 Address found.. Skipping
Jul  2 22:51:12 admin: ULA IPv6 Address not found.. Adding
Jul  2 22:56:15 admin: ULA IPv6 Address found.. Skipping
Jul  2 23:01:18 admin: ULA IPv6 Address not found.. Adding
Jul  2 23:06:18 admin: ULA IPv6 Address found.. Skipping
Jul  2 23:11:20 admin: ULA IPv6 Address not found.. Adding
Jul  2 23:16:20 admin: ULA IPv6 Address found.. Skipping
Jul  2 23:21:20 admin: ULA IPv6 Address not found.. Adding

 

[dave14305]

I think that the prefix is considered “changed” because your ISP is giving a /56 but nvram ipv6_prefix_len_wan is probably 64, so it’s considered different. Please check that one, or log it in your script.

Do you also see the “WAN Prefix Size Requested:” message in the syslog?

EDIT: No, I see there are 2 separate prefix length variables. So probably not the issue but I would definitely focus on the 600 seconds lifetime.

You can turn on debugging of odhcp6 with

nvram set ipv6_debug=1
nvram commit
service restart_dhcp6c

 

[dave14305]

I added a ULA about 2.5 hours ago and it’s still present, but my prefix lifetime is over 3 days, and I only request a /64 (simple network).

 

[hartacus]

I ran nvram get ipv6_prefix_len_wan and it returned 56.
I see WAN Prefix Size Requested:/56, Received:/56 in the syslog.
It does think something is updated though. The log when the script runs is custom_script: Running /jffs/scripts/dhcpc-event (args: updated 6). It's not clear what is updated. The prefix is static according to the ISP (which raises the question of why they only set a lifetime of 10 minutes).
Also, this isn't new - the script has been running on the 'updated' dhcpc-event trigger for about as long as I've used it - probably a few years now - without any issues of the ULA dropping off.

 

[dave14305]

It does think something is updated though. The log when the script runs is custom_script: Running /jffs/scripts/dhcpc-event (args: updated 6). It's not clear what is updated.

It looks like odhcp6c sends “updated” when the DHCPv6 prefix renewal is complete. You might see more if you enable the debugging I mentioned in my earlier post.

asuswrt-merlin.ng/release/src/router/odhcp6c/src/odhcp6c.c at 2bdc88d5c5445b139967ccd039d1e110ff537870 · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

 

[hartacus]

Thanks, I've enabled the debugging you mentioned (sorry I missed that), and it explains a lot about what's being renewed and why:

Jul  3 07:18:17 odhcp6c[25089]: Starting RENEW transaction (timeout 180s, max rc 0)
Jul  3 07:18:17 odhcp6c[25089]: Send RENEW message (elapsed 0ms, rc 0)
Jul  3 07:18:17 odhcp6c[25089]: Got a valid reply after 8ms
Jul  3 07:18:17 odhcp6c[25089]: Starting <POLL> transaction (timeout 300s, max rc 0)
Jul  3 07:18:17 custom_script: Running /jffs/scripts/dhcpc-event (args: updated 6)
Jul  3 07:18:17 admin: Triggering: DHCPC Event / Status Change
Jul  3 07:18:17 admin: ULA IPv6 Address found.. Skipping
Jul  3 07:23:20 odhcp6c[25089]: Starting RENEW transaction (timeout 180s, max rc 0)
Jul  3 07:23:20 odhcp6c[25089]: Send RENEW message (elapsed 0ms, rc 0)
Jul  3 07:23:20 odhcp6c[25089]: Got a valid reply after 11ms
Jul  3 07:23:20 odhcp6c[25089]: Starting <POLL> transaction (timeout 300s, max rc 0)
Jul  3 07:23:20 custom_script: Running /jffs/scripts/dhcpc-event (args: updated 6)
Jul  3 07:23:20 admin: Triggering: DHCPC Event / Status Change
Jul  3 07:23:20 admin: ULA IPv6 Address not found.. Adding
Jul  3 07:28:23 odhcp6c[25089]: Starting RENEW transaction (timeout 180s, max rc 0)
Jul  3 07:28:23 odhcp6c[25089]: Send RENEW message (elapsed 0ms, rc 0)
Jul  3 07:28:23 odhcp6c[25089]: Got a valid reply after 15ms
Jul  3 07:28:23 odhcp6c[25089]: Starting <POLL> transaction (timeout 300s, max rc 0)
Jul  3 07:28:23 custom_script: Running /jffs/scripts/dhcpc-event (args: updated 6)
Jul  3 07:28:23 admin: Triggering: DHCPC Event / Status Change
Jul  3 07:28:23 admin: ULA IPv6 Address not found.. Adding

There's still no log that directly points to flushing the global scope addresses. However, I did a test using ip addr show br0 inside a while sleep 10 loop to try to figure out when the address was removed. It is being removed at pretty much the exact time of the renewal, even where the dhcpc-event script finds that the address is present. So it seems like dhcpc-event is racing with whatever removes the ULA and producing the unreliability.
I'm currently testing a workaround of using a 1 second sleep before checking for the ULA in the dhcpc-event script. So far, it looks like it lets the address be removed each time the renewal occurs, but then can more reliably add it again.

Following the code through further, in udhcpc.c, bound6() calls add_ip6_lanaddr() when it has an address, which then calls _add_ip6_lansdnaddr() which flushes IPv6 global scope addresses before re-constructing the router address from the delegated prefix and then adding it again. That last call is new to version 3006 I believe. So I think that explains why this has started happening at each DHCPv6 renewal after the upgrade.

I'm not sure it's really required to flush global scope addresses in this way for this purpose. It's constructing the address from its basic elements and then just adding it to the interface via shell commands, so it should be possible to construct the address and then test whether it's already assigned to the interface, or specifically remove and re-add that address if it's stale, using very similar methods. That would avoid interfering with ULAs or any other manual configurations.

 

[dave14305]

However, I did a test using ip addr show br0 inside a while sleep 10 loop to try to figure out when the address was removed.

You can also run ip monitor address dev br0 in another window to watch the removal happen in real-time.

So it seems like dhcpc-event is racing with whatever removes the ULA and producing the unreliability.

Yes, as you have discovered the hard way, the dhcpc-event script runs before the actual action is taken to update the prefix/address from DHCPv6.

 

[SomeWhereOverTheRainBow]

You can also run ip monitor address dev br0 in another window to watch the removal happen in real-time.

Yes, as you have discovered the hard way, the dhcpc-event script runs before the actual action is taken to update the prefix/address from DHCPv6.

Would a simple wait until the action happens fix the issue?

 

[RMerlin]

Yes, as you have discovered the hard way, the dhcpc-event script runs before the actual action is taken to update the prefix/address from DHCPv6.

It was done this way because to have it run at the end would have required me to completely rewrite Asus's event handler, and I always aim for the minimum amount of changes to avoid issues when merging new GPLs. Is there any good rationale for going ahead with it to have the event script run afterward instead? One potential reasons for not wanting to change this is someone might want to actually prevent the event, or change something before the event gets applied.

 

[dave14305]

It was done this way because to have it run at the end would have required me to completely rewrite Asus's event handler, and I always aim for the minimum amount of changes to avoid issues when merging new GPLs. Is there any good rationale for going ahead with it to have the event script run afterward instead? One potential reasons for not wanting to change this is someone might want to actually prevent the event, or change something before the event gets applied.

I wouldn’t be in a hurry to change the existing behavior, for the reasons you mention. I do think it’s not obvious to a user that the script will run before the dhcpc event is completed. I used to use it to know when the WAN IP was bound, but eventually switched to wan-event “connected”.

Oh, I see the difficulty since every event type returns with a separate function, so no single place to add a dhcpc-event-end, for example.

I still prefer the option of having the Pi-Hole advertise the ULA prefix.

 

[hartacus]

I'm fine with the current behaviour of dhcpc-event now that I know about it, and agree it's better to accommodate as many use cases as possible with as few changes as possible. In my case, it was pretty easy to add a short sleep to the script for adding my ULA after the dust has settled. Still working fine today.

I think it's more problematic to flush addresses that aren't within the context of the DHCPv6 renewal, and that this behaviour seems to have been introduced to accommodate Guest Network Pro but applies even when the guest network is not enabled.

I could probably get the ULA prefix advertised from one of the Pi-holes, but routing is a concern. From traces in my Pi-hole config files, I did actually try this at one point, back when I was using two routers in a media bridge configuration with the Pi-holes on the far side of the bridge, and it looks like I reverted pretty quickly. I'm using AiMesh now with the Pi-holes on the near side but still some devices don't talk to each other across the mesh. So I'm persisting with configuring it on the router, similarly to how I'm continuing to use the router for DHCP instead of using a Pi-hole which might have avoided the local name resolution issue in the first place if it wasn't for inconsistent behaviour across the bridge/mesh.

 

[RMerlin]

Oh, I see the difficulty since every event type returns with a separate function, so no single place to add a dhcpc-event-end, for example.

Exactly. It's technically doable, but would require rewriting the two functions (IPv4 and IPv6 event handlers), which means any future GPL change to either functions would require manual merges.

if (event == "blah")
     result = bind(blah,0);
     run_custom_script();
     return result;
} else if (event == "bleh")
    ....
}

 

[dave14305]

Exactly. It's technically doable, but would require rewriting the two functions (IPv4 and IPv6 event handlers), which means any future GPL change to either functions would require manual merges.

if (event == "blah")
     result = bind(blah,0);
     run_custom_script();
     return result;
} else if (event == "bleh")
    ....
}

Or wrap the existing invocations of /tmp/dhcp6c and /tmp/udhcpc_wan in your own function that calls a custom script at the end and update the script path being passed to odhcp6c and udhcpc.