IoT Devices falling off nodes (only) always requires reboot to fix

[jksmurf]

I reported this to my open Asus support ticket. Hopefully, if this does indeed turn out to be the cause of some of our IoT issues, that the source can be found by Asus and resolved, allowing our routers to be used to their full advertised capabilities.

So to be clear; yours was observed on an ASUS Stock FW System, not a Merlin FW one?

 

[penguin22]

So to be clear; yours was observed on an ASUS Stock FW System, not a Merlin FW one?

Correct with one exception. I observed with both stock and Merlin running on the BE86U node. I am currently on the most recent stock version due to the ticket I have open with ASUS. My BE88U router is running the current Merlin firmware.

 

[jksmurf]

Just an update since I removed the Access mode setting from all ports on my attached AiMesh nodes, via the LAN/VLAN settings page on my main router on the 25 Feb 26 and reverted to All (default).

The post in which I noted I did this was in the FW release thread, so for completeness I am reporting the current status here in my original thread.

I am very happy to report the devices on BOTH nodes experiencing drop offs from the wireless IoT devices have completely disappeared, they came up straight away on reboots, are all pingable and accessible and it’s been this way for almost 10 days now. Very, very happy with the stability.

Note there is a known user who (pointed this solution out in the first instance) has had an identical issue and solution and another user who uses the Access or Trunk mode and does not.

Until a hoped for FW update addresses this unfortunately I cannot use the advertised VLAN capability of my specific models to assign wired devices attached to these nodes IP addresses in the subnet of my choosing. However at least my more important wireless devices are stable.

 

[Tech9]

One of the nodes on each network is not GNP compatible. Perhaps related to the issue?

 

[jksmurf]

One of the nodes on each network is not GNP compatible. Perhaps related to the issue?

Maybe. But at this juncture it’s just guessing.

The Main and two nodes are both GNP AND VLAN capable. The RT-AX58U is not VLAN capable but is ostensibly supported by AiMesh.

Your suggestion is that devices are falling off the mesh (and off fully VLAN and GNP capable nodes) because a third node (which has the fewest number of devices falling off it) is not GNP (nor indeed VLAN) capable.

I struggle with that possibility but cannot rule it out.

 

[Seth Harman]

Just an update since I removed the Access mode setting from all ports on my attached AiMesh nodes, via the LAN/VLAN settings page on my main router on the 25 Feb 26 and reverted to All (default).

The post in which I noted I did this was in the FW release thread, so for completeness I am reporting the current status here in my original thread.

I am very happy to report the devices on BOTH nodes experiencing drop offs from the wireless IoT devices have completely disappeared, they came up straight away on reboots, are all pingable and accessible and it’s been this way for almost 10 days now. Very, very happy with the stability.

Note there is a known user who (pointed this solution out in the first instance) has had an identical issue and solution and another user who uses the Access or Trunk mode and does not.

Until a hoped for FW update addresses this unfortunately I cannot use the advertised VLAN capability of my specific models to assign wired devices attached to these nodes IP addresses in the subnet of my choosing. However at least my more important wireless devices are stable.

Are you going to take another crack at using managed switches to tag wired VLAN clients?

 

[jksmurf]

Are you going to take another crack at using managed switches to tag wired VLAN clients?

No. The reason being is that I know those switches work and is wholly independent of AiMesh, GNP and VLAN propagation to Mesh nodes.

Absent ASUS acknowledging this is an AiMesh flaw and providing a FW update OR that I (we) discover “the very special set of combinations or settings” that causes it to fail for a couple of us, I see no merit in applying that workaround but would rather really like to run it to ground.

Unfortunately there’s only a very small sample size of folks with similar networks who are inclined, motivated, knowledgeable or bloody-minded enough to do that.

 

[degrub]

Maybe. But at this juncture it’s just guessing.

The Main and two nodes are both GNP AND VLAN capable. The RT-AX58U is not VLAN capable but is ostensibly supported by AiMesh.

Your suggestion is that devices are falling off the mesh (and off fully VLAN and GNP capable nodes) because a third node (which has the fewest number of devices falling off it) is not GNP (nor indeed VLAN) capable.

I struggle with that possibility but cannot rule it out.

Have you run a layout where the AX58U is removed as a node, powered down for a few days, and the rest (main + nodes) re-booted to see what happens with the IOTs dropping off of the GNP/VLAN capable nodes ?
i recall several threads/cases where older , AC based IIRC, were causing issues apparently and the issue went away when they were removed. Don't know about less capable AX devices.

 

[Tech9]

Your suggestion is that devices are falling off the mesh (and off fully VLAN and GNP capable nodes) because a third node (which has the fewest number of devices falling off it) is not GNP (nor indeed VLAN) capable.

I don't know what causes your issues. On top of everithing your nodes run Asuswrt-Merlin with most likely additional customizations. This is not the best setup to evaluate AiMesh performance the way it was designed by ASUS. Too many variables in your case. I would agree there is AiMesh issue if your routers were all running original firmware where AiMesh comes from.

 

[GoNz0]

Given up on nodes for IOT, I have since set the IOT just to the router and moved my problem device to that network as the rest are happy enough on the main one

 

[jksmurf]

Have you run a layout where the AX58U is removed as a node, powered down for a few days, and the rest (main + nodes) re-booted to see what happens with the IOTs dropping off of the GNP/VLAN capable nodes ?

No, not yet. It’s obviously the next step. Unfortunately this is my (very) remote network, so it’s hard to test it and even when I’m there once, max twice a year it’s not for long. But certainly something to test.

Whilst the RT-AX58U is not GNP capable as a main router, as a node (in an AIMesh network) it is perfectly capable to work with a GNP Main router, albeit with the known limitations on Wi-Fi interfaces. It is not VLAN capable like my other two nodes. I only use one Wi-Fi interface on it anyway, my IoT 2.4 GHz one (the main Wi-Fi obviously also gets propagated to it by default).

However you’d need to look at @penguin22 hardware (all BE class AND all Stock Nodes) to see if there’s a node with the same lack of features as mine causing his issues. And equally whether @visortgw successful setup includes only GNP and VLAN capable nodes. Both use Merlin. I do not know exactly what addons they use.

I don't know what causes your issues. On top of everithing your nodes run Asuswrt-Merlin with most likely additional customizations. This is not the best setup to evaluate AiMesh performance the way it was designed by ASUS. Too many variables in your case. I would agree there is AiMesh issue if your routers were all running original firmware where AiMesh comes from.

I don’t know either, I only know what solves it, which if nothing else is worth reporting.

As regards whether it will work on stock, maybe, but my gut feeling is no. I’m not in a position to revert them all to stock and test it. The reason for my belief that it is not AsusWRT Merlin related is due to the closed source nature of the AIMesh components of the FW, which is common to either stock or Merlin. I’ve stated this before but my reading of the Merlin FW is that it does not make wholescale changes in this regard.

I cannot prove it and I am not in a position to try. What I can prove and have done so and have reported it to see if others have similar issues and solutions, is that reverting that one feature, VLAN propagation to Ethernet ports on the nodes (an ASUS implementation) from enabled (Access/Trunk) to All (Default), solves the instability; completely.

This is rare, IMO. It’s a solution I can live with for the moment. It’s not how it should work but I and a couple of others have done the hard yards and actually reported the results. I could have just complained ad-nauseum about how rubbish ASUS products are (and no they’re not perfect and yes their marketing could be considered to be more elastic than Jim Carrey’s face), but I live in hope.

My hope is that if sufficient users, on Merlin or stock, trial this ASUS feature we will eventually have a better understanding of what HW combinations, what configurations, what firmware (even) works and what doesn’t. I don’t think this is approach is unreasonable.

 

[visortgw]

However you’d need to look at @penguin22 hardware (all BE class) to see if there’s a node with the same lack of features as mine causing his issues. And equally whether @visortgw successful setup includes only GNP and VLAN capable nodes. Both use Merlin. I do not know exactly what addons they use.

1. All of my AiMesh nodes are Network (or Guest Network Pro) capable.
2. All of my AiMesh nodes are VLAN capable, except for the ZenWiFi BQ16 Pro.
3. All of my nodes use wired backhaul (either 2.5 or 10 Gbps), except for the ZenWiFi BQ16 Pro which uses 6 GHz/320 WiFi.
4. Managed and unmanaged switches sit between router and AiMesh nodes.
5. Skynet, scribe, MerlinAU, connmon, RTRMON, BACKUPMON, scMerlin, spdMerlin, and uiScribe add-ons are installed — Skynet is configured to filter inbound traffic only.
6. Pi-hole and unbound are running on a dedicated Raspberry Pi 5.

 

[jksmurf]

1. All of my AiMesh nodes are Network (or Guest Network Pro) capable.
2. All of my AiMesh nodes are VLAN capable, except for the ZenWiFi BQ16 Pro.
3. All of my nodes use wired backhaul (either 2.5 or 10 Gbps), except for the ZenWiFi BQ16 Pro which uses 6 GHz/320 WiFi.
4. Managed and unmanaged switches sit between router and AiMesh nodes.
5. Skynet, scribe, MerlinAU, connmon, RTRMON, BACKUPMON, scMerlin, spdMerlin, and uiScribe add-ons are installed — Skynet is configured to filter inbound traffic only.
6. Pi-hole and unbound are running on a dedicated Raspberry Pi 5.

Thank you for taking the time to jot these down, that’s very much appreciated .

 

[penguin22]

I only have BE devices and was experiencing the same issues prior to disabling the wired VLAN on one node. I tried introducing several other changes thereafter, such as adding a BE58 Go in my garage to address a wireless IOT network for cars but ended up with IOT device instability again, so removed it as a node. This led to me deciding to perform a full reset from scratch to clear any gremlins in the system.

Fast forward to two nights ago, I take all my backups and prepare for a full manual setup, only adding back in client names before setting up the GNP VLANs.

Node 1, BE88U (3006.102.7) as router, get everything configured with just my laptop wirelessly connected, with all nodes powered off and all wired devices unplugged. Configure with a different SSID to prevent any other devices from connecting before adding client names back via MAC addresses. Once name<-->MAC list has been imported and all other configs are mostly in place, change SSID to main network (~5 devices all that will connect). Add initial GNP IoT VLAN (2.4 GHz) that are hosted on this router and devices come on happily and stable.

Node 2, BE82U (3.0.0.6.102_39099) added, configure GNP with dedicated IoT2 VLAN (2.4 GHz); all devices come back online, things are happy and stable.

Node 3, BE86U (3.0.0.6.102_39125) added, configure GNP with dedicated IoT3 VLAN (2.4 GHz); all devices come back online, though are showing signs of flapping, which I figure I would monitor while getting other networks that span all nodes configured.

Add remaining IoT VLANs and Guest networks, assigned to nodes as appropriate. It's 2am at this point and though I see instability, I decide to wait until morning to look at it further.

Day 1 Morning; clearly have some instability, even with no wired VLAN configured on nodes, though I do have wired Access VLANs for several devices on the BE88U router as needed. Troubleshoot throughout the day, figure that network overlap may be a factor, so turn down power to Balance on 2.4 & 5 GHz, most devices have clear delineation and can determine strongest node where available on multiple nodes, also reducing overlap for common 2.4 GHz band (20 MHz channel 11) for IoT dedicated VLANs per node. Disabled Roaming Assist on both bands to ensure not causing client disconnects; instability remains. Recognize that with the network separation that GNP offers (coming from AX non-Pro series routers without the capability previously), that all clients on the main SSID fully support 5 GHz and 2.4 GHz is undesirable. I wondered if setting the main SSID to 5 GHz only would have a positive impact as Smart Connect would no longer be in use; set to 5 GHz, upon propagation to all nodes, 100% stability. Re-added the BE58 Go (3.0.0.6.102_38978) in the garage for extended support and everything has been flawless now for 2 days. Logs have remained completely calm, ~85 devices across all networks have been stable, and following a week, I will try reconfiguring the wired VLAN on the BE86U to see if that is no longer an issue as well.

While disabling the wired VLAN did work for a period of time at first, it seemed short-lived and came back when I tried making network changes (adding the BE58 Go). This appeared as ARP table completely filling up at one point, which caused overall network instability, even with connected devices. At this point, though I don't believe that Smart Connect on the main network should be causing issues with networks configured with a single dedicated band on individual nodes, it clearly was causing massive issues from my observation. It was shocking how immediate and stable things became when the main SSID was configured with only a single band (5 GHz in my case) with Smart Connect disabled as a result.

Of note, I have two Guest networks still configured with 2.4 & 5 GHz to support clients that require both and have seen no issues with those networks or clients connected at all. I don't have any Smart Connect configuration available for these networks, so believe it is disabled the moment you set the primary SSID to a single band. Second note, doing so will require you to turn off MLO fronthaul for clients, though the main MLO setting can remain enabled. In my network, I only have 2.4 & 5 GHz, no 6 GHz support, so this was a non-issue for me.

Not sure if this helps anyone, though figured I would add a comprehensive overview from prior updates to now in case it does.

 

[jksmurf]

@penguin22 you’ve gone above and beyond! What a write up, it’ll take me 3 days to decipher it ! Thank you very much.

I’m not even sure where you ended up TBH, whether you totally disabled 2.4Ghz and whether you kept the All (Default) or used Access / Trunk. I’ll read it a few times.

How are you adding client names in your system btw? dnsmasq-x.conf.add, YazDHCP or something else ?

 

[Seth Harman]

What's interesting about all of this is I haven't done anything other than create two wireless VLANs under GNP with default settings (other than using different IP blocks): both VLANs are using 2.4/5GHz on the main router and both nodes, I haven't gone in and changed any of the default settings for things like MLO, I haven't played with power output, etc...and my setup works with no noted instability. This problem is likely going to be a real bear to isolate and diagnose.

 

[jksmurf]

What's interesting about all of this is I haven't done anything other than create two wireless VLANs under GNP with default settings (other than using different IP blocks): both VLANs are using 2.4/5GHz on the main router and both nodes, I haven't gone in and changed any of the default settings for things like MLO, I haven't played with power output, etc...and my setup works with no noted instability. This problem is likely going to be a real bear to isolate and diagnose.

I can’t recall if I’ve asked you before but have your routers and nodes both the VLAN capable Ethernet ports and are you using that function and with what settings Access? Trunk? and what Ethernet devices do you have in those VLAN capable Ethernet ports in nodes specifically?

My wireless VLAN also works perfectly as yours does, but only without the nodes VLAN ports being enabled. As soon as I enable those I get the instability. As soon as I disable again I get perfect stability.

 

[Seth Harman]

I can’t recall if I’ve asked you before but have your routers and nodes both the VLAN capable Ethernet ports and are you using that function and with what settings Access? Trunk? and what Ethernet devices do you have in those VLAN capable Ethernet ports in nodes specifically?

My wireless VLAN also works perfectly as yours does, but only without the nodes VLAN ports being enabled. As soon as I enable those I get the instability. As soon as I disable again I get perfect stability.

My nodes are both RT-BE58U which is a model that doesn't support wired VLAN tagging which is why I ended up having to use managed switches for wired devices I want to use within a GNP VLAN. For completeness, both of my nodes are wired up like this:

Wired devices I want tagged for VLAN <--> managed switch <--> AiMesh node <--> wired backhaul <--> main router

I am not using any of the VLAN-capable Ethernet ports on the main router to put any wired devices onto either of my GNP VLANs so they're all set to "All (Default)".

 

[jksmurf]

My nodes are both RT-BE58U which is a model that doesn't support wired VLAN tagging which is why I ended up having to use managed switches for wired devices I want to use within a GNP VLAN. For completeness, both of my nodes are wired up like this:

Wired devices I want tagged for VLAN <--> managed switch <--> AiMesh node <--> wired backhaul <--> main router

I am not using any of the VLAN-capable Ethernet ports on the main router to put any wired devices onto either of my GNP VLANs so they're all set to "All (Default)".

Ah ok thanks. I recall the cheap managed switches discussion.
So yeah, it will be difficult to isolate as the sample size of those with my specific setup, so far is… maybe 2 or 3 including me.

@visortgw has a working system, the difference his to mine appears to be primarily HW.

 

[Seth Harman]

Ah ok thanks. I recall the cheap managed switches discussion.
So yeah, it will be difficult to isolate as the sample size of those with my specific setup, so far is… maybe 2 or 3 including me.

@visortgw has a working system, the difference his to mine appears to be primarily HW.

From what I've seen so far in this thread maybe I'm lucky my nodes don't support wired VLAN tagging and I had to go out and got some managed switches.