AC68U occasionally unresponsive with AiMesh enabled

[Sinbios]

I have two AC68Us, one original and one converted from a TM1900. I'm using the original as the main AiMesh node and have it connected to my main computer via ethernet. All the wireless settings are default, and I have NAT acceleration set to auto. Both nodes are on the latest firmware 3.0.0.4.384_21140.

The problem I'm having is occasionally the main node would become unresponsive to pings, even through ethernet. This happens once every few hours and lasts for 5-10 seconds. The secondary node also drops out occasionally every few hours for maybe 10-20 seconds, though I haven't confirmed whether it's unresponsive to its ethernet clients as well or it's just unreachable through the wireless link.

I have the secondary node turned off now and haven't seen any more packet loss to the main node.

 

[Sinbios]

Here is the syslog from the secondary node for a couple of dropout events.

Aug 8 06:45:13 rc_service: cfg_client 4476:notify_rc restart_cfgsync
Aug 8 06:46:23 rc_service: amas_wlcconnect 20917:notify_rc restart_wireless
Aug 8 06:46:24 rc_service: watchdog 293:notify_rc start_amas_bhctrl
Aug 8 06:46:24 rc_service: waitting "restart_wireless" via ...
Aug 8 06:46:27 kernel: dpsta enabled
Aug 8 06:46:32 rc_service: cfg_client 8304:notify_rc restart_cfgsync
Aug 8 06:46:32 rc_service: waitting "restart_wireless" via ...
Aug 8 06:46:37 rc_service: watchdog 293:notify_rc start_cfgsync
Aug 8 06:46:42 rc_service: amas_bhctrl 8683:notify_rc stop_amas_wlcconnect
Aug 8 06:46:42 rc_service: amas_bhctrl 8683:notify_rc start_amas_wlcconnect
Aug 8 06:46:42 rc_service: waitting "stop_amas_wlcconnect" via amas_bhctrl ...
Aug 8 06:46:50 rc_service: udhcpc_lan 8740:notify_rc stop_httpd
Aug 8 06:46:50 rc_service: udhcpc_lan 8740:notify_rc start_httpd
Aug 8 06:46:50 rc_service: waitting "stop_httpd" via udhcpc_lan ...
Aug 8 06:46:51 RT-AC68U: start httpd:80
Aug 8 06:46:52 rc_service: udhcpc_lan 8740:notify_rc start_dnsmasq
Aug 8 06:46:52 rc_service: waitting "start_httpd" via udhcpc_lan ...
Aug 8 06:46:54 LAN network changes (%s/%s --> %s/%s). : 192.168.1.80
Aug 8 06:46:54 rc_service: udhcpc_lan 8740:notify_rc stop_samba
Aug 8 06:46:54 rc_service: udhcpc_lan 8740:notify_rc start_samba
Aug 8 06:46:54 rc_service: waitting "stop_samba" via udhcpc_lan ...
Aug 8 06:46:55 Samba Server: smb daemon is stoped
Aug 8 06:46:55 kernel: gro disabled
Aug 8 06:46:55 kernel: gro disabled
Aug 8 06:52:06 rc_service: cfg_client 8793:notify_rc restart_cfgsync
Aug 8 06:53:14 rc_service: amas_wlcconnect 8713:notify_rc restart_wireless
Aug 8 06:53:19 kernel: dpsta enabled
Aug 8 06:53:27 rc_service: cfg_client 10105:notify_rc restart_cfgsync
Aug 8 06:53:29 rc_service: amas_bhctrl 10435:notify_rc stop_amas_wlcconnect
Aug 8 06:53:29 rc_service: amas_bhctrl 10435:notify_rc start_amas_wlcconnect
Aug 8 06:53:29 rc_service: waitting "stop_amas_wlcconnect" via amas_bhctrl ...
Aug 8 06:53:38 rc_service: watchdog 293:notify_rc start_cfgsync
Aug 8 06:53:51 rc_service: udhcpc_lan 10578:notify_rc stop_httpd
Aug 8 06:53:51 rc_service: udhcpc_lan 10578:notify_rc start_httpd
Aug 8 06:53:51 rc_service: waitting "stop_httpd" via udhcpc_lan ...
Aug 8 06:53:53 RT-AC68U: start httpd:80
Aug 8 06:53:53 rc_service: udhcpc_lan 10578:notify_rc start_dnsmasq
Aug 8 06:53:54 LAN network changes (%s/%s --> %s/%s). : 192.168.1.80
Aug 8 06:53:54 rc_service: udhcpc_lan 10578:notify_rc stop_samba
Aug 8 06:53:54 rc_service: udhcpc_lan 10578:notify_rc start_samba
Aug 8 06:53:54 rc_service: waitting "stop_samba" via udhcpc_lan ...
Aug 8 06:53:54 Samba Server: smb daemon is stoped
Aug 8 06:53:54 kernel: gro disabled
Aug 8 06:53:54 kernel: gro disabled

 

[Sinbios]

Another slightly different one:

Aug 8 11:15:55 rc_service: udhcpc_lan 11243:notify_rc stop_httpd
Aug 8 11:15:55 rc_service: udhcpc_lan 11243:notify_rc start_httpd
Aug 8 11:15:55 rc_service: waitting "stop_httpd" via udhcpc_lan ...
Aug 8 11:15:56 RT-AC68U: start httpd:80
Aug 8 11:15:57 rc_service: udhcpc_lan 11243:notify_rc start_dnsmasq
Aug 8 11:15:57 LAN network changes (%s/%s --> %s/%s). : 192.168.1.80
Aug 8 11:15:57 rc_service: udhcpc_lan 11243:notify_rc stop_samba
Aug 8 11:15:57 rc_service: udhcpc_lan 11243:notify_rc start_samba
Aug 8 11:15:57 rc_service: waitting "stop_samba" via udhcpc_lan ...
Aug 8 11:15:58 Samba Server: smb daemon is stoped
Aug 8 11:15:58 kernel: gro disabled
Aug 8 11:15:58 kernel: gro disabled
Aug 8 11:16:10 rc_service: cfg_client 11297:notify_rc restart_wireless
Aug 8 11:16:14 kernel: dpsta enabled
Aug 8 11:16:24 rc_service: amas_bhctrl 11399:notify_rc stop_amas_wlcconnect
Aug 8 11:16:24 rc_service: amas_bhctrl 11399:notify_rc start_amas_wlcconnect
Aug 8 11:16:24 rc_service: waitting "stop_amas_wlcconnect" via amas_bhctrl ...

 

[Sinbios]

Something strange I've noticed is that when these dropouts happen, sometimes the main router doesn't respond to ping but the secondary node does, which should be impossible. It's almost like the secondary node is trying to take over as the main router for a bit.

 

[Grisu]

first I would try load latest 32738 firmware.

 

[Sinbios]

first I would try load latest 32738 firmware.

Yeah I have 32738 on both. I'm on a repeater setup now and all the problems went away. Unfortunately the bandwidth is halved but it's an improvement over getting these dropouts constantly.

 

[bachastain]

I'm having the same problem as the OP. In my case it is 2 AC66U B1 boxes setup as Mesh router and Mesh node. Everything runs fine for many hours, but about twice per day at random times, those clients using the node will start to get errors and time outs. If I try pings from an affected client I can ping the Node but not the main router. Pings of the router simply time out. Pings of the Node are fine. It stays like that for 30 second to maybe 2 minutes and then all is fine again without me doing anything. Pings return to normal. A few hours later it happens again.

I've turned off all the settings I've seen in this forum that can cause weird problems like Airtime Fairness and Roaming Assistant.

I've telneted in to the boxes and watched the temperatures when the boxes are under load and the temperature never gets above mid-70s F. "cat /proc/dmu/temperature".

If I monitor uptime (using telnet) both boxes show no signs of reboots.

The client I'm testing with is a Asus USB-AC68 USB 3.0 Wi-Fi Adapter. Other clients currently using the Node show the same symptoms.

If I simply power down the Node, the clients flip back to the router and all returns to normal. Everyone works reliably off the router. The Node was added to the Mesh right out of the box with no changes to it.

32738 on both.

 

[sfx2000]

I have two AC68Us, one original and one converted from a TM1900

Your TM-1900 is not supported - try replacing it with a regular consumer unit, and ask your question again once you've done that.

 

[Sinbios]

I'm having the same problem as the OP. In my case it is 2 AC66U B1 boxes setup as Mesh router and Mesh node. Everything runs fine for many hours, but about twice per day at random times, those clients using the node will start to get errors and time outs. If I try pings from an affected client I can ping the Node but not the main router. Pings of the router simply time out. Pings of the Node are fine. It stays like that for 30 second to maybe 2 minutes and then all is fine again without me doing anything. Pings return to normal. A few hours later it happens again.

I've turned off all the settings I've seen in this forum that can cause weird problems like Airtime Fairness and Roaming Assistant.

I've telneted in to the boxes and watched the temperatures when the boxes are under load and the temperature never gets above mid-70s F. "cat /proc/dmu/temperature".

If I monitor uptime (using telnet) both boxes show no signs of reboots.

The client I'm testing with is a Asus USB-AC68 USB 3.0 Wi-Fi Adapter. Other clients currently using the Node show the same symptoms.

If I simply power down the Node, the clients flip back to the router and all returns to normal. Everyone works reliably off the router. The Node was added to the Mesh right out of the box with no changes to it.

32738 on both.

Do you have any TM-1900s in your setup?

 

[bachastain]

Do you have any TM-1900s in your setup?

Nope. Just the hardware I mentioned. Twin AC66U B1's (same hardware as AC68s). But the OP's description of his symptoms sound very similar. The OP says he changed to repeaters so we might not hear from him again, though since his bandwidth was halved I'd guess he'd love to get his mesh working better. I have no experience with TM-1900s so I can't say if that might be a factor or a dead end.

 

[Sinbios]

Nope. Just the hardware I mentioned. Twin AC66U B1's (same hardware as AC68s). But the OP's description of his symptoms sound very similar. The OP says he changed to repeaters so we might not hear from him again, though since his bandwidth was halved I'd guess he'd love to get his mesh working better. I have no experience with TM-1900s so I can't say if that might be a factor or a dead end.

I'm OP. sfx2000 said my issues might be because TM-1900 isn't supported, but that doesn't explain you having the same problems on unmodified hardware.

 

[Grisu]

I'm OP. sfx2000 said my issues might be because TM-1900 isn't supported, but that doesn't explain you having the same problems on unmodified hardware.

Its only that TM is not allowed to be supported here with non TM-firmware (and with TM-firmware there will be no help either, maybe some basic help). If its done correct there is no difference to a 68U!

 

[bachastain]

Ok, well then I, with my pure Asus hardware and firmware, would love to hear ideas and suggestions on the symptoms mentioned in this thread.

 

[OzarkEdge]

Ok, well then I, with my pure Asus hardware and firmware, would love to hear ideas and suggestions on the symptoms mentioned in this thread.

I would rebuild your 32738 AiMesh from scratch, being sure to factory default reset/initialize first.

If no joy, I would downgrade the firmware to 21140, reset, and rebuild.

I would only disable Airtime Fairness... leave the roaming assistant and smart connect (if present) at defaults.

And make sure your node is within range of the router's 5.0 GHz band.

OE

 

[Sinbios]

I would rebuild your 32738 AiMesh from scratch, being sure to factory default reset/initialize first.

Yep, tried that, it would behave for a day or two and then start acting up again.

If no joy, I would downgrade the firmware to 21140, reset, and rebuild.

I might give that a try today, is 21140 considered more stable?

 

[OzarkEdge]

Yep, tried that, it would behave for a day or two and then start acting up again.

I might give that a try today, is 21140 considered more stable?

Not particularly... that's a broad question. However, I have reported here a specific throughput/Internet connectivity instability at the 32738 node on two separate 2xRT-AC86U AiMesh systems. It was reproducible and unacceptable and avoidable by downgrading to 21140. And, it coincided with a release introducing significant changes... Blue Cave support, multiple security fixes, and a fix for a longstanding QoS defect. And, the ASUS rep here has showed no interest in it, so assuming the best of them, they probably already know about it.

As bachastain was asking for any suggestions... I offered mine for his issue, not for the OP's misbehaving hacked TM-1900.

OE

 

[bachastain]

I would rebuild your 32738 AiMesh from scratch, being sure to factory default reset/initialize first.

If no joy, I would downgrade the firmware to 21140, reset, and rebuild.

I would only disable Airtime Fairness... leave the roaming assistant and smart connect at defaults.

And make sure your node is within range of the router's 5.0 GHz band.

OE

I've been working on this for several weeks. Both the router and new mesh node were on 21140 when I first built the mesh, and had the same symptoms. I updated both to 32738 hoping it would help and it didn't.

I'll try to budget some time to try and rebuild. At the moment my family is unhappy with the weeks of problems and I've had to power off the node. I only started this trying to fix a couple of poor reception areas (without creating roaming problems or slow speeds).

 

[bachastain]

Since it was off anyway, I factory reset the node AC66 B1, and then added back to the Mesh. No change in symptoms. It only took about an hour after attaching to the node before my pings (and all activity) to the router (192.168.1.1) died. It took 3-5 minutes before they worked again.

During that time the router logged nothing of interest:

Sep 9 16:03:22 rc_service: cfg_server 28876:notify_rc restart_obd_monitor
Sep 9 16:05:43 rc_service: cfg_server 29149:notify_rc restart_obd_monitor
Sep 9 16:05:57 rc_service: cfg_server 29241:notify_rc start_wps_method
Sep 9 16:07:13 rc_service: cfg_server 29289:notify_rc restart_obd_monitor
Sep 9 16:07:27 rc_service: cfg_server 29387:notify_rc start_wps_method
Sep 9 16:19:22 rc_service: cfg_server 29678:notify_rc restart_obd_monitor
Sep 9 16:19:31 rc_service: cfg_server 29750:notify_rc start_wps_method

All of that is due to the things I did to bring the node back up from the reset. If the node has a log I don't know how to access it.

I'm hoping to try resetting the router tonight. If that doesn't fix it I won't be going back to 21140. The node will remain off and be parked in a closet somewhere until the next release comes out.

Bruce.

 

[bachastain]

Ok, status report. As I mentioned, doing a factory reset on the Mesh Node by itself did nothing. Overnight I did the Mesh Router too. As suggested I did a full factory reset and rebuilt it from the ground up. I changed the absolute minimum number of settings away from their defaults, except Airrime Fairness is off. I left Roaming Assistant on. Both router and node are still on 32738.

After rebuilding the router, I had to do an additional reset on the node (not factory reset) before the router recognized it as a node and added it back in.

It's been up since 5am (6 hours so far) under load testing and so far it's been flawless. It also seems to be acting more predictably, with less weird stuff happening. Clients initially pick the strongest box and stay there (I'm not testing roaming) instead of unpredictably flipping back and forth for no apparent reason.

Before the resets and rebuilds I'm confident that I would have already seen symptoms by this point in testing, but it's cooking right along.

It took me about 5 hours to do the rebuild so I'm a bit sleep deprived, but encouraged.

Bruce.

 

[OzarkEdge]

Ok, status report. As I mentioned, doing a factory reset on the Mesh Node by itself did nothing. Overnight I did the Mesh Router too. As suggested I did a full factory reset and rebuilt it from the ground up. I changed the absolute minimum number of settings away from their defaults, except Airrime Fairness is off. I left Roaming Assistant on. Both router and node are still on 32738.

After rebuilding the router, I had to do an additional reset on the node (not factory reset) before the router recognized it as a node and added it back in.

It's been up since 5am (6 hours so far) under load testing and so far it's been flawless. It also seems to be acting more predictably, with less weird stuff happening. Clients initially pick the strongest box and stay there (I'm not testing roaming) instead of unpredictably flipping back and forth for no apparent reason.

Before the resets and rebuilds I'm confident that I would have already seen symptoms by this point in testing, but it's cooking right along.

It took me about 5 hours to do the rebuild so I'm a bit sleep deprived, but encouraged.

Bruce.

Good work!

A few thoughts...

Until AiMesh firmware upgrades become simple fixes instead of AiMesh development additions, resetting all after an upgrade is recommended, imo. Likewise, it helps to keep the router configuration to a minimum for now to simplify reconfiguration.

Resetting the node before adding it to the AiMesh is required. I'm not sure I appreciate your distinction "I had to do an additional reset on the node (not factory reset)". Either a button reset or GUI reset or GUI Initialize (clears logs) should do it, all factory resets... I prefer the GUI Initialize followed by a button reset for good measure... maybe even throw in a power cycle, too. Whatever it takes to preclude unnecessary distractions from the real AiMesh defects.

Finally, recent news here from ASUS suggests Smart Connect node band steering has still not (and may not) be added to the 68U (and related builds like your 66Us?). IMO, AiMesh without node band steering is not very worthwhile except for the centralized management and improved wireless backhaul rates. Me, I would not live with it... I sold my 2xRT-AC68Us and got 2xRT-AC86Us (on sale plus rebate).

Without Smart Connect, wireless clients may not connect to the desired band. As I see it, the only recourse for this is to define separate SSIDs, such as OE-24 and OE-50, and configure your clients to connect to the desired band. This may be preferable for some users with specific wireless clients like IoT clients.

Now, I wonder how your node will perform when you connect a wireless PC to it and run http://www.speedtest.net/ or
https://www.dslreports.com/speedtest? repeatedly (until it slows down or completely stops?)...

OE