Loss of Internet and Red Light after a few days when one particular computer is on

[cgrey8]

I've got an Asus RT-AC88U that intermittently has an issue where it essentially locks up and has to be power cycled.
While locked up:

• Router displays a Red Led
• Ethernet connected devices do not have Internet access
• Wifi SSIDs (main and guest) are being broadcast but no device can actually connect

Here's a rather typical snippet from the router's logfile:

...
May  5 01:05:24 kernel: xhci_hcd 0000:00:0c.0: Failed to enable MSI-X
May  5 01:05:24 kernel: xhci_hcd 0000:00:0c.0: failed to allocate MSI entry
Jan 24 02:02:55 crond[524]: time disparity of 3009717 minutes detected
Jan 26 02:00:21 Mastiff: Got SIGTERM
Jan 26 02:00:21 Mastiff: Got SIGTERM
Jan 26 02:00:21 Mastiff: Got SIGTERM
Jan 26 02:00:22 Mastiff: Select error
May  5 01:05:12 kernel: pflash: found no supported devices
May  5 01:05:12 kernel: bcmsflash: found no supported devices
May  5 01:05:19 acsd: selected channel spec: 0x1009 (9)
May  5 01:05:19 acsd: Adjusted channel spec: 0x1009 (9)
May  5 01:05:19 acsd: selected channel spec: 0x1009 (9)
May  5 01:05:19 acsd: acs_set_chspec: 0x1009 (9) for reason APCS_INIT
May  5 01:05:20 acsd: selected channel spec: 0xe09b (149/80)
May  5 01:05:20 acsd: Adjusted channel spec: 0xe09b (149/80)
May  5 01:05:20 acsd: selected channel spec: 0xe09b (149/80)
May  5 01:05:20 acsd: acs_set_chspec: 0xe09b (149/80) for reason APCS_INIT
May  5 01:05:20 Mastiff: init
May  5 01:05:25 kernel: xhci_hcd 0000:00:0c.0: Failed to enable MSI-X
May  5 01:05:25 kernel: xhci_hcd 0000:00:0c.0: failed to allocate MSI entry
Jan 26 02:03:13 crond[526]: time disparity of 3012597 minutes detected
Jan 26 17:53:54 miniupnpd[1031]: upnpevents_processfds: 0x2f378, remove subscriber uuid:e17e7ea4-bc72-4606-b8a1-d288901315df after an ERROR cb: http://192.168.1.130:2869/upnp/eventing/bndasswcnc
Jan 27 02:00:06 Mastiff: Got SIGTERM
Jan 27 02:00:06 Mastiff: Got SIGTERM
Jan 27 02:00:06 syslog: Got SIGTERM
May  5 01:05:11 kernel: pflash: found no supported devices
May  5 01:05:11 kernel: bcmsflash: found no supported devices
May  5 01:05:18 acsd: selected channel spec: 0x100b (11)
May  5 01:05:18 acsd: Adjusted channel spec: 0x100b (11)
May  5 01:05:18 acsd: selected channel spec: 0x100b (11)
May  5 01:05:18 acsd: acs_set_chspec: 0x100b (11) for reason APCS_INIT
May  5 01:05:19 acsd: selected channel spec: 0xe09b (149/80)
May  5 01:05:19 acsd: Adjusted channel spec: 0xe09b (149/80)
May  5 01:05:19 acsd: selected channel spec: 0xe09b (149/80)
May  5 01:05:19 acsd: acs_set_chspec: 0xe09b (149/80) for reason APCS_INIT
May  5 01:05:20 Mastiff: init
May  5 01:05:25 kernel: xhci_hcd 0000:00:0c.0: Failed to enable MSI-X
May  5 01:05:25 kernel: xhci_hcd 0000:00:0c.0: failed to allocate MSI entry
Jan 27 02:03:02 crond[524]: time disparity of 3014037 minutes detected
...

There are numerous entries that are repeated at rather patternistic intervals. The thing I've noticed that isn't quite as "rhythmic" are the entries making mention of miniupnpd. Sometimes there's a single entry as shown above, but there are a few places in the log that have 10 or so entries back-to-back.

The other thing worth mention is I suspect this lockup is somehow related to a specific laptop I have that stays powered off most of the time. I say this because the router can go MONTHS without having an issue at all. However last week, the router did this lockup twice, and that laptop was powered and online both times it happened. And I suspect that the previous events of this happening were also when this laptop was powered. Add to this, the IP address indicated in that miniupnpd entry is that laptop's IP when powered. So I suspect this being related to UPnP. However I have to admit ignorance to UPnP. I'm aware that it's a thing, but I don't really know what services or features it enables that I would notice or miss if I disabled UPnP support in the router. I do see there are switch controls to disable it on the WAN page. But before I tried that, I thought I might should ask others more familiar with these log messages, this issue, and/or those settings before I proceeded.


Other things of note, I've confirmed the issue is not my cable modem. Each time this has happened, a restart of the router fixes the issue until it happens again. And while the router is in its locked-up state, the cable modem shows online and even responds to remote-commands to restart via the Xfinity app.
I also have an RT-AC68U as an AIMesh Node however I haven't come across anything so far that makes me suspicious of it yet.

Anybody have an idea as to what this issue might be?
And more importantly what setting(s) I can change to possible make it go away?

 

[OzarkEdge]

The AC88U has a history of LAN port trouble, often the upper bank 5-8 fails in some way... could be a factor.

OE

 

[cgrey8]

I'm aware of that issue. All my Ethernet connections are in 1-4. My router's 5-8 ports do work, but don't connect faster than 100BaseT. Only the 1-4 actually connect at 1GbaseT.

I've got a replacement (used) RT-AX88U on the floor that I got off eBay, and its 5-8 ports are completely dead (sold that way). Ports 3-4 only connect at 100BaseT, and 1-2 work to 1GBaseT. I haven't swapped it out yet but if there's a chance this issue could be fixed by that, I would put more priority on it. I just have a lot of IoT devices (cameras, printers, TVs, Tstat, etc) that I really don't want to run around the house re-joining them all so I've been putting it off.

I forgot to also mention the loss of time that shows in the timestamps in the log. It'll get time, and entries will be correct for a while, then it'll revert back to May 5, then go back to correct dates...no clue what that is. But that hasn't been a pressing issue, just a minor annoyance at nailing down when exactly events happen.

 

[L&LD]

Try the easiest things first.

Power down your entire network. Anything AC-powered modems, ONTs, etc. All routers. All Switches. All printers. All scanners. All NAS'.

For the computers, do not only power off but also Restart them after they've turned on again (not the same thing as a shutdown - power up)

Wait for a minute or two.

Now, power up the modem/ONT. Power up your main router. Power up the node. Power up the switches, printers, scanners, NAS, etc.

Leave the suspect laptop 'on', and test again.

 

[Tech9]

Add to this, the IP address indicated in that miniupnpd entry is that laptop's IP

What happens if you disable UPnP? Do you need it enabled for something?

 

[OzarkEdge]

I'm aware of that issue. All my Ethernet connections are in 1-4. My router's 5-8 ports do work, but don't connect faster than 100BaseT. Only the 1-4 actually connect at 1GbaseT.

I've got a replacement (used) RT-AX88U on the floor that I got off eBay, and its 5-8 ports are completely dead (sold that way). Ports 3-4 only connect at 100BaseT, and 1-2 work to 1GBaseT. I haven't swapped it out yet but if there's a chance this issue could be fixed by that, I would put more priority on it. I just have a lot of IoT devices (cameras, printers, TVs, Tstat, etc) that I really don't want to run around the house re-joining them all so I've been putting it off.

I forgot to also mention the loss of time that shows in the timestamps in the log. It'll get time, and entries will be correct for a while, then it'll revert back to May 5, then go back to correct dates...no clue what that is. But that hasn't been a pressing issue, just a minor annoyance at nailing down when exactly events happen.

Equipment that has partial hardware faults would be removed from my network asap.

FYI when the router reboots, the startup log entries get the default date stamp until the router updates its clock via the Internet. Then all entries should have the current date. Your log above shows the date switching back and forth in short order... this is not normal. I would recommission the router with nothing connected... if it still exhibits weirdness (likely)... I would replace it.

OE

 

[Tech9]

Your log above shows the date switching back and forth

This router actually restarts at the same time for some unknown reason.

 

[OzarkEdge]

This router actually restarts at the same time for some unknown reason.

I'm thinking it's not even restarting/rebooting... not enough May 5 entries for that... it's just misbehaving in more places than one... it's toast!

OE

 

[Tech9]

I believe the log above is not a full log, but extracted messages to show the reboot moments only.

 

[OzarkEdge]

I believe the log above is not a full log, but extracted messages to show the reboot moments only.

Could be... so logging is ok... just need to figure out those pesky reboots and other hard faults.

OE

 

[Tech9]

I personally would throw away both defective routers and get a new one. The "replacement" router with 6x damaged LAN ports was probably zapped by high voltage in his previous life. The old one is not only handicapped, but also End-Of-Life model. If the budget is tight there are new AX-class routers starting at around $100 with some good ones at around $150. Doesn't have to be Asus, doesn't have to be high-end model, but it has to be good working one. Otherwise the tail chase may continue forever.

 

[cgrey8]

Good responses. And it's good to know that restarts are why the May 5th entries keep happening. And I do have it setup in the configuration to reboot on a schedule. I did that thinking the lockup was memory leaks or some accumulated issue that periodic/regular reboots would avoid. So, that explains why there are so many May 5th entries and why SIGTERM signals are being sent so often, because Linux is sending them to reboot the system. So I can safely ignore all that.

As for the UPnP disabling, I was hoping to get some insight as to what it is UPnP enables/makes possible so I'd know whether its a feature I can live without. But I didn't get that. So I'll try just disabling UPnP and see how that goes. Google-searching UPnP, it seems like something that should be disabled. It enables devices, especially IoT devices, to find each other and work together more easily, however I put every IoT device I have on my Guest network to intentionally isolate them from the rest of the network (and each other). The only exception is the printer, which relies on local connections between phones/laptops to function. So it seems UPnP is definitely something I can live without and probably should've had disabled all along.

 

[cgrey8]

I've disabled UPnP, turned the reboot-scheduler off, and cleared the log. The computer that I suspect MIGHT have been contributing this is on. I'll leave it like this a few days and if it makes it a week or so without locking up, I'll call this fixed.

For those that know, is there any chance the AX series routers have realtime clocks in them that preserve the time through reboots unlike the AC series? I don't have nearly enough experience with them to know what improvements were made in their hardware.

All of my Asus hardware is used off eBay...
AC88U $88
AC68U $45 (bought a few months later explicitly to be an AIMesh Node when I learned that was a thing)
AX88U $70 (just recently got, and not yet deployed)

All 3 combined cost me less than what I would've paid for the AC88U new back when I got it a few years ago (2020ish?). I feel I've gotten my money's worth out of the 2 AC units. I bought the AX88U just as a spare for the AC88U.

So, I'm not convinced that had I bought the AC88U new that this issue wouldn't have occured. While this may be related to the AC88U's age and/or known hardware issues, with the info I have so far, I can't claim that as the cause (yet). But more to the point, I don't want to discourage others reading this from taking a chance on used stuff. Buying used certainly has risks and I've gotten burned once or twice buying used stuff off eBay, but by far the VAST MAJORITY of the used stuff I've gotten off eBay has kept working stuff out of the landfill, saved me money, and gave a little back to the seller.

Now if the risk isn't worth it to you and you have the funds, then buying new is the right thing for you and that's what you should do.

 

[glens]

Up to at least the GT-AX6000 there's no battery-operated clock.

 

[bennor]

is there any chance the AX series routers have realtime clocks in them that preserve the time through reboots unlike the AC series?

No. On a RT-AX86U Pro the router will default back to an earlier date/time for a short period, as indicated in the system log, during router reboot sequence before resuming the current date/time. The short gap with old date/time in the system log is normal for power on or a reboot.

 

[cgrey8]

I've had UPnP disabled for a week now and kept the suspect computer online as well. So far, no hiccups from the router at all. I've also not noticed any "missing" functionality from disabling UPnP either. So I think it's definitely a feature I can live without.

I also increased the severity of logging to error so I don't get noise entries in the log. Then I cleared the System Log. I didn't get any messages in it at all. So perhaps setting the log level to error is too restrictive? For those familiar with using the logs to diagnose problems on these routers, is error too restrictive to be useful?

TBO, I was suspicious that MAYBE the UI wasn't actually showing me the messages for some reason. So I triggered a reboot figuring this would generate some entries. After the reboot, there were only the expected reboot messages including the SIGTERM entries from Mastiff. So yeah the log just had no entries that rose to the level of error. I consider that a good thing.

Until I get more info from another incident, I'm calling this fixed via disabling UPnP. I'll report back on this thread if it happens again.

 

[cgrey8]

Nearly a month later, and no issues.
Right now, the AC88U has an uptime of 13 days. The last restart was manually triggered, not due to a mishap. And NO entries in the System Log that are of the level of ERROR.