Mystery LAN failures

[f00dl3]

I'm having a mystery problem of sorts. This is not specifically tied to Asus or even Merlin firmware, but it's happening on it right now and I was wondering if there was any ideas on how I can further isolate what is happening.

This has been happening for several weeks now. It has persisted through 3 routers of 2 brands. This started 2 weeks ago when I started doing some weather data fetches, going from downloading about 5 GB/day to downloading about 20 GB/day. I started experiencing random LAN drops, from what at that time was a Netgear WNR2000v3 (N300 series) router. I upgraded to a Netgear RT AC6300 series router, and had the same problem with the latest firmware. I took that back as it was within the 14 day return period, and got a Asus AC3200 router. I put the Merlin firmware on, and just experienced the same problem.

LAN drops. All LAN connections under the immediate LAN segment drop. WiFi & Wireless. If I have a separate VPN router dangled off of the home router, the VPN tunnel stays up and the 2nd router's LAN is not impacted, like nothing happened. The 192.168.1.1 is unreachable from my desktop computer.

It does not restore until I reboot the router.

I checked into the syslog.log file on the new Asus router, and it shows nothing indicating the failure. The last log is the last SSH connection my desktop makes to the router to get system info off of it, and then there are no further log entries until I rebooted the router to get it back online.

The only 3 things I can think that could be happening:
- I have a wired IP camera that goes through my garage. Maybe this is picking up EMI?
- Pretty sure I can rule out someone trying to hack me as I have had 4 public IP addresses, one with each new router.
- My ISP Charter Spectrum formerly Time Warner could be sending "death packets" to control my data use since again this started right after I started pulling ~20 GB/day (~650 GB/mo). To my ISP's fairness, I called them up several times and they claim they have no sort of data abuse list as it's a completely uncapped plan. This being said, I signed up for Google Fiber in December and they will be installing tomorrow.

 

[Fitz Mutch]

... The 192.168.1.1 is unreachable from my desktop computer. ...

Is this a problem with only your desktop computer? If so, are you cloning your desktop computer's MAC address to the WAN?

 

[f00dl3]

Is this a problem with only your desktop computer? If so, are you cloning your desktop computer's MAC address to the WAN?

I am not cloning any MAC addresses of any machines on the network. Impacts anything on the network - IP cameras go down, wired & wireless. Of course, my desktop computer runs the shell scripts that wget from the IP cameras... so it's possible. I may try to test and hook up a laptop to see if I can't get through with the laptop when the issue happens.

 

[sfx2000]

LAN drops. All LAN connections under the immediate LAN segment drop. WiFi & Wireless. If I have a separate VPN router dangled off of the home router, the VPN tunnel stays up and the 2nd router's LAN is not impacted, like nothing happened. The 192.168.1.1 is unreachable from my desktop computer.

It does not restore until I reboot the router.

I checked into the syslog.log file on the new Asus router, and it shows nothing indicating the failure. The last log is the last SSH connection my desktop makes to the router to get system info off of it, and then there are no further log entries until I rebooted the router to get it back online.

The only 3 things I can think that could be happening:
- I have a wired IP camera that goes through my garage. Maybe this is picking up EMI?
- Pretty sure I can rule out someone trying to hack me as I have had 4 public IP addresses, one with each new router.
- My ISP Charter Spectrum formerly Time Warner could be sending "death packets" to control my data use since again this started right after I started pulling ~20 GB/day (~650 GB/mo). To my ISP's fairness, I called them up several times and they claim they have no sort of data abuse list as it's a completely uncapped plan. This being said, I signed up for Google Fiber in December and they will be installing tomorrow.

Draw a picture of your topology - include the elements (for example, Spectrum's Modem (is this a gateway or a pure modem), other items in your LAN.

 

[f00dl3]

I have some IPs reserved as my desktop runs shell scripts on them/SNMP requests. Asus syslog does not alert of any IP conflicts when retroactively pulling the log.

No relationship with devices being powered on /etc. It's happened in the middle of the night.

 

[sfx2000]

Hmmm...

First thing that comes to mind is possible signalling storms - I used to have an Aruba RAP2 SSL-VPN device, and if plugged into the wrong port (it had two 100BaseT ethernet ports), it would cause problems (safe bet there is use the WAN port, which with the RAP2 config we had was E0 - newer RAP's color code the ports, but the older ones didn't).

I'm assuming that Windows VM is bridged right?

The other thing perhaps is power related... since all three routers had basically the same problem, the common element there is power.

Last thought would be the SB6121 Cable Modem - push too much upstream traffic, and it can get unstable - if I recall TWC/Charter/Spectrum still enable the modem status page, which can be found at http://192.168.100.1 - see the log entries and if there are significant errors that might cause the modem to reboot (for example T4 timeouts).

 

[f00dl3]

Hmmm...

First thing that comes to mind is possible signalling storms - I used to have an Aruba RAP2 SSL-VPN device, and if plugged into the wrong port (it had two 100BaseT ethernet ports), it would cause problems (safe bet there is use the WAN port, which with the RAP2 config we had was E0 - newer RAP's color code the ports, but the older ones didn't).

^ Aruba is powered off. Usually only use it 1 day a week. Aruba WAN goes to LAN 4 on Router.

I'm assuming that Windows VM is bridged right?
^ Yes

The other thing perhaps is power related... since all three routers had basically the same problem, the common element there is power.

^ Not power. The Asus router is actually master controller on my APC BackUPS XS1500 - so it's the last thing that would be impacted by a power surge/outage.

Last thought would be the SB6121 Cable Modem - push too much upstream traffic, and it can get unstable - if I recall TWC/Charter/Spectrum still enable the modem status page, which can be found at http://192.168.100.1 - see the log entries and if there are significant errors that might cause the modem to reboot (for example T4 timeouts).

I actually own the cable modem.. either way I checked the Log tab and nothing in there except when I called the ISP after the event to try to reset the modem remotely to check if that was the issue as my SSH connection to the Desktop was getting a No route found message.

I have enabled SSH on the Asus router so I can isolate the Desktop part of the connection to it's segment, if I can still get in the router I know it's between the router and the desktop. Going to also fire up the laptop as it has SSH and I can remote in there from outside too - segment it individually, and run tcpdump in promiscuous.

Again - oddest thing is until I started pulling 5 Gigs twice/day and 4 Gigs twice/day - computer model data from the National Weather Service as GRIB2 files to be exact - this was never an issue. Almost like it's bandwidth related. The script I wrote pulls about 25 MB/sec with 8 active connections (1 per CPU core, the Weather Service request for you not to use more than 20)... also downloading some heavy data / Reanalysis data / that I've been pulling 150-240 GB/day on - but again, that peaks at 15-20 MB/sec. (25 mb /sec during model data pulls.) - and this is only until I get the data I want which is about 2 TB.

IP cams pull about 150 KB/sec combined.. but only on the LAN.

All combined about 25 MB upstream to modem, 30 MB/sec peaks on LAN which should be well under the aprox. 121 MB/sec that 1 Gig router should handle.

 

[sfx2000]

Try putting a rate limit on the upstream - the SB6121/SB6141 may run into issues when putting a lot of traffic that is not provisioned on the device...

Starting to make sense here...

FWIW - I don't think it's the router - I think it's the modem...

I don't run AsusWRT - but look at the QoS settings, and one should be able to set both upstream and downstream characteristics - or just use their dynamic stuff, as it considers that many broadband connections are asymmetric...

 

[f00dl3]

Try putting a rate limit on the upstream - the SB6121/SB6141 may run into issues when putting a lot of traffic that is not provisioned on the device...

Starting to make sense here...

That makes sense. 25 MB/sec is ~200 Mbyte/sec - not sure what upload is pushing.
But could that crash the LAN but keep the modem online with nothing to show in modem or router logs?

And if that's the root cause, not sure if I really care to invest much time since it's bye bye Tomorrow when I get Google Fiber.

 

[sfx2000]

That makes sense. 25 MB/sec is ~200 Mbyte/sec - not sure what upload is pushing.
But could that crash the LAN but keep the modem online with nothing to show in modem or router logs?

And if that's the root cause, not sure if I really care to invest much time since it's bye bye Tomorrow when I get Google Fiber.

Wait for the Google Fiber install - and perhaps even consider their end-point as the primary...

They do a decent job there.

 

[sfx2000]

FWIW - I don't think the router has anything to do with the problems you're having, since you've swapped out across multiple vendors...

Ruling out power, then it has to be the modem itself...

(or the client PC, but that's a bit of a stretch...)

But knowing that there is an Aruba RAP somehow inside the network - just check it...

 

[f00dl3]

Yeah so not a single reset since I went live on Google Fiber 1000 Saturday afternoon. W00T!