And it was Stink Bugs that Knocked Down Our Network …

[Klueless]

Our main building is anchored with a dual band Asus N66u. Most PCs are on Ethernet, the rest wireless. The building next door ties in through a dual band Netgear 6150 wireless range extender. Line of sight, window to window, we use the 5 GHz as the back-haul between the two buildings and 2.4 GHz for clients in bldg. 2.

So as I’m making my coffee run I see two of the guys standing by the router in the window arguing. One wants to reboot the router because he has no Internet. The other doesn’t want him to because he is working fine and because the last time he rebooted the router they were down for hours.

Because I’m old they said they’d rather blame me than each other so I started looking around. I noticed all wired nodes were working and all wireless (both bands, both buildings) were down.

I shutdown the range extender in bldg. 2 (it was hot), still no wireless network. Reluctantly rebooted the router in bldg. 1 and everything came back up; wired and wireless. Went back to bldg. 2 and turned on the range extender; everything came up.

I turned on the Asus traffic monitor and Ping Plotter. Everything looked good; no packet loss between me, range extender, router, etc.

Couple of hours later I noticed things were running a little slow. Looked at Ping Plotter, it was showing packet loss between me/range extender and the router. By the time I went next door wireless was down hard. Shut down range extender. Wireless did not come back up so I rebooted router and everything came back up.

This time I removed the range extender and started looking at it. It was still hot to the touch. I noticed a stink bug had welded himself to the front of the plastic housing and another over some air vents. I scrapped them off, cleaned everything up, plugged it back in and bldg. 2 came back up.

Two days later and we’re still running. Stink Bugs - so not the computer bug I expected!

 

[degrub]

Looks like you need an IDS with periodic sweeping !

 

[Klueless]

Looks like you need an IDS with periodic sweeping !

<LOL> Great Idea!

(Still, in the back of my mind, I'm left pondering why a compromised range extender would take down the wireless in the main building?)

 

[VisionxOrb]

BTW this thread inspired me to find some stories and allowed me to find https://www.reddit.com/r/talesfromtechsupport/
SO AWESOME.

 

[Klueless]

BTW this thread inspired me to find some stories and allowed me to find https://www.reddit.com/r/talesfromtechsupport/
SO AWESOME.

Thanks. Reading through it now. Fun stuff. Brings back some memories.

This really Grumpy lady I used to work with was in charge of a Data Center Consolidation project. Instead of a bunch of little data centers they were going to move everything into one big data center.

Everything in the data center tied to a router and the router tied to the backbone with 10 Mbps half duplex fiber. (Wow, that must be longer ago than I care to admit!) Key to the project was to install a card into the router so it could connect to the backbone at 100 Mbps.

Apparently the card didn't arrive. She blew a gasket and stormed off the job. That Friday afternoon on my way out the door my adult supervision tells me I'm to help out with the data consolidation project.

Saturday night I go to a party. About 2AM I take a handful of aspirin in anticipation of Sunday's hangover. At 4AM I get a phone call.

Network Team: "Data Center is down!"

Me: "So?"

Network Team: "You're networking support. You need to get down here. Now!"

Me: "Oh yeah. But I can't, I'll get a DWI. What's going on?

Network Team: "The data center team moved all the little data centers into the big one and the network crashed."

After some conversation I learn the pipe to the backbone is saturated.

Me: "I thought everyone decided that 100 Mbps would be adequate?"

Network Team: "The upgrade to the router didn't arrive."

Me (sobering up now): "So, why weren't Saturday's moves cancelled?"

Network Team: "The project lead works in your group, why don't you ask her?"

Me: "What are you hoping I can do for you?"

Network Team: "There's only one thing you can do. Get the data center team in and have them move everything back."

Me (shuddering at the logistics of pulling that off): "Uh, yeah ... Hey, it's a long shot. Why don't we set the 10 Mbps link to full duplex?"

Network Team: "It's against company standards. You need to get the data center team in."

Me: "Uh, I think we might have good reason to violate that standard."

More conversation, much more. They tried it. It worked!

Few days later at our staff meeting there were no "thanks, nice job". Instead,

Grumpy Lady: "Klueless knowingly violated one of our cabling standards."

Me: "What? Wait a minute, let me tell you ..."

Boss: "Stop. Just fix it."

Me: "But she's wrong. Here's what happened ..."

Boss: "Don't. She's senior. You - have it fixed."

Happily the 100 Mbps card came in so we "fixed" it with that.

Talk about a toxic environment to work in.