TP-link AV2000 packet storms?

[kab]

Symptom: TP-Link AV2000 (TL-PA9020) causes some sort of packet storm if two cables are connected from switch to first unit and nothing at all is connected to the second unit. I have no idea what packets are being sent because the router pegs at 100% CPU usage and nothing on the network (including the router) will respond, and I don't have a dedicated sniffer any more. All I get from post-mortem router logs (debug level enabled) is:

Feb 21 07:36:20 kernel: net_ratelimit: 57400 callbacks suppressed
Feb 21 07:36:25 kernel: net_ratelimit: 62262 callbacks suppressed
Feb 21 07:48:36 kernel: net_ratelimit: 36429 callbacks suppressed
Feb 21 07:48:41 kernel: net_ratelimit: 83498 callbacks suppressed

The gap in that timeline is unplugging one cable at the switch, waiting 5 minutes; unplugging second cable for one minute and replugging the first, waiting 5 minutes - everything is fine at the router during that time; plugging second cable back in and storm restarts.

More details:

• Comcast Arris cable modem in bridge mode
• AC86U router in paranoid vanilla state (no external ports open except Asus DDNS, minimal config); WAN connection to Comcast is fine;
• Netgear GS724T switch with nothing configured (reset for network rebuild, no VLANS, LAG, etc).
  • AC86U on port 1
  • TP-Link AV2000 on ports 19 and 20
  • Nothing else connected during test point above
  • Haven't set the 724 to suppress the storm yet - more interested in whether known issue/why?
• TP-link AV2000 in wallplug near switch, 2nd in wallplug in upstairs hallway (had been connected to AP, worked fine).
• Verified/swapped all the individual cables involved.

Everything (full network) had been working fine until I moved the Ac86U from the basement to a new set of ethernet plenum cables to the center of the house, at which point I disconnected the existing AP from the second AV2000 - and chaos ensued. Unplugging/resetting the second AV2000 made no difference (just realized I haven't reset the first one) - if AV2000 has nothing plugged in then I get a storm; and now, if I have 2 cables from switch, I'm getting a storm even with one cable plugged in to something alive at the other end (eh, another test cast - two cables at both ends).

Resetting the AV2000's doesn't seem to have fixed anything. I just got the TP-link utility, but it requires a wired connection to the device (when I get home....).

My google-fu has failed me so far. Familiar to anyone? Suggestions on how to proceed?

TIA

 

[degrub]

is there a loop in your ethernet network ?

 

[kab]

Well, yes, there may be a loop - but that would seem to be the Av2000!

Anyway, no, not when only one cable is attached. I finished everything (2 NAS, 2 game machines, a build host, all the wireless devices, VOIP...) and it's all peachy.

Then shut everything down.

Updated AV2000 ROMS, reset to factory, and reconnected.

Spent an hour w/ TP-link support and the normal "are the LED's on?" silliness until I explained that I could repeat the situation with exactly 4 devices: cable modem (optional), router, and the the AV2000 connected directly to the router (remove any chance of the switch doing something funky) - and nothing else at all on the LAN. And then replaced the 68U with another router (netgear700 w/ stock ROMs), just to be sure.

Kinda think the HW is bugged, and question is whether it's just the pair I got? Cannot imagine that this would not have been reported widely if common.

Oh, and in the giggles dept: The TPlink Utility was happily reporting 1083Mbps with a single cable connected (to a plug at the other end of my house?!?), but after connecting two cables, then disconnecting one and recovering, the Utility would report only 585Mbs -- without having done anything other than remove that cable.

 

[sfx2000]

Do you have a Sonos system in house?

 

[kab]

No Sonos, nor anything similar (no Alexa &c); typical TV/DVR in two rooms, stereo was completely off. Any of that known to cause EoP issues?

Replacement works as expected. Even has somewhat more realistic thru-put numbers: reports 870 Mbps vs 810 Mbps simple-minded copy test laptop<->NAS from across the house. Now I *can* generate a storm if I plug both of the cables from the far-end AV into a dumb switch (just checking!), but it's no longer the behavior when just plugging two cables into the first AV unit itself.

So, bad hardware batch.

 

[Samir]

is there a loop in your ethernet network ?

Yes, there most definitely is...and it's right here:

Symptom: TP-Link AV2000 (TL-PA9020) causes some sort of packet storm if two cables are connected from switch to first unit and nothing at all is connected to the second unit.

TP-Link AV2000 on ports 19 and 20

So each of the units has a built-in 2-port switch. And what happens when you plug two ports from an unmanaged switch to another unmanaged switch? Packet storm!

Plug in only one cable to the switch--either to port 19 or 20. And then plug something in the other PL unit and see if everything works normally. You can also plug in other devices into the other PL Ethernet ports and they should connect too--just don't connect any more connections than one hard wire back to the main switch.

 

[kab]

Yeah, thank you, I figured that out eventually. I was misled by TP-link tech support into butt-u-me-ing that the AV2000 was something other than a dumb 2-port switch (they also make a 3-port version). They **directed** me to plug both ports directly into my router. I even asked if that would work, and was told "Yes."

Way back before the original post, there was a real but very different problem causing storms even when connected "correctly". Short version, found a week after my original post and after a couple visits from my ISP about large signal spikes, was that 3m of unterminated coax, running behind a bookcase, was picking up the EoP signal and causing feedback through a MOCA connection to the cable modem and back to the router -- and to the ISP. Once found, we could watch the frequency and amplitude of the spikes change by holding the coax at various angles; and the problems all went away when that that "antenna" was terminated or unplugged from the wall.

The multiport AV2k units connect devices directly: e.g., a computer and a TV at one end to a computer and a router at the other, or similar. Why anyone would pay ~$30 for one extra port, vs $8 for a 5-port switch, is left as a exercise for the reader. I would recommend the single-port AV2k at $60-70 for a quick 200-400 Mbps solution (I typically get @600 Mbps in current location, but it's unpredictable from one plug to the next and also by phase of the moon, etc).

 

[Samir]

Ah yes, lovely 'support' that knows less about networking than you do. Too bad this is par for the course vs the exception.

Odd issue, but that does explain why there are these terminator plugs that I see the cable company now putting on taps--it must play quite a bit of havoc if left open.

Sometimes that one extra port would be good. I had a situation once where I needed an extra port and no other wiring/power/mess that someone could play with. So I ended up using two powerlines to get my two ports. In certain use cases it makes sense, but in others it would not. Either way, if you have a switch in the adapter, that's another point of failure. I actually have a PL adapter that the switch failed (or vice versa) so now it's useless.

 

[System Error Message]

the av2000 is not switched. the 2 ports are CPU connected and the best way to utilise them is to combine them (LAGG?). To do this you need a managed switch that also supports the LAGG type used.

You can ofcourse connect 2 devices to it instead or connect it to 2 different networks as theres no way you're getting above 1Gb/s at best. For me the best i ever got was 200Mb/s for the tp link av2000. Still, its always fun to combine ports for no reason.

When ports arent combined it creates a loop. So either set one port as fall back or combine them.

 

[Samir]

I'm a bit confused--how would you set up a LAG to the PL adapter unless the PL adapter can also set up a LAG.

 

[sfx2000]

I'm a bit confused--how would you set up a LAG to the PL adapter unless the PL adapter can also set up a LAG.

The multi-port TP-Link adapters are dumb switched - no way to LAGG them.

The extra port/ports are handy for certain situations - really nice for a Home AV setup.

 

[Samir]

The multi-port TP-Link adapters are dumb switched - no way to LAGG them.

The extra port/ports are handy for certain situations - really nice for a Home AV setup.

That was my thinking as well, but that contradicts what System Error Message is saying.

 

[System Error Message]

The multi-port TP-Link adapters are dumb switched - no way to LAGG them.

The extra port/ports are handy for certain situations - really nice for a Home AV setup.

That was my thinking as well, but that contradicts what System Error Message is saying.

I have this exact device you know. It is not switched, rather it is bridged. The device itself is also networked so you get the device advertising itself too (which is why we have the crappy tp link utility). If it were switched they'd have used 3 ports rather than 2, as the 2 ports are there because the theoratical max is 2Gb/s requiring 2 gigabit ports theoratically, and tp link utility should allow setting up LAGG if they havent ruined it even more.

The problem you have is a bug in the firmware as the device when not having anything connected to on the other side sends the L2 back and forth. If you did connect 2 ports to a switch, that would also be a cause for loops. Loops can be physical, or even virtual. Mikrotik switches have a layer 3 issue (not loop) when in layer 3 mode and the physical port which the gateway is on changes for example. So i wouldnt put the av2000 device out of the equation of the problem yet. What you have is a layer 2 loop, likely to be both physical and virtual.

 

[Samir]

Thank you for the great additional information on these.

So knowing what you know about them, would you avoid them and use a single port or 3-port PL adapter instead? Or is this issue just because of TP-Link's particular design?

 

[System Error Message]

Thank you for the great additional information on these.

So knowing what you know about them, would you avoid them and use a single port or 3-port PL adapter instead? Or is this issue just because of TP-Link's particular design?

i'd avoid getting a 3 port PL adapter, you're just adding too many chips in between. I never recommend to get PL with switch/wifi included, only passthrough for plugs.

I myself have used the tp link AV2000 a lot in the past, i only used 1 port though because the practical bandwidth does not necessitate this. If you connect them directly like through a wall (houses arent wired like this but if you are this anywhere is something for you to consider), you could get more than 1Gb/s but this is a rare setup you would only find in places like offices where the plugs if included as part of the setup will be on the same circuit with others.

As far as latency goes, not great either. There are times to use powerline and times to use wifi. For instance you could use power lines to wire up your passive network devices (sensors, smart devices, etc) because of the convenience it offers (plug + ethernet) which is a great way to reliably keep things connected if you dont get packet drops. Wifi on the other hand does much better when it comes to practical performance if you use 5Ghz. If the device has a 2.4Ghz wifi only you're better off with powerline especially the AV2000 or the best around as the practical speeds do increase the higher you go and the newer it is.

So if your device does not have wifi AC, powerline helps, if it does, use wifi AC unless you are wiring something like sensors, smart devices, alarms, etc where reliability matters more than practical throughput. This is because even on wifi AC under load you can lose the occasional packet while with powerline as long as you dont reach the practical bandwidth you will not get drops if the copper cabling is good and you dont get issues crossing fuses.

In summary i would still recommend the tp link av2000 or the zyxel variant for your powerline needs where the device does not come with wifi AC or you simply need reliability and the copper cabling is good. Other brands do have their products but i'd avoid dlink, netgear and weird branded ones (like BT for those in UK) as you're better off with wifi. I believe thiggins has a very in depth review of these where as long as the chips are good, the brand is ok and they use the latest standards (MU-MIMO stuff or similar for powerline), they are worth getting. With tp link i would not expect good advanced functionality but as far as basic functionality of connecting stuff it usually does well at that.