Re-thinking my home network

[ncfoster]

Hi all,

I have been having a few network-related problems lately, and it has me thinking about how to re-design things.

My current network architecture is this (all physical hardware currently in my basement, unless noted):

Comcast service coming in through a Motorola SB6120, which feeds directly to-->
Primary router: Asus RT-N56U, which feeds to:

1. Secondary wireless router: Asus RT-N16U (currently serving most of my WiFi needs, because the RT-N56U has been so unreliable recently)
   
2. D-Link DGS-1016D Switch, which feeds to:
   1. A distributed wiring panel that feeds gigabit throughout the house
   2. Two Ooma VoIP boxes
   3. 3 PCs in the basement, including a media server, a desktop machine and an HTPC.
   4. 2 printers in the basement

Everything in the house has its own feed from the DGS-1016D, except for the living room, where an Asus GS-D1081 switch feeds my TV, PS4, Tivo Roamio and an Oppo Blu-ray player.

My two main problems are general wireless connectivity and performance for games on my PS4. I believe that most of the wireless problems stem from trying to use 3rd party firmware, where the wireless drivers seem to be sub-par. However, in my experience, this firmware does make my wired connection perform better for my PS4. I have been through many iterations of Padavan, Merlin and OpenWRT. I keep tinkering, hoping to make my PS4 work better, but none of them seem to get it all of the way there, and the wireless problems just aren't worth it.

Previously, the RT-N16U was the main router, and it was starting to show its age. Also, I have tried running old computers with various routing distros of Linux on them as the main router as well.

My current thought process is that I want to split up the wired and wireless duties. The two general scenarios would be:

1. Get something new to handle the main routing duties. While I think that the performance of the RT-N56U is good overall, there is just something about it that seems not to be up to the task for online gaming. I don't know what it is, nor how to best narrow it down. If I did this, I would put the RT-N56U back on either stock firmware, or at least something that is close enough that it works.
2. Use the RT-N56U as the main router, and turn off wireless. Buy an AC router of some sort to handle wireless duties, and potentially place it somewhere other than the basement in order to optimize placement.

I am also open to the possibility that either of my other gigabit switches could have something to do with it. Replacing those wouldn't be my first choice, but if I had obvious reason to do so, I would. Running with no switch near the entertainment center would be the most difficult, since I would have to add significant additional wiring from the basement.

I would like to have something that is easy to keep up to date with current advances, whether that is vendor-provided or 3rd-party firmware. However, I don't want something that requires a lot of management on an everyday basis. I need something stable, since my wife works at home a lot.

As you can probably tell, I am not a networking expert by any stretch, but I am not afraid to get my hands dirty setting things up. I just don't want to have to manage it often.

Thanks, in advance, for any and all suggestions.

 

[thiggins]

What specifically is wrong with gaming on the PS4?

 

[ncfoster]

What specifically is wrong with gaming on the PS4?

I play mostly the NHL series of games. I have more than my share of laggy games. It can be okay sometimes if I have the connection all to myself, but especially when I am trying to compete with things like my wife or kids streaming Netflix in another room, it is pretty much a no-go. I do think when I started running with QoS on, it was an improvement, but I have never been able to get the two to co-exist nicely.

I do know that EA's servers are not always the most reliable, but I do think that there are issues on my end.

 

[thiggins]

what are your downlink / uplink speeds?

 

[ncfoster]

what are your downlink / uplink speeds?

Speedtest is giving me 90/12 on stock firmware. I think it is probably supposed to be more like 75/10. I was getting much lower numbers on OpenWRT.

 

[thiggins]

Speedtest is giving me 90/12 on stock firmware. I think it is probably supposed to be more like 75/10. I was getting much lower numbers on OpenWRT.

Is there a pattern to the problem? Certain times / days of the week? With or without Netflix running? How many Netflix streams?

Has your tinkering included QoS? If yes, what have you tried?

Since you are on cable modem, your connection is affected by others on your node. So you should take a look at your internet connection quality.

DSLreports.com has a few tools that can help. Smokeping and LineMonitor in particular will provide a better idea of long-term latency.

 

[ncfoster]

Is there a pattern to the problem? Certain times / days of the week? With or without Netflix running? How many Netflix streams?

Has your tinkering included QoS? If yes, what have you tried?

Since you are on cable modem, your connection is affected by others on your node. So you should take a look at your internet connection quality.

DSLreports.com has a few tools that can help. Smokeping and LineMonitor in particular will provide a better idea of long-term latency.

My wife has only recently started using Netflix upstairs, but my problems have gone back before that. I have tried QoS, yes. I just went back to stock, and I don't have a record of what my settings were. I am open to any suggestions, as I have always found QoS a bit of a black art.

I tend to play online games late at night, often into the very early morning hours. It tends to come in spurts, but I don't know that there is a pattern in terms of time of day.

After going back to stock firmware, my overall experience seems better, but not completely trouble-free. It is possible that I am playing against others with connection problems, but it still feels like it could be better.

I will try those tools and see what I find out. Thank you.

 

[coxhaus]

Can you plug your laptop or machine into the same wire as the PS4 and run DSLreport speedtest. Do you still get 90/12? We need to see if there is a bad wire or connection in your network. Trouble shooting is just a process of testing everything one at a time.

 

[Trip]

First off, the modem. Connect directly and test for consistent passage of traffic. This should rule out any issues with the 6120.

Next, Asus firmware cleanup. Good start with RMerline, but he's still bound by the confines of AsusWRT and any possible concessions it may still entail. I don't have experience with Padavan, but OpenWRT and DD-WRT sub-versioning can be a joke as far as reoccurring bugs go and I personally don't have the patience. YMMV. My preference in most cases is AdvancedTomato (Shibby's TomatoUSB with a much nicer GUI). I would flash both from recovery, do a proper NVRAM clear-out and redo your configs from scratch. Tomato usually improves things a fair amount. This still doesn't dying hardware itself, but if you individually test each Asus unit and the other pieces in your topology with some of the tools mentioned above and they check out, I think it's safe to assume you're "ok" as far as baseline functionality of your network hardware.

Moving on, latency spikes and other performance "blips" for LAN-WAN activity most likely come down to improper topology design causing preventable bottlenecks, and/or lack of a proper QoS schema. The easiest way forward here would be to first make sure you're not loading one particular switch, AP, etc. with too much traffic right off the back. For example, a "gigabit" switch whose actual backplane capacity is a not enough, or an AP radio that just can't keep up with client number or wifi bandwidth. I highly doubt the above scenario is the case, but you want to rule that out first. Next, QoS. I'd try classing/prioritizing in Tomato or whatever firmware, then perhaps move to a firmware with fq_codel. You could use DD-WRT Kong, but like I said, I've found it to be a total gamble as to whether it will be stable or not, and I personally don't have the patience. Another option would be a $50 EdgeRouter-X with smart-queues active (fq_codel + HTB). It will most likely solve 99% of any previous router-based bottlenecking, but you have to have some skills to set it up, and Ubiquiti "support" really is just wikis and a somewhat stuckup user form. If setup properly, though, it will be set-and-forget.

If you see bottlenecking with just wifi, then either or Asus units may just be underperforming regardless of firmware, and you might need to replace them newer AIOs set as APs or purpose-built products to get the load levels and coverage that are acceptable to you.

 

[ncfoster]

I am in the process of a 24-hour smokeping test, which doesn't seem to be revealing anything to my eye. Image attached.

Going to test the cable some time tonight.

Testing the cable modem is much more annoying, but will see what I can do.

I could be missing something about Advanced Tomato, but I don't see that it supports the RT-N56U. If it does, I am happy to go that route. Shibby's Tomato is still running on the secondary router, and it is my preference. I am not familiar with the "Kong" variant of DD-WRT, but I don't think that there is RT-N56U support there either. Of course, that isn't really a ringing endorsement from you.

I have done a lot of reading about codel, and I am intrigued that it could be part of the solution, but it is my impression that it is only implemented in newer versions of the Linux kernel, and that somehow this doesn't work with any reliable version of the wireless driver on the RT-N56U. This is somewhat how I ended up going down the rabbit hole of OpenWRT, etc., which I now think was a bad road.

EdgeRouter-X looks intriguing at a pretty low price-point. It could be what I am looking for if I just want to move the RT-N56U out of the primary router position. Whether I have the "skills" necessary to set it up right is an open question.

Thanks all for the input.

 

[Trip]

Curious as to how your smokeping testing is going...?

TL;DR - I was perhaps a bit harsh on OpenWRT; it actually runs pretty well on a lot of stuff, especially now that Chaos Calmer is out and largely solid on most Atheros AR7161 and AR9x series stuff, but DD has been quite the conundrum with several pieces of gear I've had... As I said, YMMV.

And I can confirm that unless part of your config is built on something highly specific to one of the *WRTs (AsusWRT included), the ER-X is a pretty nice little stand-in, with fq_codel baked in and working out of the box (after a quick update to 1.7.0, of course).

 

[ncfoster]

Curious as to how your smokeping testing is going...?

TL;DR - I was perhaps a bit harsh on OpenWRT; it actually runs pretty well on a lot of stuff, especially now that Chaos Calmer is out and largely solid on most Atheros AR7161 and AR9x series stuff, but DD has been quite the conundrum with several pieces of gear I've had... As I said, YMMV.

And I can confirm that unless part of your config is built on something highly specific to one of the *WRTs (AsusWRT included), the ER-X is a pretty nice little stand-in, with fq_codel baked in and working out of the box (after a quick update to 1.7.0, of course).

After 24 hours, I saw nothing that looked at all remarkable. Not an expert, but it seemed pretty straightforward that it was mostly a straight-line. Just to be clear about fq_codel, if you have that at the entrypoint to your home network, that is pretty much all that you need, right? Nothing else inside the network really needs to support it at all, right?

 

[ncfoster]

Line quality test results similarly unremarkable. File attached.

 

[Trip]

Re- fq_codel, pretty much yes, as long as you're supplying enough switching and wifi headroom for your endpoints coming into the router to get there without bandwidth contention.

 

[ncfoster]

Edgerouter X has arrived. Now, the next question is how to deploy it. Obviously, it will be the first link in the chain. With five ports, I could directly attach up to 4 things downstream, I would think. Am I best off just passing everything to the large switch first? Should I pass to the large switch, the RT-N56U and the RT-N16U, and the link to the entertainment center switch? Something else? Is there anything else obvious that I should consider in making this decision?

I am hoping to retire the RT-N16U shortly, presuming that the RT-N56U proves reliable in a solely Access Point capacity. I am merely including it for now for completeness purposes.

Edit: I think that I came to the conclusion (without trying the alternatives) that the large switch should remain the center of the network, and that I should not attempt to branch off of the Edgerouter X, as that might unnecessarily complicate matters. Sound right?

 

[ncfoster]

Update: Edgerouter X installed. I have taken the following steps so far:

1. Ran WAN+2LAN2 Wizard (initially used WAN+2LAN, but I have my local network setup to be on 192.168.27.x, which I want to maintain).
   
2. Setup a new user account and deleted default UBNT user account.
3. Ran UPnP Wizard for simpler PS4 setup.
4. Enabled a Smart Queue to get fq_codel + HTB working (setting the bandwidth numbers, but using no advanced settings).
5. Put Asus RT-N56U in Access Point mode for wireless clients.

I have also tested that the cable used for the PS4 also provides the same speedtest.net results on my laptop as I get wirelessly. These results stay consistent from RT-N56U as the main router to Edgerouter X as the main router. In each case, they are approximately 90/12. Turning on Smart Queue in the Edgerouter causes those numbers to be slightly worse, especially the download numbers. I believe that this is an expected result with QoS.

I have played a few games of NHL 16. To my understanding, when I am designated as the home team, I am hosting the game session, and when I am away, the other side is. I played one each way. Neither game was horribly laggy, but neither was as smooth as I would have hoped. In neither instance was anything major going on elsewhere on the network, like Netflix. I will test that scenario later, but I want to make sure that I am off to the right start. Any thoughts? Thank you for all the input thus far.

 

[Trip]

Nicely done so far. Re- QoS, yes, the actual value used for the queueing discipline is like 90-95% of what you specify in the fields, so of course the larger the number, the bigger the reduction -- ie. a 90/12 would be reduced to, say, 80/10. In terms of you noticing a continued bit of lag in the scenario you described, there are tons of factors out of your control that could still be involved in making the overall experience feel that way; in no particular order: the quality of the connection and equipment of your opponent, especially when he/she is the one hosting the gameplay, the routing and peering latency/jitter/packetloss at any/all hops between you, the mediating PS4 network, and your opponent, and other external factors unrelated to just the ER-X and your gear.

Sounds like for your own LAN and LAN-to-WAN, you've got things set as needed. In terms of assuring quality of service and airtime fairness to the router, as long as you've got enough backplane capacity on your switches to accomodate max simulatenous wired traffic and enough channel bandwidth and processor capacity on the AC56U and whatever other APs to accommodate as many wireless clients as you might normally have, then I don't see your LAN components being a bottleneck.

If you're inclined to troubleshoot network segments, then wiring clients straight to the ER-X, for example, would be a good way to rule out a distribution switch that you think might be flaking out on you, or an access point-generated latency/jitter, etc.