Expanding Network, Router Advice

[jrfuda]

New user here. Much of this is a repost from my intro post on ubnt’s forum, but is relevant to what I’m asking here as well, and part of the discussion may need to move to the VPN category.

I'm in the process of updating my home network - actually nearly every wire in the house (low voltage and electrical). My home was built in the 1860s and is 3800 SF, with another 800 SF of unfinished, but finishable, attic and a 600 SF garage on 1.5 acres in the middle of town where most yards are 1/2 acre or less. I've been into home automation since 2001, back when I only had two computers connected with a crossover cable.

Right now, my current router, an ASUS RTN66U shows 38 clients connected - a mix of wired and wireless clients, there are probably another 5-7 clients that are actually sleeping or otherwise disconnected because of the remodel, so let’s say 45. My goal is to get nearly everything wired except laptops and things without wired connections (tablets, phones, WiiU, etc.).

The size and construction of the house (still several plaster and wood and/or metal lath walls and ceilings, some multiple layers) really sucks up wifi signal, and the size of the yard (~250' x 250') make reception not so good. I'm currently using the router's wifi plus another router configured as an AP only to cover most of the house, but of course, I have to use different SSIDs since they can't do zero handoff , and there are still fringe areas. When I tried same SSIDs, I had clients connected to the more distant AP with bad connections instead of the nearer... my Squeezeboxes were notorious for this and would buffer like crazy.

Right now I'm in the middle of a major remodel, the second phase of probably 4 that I've been doing over the past 5 years as time and money has allowed. This phase had a lot of walls come down, tons of rot and termite damage be repaired, replacement of windows and doors that were not replaced in phase 2 and spray foam insulation of all exterior walls (also done during phase 1).

I've been running flexible conduit (Carlon Riser guard, 1.5" and 1", depending on location) as homeruns to a 4' x 8' space we discovered between walls, which is now the server room/ wiring closet. The conduit will carry all my CAT6, RG6 and other low voltage wire to locations in each room.

I recently received 3 x Unify APs to replace my router’s wireless and get rid of the second AP. I also have a cloud key due in anytime. Everything is currently on the test bench, which is my 22U wall mount rack temporarily mounted to some 2x4s on the floor in one of the rooms.

The test rack currently has a Dell Powerconnect 2748 48-port gigabit switch and a Powerconnect 3424 10/100 POE switch with gigabit uplink. Both switches are working perfectly with the ASUS router, and the gigabit uplinks are working between the two switches (just using cat6, as I have not figured out how to use the SFP ports yet, and assume I don't need to use them). I got the Powerconnects for about $70 each on ebay. I was worried about power draw when I first got these, but right now the whole rack (2 switches and an LTS LTN8816 NVR) only draw 1.6 amps (measured where the UPS they’re connected to connects to power), though I’m sure it will go up a bit as I only have 4 POE devices connected at the moment.

The LTN8816 NVR with 12 LTS IP cameras (16 camera capacity, will expand later) on the bench as well (4 connected for testing at the moment). I'm installing them as part of my rewire. The cameras and the UAPs are being powered by the POE switch (UAPs with 80211AF adapters) and the cloud key will be powered by it as well.

The POE switch will have:
16 x IP cameras
3 x UAPs w/ 80211af adapters
1 x outdoor UAP, (still need to get, not sure if unsheilded CAT5e I ran to yard 2 years ago is still OK, won’t know if I actually need it until the UAPs are up)
1 x Unifi Cloud key
2 or 3 x in/on wall touchscreen PC/tablet (still trying to find good value in POEable tablets)
1 x uplink to 48-port switch (or router, whichever is more efficient, need advice on this)

The 48-port gigabit switch will have:
2 x R Pi Minecraft Servers
8 x Logitech Squeezebox
1 x Plex Server PC
1 x Logitech Music Server PC
1 x Windows Home Server
1 x Homeseer Z-Net
1 x Homeseeer Hometroller
1 x Quatech Serial Device Server
4 x Tivo (might add 2 more later)
1 x Tivo Stream
2 x Roku
1 x LTS LTN8816 NVR
7 x smart TVs
1 x uplink to POE switch (or router if that's more efficient)
All remaining ports as extra runs (1 or 2 to each room) for expansion

Wireless connections
4 x laptops
2 x phones
2 x tablets
1 x WiiU
2 x Squeezebox Duet controllers
2 x Roku sticks
1 x Chromecast

The router will be connected to my DSL modem and either both switches or to one switch, whichever is supposed to work better (could use advice on this). Two UAPs will be on opposite ends of the 1st floor with the third in the center of the attic (3rd floor, unfinished but insulated and conditioned, so no heat issues).
Now, given all this equipment I have some questions.

- Will my ASUS RTN66U handle all of this OK. It's been doing great so far. I've never had to reboot it - it always just works and has for ~3 years. I also installed one at my in-laws after they were having problems, and it's never missed a beat there either. So my assumption is yes, it will be fine.

- Right now I'm forwarding ports, using non-standard private ports to make it a bit more difficult to mess with my network, but I know that's not anywhere near the safest thing in the world. I'm interested in setting up VPN. I’m considering 3 courses of action:

1. Use VPN built into the router (I’m using one of the Merlin Builds) or load OpenVPN on the router. Is the router robust enough to do this? How will I generate keys? I’m new to VPN so fuzzy on the term... does the router take care of all authentication or does a RADIUS or other authentication server have to be running.

2. Set up another Raspberry Pi 2 as per this tutorial: http://readwrite.com/2014/04/10/raspberry-pi-vpn-tutorial-server-secure-web-browsing which seems to cover everything including install, key gen and authentication

3. Get a more security oriented router that has robust VPN support. So far, other than old ebay stuff I don’t have a clue about, some of the routers listed in the router ranker and Ubiquiti’s Unify Security Gateway and EdgeMax routers, I haven’t looked at much. Are these any better than my RTN66U for achieving my goals? I was leaning towards the Ubiquiti products, especially the USG because of being able to use the controller software to view my whole network, but it appears that most of the real settings in these items are done by CLI and – for now – the controller software (USG) or EdgeOS interface (Edgemax) seem more of a novelty.

My ISP is going to be a bottleneck. I have DSL that’s stuck at 3M down and 768K up becuase of line issues in the neighborhood (people down the street get 10M down). Getting cable will require some excavation in the yard that I’m not ready to deal with yet, but I will eventually get it installed unless CenturyLink gets off their backside and gets the DSL improved.

Based on the above, any advice on where to go with router and/or VPN setup?

 

[System Error Message]

you have too many devices and your IP cameras are bottlenecked. The router matters only for internet speeds. In your case it has plenty of CPU for DSL and vpn at dsl speeds. With RMerlin's firmware you can get the same security as a configurable router. A lot of tasks can be combined, plex media server is just a software that runs on the machine, you can run other things on it.

If you use ubiquiti wifi, switches, routers and such than go get a USG but if your network consists of devices from varying manufacturers the USG's controller software is useless.

When you upgrade to cable and i assume you are going to be using QoS and firewall you may want to consider mikrotik instead as the CCR1009 is well capable of performing NAT, QoS and firewall at gigabit speeds and is quite fast for vpn, compared to the best ubiquiti has to offer the CCR1009 trumps the edgerouterpro when it comes to software speeds. Hardware acceleration doesnt work if you start adding rules which is why the speed stats shown by ubiquiti are usually worthless. The CCR1009 does layer 3 routing in software at wirespeed wheres the edgerouter series relies on hardware acceleration to achieve wirespeed.

If you really want a security based router, dont go with any of the embedded products, install a UTM distribution on an x86 machine with 2 NICs and use that as the router as well.

If you use vpn built into the router you just set the username and password but mikrotik has RADIUS server and lets you choose from various sources where the authentication is from. Since you have x86 machines or even the RPi you can use them as the RADIUS server.

You will want managed switches. If your cameras use gigabit ethernet ports than use a gigabit POE switch otherwise the 100M poe switch is fine. You will need to combine 2 gigabit ports to the 48 port switch as that will eliminate bottlenecks and your file server which the cameras store their images will also need dual NICs so you will need a semi managed switch for combining ports. IF your poe switch is gigabit than connect your server which your camera uses to it. If you are looking at stacking switches than you will need to calculate how much of a bottleneck are you willing to live with. a 24 port switch with 2 10Gb/s ports can stack without bottlenecks to only 1 other switch like it or half 50% maximum bandwidth by stacking using a bus like network. Since your 48 port switch is the central switch it doesnt need 4 10Gb/s ports but it needs enough ports to stack with the other switch depending if you're stacking with 10Gb or 1Gb.

 

[jrfuda]

you have too many devices and your IP cameras are bottlenecked.

The POE switch is 100M but has Gb uplink (2 RG45 and 2 SFP) ports. According to an online IP camera bandwidth calculator:

16 2MP cameras at 30fps = 150 Mbps, or 100 Mbps at 20 fps (I'm currently using 20 fps), so 1 camera is 6.27 Mbps.

Since the POE switch, while 100M, has Gb uplinks, does that mean the backplane is at least a Gb, or do I not understand how this works, which is likely. If it does have a Gb backplane, and is connected by Gb to the other switch, does that relieve the bottleneck here?

A lot of tasks can be combined, plex media server is just a software that runs on the machine, you can run other things on it.

The machine running plex is maxed out once it starts transcoding, and is still stressed when direct streaming (It's on a Lenovo Q190 with a Celeron 1017U, so it's not powerful at all, need to update). The Logitech Media Server is on a Quantum Byte with and Atom Z373, so again ,little power to spare. They both sip power though!

If you use ubiquiti wifi, switches, routers and such than go get a USG but if your network consists of devices from varying manufacturers the USG's controller software is useless.

I only have Ubiquiti APs, the switches are Dell PowerConnects.

you may want to consider mikrotik instead as the CCR1009 is well capable of performing NAT, QoS and firewall at gigabit speeds and is quite fast for vpn, compared to the best ubiquiti has to offer the CCR1009 trumps the edgerouterpro when it comes to software speeds...

I will keep this in mind, the price is not too bad either.

If you really want a security based router, dont go with any of the embedded products, install a UTM distribution on an x86 machine with 2 NICs and use that as the router as well.

I need to read more about UTM.

You will want managed switches.

The PowerConnects are managed, but I'm not currently using any of the managed functions. I reset to defaults both of them and took the 2748 completely out of managed mode. The 3424 (POE) cannot be taken out of managed mode, which is fine, but other than possibly using the web interface to power cycle devices connected to it (or do as you advise further down), I haven't found a need to use the features yet.

If your cameras use gigabit ethernet ports than use a gigabit POE switch otherwise the 100M poe switch is fine.

The cameras are 100M.

You will need to combine 2 gigabit ports to the 48 port switch as that will eliminate bottlenecks and your file server which the cameras store their images will also need dual NICs so you will need a semi managed switch for combining ports. IF your poe switch is gigabit than connect your server which your camera uses to it. If you are looking at stacking switches than you will need to calculate how much of a bottleneck are you willing to live with. a 24 port switch with 2 10Gb/s ports can stack without bottlenecks to only 1 other switch like it or half 50% maximum bandwidth by stacking using a bus like network. Since your 48 port switch is the central switch it doesn't need 4 10Gb/s ports but it needs enough ports to stack with the other switch depending if you're stacking with 10Gb or 1Gb.

When I bought the switches I knew even less than I do now, so I did not even know stacking was an options. From what I've read so far, I cannot stack the switches (stack in the way where they are managed as one switch) since they are not the same series. Are you referring to stacking in a way where they still remain independent of each other? I know I'm showing my newbness here. Is there any advantage to using the SFP uplinks? I'm having trouble figuring out which modules will work with these two switches, and so far have only found an RG45 module that will work with the 3424 (POE) but nothing for the 2748. They're both older switches, which is why they were so inexpensive.

My NVR (not using a PC or file server) has 16TB storage and a single Gb connection. By the above calculations it should be getting a steady stream of 150M once all the cameras are installed, plus whatever overhead and additional bandwidth from when I or my home automation software access it. Given your advice to place the NVR on the same switch, what would be better overall, to use the second Gb connection on the 100M POE switch to attach the NVR or keep the NVR on the 48port Gb switch and use port combining to connect them?

All that said, once everything is running, what is the best way to actually test throughput to test various configurations. Since I can't install software on the NVR, I could place PCs on different parts of the network to test transfers between them. Is there a good program for this?

 

[coxhaus]

The backplane is always bigger than the port speed.

 

[System Error Message]

stacking is a concept and what you need is a switch with higher bandwidth ports or the ability to combine ports and RSTP at least. RSTP is important and set your 48 port switch as the root. Than combine 2 gigabit NICs on your 100M switch to your main switch (the ports need to be combined on both sides). This will allow 2Gb/s of data from your various devices to transfer without bottlenecks for the other stuff you have connected. It means your cameras can use the highest quality settings.

Since you have ubiquiti UAPs you can get the USG if you find the controller will help with the UAP management and administration.

x86 UTMs are usually better than embedded because they are faster and have more features or flexibilities. License depends on the distribution and some are free. Ive seen intel NUCs on amazon uk between £200-£300 for mobile i3-i7s that have wifi and dual NIC.

I was reading that the new celerons are intel atoms, however it has always been the case that a celeron is a cut down version of a full CPU. Either use a pentium or higher or AMD phenom ii or higher (not their APUs) for plex. If you are worried about power usage you can use standby with wake on lan. You can do this with a lot of your servers or machines you keep on all the time except for networking equipment. If you use a good x86 CPU you could use it for storing camera videos and using plex on them as well.

 

[abailey]

Since your Cameras are using less than 200Mbps I would use the two Gig ports to connect the two switches and keep the NVR on the 48port switch. What UAP's did you get? If you got UAP-AC then you are limiting them by putting them on a 100Mbps switch. Also don't know if you are planning on using zero hand-off, but if you are just know (at least last time I tried - less than a year ago) the AC class UAp's could not do zero hand-off and all zero handoff could only use 20mhz wide channels, whether 2.4 or 5ghz. This made me abandon zero hand-off, and ultimately Ubiquiti AP's.
As far as UTM's go. I use the Zyxel USG40 at my house and really like it.

 

[jrfuda]

I just have regular UAPs. We dont use wireless in a way that we need a lot of speed as most everything bandwidth hungry is hardwired. The exception isnthe occasional file transfer, like moving a video I just encoded, from my laptop to Plex, but it only takes a couple of minutes.

 

[jrfuda]

I believe I've got the bonding/stacking working. Both switches had Link Aggregation Groups to which ports could be assigned. I assigned the 2 Gb ports on the 100M POE switch to a LAG and 2 on the 48 port Gb switch to a LAG. It appears that only the 24 port switch has RSTP. The 24 port has a more robust feature set than the 48 port switch, probably a correlation between model series (27xx for the 48 port and 34xx for the 24 port), with higher numbers being more powerful/newer. The 3424P has a terminal connection but the 2748 does not and is only configurable through the web interface or can be set to unmanaged mode via a button. I've also seen that making settings stick through a power cycle in these is not a one step deal as it is in the consumer hardware I'm used to. I had to apply setting, then go to file management and copy current settings to a startup_config file. Guess I'll figure it out eventually.

I assume the LAG is working. I tried pinging hardware on the 24 port, pulled one of the cables, saw a lost packet and then recovery, same happens after reconnecting, same happens again after pulling the other cable. Both ports show data passing through, at least on the 24 port where I can actually monitor bandwidth, which, with 4 cameras currently feeding the NVR, is about 16Mbps, which is less than what the online calculator said it would be (25Mbps).

 

[System Error Message]

that means the cameras are not giving you high quality feeds as there are a few bandwidths depending on quality. For example 720p streams has between 1-4Mb/s depending on the quality while 1080p has 8-16Mb/s again depending on quality. If you've used something like plex and looked at the quality setting of a HD video you will see this.