Looking for new routers for FIOS Gig with emphasis on VPN Throughput

[Roveer]

Thank you @Roveer !!

I have been eyeing the qotom boxes. But some users on the pfsense qotom thread
https://forum.pfsense.org/index.php?topic=132528.0
have concerns with heat and a backward port assignment:

Mac   Phys port   BSD name
-----------------------------
xx6F    1          igb0

xx70    4          igb1

xx71    2          igb2

xx72    3          igb3

Your build appears to be a viable alternative with a lower cost. My pfSense appliance was purchased in BK thru a pfSense reseller. It has a Quad core Atom D525 CPU installed. It does not support AES-NI. I did not know enough at the time for this to cause me concern. I will want to replace it when pfSense 2.5 is out. Please keep me posted on your progress.

I am convinced that you can NOT achieve what I am looking to do (ipsec vpn site to site) at line speeds with anything less than Intel core processor iron. I would love, love to have a non pc solution, something that works off of a 12v brick, an appliance, but everything I've learned (and I still have a lot to learn) tells me there's not enough oomph in those lesser processors to do what I'm trying to do not to mention the lack of AES-NI.

In my lab test when I had the i5 and i7 running both sides of my system I was unable to get more than 500mbps with AES-NI disabled. At that point I wasn't sure what it was going to take. Then I configured AES-NI and set up the correct encryption parameters (yes, it does matter what you use). I'm still in the testing phase on that. And all of a sudden, instant jump to 900 mbps over the vpn tunnel. Now, that was connected via a gig switch, but at least the encryption overhead processing was there. The i7 was probably somewhere in the 15-20% utilization (during full out iperf extended 600 second test) and the i5 was probably 30-40%. And these were old core processors. The i7 was a 3770 and I don't even know what the i5 was, but thinking it was probably the same vintage.

Part of what has convinced me that big iron is needed is that I bought a CheckPoint 4600 appliance that is rated for huge numbers 1.5 gb ipsec VPN. It puked at 350 mbps. No HW AES acceleration on an on P4 dual core. Don't get me wrong, its a nice box and will be my lab box, but it won't do what I want. I determined if I wanted to try and use a Check Point hardware solutions I'd have to go to a 12200-12400 series which is considered an enterprise box and not cheap. So all of this basically led me to what I am doing now.

I am definitely jonesing to build my own appliance and eventually will. This will fit the bill nicely: https://www.amazon.com/dp/B01KP8GOXI/?tag=snbforums-20 My only problem is that when I was researching processor TDP spec's it will be heard to keep a K series processor cool enough in this type of enclosure. That could be a problem during high workloads.

Understand that I also wanted overhead to consider pfSense packages like AV, IDS, ADBlock etc all while having my super fast ipsec vpn.

So we'll see what I end up with when the other i7 arrives next week. I'm excited to see how fast I can push data between the two locations.

Roveer

 

[sfx2000]

I went with used Dell Optiplex 7010 Sff (small form factor). A fairly small box. I7-3770 4gb ram, and I put a laptop hdd in for now. Staying away from ssd on pfsense until I explore the whole "embeded" thing which wasn't obvious to me on the latest version.

I paired that with a 4 port intel server nic. Here are links to the pieces I used:

Sounds like a nice pfSense box - clock speed is important, perhaps more than number of cores once one hits a certain point. A third gen i7 desktop CPU is more than enough - even an i5 desktop CPU is going to perform well.

Check performance with HT disabled on the i7, might actually see better performance there.

 

[sfx2000]

Part of what has convinced me that big iron is needed is that I bought a CheckPoint 4600 appliance that is rated for huge numbers 1.5 gb ipsec VPN. It puked at 350 mbps. No HW AES acceleration on an on P4 dual core. Don't get me wrong, its a nice box and will be my lab box, but it won't do what I want. I determined if I wanted to try and use a Check Point hardware solutions I'd have to go to a 12200-12400 series which is considered an enterprise box and not cheap. So all of this basically led me to what I am doing now.

The Checkpoint is good for the bandwidth as a collection of connections, but as you noticed, each individual connection over VPN takes a hit.

We have a Xeon-D box that can do wire speed on L2TP/IPSec for site to site tunnels (along with many other things) on a Gigabit connection - but we're well out of most home network price ranges - not pfSense, but our own SW. We've got some special sauce with a tightly optimized linux kernel and we definitely leverage the heck out of QAT and DPDK - our big boxes do much more, including 40Gb and 100Gb... but those are enterprise/carrier grade boxes, and priced accordingly.

I've been tempted to grab one of the lab boxes and spin up pfSense to see how it would work...

 

[Roveer]

The Checkpoint is good for the bandwidth as a collection of connections, but as you noticed, each individual connection over VPN takes a hit.

We have a Xeon-D box that can do wire speed on L2TP/IPSec for site to site tunnels (along with many other things) on a Gigabit connection - but we're well out of most home network price ranges - not pfSense, but our own SW. We've got some special sauce with a tightly optimized linux kernel and we definitely leverage the heck out of QAT and DPDK - our big boxes do much more, including 40Gb and 100Gb... but those are enterprise/carrier grade boxes, and priced accordingly.

I've been tempted to grab one of the lab boxes and spin up pfSense to see how it would work...

I figured there had to be "special sauce" in the checkpoint systems in order to produce some of the speeds they report on their spec sheets. Thing is, I even tried my ipsec/vpn tests using their software and still got way below the rated speeds. Not just a little, but more like 75% less than stated spec's. As far as I'm concerned unless you have some hardware acceleration going on you'll never produce the kinds of speeds I'm looking for and the 4700 box I have only had a Pentium dual core processor (no AES-NI).

Most of those boxes are capable of service a bunch of users, but individual connections may disappoint. As usual, I'm in a unique situation with my "demands" and from everything I've read and tested, I just need more processing. For now I'm going to stick with my little dell 7010 SFF boxes which are i7-3770 and see how I do. I'm not really happy about having to run another full computer 24/7 but if that's what it takes to fully utilize my FIOS GB then so be it.

I'll report back on my outcome once I have the 2nd 7010 running and have performed some iperf tests across the vpn.

Roveer

 

[Audio-catalyst]

This is a copy of a post I made over on CPUG.COM about my Checkpoint 680's. Since I see there is a lot of knowledgeable people here I figured I'd share in case anyone had and comments. Thanks.

This week I upgraded both sides (work & home) to FIOS gigabit internet. It's supposed to top out around 800 Mbps or so.

My 680's don't use any of the intrusion or AV blades, just Firewall and IPSEC/VPN.

First observation: Speedtest.net and FIOS speed test to internet top out around 450 Mbps in both directions at both locations. Processor pegged and router gui sometimes stops responding til test finish. Specs for 680 say Firewall (Gbps): 1.5, so I'm getting 1/3rd of spec'd speed.

Second Observation: IPSEC/VPN is giving me 112 Mbps between devices. Specs say 220 Mbps so I'm getting 1/2 of spec'd speed.

I've tried setting IPSEC/VPN encryption settings about as low as they can go (AES 128, DH 768) with no real change in speed results.

Third observation: Iperf3 results 2 machines either side of link: 122 Mbps. Was expecting at least 200.

I wasn't expecting a miracle, but this is a bit worse than I expected. Is there anything I should look at or tune? Looking now for devices that would handle the new internet speeds, with the emphasis on IPSEC/VPN throughput. Any suggestions (for a small business on a limited budget).

Thanks,

Roveer

If you like the cli of the edgerouters, take a look at vyos
Both are derived from vyatta, but vyos is build for x86, is a proper router distrubtion, used by large providers, and best of all completely free to build your own

Verstuurd vanaf mijn SM-G850F met Tapatalk

 

[paraplu]

I am using vyos 1.2 in a VM on ESX 6.5 since a month and am very happy; very low lag time.
Using codel QOS, ssh and OpenVPN on a 3.5ghz intel core i3 with AES.
OpenVPN consumes <20% cpu with 100mbps (my test case), a 500mbps OpenVPN should be no problem with this kind of setup.

 

[Roveer]

So tonight I was able to install the other Dell 7010 and get it configured. Here is a copy of the post I made from the pfSense forums: Hoping we can add to the discussion and come up with some ideas on how to improve machine to machine speed.

So I've been posting over the past few weeks about wanting to put together pfSense boxes on either end of two FIOS gigabit services and connect them via ipsec vpn.

I bought 2 Dell 7010 I7-3770 machines and put in Intel 4 port NIC's. Installed pfSense 2.4.1 and configured for AES-NI operation.

The results are both good and bad. Not exactly sure what's happening here, but I have a lot of data to share. Hopefully we can figure something out.

Here are the iperf results of the WAN interfaces across the internet:

Very happy with that speed. Pretty much full subscription rate.

Here is a traffic graph of the previous iperf run:

Notice the traffic is only showing up on WAN, no LAN or IPsec. This is wan port to wan port across the internet.

Next up is iperf results of the LAN interfaces across the ipsec vpn:

Here is the traffic graph of the vpn iperf run:

You see traffic showing up on WAN and IPsec. I'm very happy with these results 872 mbps. On my previous non AES-NI setup I was only getting 250 mbps.

But here is where all the joy ends. When I iperf two computers connected to networks on either side of the tunnel the results drop down hugely. Not sure why. If I iperf from machine to local WAN interface I get 900+ mbps so I know I have a fast enough computer and it's getting packets to the WAN quickly (both computers on both sides can iperf to their respective WAN interface at 900+ mbps. But when I iperf between the two computers it drops all the way down to 274 mbps. I can't for the life of me figure out what's going on. Here it is:

So a little more information.

The first two iperf tests were done from shell's on the firewalls. iperf commands are very simple: iperf -B 192.168.0.1 -c 172.16.1.1 no other switches used -B binds to a particular interface which is how I force it to use the ipsec or wan ports.

On the computers I open command prompts and do very simple iperf -c 172.16.1.117 commands.

Windows smb file copies are 35-38MBps : I was shooting for 70-90MBps

So any input or ideas are greatly appreciated and hopefully I can somehow improve these speeds otherwise I succeeded and failed at the same time.

Many thanks,

Roveer

 

[kvic]

@Roveer

you may want to experiment with different MSS size which is part of CLI option in iperf.

 

[Roveer]

@Roveer

you may want to experiment with different MSS size which is part of CLI option in iperf.

I will, but with my windows copy speeds of only 35-38 (peaking at 40 after some NIC tuning) I'm only seeing 320 mbps of throughput when the tunnel is actually capable of pushing 800+ mbps (measured). The CPU is only running up to 14-16% during these transfers so I'm wondering where it's all going wrong.

Tweaking the iperf test might show better results, but the file transfers are still not anything close to the line speed.

Roveer

 

[MichaelCG]

What is the latency across the VPN?

Sent from some device using Tapatalk

 

[kvic]

Tweaking the iperf test might show better results, but the file transfers are still not anything close to the line speed.

The intention of adjusting MSS is not to show better result in iperf. But rather to help you diagnose if it's a MSS issue. Once proved so, you can tune the system MSS accordingly over the VPN.

 

[Roveer]

What is the latency across the VPN?

Sent from some device using Tapatalk

4ms

 

[Roveer]

The intention of adjusting MSS is not to show better result in iperf. But rather to help you diagnose if it's a MSS issue. Once proved so, you can tune the system MSS accordingly over the VPN.

Big result!!!

I was having a hard time adjusting the MSS in my command line (I think it has to be done with an environment variable in windows)

BUT

I was looking at some other websites and came across a iperf syntax that I tried. The result is windows pc at home to windows pc at work (across the vpn)

iperf command line was: iperf -c 172.16.1.117 -u -b 1000m

Results are pretty telling: I'm not sure what these swithes do (-u says use UDP not TCP and I'm not understanding -b much at all) but I'm getting full line speed. Hopefully this can tell us something which in turn I can tune on my firewalls. If I lower the -b to 900 800 700 the speed starts to decrease.

------------------------------------------------------------
Client connecting to 172.16.1.117, UDP port 5001
Sending 1470 byte datagrams, IPG target: 11.76 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.55 port 58746 connected with 172.16.1.117 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.11 GBytes 953 Mbits/sec
[ 3] Sent 810345 datagrams

I just did the same test except -b 3000m from lan interface to lan interface (on each router), and got 1.5gbps throughput. What's going on here. How do I unleash this beast? (no graph) bad command line, it never sent any data across the network.

 

[MichaelCG]

I just did the same test except -b 3000m from lan interface to lan interface (on each router), and got 1.5gbps throughput. What's going on here. How do I unleash this beast? (no graph)

I thought you only had 1Gbps NICs....how did you manage to get to 1.5Gbps?

Also to note...I don't understand your continued testing with UDP based iPerf? You have already proven that with UDP, you can peg out the VPN. UDP traffic behaves very differently than TCP traffic and your "real-world" test cases are all TCP based. So any further testing should be using something with TCP so you can fully understand any impact that latency will have. At 4ms latency, you shouldn't expect a single TCP stream to go past 150Mbps....and this doesn't take into account any packet loss. With a single file transfer operating as a single stream/session, you will never achieve the same speeds over the WAN that you will get over the LAN due to latency. Now you can open multiple streams at the same time and eventually you will be able to saturate the connection.

If you want some quick and dirty numbers on how much impact latency has, check this calculator out:
https://www.silver-peak.com/calculator/throughput-calculator

Of course take the numbers from this specific one as an "idea" of where it will be...this is a vendor page that wants to sell you a WAN accelerator....but over the years I have found this one to be useful in giving a baseline of what to expect when dealing with some of our remote sites when they complain about transfer speeds.

 

[kvic]

iperf command line was: iperf -c 172.16.1.117 -u -b 1000m

udp is less useful. I would stay with tcp for your diagnosis.

EDIT:

iperf is same on all platform. you can use "-M" followed by an integer to specify MSS size in bytes e.g. 587, 1300, 1400. You need to specify on both ends.

 

[Roveer]

Guys,

First, thank you for lending your knowledge and support to my issue. I'm learning a lot as I go along here.

I'm not stuck on UDP for my testing I just wanted to report that I got a much different results using that command line on iperf. It was the first time I saw anything close to line speed over the VPN. From this point forward I'll ignore that result and concentrate on TCP.

Attempts to use the -M switch on iperf (whether it be on my Netgear ReadyNAS which is based on some form of unix, or my windows 7 machine produce a message that I am not fully understanding and produced no change in transfer speeds. I'm assuming that iperf is NOT setting my MSS to 1400 after this message.

I'm also looking at some very interesting tuning details at the following website. Specific to FreeBSD. https://calomel.org/freebsd_network_tuning.html

 

[Roveer]

I thought you only had 1Gbps NICs....how did you manage to get to 1.5Gbps?

I have to ignore that result. I tried it again and what ever value I set the -b to it would display that in gb's not even sending anything over the interface. -b 8000m would report back 8 gigabits. So now we are sticking with TCP.

To your point about single threaded transfers. From my work win7 pc I can send files up to the server (Mapped drive copy) on the LAN at 100+ MBps. When I do that across the VPN (mapped drive copy) it's 30-40 MBps. Is there any way to improve that transfer?

Also, I fired up 2 iperf's and 2 file copies from different machines and was able to saturate the VPN. Now I just want to make single machine copies as fast as I possibly can.

 

[Roveer]

After reading a few more iperf thread I tried using the -P option which will open multiple streams to send data. So from my computer at work I did iperf -c 192.168.0.101 -P 3 (101 being my NAS on the other side of the vpn), and it fully saturated the line, 890 mbps.

So what's that telling me? My windows file copies are single stream and 280+ mbps is the most I'm going to get out of one stream? (as one post suggests). Are their copy programs that will do multiple streams? I've been searching and haven't come across anything.

My eventual need would be to be able to move data from the computer at work to the NAS on the other side of the vpn at line speeds. iperf just showed I can do it from machine to NAS, now I just have to find a program that can make it happen.

Roveer

 

[MichaelCG]

CIFS is not a fast nor light protocol and is extremely sensitive to latency.

Only thing you can really do is move to a different protocol or method that will do multi-stream. Or buck up and buy WAN accelerators.

Sent from some device using Tapatalk

 

[Roveer]

So today I have been looking at catapult gridftp and some others. Some are very costly. Some are very complex. I feel like I'm missing something that's obvious, but maybe not.

So wouldn't it be great if I could find a copy program (let's call it MCOPY) that would open multiple streams to copy files. I think that would allow me to saturate my vpn. doesn't TFTP use UDP to copy? Any other ideas?

Roveer