Looking for new routers for FIOS Gig with emphasis on VPN Throughput

[Roveer]

I just upgraded both sides of my work-to-home network link to FIOS Gigabit. Previously had been 150/150 at home and 75/75 at work but now both will big FIOS Gig.

Was using Checkpoint 680 appliances with IPSEC/VPN. Did the job well. iperf3 testing across the link showed full 75 mbit speeds which was my slowest link.

I knew the checkpoints wouldn't keep up with the new speeds but I was surprised to see that they won't even keep up with WAN speeds either just as a firewall. Speed tests showing 450/450 and it's rated at 1.5gb firewall speed. Not using any of the "blades", just firewall.

Tested in my lab (not vpn, just straight firewall connections) and was only getting 500/500 with utilization pegged so these boxes just won't keep up. Again disapointing considering the specs say 1.5Gb/s. For kicks I replaced my test checkpoint box with a Verizon G1100 and did same test and was seeing 800-900 which is about all I figured I was going to get out of a gig connection.

So I've been researching new routers (don't need wifi, handled elsewhere) but they must have ipsec/vpn capability so I can connect both locations. My emphasis needs to be on ipsec/vpn throughput. I really want to saturate the link as much as possible and I understand that ipsec/vpn has all the added overhead of encryption.

One box that I'm going to look at is the ubiquiti ER-4 (or ER-4P as I'm seeing it called in the production release) which is due out this month. Say's it's 4 times faster than the ERPro-8 which is their previous model. I tried to get some VPN tests from the beta testers but nobody had done that testing.

Looking for other recommendations for routers that can handle gigabit internet access and will have fast processors for ipsec/vpn encryption.

I Don't want to have to build 2 pfSense Xeon boxes that will cost close to $1,000 bucks each, but it doesn't seem like any appliance is powerful enough to handle all the processing.

Also seems like having a processor with AES-NI capability is a must which I don't think any of the "appliance" routers have.

Ideas?

 

[MichaelCG]

The 680 is rated at 100Mbps with IPS...which is generally the rating you should look at for close to real-world performance ratings for Checkpoint appliances. Their firewall throughput ratings have always been perfect use case models and mostly meaningless.

My vote will be pfSense...and doubtful you have to build Xeon boxes...should be able to get by with some of the newer consumer CPUs with AES-NI get decent speeds.

I don't know what other "appliance" out there exists that can easily handle 500Mbps+ routing and/or VPN traffic without getting into the $1000+ price points.

 

[Roveer]

The 680 is rated at 100Mbps with IPS...which is generally the rating you should look at for close to real-world performance ratings for Checkpoint appliances. Their firewall throughput ratings have always been perfect use case models and mostly meaningless.

My vote will be pfSense...and doubtful you have to build Xeon boxes...should be able to get by with some of the newer consumer CPUs with AES-NI get decent speeds.

I don't know what other "appliance" out there exists that can easily handle 500Mbps+ routing and/or VPN traffic without getting into the $1000+ price points.

So I'm going to do some more testing. I've got a checkpoint 4600 that I've put pfSense on and I'm going to VPN it to my computer (i7) which will run pfsense as well. I'll be able to get some "lab" testing out of that as I assume the CP box will be the slower of the 2 in that test. If I can get the pfSense VPN'd with the CP running GUIA I'll try that but I was having a hard time getting it to connect.

The ubiquiti ER-4p looks promising at 150 bucks a side but I think it will fall short on the vpn performance as well.

I'd be interested in hearing what boxes would handle that much throughput on ipsec/vpn even if the cost is higher than I was expecting. I'd like to know what's out there and now I'm very leery of published specs.

 

[MichaelCG]

Keep in mind that my Checkpoint performance ratings views are based on the bigger CP boxes...I have never used the 6xx or 7xx series myself. The smallest we generally have used are the 4800 for sites that need just over 1Gpbs of routed traffic. Otherwise we are in the 12xxx/14xxx chassis mostly.

Which is slower, the 4600 or the i7? Well...I think that will really come down to what CPU is actually in the 4600. I haven't poked around in those to know for sure which CPU is in there. A google search is showing it may be a E6500....so yeah, it will probably be slower than the i7 unless there is some type of crypto card on-board since the CPU doesn't support AES-NI.

 

[Roveer]

Keep in mind that my Checkpoint performance ratings views are based on the bigger CP boxes...I have never used the 6xx or 7xx series myself. The smallest we generally have used are the 4800 for sites that need just over 1Gpbs of routed traffic. Otherwise we are in the 12xxx/14xxx chassis mostly.

Which is slower, the 4600 or the i7? Well...I think that will really come down to what CPU is actually in the 4600. I haven't poked around in those to know for sure which CPU is in there. A google search is showing it may be a E6500....so yeah, it will probably be slower than the i7 unless there is some type of crypto card on-board since the CPU doesn't support AES-NI.

The 4600 is definitely slower even know the spec sheets show wildly big numbers. I think it's a Pentium dual core cpu without AES-NI. I'm assuming in order to handle anything along the lines of what I'm looking for I'd need AES-NI cpu at very least. I think the first CP box that has a AES-NI cpu was the 12000 series.

 

[HeMaN]

Something like a Qotom box with an i5 or i7 together with pfSense?
(sorry, no experience with this setup. Using a j1900 box myself)

 

[kvic]

One box that I'm going to look at is the ubiquiti ER-4 (or ER-4P as I'm seeing it called in the production release) which is due out this month. Say's it's 4 times faster than the ERPro-8 which is their previous model. I tried to get some VPN tests from the beta testers but nobody had done that testing.

My ER-X can do 150-250mbit ipsec/vpn with hw accelerator enabled. Mikrotik's hEX r3, the same SoC, can do close to 500mbit in their firmware. That's near the hw accelerator's spec.

Put it into perspective, I won't be surprised ER-4's new Cavium SoC can do 1Gbit ipsec. The software makes a difference though.

I haven't upgraded my ER-X since v1.9.1.1 (and it's been up >120 days and counting). Versions after that include more and more codes by the new team since Ancheng and co left the firm.

Hopefully Edgerouters can restore their reputation with v1.10 and v2.0 releases.

edit:

eth0 is my WAN. It has processed over 3.4TB of packets including IPsec VPN in the 120 days. As an aside, the MediaTek SoC inside ER-X is rock solid. More than an excellent steal for its price.

 

[Roveer]

Put it into perspective, I won't be surprised ER-4's new Cavium SoC can do 1Gbit ipsec. The software makes a difference though.

Thanks for the info. I'm leery that any "appliance" device, passively cooled will perform well on a ipsec/vpn gig connection. It does not appear that the Cavium has any special AES acceleration capabilities for encryption processing. I'm a little tainted by the wildly inaccurate figures that CP publishes. Their specs for the 680's that I'm using show 1.5 Gbps on firewall I'm getting 420 max. No IPS, just firewall. It shows 230 Mbps IPSEC, I'm getting half that. So basically half the rated spec for real world and the processor is pegged when transferring files and often doesn't respond to other requests.

with the ER-4 being 150 bucks if it were to perform as good as we wish it would, it would be an astounding level of performance for the price. I've been looking at this stuff for a while and trying to get something to 1. handle gig connection and 2. give nearly line speed on ipsec/vpn is a very tall order. To see something in that price range do that well would be as Ubiquiti puts it... Disruptive. I can only hope. Shouldn't be long. Mid-November availability and I'll most likely be ordering 2 the minute they are available. I have however read that they are having some issues with the software and from your post, seems you have had your issues as well. Let's see how this plays out. I always like to disrupt things!

Roveer

 

[Roveer]

This is a copy of a post I made over on CPUG.COM about my Checkpoint 680's. Since I see there is a lot of knowledgeable people here I figured I'd share in case anyone had and comments. Thanks.

This week I upgraded both sides (work & home) to FIOS gigabit internet. It's supposed to top out around 800 Mbps or so.

My 680's don't use any of the intrusion or AV blades, just Firewall and IPSEC/VPN.

First observation: Speedtest.net and FIOS speed test to internet top out around 450 Mbps in both directions at both locations. Processor pegged and router gui sometimes stops responding til test finish. Specs for 680 say Firewall (Gbps): 1.5, so I'm getting 1/3rd of spec'd speed.

Second Observation: IPSEC/VPN is giving me 112 Mbps between devices. Specs say 220 Mbps so I'm getting 1/2 of spec'd speed.

I've tried setting IPSEC/VPN encryption settings about as low as they can go (AES 128, DH 768) with no real change in speed results.

Third observation: Iperf3 results 2 machines either side of link: 122 Mbps. Was expecting at least 200.

I wasn't expecting a miracle, but this is a bit worse than I expected. Is there anything I should look at or tune? Looking now for devices that would handle the new internet speeds, with the emphasis on IPSEC/VPN throughput. Any suggestions (for a small business on a limited budget).

Thanks,

Roveer

 

[System Error Message]

mikrotik CCR can do gigabits worth of VPN.
desktop/server based x86 CPUs also can do gigabits worth of VPN. The better the IPCs and hardware acceleration, the better the throughput. For example all intel iseries and faster (focusing on the extreme/ server side due to more memory bandwidth and cores). On AMD's side the phenom ii, zen but not bulldozer. I have managed SFTP at 1G using the AMD phenom ii 4 cores at 3.2Ghz using around 80% CPU with RAID 5 software so no doubt it'd do 1Gb/s of encrypted VPN.

You must remember that with VPN you can get a fast CPU but the limits are both the internet and your client. Mobile devices arent fast enough not to mention that many laptops fail to even play my non standard video file.

 

[kvic]

Mobile devices arent fast enough not to mention that many laptops fail to even play my non standard video file.

A three-year old iPhone 6 can easily do 100/100mbit I believe: Time to Retire OpenVPN.

 

[System Error Message]

A three-year old iPhone 6 can easily do 100/100mbit I believe: Time to Retire OpenVPN.

was talking about gigabit speeds VPN. 100Mb/s is not a problem for mobile chips.

 

[Roveer]

Well, I did some extensive testing today.

Putting a i7 and an i5 on pfSense (connected via gig switch) and turning on AES-NI I was basically able to get line speed ipsec/vpn between the two firewalls on iperf testing.

Firewall to firewall (lan to lan across ipsec/vpn) on iperf 893 Mbps
Laptop to laptop (not high performers), (lan to lan across ipsec/vpn) I got ~850 Mbps.

So throwing hardware at the problem does seem to provide results.

The checkpoint 4600 device running pfSense (and also reconfigured to run CP software) gave only 350 Mbps but that's a Pentium dual core non AES-NI device. This is disappointing considering it's rated at 1.5 gbps vpn speed. I guess they have their own world which they cook up those numbers.

Hardware encryption is key here.

Without AES-NI encryption turned on, gateway to gateway iperf was only 510 Mbps and the laptops were only getting 390 Mbps.

Windows 7 file copies were between 70-90 MBps (Megabytes/second) which is probably as good as I'm going to get, and that was on these old crappy laptops.

This should all pretty well saturate the FIOS gigabit service when I'm moving data across the VPN which is what I was looking for.

The only other consideration is ubiquiti ER-4 devices which are promising lots of great things, but they are "appliances" so unless they've got some really great magic cooked up I don't see how they can keep up with i7 processing power. At 150 bucks each, I might have to just grab 2 just to see how they perform. Otherwise I'll be building me some pfSense boxes.

Roveer

 

[MichaelCG]

Also be sure to not forget the impact that latency will have on your maximum throughput on a single data stream.

I have no idea where Checkpoint gets their VPN numbers from. We have never hit the limits of their boxes for IPSEC...but then again, we don't push that much IPSEC traffic overall...at least not in raw bandwidth we don't. Pretty sure our IPSEC traffic is under 100Mbps while our total US Internet consumption is probably 5-6Gbps.

Very interested to see results on the ER-4 as well though. My pfSense box is old (no AES-NI..core2) and just not sure I want to spend the money to rebuild it yet.

 

[kvic]

@Roveer I hope you haven't abandoned the idea of buying ER-4. Would be interesting to see a first-hand test here. lol

 

[Roveer]

@Roveer I hope you haven't abandoned the idea of buying ER-4. Would be interesting to see a first-hand test here. lol

I'm pretty sure I'll be buying 2 ER-4's as soon as they hit the market. I'll be testing mostly for overall firewall throughput (now that i have gig FIOS), and ipsec/vpn throughput. Those are my 2 big wants. For a 400 dollar investment if I can get anything over 600 mbps on ipsec vpn i'll probably put them into production. my pfSense alternatives will be much more expensive so I'll be looking at the ubiquiti gear.

Roveer

 

[Roveer]

I'm pretty sure I'll be buying 2 ER-4's as soon as they hit the market. I'll be testing mostly for overall firewall throughput (now that i have gig FIOS), and ipsec/vpn throughput. Those are my 2 big wants. For a 400 dollar investment if I can get anything over 600 mbps on ipsec vpn i'll probably put them into production. my pfSense alternatives will be much more expensive so I'll be looking at the ubiquiti gear.

Roveer

Well, I've changed my mind. Went the pfSense route and have already got my two sites up and running on $180.00 small form factor i7-3700. I'm waiting for the 2nd box to come in, but I'm already getting iperf results firewall to firewall across the internet of 900+ mbps. As soon as I get the 2nd i7 in place (running a dual core Pentium on one side right now), I'll be able to complete my ipsec vpn throughput tests and will report back.

A little disappointed that the ER-4's haven't shown up in the marketplace yet, but haven't closed the door on them. I still like the ubiquiti products, but I'm pretty sure my pfSense i7 solutions will blow the doors off the ER-4.

Roveer

 

[Xentrk]

Well, I've changed my mind. Went the pfSense route and have already got my two sites up and running on $180.00 small form factor i7-3700. I'm waiting for the 2nd box to come in, but I'm already getting iperf results firewall to firewall across the internet of 900+ mbps. As soon as I get the 2nd i7 in place (running a dual core Pentium on one side right now), I'll be able to complete my ipsec vpn throughput tests and will report back.

A little disappointed that the ER-4's haven't shown up in the marketplace yet, but haven't closed the door on them. I still like the ubiquiti products, but I'm pretty sure my pfSense i7 solutions will blow the doors off the ER-4.

Roveer

Can you please provide more details on the Hardware specs or HTML link? Type and size of HD, memory, external ports, etc..Thanks! I am in the market for an AES-NI capable pfSense box.

 

[Roveer]

Can you please provide more details on the Hardware specs or HTML link? Type and size of HD, memory, external ports, etc..Thanks! I am in the market for an AES-NI capable pfSense box.

I went with used Dell Optiplex 7010 Sff (small form factor). A fairly small box. I7-3770 4gb ram, and I put a laptop hdd in for now. Staying away from ssd on pfsense until I explore the whole "embeded" thing which wasn't obvious to me on the latest version.

I paired that with a 4 port intel server nic. Here are links to the pieces I used:

https://www.ebay.com/itm/Dell-Optiplex-7010-3-4GHz-QC-i7-4GB-250GB-DVDRW-Win-10-Pro-64-SFF-Computer-B/202109814951?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

https://www.ebay.com/itm/NEW-Intel-OEM-I350T4V2BLK-Ethernet-Server-Adapter-Gigabi-4-Port-RJ45-PCI-Express/112400469306?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

I went with this hardware for the following reasons: 1. It was cheap. 250.00 per side was about as cheap as I could go with i7 hardware. 2. I had tested i7-3770 to i5 in my lab and was getting 800+mbps across a vpn while connected to a gigabit switch with <10% utilization on the i7 during extended iperf runs. I'm hoping once I get my other i7 box (should be next week sometime), and get AES-NI turned on a configured (you have to consider your vpn encryption settings to optimize the AES-NI), I'm hoping to see similar results. I've already iperf tested my Dell i7 at home to a Pentium dual core appliance at work and have achieved 900+ mbps router to router (not vpn) across the internet (both sites gb fios). That was a good sign.

Next level up while trying to keep the hardware small would be Dell 7040's which I believe are i7-6700's which on the cpu benchmark website I use: http://cpu.userbenchmark.com/Compare/Intel-Core-i7-3770-vs-Intel-Core-i7-6700/1979vs3515 show 12% gain. Of course I'd love to build a nice i7-7700k 1U appliance but that would end up costing 800+ dollars per side and could have some cooling issues in such a low profile, but it would be a really nice box. Here's what I'm talking about. This is bare bones, but can accept 7th gen cpu's. https://www.amazon.com/dp/B01KP8GOXI/?tag=snbforums-20

Let me know if you have any other questions. I learned a lot during my research for this project. I had unique needs (max vpn throughput across gb connections) and even got in a few arguments about this being an enterprise space and not possible for under 10k per side. I'm out to prove I can do it for 250 bucks a side and so far it's looking pretty good. I will share my throughput numbers once I get my other Dell 7010 and get it up and running.

 

[Xentrk]

I went with used Dell Optiplex 7010 Sff (small form factor). A fairly small box. I7-3770 4gb ram, and I put a laptop hdd in for now. Staying away from ssd on pfsense until I explore the whole "embeded" thing which wasn't obvious to me on the latest version.

I paired that with a 4 port intel server nic. Here are links to the pieces I used:

https://www.ebay.com/itm/Dell-Optiplex-7010-3-4GHz-QC-i7-4GB-250GB-DVDRW-Win-10-Pro-64-SFF-Computer-B/202109814951?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

https://www.ebay.com/itm/NEW-Intel-OEM-I350T4V2BLK-Ethernet-Server-Adapter-Gigabi-4-Port-RJ45-PCI-Express/112400469306?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

I went with this hardware for the following reasons: 1. It was cheap. 250.00 per side was about as cheap as I could go with i7 hardware. 2. I had tested i7-3770 to i5 in my lab and was getting 800+mbps across a vpn while connected to a gigabit switch with <10% utilization on the i7 during extended iperf runs. I'm hoping once I get my other i7 box (should be next week sometime), and get AES-NI turned on a configured (you have to consider your vpn encryption settings to optimize the AES-NI), I'm hoping to see similar results. I've already iperf tested my Dell i7 at home to a Pentium dual core appliance at work and have achieved 900+ mbps router to router (not vpn) across the internet (both sites gb fios). That was a good sign.

Next level up while trying to keep the hardware small would be Dell 7040's which I believe are i7-6700's which on the cpu benchmark website I use: http://cpu.userbenchmark.com/Compare/Intel-Core-i7-3770-vs-Intel-Core-i7-6700/1979vs3515 show 12% gain. Of course I'd love to build a nice i7-7700k 1U appliance but that would end up costing 800+ dollars per side and could have some cooling issues in such a low profile, but it would be a really nice box. Here's what I'm talking about. This is bare bones, but can accept 7th gen cpu's. https://www.amazon.com/dp/B01KP8GOXI/?tag=snbforums-20

Let me know if you have any other questions. I learned a lot during my research for this project. I had unique needs (max vpn throughput across gb connections) and even got in a few arguments about this being an enterprise space and not possible for under 10k per side. I'm out to prove I can do it for 250 bucks a side and so far it's looking pretty good. I will share my throughput numbers once I get my other Dell 7010 and get it up and running.

Thank you @Roveer !!

I have been eyeing the qotom boxes. But some users on the pfsense qotom thread
https://forum.pfsense.org/index.php?topic=132528.0
have concerns with heat and a backward port assignment:

Mac   Phys port   BSD name
-----------------------------
xx6F    1          igb0

xx70    4          igb1

xx71    2          igb2

xx72    3          igb3

Your build appears to be a viable alternative with a lower cost. My pfSense appliance was purchased in BK thru a pfSense reseller. It has a Quad core Atom D525 CPU installed. It does not support AES-NI. I did not know enough at the time for this to cause me concern. I will want to replace it when pfSense 2.5 is out. Please keep me posted on your progress.