Slow site-to-site VPNs across numerous routers, ISPs

[train_wreck]

Hello all.

So me and the guys I work with have been learning about IPsec VPNs, and in particular we have purchased a number of different VPN routers for testing. Models used include Cisco RV042G & RV320, Ubiquiti EdgeRouter Lite, TP-Link TL-R600VPN, and Netgear FVS336Gs. We have noticed one commonality among all of them; we can set up a site-to-site VPN between 2 of them plugged directly into the same switch, and using iperf or FTP transfers across the tunnel we can usually get close to the speed that each device is rated to. When we take the same routers with same configurations and put them over an ISP (we've tried Comcast, Mediacom and AT&T so far), we never get close to the speed the ISP provides. In other words, we can set up a VPN between 2 Cisco RV042Gs on 100/20 Comcast connections, and all non-VPN traffic can transfer at ~20mbps, but the VPN traffic never really gets above 8-10mbps. If we take those same routers and plug them directly together again, we again see full speed (in particular, we have measured IPsec traffic between the RV042Gs to be around ~70-80mbps.) We have noticed that if we force NAT-T on the IPsec tunnels, we see dramatically improved performance, but some routers don't allow that. We have tried gradually decreasing MTUs on the WAN interfaces down to 1000, and also have tried lower MSS clamping, but both of those only made performance worse.

So why is there such a performance hit when going over ISPs? It feels like they are almost de-prioritizing ESP packets or something.....

 

[yorgi]

Hello all.

So me and the guys I work with have been learning about IPsec VPNs, and in particular we have purchased a number of different VPN routers for testing. Models used include Cisco RV042G & RV320, Ubiquiti EdgeRouter Lite, TP-Link TL-R600VPN, and Netgear FVS336Gs. We have noticed one commonality among all of them; we can set up a site-to-site VPN between 2 of them plugged directly into the same switch, and using iperf or FTP transfers across the tunnel we can usually get close to the speed that each device is rated to. When we take the same routers with same configurations and put them over an ISP (we've tried Comcast, Mediacom and AT&T so far), we never get close to the speed the ISP provides. In other words, we can set up a VPN between 2 Cisco RV042Gs on 100/20 Comcast connections, and all non-VPN traffic can transfer at ~20mbps, but the VPN traffic never really gets above 8-10mbps. If we take those same routers and plug them directly together again, we again see full speed (in particular, we have measured IPsec traffic between the RV042Gs to be around ~70-80mbps.) We have noticed that if we force NAT-T on the IPsec tunnels, we see dramatically improved performance, but some routers don't allow that. We have tried gradually decreasing MTUs on the WAN interfaces down to 1000, and also have tried lower MSS clamping, but both of those only made performance worse.

So why is there such a performance hit when going over ISPs? It feels like they are almost de-prioritizing ESP packets or something.....

A lot of ISP's seem to be throttling VPN connections because they know most people are using them to download torrents.
There are also many limitations with routers and their CPU power. In this forum we have tested many routers from ASUS and other routers and have found that none of them can do more then 50-60mbps.
Encryption plays a big role for the Routers.
If you use AES-256 its the slowest in speeds.
AES-128 is the fastest and no compression will get equal speeds as your local ISP.
I have seen guys with 100mbps and they cant get better then 40mbps when on their routers VPN client.
So from my experience check encryption, cpu power, location of VPN server,

 

[yorgi]

one more note. Testing the system through the LAN and testing it with internet is not the same thing at all.
But I am pretty sure that there are VPN providers out there that throttle and ISP companies that throttle.
so it could be a number of reasons altogether for reduced speeds.

 

[sfx2000]

A lot of ISP's seem to be throttling VPN connections because they know most people are using them to download torrents.

Maybe - depends on the ISP - but OpenVPN, like TOR, is pretty easy to spot and manage at the operator level using deep packet inspection.

Most of this is a policy based decision - I don't have problems with Cisco Connect or Juniper SSL end-points, nor do I with L2TP/IPsec, they run fine, but I've heard of issues with OpenVPN on this particular provider

Note - I don't run OpenVPN, as I typically use ssh and l2tp myself, and on the work PC, OpenVPN is specifically blacklisted by policy controls on the machine (even with Local Admin, OVPN is blocked at multiple levels).

 

[System Error Message]

ISPs dont throttle VPN, VPN is used a lot for things like work. The reason why VPN over internet is slower because NAT requires processing. Another thing you may want to consider is the mikrotik CCR, even the mikrotik CCR1009 is way faster than any of those VPN routers when it comes to VPN performance. All those VPN routers use the same MIPS CPU. The TILE CPU in the CCR has IPSEC acceleration for AES encryption. So not only does the CCR have hardware acceleration like the VPN routers do but it has 9 cores highly clocked compared to those dual core MIPS they use. I even say shame on ubiquiti for not even considering the 8 or 16 core variants of that MIPS CPU they use, they really just arent interested in performance.

You also have to remember that VPN is layer 4 so using a VPN means packing a packet into a layer 3 packet which gets packed into a layer 2 packet. If your ISP uses a some method of connection such as PPPOE (which works similar to VPNs but only on layer 2) this means that you have a VPN packet packed into an IP packet packed into a PPP packet packed into a layer 2 packet so you could end up with fragmanted VPN packets instead. When testing over LAN the frames are usually bigger and there is less encapsulation going on.

I have both the ubiquiti edgerouter pro and CCR1036 and as i say many times as a router i hate using the edgerouter. The edgerouter's best task is as a mini linux server so i run squid and clamav on it and use it as a UTM instead combined with mikrotik's performance, routing and configuration capabilities. If squid can use both cores the edgerouter pro is capable of 160Mb/s of proxy (without cache).

When looking for a router capable of VPN those VPN routers are now outdated. In the past they offered VPN features other routers didnt have but now all routers have VPN capabilities and performance. Even using openVPN on a dual core ARM router it is still faster than what you're getting over internet. Its time you look at other solutions for your VPN needs such as x86, TILE, PPC as these CPUs suffer less performance hits when used in the field. You will see ubiquiti quoting impressive figures but in reality when used in a non consumer environment you will not get those figures. Mikrotik on the other hand provides more realistic figures. NAT performance is usually equivalent to the 5 rule routing benchmark but i would look at the lowest figure for 1500byte packets as a realistic speed if you are expecting heavy configs.

 

[sfx2000]

ISPs dont throttle VPN, VPN is used a lot for things like work. The reason why VPN over internet is slower because NAT requires processing.

Actually - many have the ability to detect and quantify many types of traffic - Sandvine is a common vendor for DPI at the carrier level, and they're quite good at what they do.

While I have not had personal issues with VPN - I use Cisco Connect these days, and Juniper's Pulse SSLVPN, both of which are enterprise focused packages on my work machine (I'm a remote worker these days) - I use L2TP for my own use on personal machines when OpenSSH is not a good choice - again no issues noted..

OpenVPN on the other hand - it's not used much in the enterprise environment as not CIO/CTO/Director of IT is going to use it when they can dial up their rep and get an end-to-end solution that totally integrates with their IT back-end. It's a support issue, and a compliance issue for many publicly held companies.

That being said, most of the OpenVPN traffic peaks right around 10PM local time - that ain't work my friends... and the operator has the capability, if they so desired, to throttle it back to preserve traffic for other purposes...

Another thing you may want to consider is the mikrotik CCR, even the mikrotik CCR1009 is way faster than any of those VPN routers when it comes to VPN performance. All those VPN routers use the same MIPS CPU

Another consideration is not running OpenVPN on a router - period...

Run it on a box behind the router, and you'll - and it's very possible to work with iptables to do the same thing with an external OpenVPN host as it is for one hosted by the router itself.

a) Be safer
b) faster performance
c) better router performance as OpenVPN isn't sucking up 100 percent of one of your two CPU cores...

OpenVPN on a Broadcom based router - it's like trying to run on a cpu that isn't even as capable as the mid-range smartphone these days.

Also consumer grade routers have no concept of access/privilege separation, unlike other platforms - so everything runs as root - pretty scary if you ask me...

 

[System Error Message]

Actually - many have the ability to detect and quantify many types of traffic - Sandvine is a common vendor for DPI at the carrier level, and they're quite good at what they do.

While I have not had personal issues with VPN - I use Cisco Connect these days, and Juniper's Pulse SSLVPN, both of which are enterprise focused packages on my work machine (I'm a remote worker these days) - I use L2TP for my own use on personal machines when OpenSSH is not a good choice - again no issues noted..

OpenVPN on the other hand - it's not used much in the enterprise environment as not CIO/CTO/Director of IT is going to use it when they can dial up their rep and get an end-to-end solution that totally integrates with their IT back-end. It's a support issue, and a compliance issue for many publicly held companies.

That being said, most of the OpenVPN traffic peaks right around 10PM local time - that ain't work my friends... and the operator has the capability, if they so desired, to throttle it back to preserve traffic for other purposes...



Another consideration is not running OpenVPN on a router - period...

Run it on a box behind the router, and you'll - and it's very possible to work with iptables to do the same thing with an external OpenVPN host as it is for one hosted by the router itself.

a) Be safer
b) faster performance
c) better router performance as OpenVPN isn't sucking up 100 percent of one of your two CPU cores...

OpenVPN on a Broadcom based router - it's like trying to run on a cpu that isn't even as capable as the mid-range smartphone these days.

Also consumer grade routers have no concept of access/privilege separation, unlike other platforms - so everything runs as root - pretty scary if you ask me...

Im just mentioning why VPN routers are outdated as all other routers have surpassed them already. While broadcom routers arent ideal to run openVPN they do surpass VPN routers already.

My suggestion would be similar to yours in running it on x86 or even a CCR mainly because of performance, configurability and features not to mention those VPN routers arent worth their price (even if cheap) anymore. Ubiquiti performance figures are very misleading as they dont have any sort of real world test in a sense of setting up how the routers would be used in the real world. Ubiquiti could learn from Thiggins about router testing and figures. Using those dual core MIPS at low frequencies are an outdated practice and im surprised companies still build them instead of considering the better CPUs available from the same supplier such as more MIPS cores and higher frequencies.

 

[sfx2000]

I think we're pretty much on the same page - on the consumer grade routers, it's a checkbox on the spec sheet, much like USB drive sharing...

Yes, it'll do it, but not as well as a more specialized appliance could do.

I always worry about the security aspects of putting all of these services on a single device - yes, it's convenient, but at the same time, these companies are more focused at pushing boxes out the end of the factory and security takes a back set (we'll fix it after it ships...)

To really have good security and performance with VPN - one really needs to take a step up from the consumer routers into a platform like the Microtik or Edgerouter (or similar).

I run pfSense on X86, and even then - I don't run VPN of any kind on it - my SSH and LT2P/IPSec host stays behind the pfSense box. And that's for security purposes - there's no binding of accounts from pfSense to the Linux box running my external access servers. Again, security..

My NAS is specifically fire walled against the VPN IP range - again, security... I can ssh into it, but again, separate account from anything else, and there I can only ssh into it from a specific IP inside my LAN (even though the LAN is the trusted side, I'm still a bit more paranoid that the average person).