1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Converted a PC to a pfSense Router to test OpenVPN performance

Discussion in 'Routers' started by Xentrk, Feb 7, 2018.

  1. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,521
    Location:
    The Land of Smiles
    VPN in a must have for my use case. But as we all know, the CPUs inside most consumer routers struggle in this regard.

    I have been keeping eyes open for a PC to become available that had a CPU with AES-NI support so I could flash it with pfSense to see how OpenVPN performance compared with the AC88U. One became available yesterday. So I ran out to the store and purchased an extra Network Adapter. After installing the NIC, I installed pfSense using a USB. I used the config backup from my current pfSense appliance so setup was a non-event except I had to reinstall the pfBlockerNG package.

    The specs of the CPU are:

    Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    It has been up and running under 24 hours. I am in SE Asia and connected to a VPN Server on the west coast of USA.

    I am surprised by the increase in performance, especially when using an ethernet (ETH) connection. The other surprise is the difference between ethernet and wireless performance. I use a D-Link 880L flashed with DD-WRT as the Access Point.

    Numbers are Mbps

    upload_2018-2-8_10-47-25.png

    upload_2018-2-8_10-47-49.png

    I think I just found a new router for my home network!
     
    Last edited: Feb 7, 2018
  2. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,521
    Location:
    The Land of Smiles
    This contains more information on the VPN performance of the AC86U and the pfSense router.

    I also compared the WAN Ethernet and WiFI performance of the pfSense PC paired with a D-Link 880L as the AP and the AC88U.

    openssl speed of AC-86U per @RMerlin post

    Code:
    [email protected]:/tmp/home/root# openssl speed -evp aes-128-cbc
    Doing aes-128-cbc for 3s on 16 size blocks: 34942605 aes-128-cbc's in 2.98s
    Doing aes-128-cbc for 3s on 64 size blocks: 24912812 aes-128-cbc's in 3.00s
    Doing aes-128-cbc for 3s on 256 size blocks: 11306808 aes-128-cbc's in 3.00s
    Doing aes-128-cbc for 3s on 1024 size blocks: 3619044 aes-128-cbc's in 2.99s
    Doing aes-128-cbc for 3s on 8192 size blocks: 490938 aes-128-cbc's in 2.97s
    OpenSSL 1.0.2j  26 Sep 2016
    built on: reproducible build, date unspecified
    options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
    compiler: /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.22-binutils-2.25/usr/bin/arm-buildroot-linux-gnueabi-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_NO_HEARTBEATS -DL_ENDIAN -Os -march=armv8-a -fomit-frame-pointer -mabi=aapcs-linux -marm -ffixed-r8 -msoft-float -D__ARM_ARCH_8__ -ffunction-sections -fdata-sections -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k
    pfSense metrics

    Code:
    openssl speed -evp aes-128-cbc
    Doing aes-128-cbc for 3s on 16 size blocks: 115298666 aes-128-cbc's in 3.09s
    Doing aes-128-cbc for 3s on 64 size blocks: 30287051 aes-128-cbc's in 3.05s
    Doing aes-128-cbc for 3s on 256 size blocks: 7651089 aes-128-cbc's in 3.02s
    Doing aes-128-cbc for 3s on 1024 size blocks: 1901163 aes-128-cbc's in 3.00s
    Doing aes-128-cbc for 3s on 8192 size blocks: 243243 aes-128-cbc's in 3.07s
    OpenSSL 1.0.2m-freebsd  2 Nov 2017
    built on: date not available
    options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: clang
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k
    
    Side by side comparison. Second row is pfSense router. Third row is AC-86U
    Code:
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k
    aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k
    Ethernet speeds are nearly the same across the two routers. From this data, I need to look at tuning the Wifi in the D-Link AP. I plan to also test using the AC88U as an AP to see if that improves WiFi performance when it is paired with the pfSense router.

    upload_2018-2-8_18-23-49.png
     
    Last edited: Feb 8, 2018
  3. MichaelCG

    MichaelCG Very Senior Member

    Joined:
    Jan 4, 2017
    Messages:
    581
    Location:
    Central US
    So looking at the openSSL numbers only...the AC-86U should be able to easily out perform the pfSense i5 box in VPN performance for 256B or larger block sizes? I am purposely ignoring network bandwidth at the moment and just looking at raw VPN performance.

    Holy crap...I need a new pfSense box...the numbers from both of the systems you posted just make my current system look pretty bad.

    Intel Core2 DUO E4600 @ 2.4GHz
    Code:
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128-cbc      90379.12k   103593.85k   108091.19k   108606.46k   109062.80k
    
     
  4. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    6,096
    Location:
    United States
    Thought I'd pull all the previous data into a single table and add a few entries for comparison
    Code:
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    
    Merlin (AC86U @ 1.8GHz)
    aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k
    
    pfsense (Intel i5-3450 CPU @ 3.10GHz)
    aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k
    
    opnsense (Intel N3700 @ 1.6GHz)
    aes-128-cbc     200326.87k   322785.00k   384672.70k   405475.07k   411444.68k
    
    pfsense (Intel Core2 DUO E4600 @ 2.4GHz)
    aes-128-cbc      90379.12k   103593.85k   108091.19k   108606.46k   109062.80k
    
    LTS Fork (AC68P @ 1200MHz - overclock)
    aes-128-cbc      36018.40k    41274.95k    43366.49k    43291.23k    44250.54k
     
    Last edited: Feb 8, 2018
    Clark Griswald, tyspeed42 and Xentrk like this.
  5. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,521
    Location:
    The Land of Smiles
    Thanks for posting your metrics. I came to the same conclusion about the pfSense i5 vs the AC86U as you. The PC is a custom build and is approximately six years old. I was able to obtain another box that used to be a Windows Server 2008. I may flash with pfSense to test with. It has an Intel i7 CPU. It stopped booting up the other day so I have to fix that issue first.
     
  6. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    pfSense 2.4.2_1 - Netgate SG-2440 (Intel Rangeley C2358 @ 1.74GHz)

    Code:
    [2.4.2-RELEASE][[email protected]]/root: openssl speed -evp aes-128-cbc
    Doing aes-128-cbc for 3s on 16 size blocks: 665810 aes-128-cbc's in 0.38s
    Doing aes-128-cbc for 3s on 64 size blocks: 640868 aes-128-cbc's in 0.23s
    Doing aes-128-cbc for 3s on 256 size blocks: 527398 aes-128-cbc's in 0.19s
    Doing aes-128-cbc for 3s on 1024 size blocks: 325775 aes-128-cbc's in 0.20s
    Doing aes-128-cbc for 3s on 8192 size blocks: 69190 aes-128-cbc's in 0.05s
    OpenSSL 1.0.2m-freebsd  2 Nov 2017
    built on: date not available
    options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: clang
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128-cbc      34722.97k   155002.76k   702903.75k  3514848.60k 10336263.02k
    
    Putting things thru the Turbo Entabulator... means these numbers are basically useless for what is really OVPN performance in general - one has to look beyond things..

    Code:
    Merlin (AC86U @ 1.8GHz)
    aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k
    
    pfSense SG-2440 (Intel C2358 @ 1.74GHz)
    aes-128-cbc      34722.97k   155002.76k   702903.75k  3514848.60k 10336263.02k
    
     
    Last edited: Feb 8, 2018
  7. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    So these days with OpenSSL speed testing

    Add the -elapsed flag... provides for much more accurate numbers. It removes some of the silliness I pointed out above - in any event, it's not just OpenSSL that make a difference with OpenVPN, it's the hardware and the platform.

    intel C2358 @ 1.7GHz w/pfSense 2.4.2

    Code:
    aes-128-cbc       3528.67k    13660.13k    45295.19k   108952.23k   188904.79k
    aes-128-gcm      83243.74k   163860.12k   231950.25k   258325.55k   264888.32k
    
    intel N3700 @ 1.6GHz... Intel NUC on Ubuntu 16.04LTS

    Code:
    aes-128-cbc     212792.71k   321387.71k   385178.03k   403900.42k   411178.33k
    aes-128-gcm     125464.67k   235635.99k   322191.87k   358735.53k   368831.15k
    
    This one is fun - Asus Tinkerboard - Rockchip RK3288 (Quad [email protected]) - same command line, but Tinkerboard has OpenSSL 1.1.0f

    Code:
    aes-128-cbc      63643.25k    81057.79k    88047.70k    89871.36k    90565.29k    90587.14k
    aes-128-gcm      47259.19k    58618.75k    66093.14k    70863.87k    71832.92k    71636.31k
    
    And MacOS 10.13.4 on Core [email protected] - IvyBridge - MacMini 2012 - running OpenSSL 1.02.n there via HomeBrew...

    Code:
    aes-128-cbc     586287.55k   624049.90k   635853.23k   637082.97k   637580.63k
    aes-128-gcm     294048.12k   749649.66k  1104695.21k  1216231.34k  1258162.86k
    
     
    Last edited: Feb 8, 2018
    Xentrk likes this.
  8. st3v3n

    st3v3n Very Senior Member

    Joined:
    Feb 24, 2016
    Messages:
    509
    Location:
    Central US
    sfx2000, Dazzle vs Baffle: my guitarist always said that prior to going onstage in the late 60's. He was brilliant and dazzling, but also threw in a pinch of the BS**. I'm always dazzled by the senior member's brilliance, so excuse my polite babble lest I embarrass myself:)

    Xentrx, After reading your post re the i5-3450 and data from Michael and John, and sfx, I wondered if your box/project relates to mine, and if the performance would be similar for OPNSense, if anyone wants to share. I built the PfSense box in late Nov, preliminary testing with a Xeon 3450/ 4-core w/8 GB ram, the MB NIC and a 4-pt NIC for LAN. The Xeon 3450 installed and ran OK but obviously wasn't going to provide VPN AES going forward, so I dropped an i5-660 in and retested for a couple of days until work interrupted. The i5-660 was surprisingly faster than the 3450.

    There aren't many valid comparisons out there for PfSense vs OPNSense and I didn't find much in the way of technical data or videos relating to OPNSense. I discounted flames about who did/didn't do what before, much less to whom. I had read enough about OPNSense to decide to install on our box before finally committing to PfSense. In late January I installed OPNSense and immediately was impressed with the speed, interface, menus, etc https://opnsense.org/users/get-started/#hardware-requirements

    Seldom has a firewall/router felt 'right' so quickly. The dev seems clear about the fork, the way forward and how they're going about it: https://opnsense.org/about/about-opnsense/#so-why-did-we-fork-pfsense It hasn't existed as long as PfSense, but regardless of the spat and parting of ways, if OPNSense relates in any way to your project or if you gents care to comment on it if you know, please do. Didn't mean to interrupt, thanks.
     
  9. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    If you can't dazzle them with brilliance - baffle them with B* - they'll never know the difference, lol...

    Seriously though - there's a lot of fun numbers flung about - but if one is serious about VPN, one does know what HW to look at, and any consumer router might flout some numbers, proof remains...

    And OpenVPN is a poor choice, there are better...

    There are reasons for this - none of them technical - most folks just pick one or the other - they're more alike than different. But that's all accepted in the BSD community...
     
    Last edited: Feb 8, 2018
  10. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    6,096
    Location:
    United States
    I thought I remembered another. better test for predicting OpenVPN throughput.....and it was from @sfx2000 :)
    https://www.snbforums.com/threads/openvpn-estimate-performance-via-openvpn.33416/

    Using that technique.....
    Code:
    (AC68P @ 1200MHz - overclock)
    aes-128-cbc  3200/43.22 = 74.04 Mbps
    aes 256-cbc  3200/46.94 = 68.17 Mbps
    
    opnsense (Intel N3700 @ 1.6GHz)
    aes-128-cbc  3200/22.52 = 142.10 Mbps
    aes 256-cbc  3200/23.06 = 138.79 Mbps
    
    Would like to see an AC86 number on this test....
     
  11. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    Rockchip 3288 @ 1.8GHz (cortex-a17) - Asus Tinkerboard

    aes-256-cbc 3200/20.54 = 155,79
    aes-128-cbc 3200/18.94 = 168.95
    aes-128-gcm 3200/18.73 = 170.85

    Good reason perhaps to consider the Tinkerboard...
     
  12. st3v3n

    st3v3n Very Senior Member

    Joined:
    Feb 24, 2016
    Messages:
    509
    Location:
    Central US
    Indeed. For the most part that used to be the case as far as Dazzle/Baffle, or fool most of the fools much of the time, but in this era, now so many are in the know such as you gents, that's the part that matters. It comes down to what one prefers whether it's Fender vs Gibson, AMD v Intel, Chevy vs Ford, or a Tesla Roadster shot into orbit rather than a Ferrari. Mr. Musk apparently has the pull, the money and the car to waste, not to mention the rocket and broke no national security laws. Since no one complained he touched them, the car looks cool out there without much chance of a head-on collision.

    As for the PfSense/OPNSense box and my serious query, the box was on hand; no spending required. The Intel box was pre-built, so free. Our Asus router, hard working and capable as it is, runs only two concurrent OpenVPN tunnels; I wanted to try for four. Commercial VPNs don't care to allow users more than two OpenVPN tunnels, running L2TP or OpenVPN from the user's IP over a private IP address contracted for by the user, regardless pf how many devices are routed over the two encrypted tunnels (I think I said that correctly; long week). If OpenVPN isn't the better of the two choices provided, enlighten away.

    It's not as extreme an achievement as launching cars, but anyone who's bent on achieving personal privacy, say streaming favorite episodes of (whatever) from Netflix in addition to other more/less encrypted data on other tunnels, so the local ISP doesn't know exactly which episodes are being watched, well, it's only one example off the top of my head. Granted, that sort of thing only goes so far so if the NSA or others wants to know what the user is watching, they'd look at IPs, the Netflix customer data, or just hack the connection and watch along with said user. I managed two concurrent tunnels using both forks and haven't gotten further, the goal remains getting four concurrent tunnels up and running well both with and without streaming video. Not a burning question, just curious and interested, and I appreciate being able to ask intelligent folks without getting flamed, always a danger in site and places I don't frequent. Such a great forum, thanks for your comments.
     
  13. MichaelCG

    MichaelCG Very Senior Member

    Joined:
    Jan 4, 2017
    Messages:
    581
    Location:
    Central US
    I fully understand the OpenSSL test is not a true test of overall VPN performance since there are so many other portions to OpenVPN....but still getting a baseline comparison across different platforms can give you an "idea" of what to expect or not to expect out of a specific CPU. Knowing that a specific CPU tops out at 80Mbps on OpenSSL pretty tells you to never expect 100Mbps out of OpenVPN from it. However what it doesn't tell you is that if you have a CPU that can do 240Mbps on OpenSSL does NOT mean it can do anywhere near that on OpenVPN. What a specific platform can do just at the CPU doesn't always take into account what happens when you try shoving the same data through the rest of the buses on the platform.

    I am going to go back and read the other thread now to see what testing methods were used there and will repeat on my well aged Core2 DUO.
     
  14. MichaelCG

    MichaelCG Very Senior Member

    Joined:
    Jan 4, 2017
    Messages:
    581
    Location:
    Central US
    Based on this other test...my previous statements don't hold much water either about being able to make the assumption that OpenVPN speeds in theory will not be able to exceed OpenSSL speeds....very confused here.

    So based on the other thread, this gives me 187.9Mbps for OpenVPN??
    Code:
    /root: time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
    16.995u 0.015s 0:17.03 99.8%    722+172k 0+0io 0pf+0w
    
    So how does this work if basic OpenSSL can only do around 100Mbps on my system, but the OpenVPN tests are reporting 180Mbps? What am I missing?
     
    Last edited: Feb 9, 2018
  15. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    There's other factors at play - the OpenSSL and OpenVPN performance tests can give some idea of relative performance, but also goes to machine architecture efficiency and bandwidth across different buses... even the Ethernet card and drivers can play a small factor...
     
  16. System Error Message

    System Error Message Part of the Furniture

    Joined:
    Oct 14, 2014
    Messages:
    4,097
    i have a thinkerboard and a x86 udoo too, so perhaps i could test them soon.

    for the asus routers, the broadcom chips they use in them does have some flaws which leads to huge performance drop. When doing VPN you also have to do NAT and routing at the same time so not being able to use hardware NAT with VPN could be what kills performance on a lot of these routers. VPN is CPU intensive even with hardware acceleration, so depending on what your router already does it could cause a significant performance loss. With x86 and even TILE they have many cores and the CPUs are complex enough that the extra processing power needed isnt an issue thanks to the CPUs being super-scalar or vector CPUs with predictions to make use of every cycle.

    When i was looking at NIC driver tutorials, intel server NICs are well supported because the NIC chip isnt a blackbox so you can skip kernel stack and even directly interface with the chip yourself. as far as NIC features go, even broadcom, atheros and other decent brands pack quite a lot of features and processing power in their NICs but dont have that driver transparency as good as intel does. This is why mellanox 10G NICs are cheap because they are a blackbox and no one wants them anymore which is why even i have issues with some of them with compatibility.
     
  17. st3v3n

    st3v3n Very Senior Member

    Joined:
    Feb 24, 2016
    Messages:
    509
    Location:
    Central US
    System, most informative. The Xeon 4-core ran the firewall fine, the but the dual-core i5-660 was faster, and necessary if we want to use AES. The 4-pt NIC is a dual-CPU HP-NC354T, which is Intel with 4GB aggregate. So far it's never broken a sweat so to speak and real world is more than we'll use day to day, experiments notwithstanding. The NIC was selling for between $40-50 on Amazon. It runs a bit warm, but a fan blowing past the heat-sink sends that out the back. One of these days a tinkerboard would be fun to play with. We have an Android tablet that has an 8-core Rockhip in it; I don't remember which Rockchip but considering the lack of performance even though it supposedly is a 1.8 Ghz 8-core, it can't be the same Rockchip referred to. Thanks for the great post. Cheers.
     
  18. Xentrk

    Xentrk Part of the Furniture

    Joined:
    Jul 21, 2016
    Messages:
    2,521
    Location:
    The Land of Smiles
    Here are my metrics:

    Code:
    aes-256-cbc 3200 9.32 343.35 Mbps
    aes-128-cbc 3200 9.15 349.73 Mbps
    aes-128-gcm 3200 8.5  376.47 Mbps
    
     
  19. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    14,290
    Location:
    San Diego, CA
    Interesting - OpenVPN 2.3 (Ubuntu 16.04LTS) on N3700... Intel Nuc box...

    Code:
    aes-128-cbc 3200/12.16 = 263.16 Mbps
    aes-256-cbc 3200/12.47 = 256.61 Mbps
    
    Something's up with the OpnSense numbers - maybe config?

    Agree - it would be interesting to see where the 86U lands - it's OpenSSL numbers were pretty interesting, but OVPN is more than just that...
     
  20. Deepcuts

    Deepcuts Regular Contributor

    Joined:
    Apr 29, 2012
    Messages:
    119
    i3 7100T @ 3.4 Ghz 35W on IPFire.

    type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
    aes-128-cbc 1151736.60k 1241351.91k 1275924.65k 1285137.75k 1295377.12k
    aes-256-cbc 861152.57k 909512.00k 930465.71k 936499.54k 938377.22k

    [[email protected] ~]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

    real 0m3.903s
    user 0m3.893s
    sys 0m0.007s

    819 Mbps