What's new

Today's delightful new problem (10gbit/s connection incoming)

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Magebarf

Regular Contributor
Okay, this day came (or will come) a lot earlier than i thought it would.

And to start this thread off, this is merely a "because I can", or rather a "because I want to", project. No necessities, only pure tech interest and general madness driving this.

Today my soon to be ISP (Bahnhof, Swedish) announced they've just finished their new round of infrastructure upgrades, which enables them to offer symmetric 10 gigabit connections for their customers.
This at the price point of roughly $60 USD a month regular price, with a initial offer of $36 a month for the first 6 months.

My expected delivery of what initially was a 1 gigabit connection from the same ISP is planned for the first of March, for slightly higher monthly cost. So of course I decided to give them a call and have the speed multiplied by 10 while having my first 6 months for a cheaper rate than expected, and the same price as my current (soon to be ex) ISP offers for 250/100mbit/s.

So, now my luxurious troubles begins; how do I best utilize this. :)

Based on the information I received when placing my change request today, I'll be shipped a new media converter soon (hopefully before my access fiber switches port in the closest station), but even the person in the customer service had not yet seen or had any information on exact what type of device this will be. All I know so far is that I will be connecting to a RJ45 connector on my side of the converter. The current one I received in preparation for my gigabit line is a simple SFP port to RJ45 converter, but I'm not yet sure if the new one is fully integrated or if it will be housing a SFP+ module.

My initial point I'd need to address is most likely also the reason I'm starting the thread in this sub-forum; My current router (Ubiquity ERL-3) ain't prepared for this.

Until I receive the new media converter and know if this has a SFP+ module I can reuse, I'm not sure whether I'm limited to routers connecting using 10GBASE-T/RJ45, so I'm investigating that playing field first and expanding once I know more.

With that limitation in mind, I have so far not found a single off-the-shelf router with a 10gigabit RJ45 connection built in. So, if going that route, I'm guessing pfSense with a custom built setup will the only path at the moment?

If instead looking down the path of SFP+ equipped routers and modules, my initial research leads me to a single main options for the router, especially if trying to keep things under something that could be called a budget: MikroTik CRS317-1G-16S+RM. Barring money, a Ubiquity EdgeRouter Infinity may be an option as well.

Do any of you have any experience with running the MikroTik smart switches with RouterOs, and is that a feasible option?

As for SFP+ modules I've seen Aquantia AQS-107 since before, but while looking at things today I realized that MikroTik S+RJ10 has been made available, which in my case will be a lot more economical (a third of the price) and if I'm guessing compatibility should be fine if going with the MikroTik unit for routing...

Any type of input or experience, new products or just good ideas and pointers for useful information is welcome!

The main limitations as for now is: 10Gbit/s RJ45 -> <new router> -> RJ45. Switching and cabling is not that big of an issue, as I already have CAT-6A cabling for my drops, and I'm already planning to replace switches to devices with 10gbit/s uplinks.

And once again, this is merely a enthusiast project. I'm not expecting any night and day differences, but I'm excitedly looking forward to what will happen in the consumer, and prosumer (even if I find that a vague label for networking equipment), market with regards to 10gbit devices, as right now (alright, in about a week's time) my bandwidth bottleneck has definitely moved within my house.
 
This at the price point of roughly $60 USD a month regular price, with a initial offer of $36 a month for the first 6 months.

My expected delivery of what initially was a 1 gigabit connection from the same ISP is planned for the first of March, for slightly higher monthly cost. So of course I decided to give them a call and have the speed multiplied by 10 while having my first 6 months for a cheaper rate than expected, and the same price as my current (soon to be ex) ISP offers for 250/100mbit/s.

So, now my luxurious troubles begins; how do I best utilize this.

Ultimately at the end of the day - and ISP's may disagree - bits are cheap, and this show this...

With a 10Gb WAN connection, you'll find that the world is slow in general - one can run a few benchmarks, but finding a public 10Gb connection to test against - even with major CDN's, is a problem.

good problem to have ;)

Some might say that some consumer devices can make something of this...

Code:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.8 GBytes  10.2 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.8 GBytes  10.2 Gbits/sec                  receiver

That's an x86 box - Intel N3700 over the loopback interface... those numbers are moving bits across DDR3-1600 RAM

no way that a consumer device is going to get that, esp. when consider NAT and SW routing for an affordable cost...

@System Error Message is working on things, and I've been recently looking into this space as well - I can't speak for SEM, but even with a userspace driver and a very well tuned kernel - one is looking at Core i7-4790 or equivalent to get there, just from a CPU horsepower perspective in SW routing in the core...

The box I'm working with is a 5 figure (USD) box, and we're competitive with silicon there...
 
Thanks for chiming in @sfx2000!

Yes, it's definitely a nice problem to have, and probably also why the ISP, at this moment at least, is able to price their 10gig connection with only marginally higher monthly rate than the 1gig one; they know that for the residential users this will only be a boasting point, and the amount of data they will have to shuffle will only differ in maybe 1 out of a thousand customers. At the same time, since they probably already have the need for the infrastructure and equipment due to their commercial customers, why not.

I think time line wise for "my project" I'm considering a roughly 5-10 year horizon before I believe consumer equipment may have transitioned so that the high end gear actually might utilize the type of bandwidth just made available. My aim is to along this timeline progressively increase how much of my connection I can actually utilize, but I'm not even sure I'd see peak speeds of above 1gbit/s other than synthetic usage, which even then I may not see in quite a few years.

My considerations since the initial post goes somewhere along the way of your thinking, pfSense seems to be the way to utilize the most of the connection per dollar amount spent on it. LAN throughput seems possible at line speeds without problems, but routing is still going to be the weak point right now.

The MikroTik switch/router I looked at does indeed have great performance numbers (according to MikroTik themselves) when looking at switching, but when bridging it seems to be choking around 3gbit/s, and that without even having NAT enabled if I've understood their measurement methods correctly from other peoples' discussions.

So, with that, I'd get 10gbit link speed one more hop on the network, possibly improving latency slightly. Then I'm unsure how much more of a performance hit NAT brings to their published numbers, but I'm kind of guessing that I'd only see peak speeds above gigabit in case I set up a DMZ for a specific device or so.

It might be a good switch for use as my central LAN switch, as I'd get a pretty inexpensive starting point where I can add additional ports for around $60 per port with today's pricing. It could also replace my ERL-3 with what I hope is comparable if not better performance.

MikroTik's CCR1036-8G-2S+ seems to be able to close in on my ISP's limits on the routing performance, but then we move into 4 digit pricing, and I'd also say Ubiquity's Unifi USG XG becomes an option, and even their EdgeRouter Infinity XG is not too far away. For some reason I anticipate this (2 or more SFP+ ports and possibly also the first 10GBASE-T routers) may very well be an area where these two companies will duke it out over the coming few years.
 
Thanks for chiming in @sfx2000!

Yes, it's definitely a nice problem to have, and probably also why the ISP, at this moment at least, is able to price their 10gig connection with only marginally higher monthly rate than the 1gig one; they know that for the residential users this will only be a boasting point, and the amount of data they will have to shuffle will only differ in maybe 1 out of a thousand customers. At the same time, since they probably already have the need for the infrastructure and equipment due to their commercial customers, why not.

I think time line wise for "my project" I'm considering a roughly 5-10 year horizon before I believe consumer equipment may have transitioned so that the high end gear actually might utilize the type of bandwidth just made available. My aim is to along this timeline progressively increase how much of my connection I can actually utilize, but I'm not even sure I'd see peak speeds of above 1gbit/s other than synthetic usage, which even then I may not see in quite a few years.

My considerations since the initial post goes somewhere along the way of your thinking, pfSense seems to be the way to utilize the most of the connection per dollar amount spent on it. LAN throughput seems possible at line speeds without problems, but routing is still going to be the weak point right now.

The MikroTik switch/router I looked at does indeed have great performance numbers (according to MikroTik themselves) when looking at switching, but when bridging it seems to be choking around 3gbit/s, and that without even having NAT enabled if I've understood their measurement methods correctly from other peoples' discussions.

So, with that, I'd get 10gbit link speed one more hop on the network, possibly improving latency slightly. Then I'm unsure how much more of a performance hit NAT brings to their published numbers, but I'm kind of guessing that I'd only see peak speeds above gigabit in case I set up a DMZ for a specific device or so.

It might be a good switch for use as my central LAN switch, as I'd get a pretty inexpensive starting point where I can add additional ports for around $60 per port with today's pricing. It could also replace my ERL-3 with what I hope is comparable if not better performance.

MikroTik's CCR1036-8G-2S+ seems to be able to close in on my ISP's limits on the routing performance, but then we move into 4 digit pricing, and I'd also say Ubiquity's Unifi USG XG becomes an option, and even their EdgeRouter Infinity XG is not too far away. For some reason I anticipate this (2 or more SFP+ ports and possibly also the first 10GBASE-T routers) may very well be an area where these two companies will duke it out over the coming few years.
For software routing, the cheapest 10G NAT firewall capable router is mikrotik CCR1036, ubiquiti's solution at the same price cant do those speeds in software. Other affordable solution include x86 based setups involving 2nd hand 10G cards, and a CPU + architecture that can handle it (CPU PCIe ports, DDR3 ram minimum, etc).
 
One area where I've failed so far myself is to map the market of devices in the range between $1000 and $5000. Or rather, I've hardly found any market of devices in that range.

The MikroTik/UBNT routing eqiupment seems to round when reaching four digits (besides the EdgeRouter Infinity around 1500 and the MikroTik CCR1072 at 3000), and and looking at the more enterprise grade hardware I'm seeing prices starting at 6 digits for most routers, with the most affordable ones still ending up a bit into the 5 digits region...

Thinking a bit outside the box I started looking at firewalls as well, but seeing as then you need to go into high-end or enterprise/carrier grade for those speeds it's not all that much that makes sense for residential use for the moment... I did however notice that it's possible to find used Palo Alto Networks models for somewhat reasonable prices, and seeing a PA-5050 on ebay at roughly $900 currently. Unsure if it would have an edge on the CCR1036 though.

Well, the search goes on. Exciting just to be waiting for the delivery next week.
 
I did however notice that it's possible to find used Palo Alto Networks models for somewhat reasonable prices, and seeing a PA-5050 on ebay at roughly $900 currently. Unsure if it would have an edge on the CCR1036 though.

Palo Alto's are fast enough, challenge is they're power hungry noisy beasts - and the licenses can be a challenge - there's a reason why they're starting to show up on ebay...

Food for thought - pfSense on Xeon-D - for $2500USD, you'll get a beast of a device...

https://store.netgate.com/pfSense/XG-1541.aspx

We've got an appliance that is very similar - and the netgate price is still competitive with our box - config is different as we built it specifically per a specific customer set of requirements - that being said, I know the netgate box will definitely handle a 10Gb connection with aplomb...

TCP quick check on our box... this is a test build* with a lot of instrumentation/debug turned on, so not as fast as Xeon-D can go, but a good feel for what the chip is capable of.

* spectre/meltdown testing, so QA has ownership of the lab box for now - source and sink are in the same rack, 100Gb cards and switch in the middle.

Code:
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.00  sec  49.2 GBytes  42.3 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  49.2 GBytes  42.3 Gbits/sec                  receiver

UDP numbers...

Code:
[SUM]   0.00-10.00  sec  40.3 GBytes  34.6 Gbits/sec  0.001 ms  0/4860160 (0%)
 
Last edited:
...
Food for thought - pfSense on Xeon-D - for $2500USD, you'll get a beast of a device...
...

Yep, this popped up on my radar as well. We have a local vendor who prepares small footprint servers with the same Xeon D as well which I saw mentions of yesterday.
The performance numbers definitely looks promising!

In addition I saw in the replacing the pfSense appliance thread that Netgate released som pre-release/pre-order info on the XG-7100 which will be interesting to see where performance testing will clock it.
 
I have not been keeping up with pfsense but not to long ago they were having problems with 10 gig cards.

Pfsense does not work well with a layer 3 switch. I gave up on it. My old Cisco RV320 router painted web pages much faster than my Xeon based pfsense rack mounted machine. My connection speed was 300 meg. The fastest I can get. No gig internet connections available for me.

For a 10 gig connection you are going to have a hard time finding a router at consumer prices. Maybe in the future they will think about building layer 3 switches with NAT. That would allow us to use a switch to connect to the ISP.

If you can not find equipment and you want to use the connection you might be able to connect with multiple routers. Instead of VLANs use separate routers.
 
Last edited:
...
For a 10 gig connection you are going to have a hard time finding a router at consumer prices. Maybe in the future they will think about building layer 3 switches with NAT. That would allow us to use a switch to connect to the ISP.

If you can not find equipment and you want to use the connection you might be able to connect with multiple routers. Instead of VLANs use separate routers.

Great idea with router per VLAN. I'm not yet sure about the allowance of IP addresses within the new transport network however, so it may or may not work.

As for pricing, I'm not strictly going for consumer cost equipment, while I'm also not ready to fess up with the costs for carrier grade equipment either. If I'd have to guess, I'll put the absolute limit somewhere near $2500 per piece of gear, while not really limiting the total budget as I'm anticipating I'll be picking up a bit of different devices to test out over the coming year or two. But yes, my understanding at the moment is just as you say, 10gbit/s routing at consumer prices definitely not available today.
 
I have not been keeping up with pfsense but not to long ago they were having problems with 10 gig cards.

Pfsense does not work well with a layer 3 switch. I gave up on it. My old Cisco RV320 router painted web pages much faster than my Xeon based pfsense rack mounted machine. My connection speed was 300 meg. The fastest I can get. No gig internet connections available for me.

Just be aware that @coxhaus has been having issues specific to his installation with pfSense... it does not speak broadly over pfSense or other elements.

With hi-performance gear and high bandwidth, results will vary... depends on who is turning the knobs.
 
@sfx2000 yes, thanks for the heads up.
As far as my understanding goes the milage varies quite a bit with how well people get it running.
Could you maybe enlighten me a bit, is there some specific licensing or packaging of pfSense which enables hardware acceleration for specific hardware, or is it just that some NICs and options give more trouble than others? Almost everywhere I read people tend to believe pfSense has an allergy, or something of a severe impairment, when it comes to 10GbE NICs, and usually the magic number being thrown around as the limit is around 3gbit/s...

@coxhaus (or maybe @sfx2000) could you refer me to the older threads where your experiences has been discussed (maybe in PM), and I can try to get a better understanding of what you've gone through and encountered without having to replay the whole topic in this thread as well?
 
Could you maybe enlighten me a bit, is there some specific licensing or packaging of pfSense which enables hardware acceleration for specific hardware, or is it just that some NICs and options give more trouble than others? Almost everywhere I read people tend to believe pfSense has an allergy, or something of a severe impairment, when it comes to 10GbE NICs, and usually the magic number being thrown around as the limit is around 3gbit/s...

Again - @coxhaus issues, mostly revolve some of his choices - he's got good HW (good big Xeon's that he underclocked), but he gave up sorting some DNS stuff (pfSense has both forwarding and resolving there). Rage quit from there, rather than solving the problem. We tried to help, but at the end of the day, if his Cisco-RV small business device works well, that's good enough, and we still appreciate his insight and contributions.

Special Licensing with pfSense - there is different levels of support, performance is the same - they've recently introduced a new level that is time based vs. ticket based, and they have a community thing - pfSense Gold - which is something different.

pfSense, like the FreeBSD that is the foundation, is sensitive to drivers... some work better than others. With 10Gb - chelsio tends to work well freeBSD on pfSense, and this is mirrored with the FreeNAS folks - The lower cost mellanox boards need some effort perhaps...
 
Food for thought - pfSense on Xeon-D - for $2500USD, you'll get a beast of a device...

https://store.netgate.com/pfSense/XG-1541.aspx

We've got an appliance that is very similar - and the netgate price is still competitive with our box - config is different as we built it specifically per a specific customer set of requirements - that being said, I know the netgate box will definitely handle a 10Gb connection with aplomb...

And FWIW - I'd be happy to sell you one of our devices... but we're in a different market, and it's well above your $2500 cap...
 
Again - @coxhaus issues, mostly revolve some of his choices - he's got good HW (good big Xeon's that he underclocked), but he gave up sorting some DNS stuff (pfSense has both forwarding and resolving there). Rage quit from there, rather than solving the problem. We tried to help, but at the end of the day, if his Cisco-RV small business device works well, that's good enough, and we still appreciate his insight and contributions.

Special Licensing with pfSense - there is different levels of support, performance is the same - they've recently introduced a new level that is time based vs. ticket based, and they have a community thing - pfSense Gold - which is something different.

pfSense, like the FreeBSD that is the foundation, is sensitive to drivers... some work better than others. With 10Gb - chelsio tends to work well freeBSD on pfSense, and this is mirrored with the FreeNAS folks - The lower cost mellanox boards need some effort perhaps...

I think if you run pfsense flat or let it handle the local routing you have a good chance of it working. Once you take the local routing out of pfsense by using a layer 3 switch it does not work well. There is nobody on the pfsense forums which uses a layer 3 switch with pfsense so it is not tested very well. They all run their layer 3 switches as layer 2. And I am not going to give up my layer 3 switch since it works so well. I just dumped pfsense.

Plus at some point I wish to run a routing protocol. From my reading this is another issue with pfsense. I asked about RIP2 and it seems like it is buggy. I would have to go into debugging mode which I don't care to do any more. There is some OSPF support but no EIGRP support but again very few if anybody runs a routing protocol with pfsense. I have not found anybody.
 
Last edited:
My new setup will be Comcast through SB8200 into pfsense into EdgeSwitch 8
EdgeSwitch 8 review. and for AP's UAP-AC-PRO or UAP-AC-HD

I don't like the CPU in the SG-4860 right know but I hopes for a new one soon. But I my have to build one something like this.

as for pfsense and a switch this my help

Or Layer 3 switch + pfsence you will see 9 comments click on that.
And you can Google: layer 3 switch + pfsense or layer 3 switch and pfsense there is a lot of info out there just look.

Processor : AMD Athlon II X4 610e Propus 2.4GHz 45watt
CPU Cooler : Zalman 9500A-LED 92mm 2 Ball CPU Cooler (fan off)
Motherboard : Asus M4A89GTD Pro/USB3 AM3 AMD 890GX
Memory : Kingston 4GB DDR3 KVR1333D3N9K2/4G
Hard Drive : Western Digital Caviar Green WD30EZRX
Power Supply : Antec Green 380 watts EA-380D
Case : Antec LanBoy Air (completely fan-less)

Network Card : Intel PRO/1000 GT PCI PWLA8391GT PCI
-OR-
Intel I350-T2 Server Adapter (PCIe x4)

what do you think
 
Last edited:
And FWIW - I'd be happy to sell you one of our devices... but we're in a different market, and it's well above your $2500 cap...
Thank you very much, but I'll have to pass on the offer for now sfx. Who knows when I'm desperate enough to give it a go, though. :)
I'm guessing as you say residential use is out of the intended scope, but what's the main client group? Carriers/ISPs/WISPs? Or something more specific?
 
Thank you very much, but I'll have to pass on the offer for now sfx. Who knows when I'm desperate enough to give it a go, though. :)
I'm guessing as you say residential use is out of the intended scope, but what's the main client group? Carriers/ISPs/WISPs? Or something more specific?

Telco, ISP, Data Centers, and hyperscalers...
 
My new setup will be Comcast through SB8200 into pfsense into EdgeSwitch 8
EdgeSwitch 8 review. and for AP's UAP-AC-PRO or UAP-AC-HD

I don't like the CPU in the SG-4860 right know but I hopes for a new one soon. But I my have to build one something like this.

as for pfsense and a switch this my help

Or Layer 3 switch + pfsence you will see 9 comments click on that.
And you can Google: layer 3 switch + pfsense or layer 3 switch and pfsense there is a lot of info out there just look.

Processor : AMD Athlon II X4 610e Propus 2.4GHz 45watt
CPU Cooler : Zalman 9500A-LED 92mm 2 Ball CPU Cooler (fan off)
Motherboard : Asus M4A89GTD Pro/USB3 AM3 AMD 890GX
Memory : Kingston 4GB DDR3 KVR1333D3N9K2/4G
Hard Drive : Western Digital Caviar Green WD30EZRX
Power Supply : Antec Green 380 watts EA-380D
Case : Antec LanBoy Air (completely fan-less)

Network Card : Intel PRO/1000 GT PCI PWLA8391GT PCI
-OR-
Intel I350-T2 Server Adapter (PCIe x4)

what do you think

If you get your Edgeswitch 8 running as a L3 switch post the config. My only thought is it does not have very many ports. It would be good for proof of concept.

I looked at your links for pfsense and an layer 3 switches and to me it mostly seems as theory and people trying to figure it out. I actually ran pfsense with a Cisco SG300-28 layer 3 switch in layer 3 mode for over a year. The config is posted on this site under my user name. My setup of pfsense for the layer 3 switch is on pfsense's forum under my username coxhaus.

The SB8200 is a good modem which I have run for almost a year. It is the best modem I used.
 
Processor : AMD Athlon II X4 610e Propus 2.4GHz 45watt

I would agree that Rangley is not a good choice for 10Gb, it's a great choice for 1Gb WAN...

An 8 year old Athlon is not a good choice for a 10Gb connection... clocks are one thing, but also IPC, which the K10 core falls behind.

Looking at the current AMD ZEN cores, but Intel still has some benefit - but these are chips well above OP's ask, just from a tray cost...
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top