New maximum connection test can be misleading.

[lstraath]

Hi, i started to test my own router with the test client and server that is used in the latest smallnetbuilder tests.

However i do not own a small router but a slightly bigger one with pfsense running as firmware on it. The ideal router to test the test app. I started with all settings reset to default the only thing i did not do was putting the client in the dmz i recon my router has plenty of cpu to spare not to be bothered by this simple test.

I also had some connections open to monitor the state table and cpu utilization of the router while conducting the test.

I run the test and as expected the maximum number of connections was just below 5000. This makes sense because the default pfsense state table is 10000 making a connection takes 1 state the ack that is send as respose takes 1 state. So on the first test nothing special.

However there are 2 settings i can change about the state table in pfsense. The fisrt is the state table size the second is the state table optimization.

Changing the size to for example 20000 would cause the test to end at slightly below 10000. And so it did when i tryed. Now i tryed to increase it even more to 40000 and expected the test to stop at 20000 but it didn't. And this was the point where i stared to get the idea that there was a loophole in the test.

I started to monitor the size of the state table realtime while starting the test again. And found that after about 25000 states where filled the number would not get any higher. I figured this had to be because the router was dropping states. To prove that this was the case i used the state table optimisation and changed the setting from normal to conservative. This means the router will wait longer before it starts erasing states that are considered old and not used for trafic.

I started the test again and yes this time it stopped just below 20000 again. So i proved this was the issue.

Now vendors of soho routers do not tell you how long they are keeping unused states in the state table. So the one that end high in the test could be faking things just by setting the state table optimisation very agressive.

To make sure this can not be the case you need to do the test a lot faster, for example just send 40000 connections or so as fast as you can and count the number of acks that you recieve. Instead of sending 1 and waiting for the ack. Altough this could create other problems. I will try to figure out a way to conduct the test without having this problem.

Router specs:
P4 @ 3.20Ghz
512mb ram
Wan nic intel E1000
Lan nic intel E1000

Test Server machine:
P4 @ 3.20Ghz
512mb ram
Nic intel E1000
Windows XP sp3

Test Client machine:
c2d E6600 @ 2.4Ghz
4Gb ram
Realtek 8111B
Windows 7 64bit

regards,

Leon Straathof.

 

[lstraath]

State table entry size

I would like to add that in pfsense the state table entries are about 1kb each so 10000 would take almost 10mb of ram. Most soho routers do not have that much ram so 10mb for just the state table seems to be a lot for a soho router wonder how big a state table entry is on other systems i think i will ask the author of dd-wrt how big it is in his firmware.... Somehow i have problems believing that a soho router can maintain more the 10000 states since most of that only have 4 mb of ram for al functions in the router.

 

[tipstir]

Newer HO/SOHO routers have 32MB, 64MB, 128MB and 256MB (more in 450mbps wireless 802.11n breed. Using the PC as a router uses just to much electric to run it 24/7, if it crashes then there goes your network. BAD hardware device an etc. I've tested the idea a few years ago. Got old P4 that I could use but I see not point doing so. MSC not much use for that that unless you use software that can take advantage of it. LAN to LAN routing is on 680MHz HO/SOHO is much faster than 384MHz to access the same data over the LAN.

Anyway good testing out the home made router concept.

 

[thiggins]

Any test can be misleading. The new test, just like the old, is just a benchmark that is applied the same way to all routers. I actually prefer the old test, since it not only opens connections, but runs traffic on them. That's more like Torrent behavior.
But 200 connections was too low a limit given today's routers.

Different routers use different TCP / UDP connection timeouts. Running the test faster would just favor those products.

 

[lstraath]

Memory size has increased a little in soho routers but as you can see in this http://www.dd-wrt.com/wiki/index.php/Supported_Devices link there are only a few that have 128mb and the ones that have 256mb are not soho routers but lowpower diy x86 solutions. I know my setup consumes a lot of power but it is not build from left overs and very stable (mostly more then 300days between down periods) which most of the time i create myself because of testing or maintainance. The reason i use this setup is because of the featureset like openvpn client and server and the trougput i can get over tunnels (vpn takes more cpu when you want to take full use of your bandwidth trough vpn tunnels).

But anyway that is not the issue i want to point to, the only thing i wanted to say is the problem found in the test and finding a way to improve the test because if many soho routers are reporting a wrong table size the test is useless and all the smallnetbuilder info about this part is not of any use either.

Don't get me wrong here i just want the database to have the best information possible, so i will try to find a better way to test altough i'm not sure yet how.

 

[tipstir]

Memory size has increased a little in soho routers but as you can see in this http://www.dd-wrt.com/wiki/index.php/Supported_Devices link there are only a few that have 128mb and the ones that have 256mb are not soho routers but lowpower diy x86 solutions. I know my setup consumes a lot of power but it is not build from left overs and very stable (mostly more then 300days between down periods) which most of the time i create myself because of testing or maintainance. The reason i use this setup is because of the featureset like openvpn client and server and the trougput i can get over tunnels (vpn takes more cpu when you want to take full use of your bandwidth trough vpn tunnels).

But anyway that is not the issue i want to point to, the only thing i wanted to say is the problem found in the test and finding a way to improve the test because if many soho routers are reporting a wrong table size the test is useless and all the smallnetbuilder info about this part is not of any use either.

Don't get me wrong here i just want the database to have the best information possible, so i will try to find a better way to test altough i'm not sure yet how.

The old PC was clients unit that failed. I just fixed it and use as router. But still not convince using PC going to be any faster that these small routers. Yes they're based on x86 still use Gig PCI Controller base, except for new 450mbps N routers Gig PCI-E Controller but still uses PCI on the PCB. The only router on PC would be effective was Windows Server 2003 running NAT, DHCP, DNS and Active Directory now that was quick until the system hardware had gone poof!

 

[lstraath]

The old PC was clients unit that failed. I just fixed it and use as router. But still not convince using PC going to be any faster that these small routers. Yes they're based on x86 still use Gig PCI Controller base, except for new 450mbps N routers Gig PCI-E Controller but still uses PCI on the PCB. The only router on PC would be effective was Windows Server 2003 running NAT, DHCP, DNS and Active Directory now that was quick until the system hardware had gone poof!

If you fix an old pc to use as router it might not be as stable for 24/7 use as you would want. My router is build from new parts, and as i already stated it has served me well the past few years.
You are right when you say that a soho router can route traffic fast enough for the internet connection you have. It's true that most new routers cpu can handle that. But when you add rules to it's firewall (if possible) and setup qos rules the cpu gets more and more load to handle. There is a point when the relative slamm cpu in the soho router cannot do the job anymore and maintain the maximum speed of your connection. However the firmware in most soho routers are limited in the number of rules you can make so it will be hard to get to this point.
A heavy x86 machine (and mine is not the heaviest) will definatly be able to do more than the average soho. Large corporate routers are not build around the weak cpu's found in soho routers either. In essential i have a small corporate router firewall at home which for almost anyone is overkill, but due to my work i have a larger network at home than most people would have including 24/7 vpn connections than enlarge my lan over then internet.

A friend also had a larger lan with heavy loads he shared his internet connection (50 down/ 10 up) with 4 other families. The total of machines that asked for an ip came to over 30 and sometimes more people where doing p2p simultaniously combined with newsgroup downloading. He tryed soho routers but they run out of sessions, did spontanious resets and other wierd things when there where extreme usage peaks. They dissapeard when he tryed the same setup i had. He also found it nice to maintain pfsense because he had more tools to monitor and control the traffic.

To make a long story short, yes bigger outers are faster but only when you more load for it then a soho can handle, which means you must think in over 20 clients and/or a lot of aditional work for the router like vpn/qos/firewall rules.