How We Test Wireless Products - Revison 8

[azazel1024]

Overall the change in process seems sounds and well thought out.

However, one thing I really don't like is that 2.4GHz 40MHz still isn't tested. I do understand a number of people can't or at least shouldn't run their routers with 2.4GHz set to 40MHz, however there are a LOT of people that CAN run it in 40MHz mode and a lot of other people who do, even if they shouldn't.

Its a shame that 40MHz testing isn't done on the routers too, as in my experience 2.4GHz 40MHz can vary wildly in performance from 20MHz mode, not simply you have a little over double the potential bandwidth, but I have seen some routers where 20MHz ran like a champ, but serious issues with 40MHz mode, or vice versa where 40MHz mode worked great, but there were odd inconsistencies and drop-offs in 20MHz mode.

It just seems like an odd inconsistency to not also throw up 40MHz numbers for 2.4GHz as there are a lot of users that will run the router that way.

 

[Razor512]

I always wondered about that boost in throughput, I noticed it a lot on wifi tests but it never seems to take place when doing a simple WAN to LAN benchmark.


For example here is a WAN to LAN benchmark (wired) using the throughput script set as described in the article.

http://i.imgur.com/N8y0UDO.jpg

But if wifi is benchmarked, especially with netgear routers, I get results like this

(AC1200 client connected to a netgear R7000 )
http://i.imgur.com/n3jHPgy.jpg

 

[thiggins]

I always wondered about that boost in throughput, I noticed it a lot on wifi tests but it never seems to take place when doing a simple WAN to LAN benchmark.

It's an IxChariot quirk that has to do with packet aggregation.

You shouldn't see it if you benchmark wireless with something other than IxChariot, like jperf, for example.

 

[thiggins]

However, one thing I really don't like is that 2.4GHz 40MHz still isn't tested. I do understand a number of people can't or at least shouldn't run their routers with 2.4GHz set to 40MHz, however there are a LOT of people that CAN run it in 40MHz mode and a lot of other people who do, even if they shouldn't.

There are also a lot of people who don't use 3x3 clients and I don't show results for that either.

 

[microchip]

I only have one device that supports 40MHz and it's my Acer smartphone. I don't own any other wireless devices, except the Acer and a 7 years old lappy which I hardly boot up

My Acer has no issues working with both 20 and 40MHz. Setting the 2.4 band on the router to 20MHz and the Acer tells the link is at 65Mbps. Setting it to 40Mhz and it says it's at 135Mbps. Doing a few speed tests on a 120Mbps connection I have, in 20Mhz it can't go above 55-56Mbps so I set the 2.4 band to 40Mhz instead to have more throughput. Not that I'll ever need that much but I like the higher number

 

[azazel1024]

I only have one device that supports 40MHz and it's my Acer smartphone. I don't own any other wireless devices, except the Acer and a 7 years old lappy which I hardly boot up

My Acer has no issues working with both 20 and 40MHz. Setting the 2.4 band on the router to 20MHz and the Acer tells the link is at 65Mbps. Setting it to 40Mhz and it says it's at 135Mbps. Doing a few speed tests on a 120Mbps connection I have, in 20Mhz it can't go above 55-56Mbps so I set the 2.4 band to 40Mhz instead to have more throughput. Not that I'll ever need that much but I like the higher number

That is odd. I know a fair number of phones and such forth don't support anything more than 20MHz, but at the same time, if you are testing the higher performance bits of a router, which seems resonable, I'd think you would still want to test 40MHz mode for 2.4GHz.

A lot of people use it. Significantly more than the number of people who have a 3:3 client I'd wager. Since most routers seem to ship with auto mode set for 20/40MHz, that means that a lot of routers DO run in 40MHz. In the few places I've run wifi snoopers, I'd would guestimate that 60-80% of the routers/APs I've seen that were 802.11n, were operating in 40MHz mode. Whether it is a friends house with only his network visible, or townhouses/apartments with half a dozen or a dozen or more SSIDs visible.

Still seems like it is missing a key test not to do 40MHz testing for 2.4GHz.

 

[azazel1024]

I always wondered about that boost in throughput, I noticed it a lot on wifi tests but it never seems to take place when doing a simple WAN to LAN benchmark.


For example here is a WAN to LAN benchmark (wired) using the throughput script set as described in the article.

http://i.imgur.com/N8y0UDO.jpg

But if wifi is benchmarked, especially with netgear routers, I get results like this

(AC1200 client connected to a netgear R7000 )
http://i.imgur.com/n3jHPgy.jpg

Do you mean WLAN to LAN benchmark?

I generally notice, at least with decent clients, that most routers you see roughly the increase in throughput that would be indicated by the increase in MCS by switching channel widths. I typically see around a 110-130% increase in performance going from 20MHz to 40MHz mode on most routers, and in some light testing with 11ac, similar. Around a 110-130% boost from 20 to 40MHz mode and around a 100-115% increase from 40-80MHz mode.

At least in environments that are not lousy with other SSIDs, or where your own APs would be stepping on each other, it is generally a no brainer to set wider channels, even with 2.4GHz.

In testing with a couple of different routers, the only time I have seen 20MHz mode on 2.4GHz perform better is with severe interference or in some unique circumstances at extreme distances. Even at long range, generally 40MHz appears to perform better than 20MHz, even if the difference is very small.

Example, one router I have 2.4GHz 40MHz provides approximately 2.5MB/sec of throughput at -85dBm signal strength, in 20MHz mode it provides 2.3MB/sec. Another router I have does around 3MB/sec in 40MHz mode and around 2.5MB/sec in 20MHz mode at similar signal strength. Closer in and the performance difference is significantly more. At close distances in 20MHz mode I get around 11MB/sec with one, but 23.5MB/sec in 40MHz mode. The other one I get around 13MB/sec, but 28MB/sec in 40MHz mode.

The only time I have really experienced 20MHz working better is when I had a ton of surrounding SSIDs that were all within 10dBm of my own network or STRONGER, in which case, 20MHz mode ranged from a few hundred KB/sec faster to nearly half again as fast as 40MHz mode. Of course if there is wifi congestion, if everyone was a good neighbor and stuck their gear in 20MHz mode, there would likely be even less congestion and then everyone would benefit even more.

 

[Razor512]

40MHz on the 2.4GHz band will become a lot more popular especially with some router makers adding the options to make it use the 40MHz mode regardless of other nearby access points

It may have been an issue in the past where unless you lived on a private island, your router would freak out about other AP's and then switch to 20MHz mode, but with the ability to suppress that, it seems like 2.4GHz can be made to perform really well for devices that support 40MHz

I will do a quick benchmark with my R7000 bridged to my Netgear WNDR4700 at a distance of about 5 feet. (Distance is more of trial and error where I start a throughput test, and then slowly move the routers apart until the speed tops out)

The 40MHz test is done with the 20/40 MHz coexistence disabled, the 20MHz test is then done with 20/40MHz coexistence enabled which is better named as the "cripple my router" button (that is enabled by default) (this test is done in a congested environment so a range test at 2.4GHz will not work well (it is bad with all of my routers (worst than 5GHz due to the 160+ other AP's in the area)

upload= data being transmitted from the WNDR4700 side, to the R7000
Download= data being transmitted from the R7000 side, to the WNDR4700

2.4GHz 40MHz download: http://i.imgur.com/uZ8Dwy7.jpg

2.4GHz 40MHz upload: http://i.imgur.com/voE1UFX.jpg

==-=-=-=-=-=-

2.4GHz 20MHz download: http://i.imgur.com/uBfmeAT.jpg

2.4GHz 20MHz upload: http://i.imgur.com/ORKVvS0.jpg

 

[microchip]

That is odd. I know a fair number of phones and such forth don't support anything more than 20MHz, but at the same time, if you are testing the higher performance bits of a router, which seems resonable, I'd think you would still want to test 40MHz mode for 2.4GHz.

A lot of people use it. Significantly more than the number of people who have a 3:3 client I'd wager. Since most routers seem to ship with auto mode set for 20/40MHz, that means that a lot of routers DO run in 40MHz. In the few places I've run wifi snoopers, I'd would guestimate that 60-80% of the routers/APs I've seen that were 802.11n, were operating in 40MHz mode. Whether it is a friends house with only his network visible, or townhouses/apartments with half a dozen or a dozen or more SSIDs visible.

Still seems like it is missing a key test not to do 40MHz testing for 2.4GHz.

I've done 40MHz test on the phone, just forgot to mention it. It maxes out at ~90Mbps. On 20MHz it maxes out at ~55Mbps. I've done a WAN -> LAN wired test too and my connection maxes out at 115Mbps, so I get the particle bandwidth of a 120Mbps subscription.

I don't really care or am interested as to why, just noticed that on 40MHz it's faster so I left it that way

 

[Razor512]

I always find it hard to test smartphones because wifi performance is very inconsistent. e.g., I have seen cheap android smartphones outperform flagship smartphones. It just seems that smartphone manufacturers are not putting enough effort into making sore that the wifi performance is performing as best as it can, instead they seem to treat it more as an afterthought.

edit:
@azazel1024, for the first benchmark, I did a WAN to LAN (100% wired test) to show the throughput boost not happening, it pretty much almost instantly jumps to the top speed, but if wifi is used, then the throughput jumps up after around 10 seconds.

 

[microchip]

I always find it hard to test smartphones because wifi performance is very inconsistent. e.g., I have seen cheap android smartphones outperform flagship smartphones. It just seems that smartphone manufacturers are not putting enough effort into making sore that the wifi performance is performing as best as it can, instead they seem to treat it more as an afterthought.

I agree. The lowest speed I've gotten from doing 10 tests and using 40Mhz is 54Mbps. Then if I play a bit with the test servers of the Speedtest app on my phone, at some servers I can easily get close to 100Mbps.

I do not know if this is due to the different test servers or due to the inconsistency you mention, but I don't care. It's not like I'll ever download GB files to my smartphone. My SD card will choke on that anyways

 

[RMerlin]

I always find it hard to test smartphones because wifi performance is very inconsistent. e.g., I have seen cheap android smartphones outperform flagship smartphones. It just seems that smartphone manufacturers are not putting enough effort into making sore that the wifi performance is performing as best as it can, instead they seem to treat it more as an afterthought.

Phones usually focus more on power saving than actual performance. A typical smartphone rarely needs to hit very high speeds, however everyone is always complaining about their battery life.

 

[azazel1024]

Phones usually focus more on power saving than actual performance. A typical smartphone rarely needs to hit very high speeds, however everyone is always complaining about their battery life.

I never complain about battery power. Opps, have to cut the post short, only 8% left. Gotta go plug it in

 

[KGB7]

There are also a lot of people who don't use 3x3 clients and I don't show results for that either.

But we still want to see the results. Why do a review in the first place, if you going to do it half way.
I, we, as consumers want to see a full picture, and not just a glimpse.

From now on, please provide us with a full picture as to what a product is capable of.

 

[KGB7]

Phones usually focus more on power saving than actual performance. A typical smartphone rarely needs to hit very high speeds, however everyone is always complaining about their battery life.

+1

Plus, each brand and model of the phone has a different focus on power savings implemented. Thus the results will vary between OS versions, model of the phone, maker, etc etc.

To test all phones at its full capability, is time and financially taxing. Its like testing a car tire on different road grades at different parts of the country, various sea levels, humidity and temps, weight and load on each tire,etc etc etc. While the website main focus is on general comfort of a vehicle.
It is doable, but it requires a very high financial support. It would take $20k just to launch the project, then another $10k a month to support a wide scale testing.

In school you are taught that anything is possible. Life teaches you that anything is possible if you throw enough money at it.

If we gave Merlin $5k per month for 6 months, we would have a firmware that would make Asus look like it was staffed with children.

 

[azazel1024]

Can't argue with that at all.

At least for different channel widths, I don't know how long it takes to run each attenuation test, but I would think just running a 20MHz and then a 40MHz test on 2.4GHz would at most add a matter of an hour or two. Which granted is an hour or two for every single product tested, but I'd still think, without knowing at all, that it would account for a relatively small expansion of the time to do a full review on a product.

Going with entirely different client combinations like 1:1, 2:2 and 3:3 would add a rather large amount of extra time, especially if you are testing at 20MHz, 40MHz plus 5GHz 11n or 11ac.

 

[sfx2000]

With all the current rage of multiple chipsets, extra radios, extended functionality, one thing to consider is the power in Watts consumed...

since routers are intended to be on 24/7, this is a peg that few reviews actually cover, and perhaps a selling point/incentive to the OEM's to improve performance here.

sfx

 

[thiggins]

sfx
I will throw my Watts Up on the NETGEAR and ASUS. I expect 7 - 10W. You think people really care? At least not the bunch that hangs around here...

 

[sfx2000]

sfx
I will throw my Watts Up on the NETGEAR and ASUS. I expect 7 - 10W. You think people really care? At least not the bunch that hangs around here...

Tim - I think so - esp at the present where the Tier 1's are throwing a lot of radio resources at WiFi and extended capabilities of the devices - and there doesn't seem to be much regards to overall power consumption...

And this is an indirect cost to the end-user - when you consider life-time expenses...

Saw a recent report/discussion on Cable Set-Top boxes, and was a bit surprised as to the vampire load they took when the TV was off...

sfx

 

[sfx2000]

sfx
I will throw my Watts Up on the NETGEAR and ASUS. I expect 7 - 10W. You think people really care? At least not the bunch that hangs around here...

I think the power consumption is higher than you think...

Case in Point - P=IE = in other words - Watts = Amps * Voltage

Apple's Airport Extreme AC (AC17500 class) - consumes 1.5 Amps Max @ 110VAC -- So doing the math 165 Watts

The Airport Express is 0.2 Amps @ 110VAC -- Math here for a dual stream N600 class router... 22 watts consumed...

So yes, I think it's relevant..

sfx