1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

How We Test Hardware Routers - Version 4

Discussion in 'Routers' started by thiggins, May 16, 2016.

  1. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    [​IMG]
    The long overdue update of our Router test process brings powerful new hardware to bear and adds functional tests.

    Read on SmallNetBuilder
     
    L&LD, RMerlin and sfx2000 like this.
  2. CDROUTER Jason

    CDROUTER Jason New Around Here

    Joined:
    May 16, 2016
    Messages:
    3
    Well, I guess that means we should make a forum account ;) Great working with you, Tim! Hopefully your readers will find this useful.
     
  3. System Error Message

    System Error Message Part of the Furniture

    Joined:
    Oct 14, 2014
    Messages:
    4,078
    How do i get one :D without losing an arm or leg and is it powerful enough to test multi gigabit routers like the mikrotik CCR line?

    Is it possible to not allow the router to recover from DoS attack? My router is pretty fast and im pretty sure it doesnt need time to recover.
     
  4. CDROUTER Jason

    CDROUTER Jason New Around Here

    Joined:
    May 16, 2016
    Messages:
    3
    Well, it's an enterprise level test product, for sure. Our main customers are the vendors themselves and service providers. Tim worked with us on getting his new test procedure in place since the good work he does here was very much like the work CDRouter does for lab-grade situations.
     
    sfx2000 likes this.
  5. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    Yes. Pretty much everything is programmable.

    As Jason said, CDRouter is not aimed at or priced for individual consumers. Like Ixia and octoScope, QACafe is partnering with SmallNetBuilder to help us provide the most comprehensive test processes you'll find from any publication.
     
    CDROUTER Jason likes this.
  6. Cloud200

    Cloud200 Senior Member

    Joined:
    Aug 16, 2012
    Messages:
    475
    Are there going to be any devices you will go back to and run the new testing process on?

    p.s.
    Thanks for all the great work over the years!
     
  7. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    Yes. I'll be retesting a selection of AC1900 and up routers. Might do a few AC1200 and AC1750, too.
     
  8. Razor512

    Razor512 Senior Member

    Joined:
    Sep 29, 2012
    Messages:
    459
    So I guess we really won't be seeing these devices at out local electronic stores for ~$300...

    Anyway, for the functional tests, for some of the latest routers, are there many differences in how many tests they pass? Can the NTA1000 automatically change settings on a consumer router?
    I have noticed that on some routers tend to have some of the less common functions disabled by default, and all of them cannot really be enabled at the same time. For example, on many TP-Link routers, the IPv6 stuff is disabled by default.

    http://i.imgur.com/x295HOV.jpg
    http://i.imgur.com/nOGEvDy.jpg
     
  9. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    CDRouter can't change router configurations. That's why I use a standard configuration for testing.

    I'm not testing IPv6, IPv4 only. Reasons are here. Maybe in a year or so things will be better.
     
  10. System Error Message

    System Error Message Part of the Furniture

    Joined:
    Oct 14, 2014
    Messages:
    4,078
    Well i was hoping to perform the functional test on the CCR while it has traffic so that if say you had gigabit level traffic and had a DoS attack at the same time or testing how fast will it perform functionally other than NAT and how secure it is.

    @thiggins i know consumer routers have flaky ipv6 or that not all 3rd party firmware have very good ipv6 implementations but for routers that arent in the consumer space or using 3rd party firmware could you perform a full security and functionality test? I think it would be good to know where a router fails.

    Just out of curiousity how much do one of these units cost?
     
  11. RMerlin

    RMerlin Super Moderator

    Joined:
    Apr 14, 2012
    Messages:
    30,245
    Location:
    Canada
    Nice :) To this date, one of my favorite reviews you've posted remain this one. It actually made me go back to look at a few things, I remember fixing one or two of the issues reported there.
     
    sfx2000 likes this.
  12. vnangia

    vnangia Senior Member

    Joined:
    Jul 2, 2008
    Messages:
    201
    Location:
    Washington, DC, USA
    Would you be able to retest some hardware that you have tested in the past (for example, the EdgeRouter?) so that we have some idea of how much more demanding this test regimen is? Cheers.
     
    Marc66 likes this.
  13. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    I'll be retesting some products, yes. EdgeRouter is not likely to be one. The performance tests don't significantly push the products harder. The functional tests are what's really new.
     
    Marc66 likes this.
  14. vnangia

    vnangia Senior Member

    Joined:
    Jul 2, 2008
    Messages:
    201
    Location:
    Washington, DC, USA
    Fair enough!

    Second question, having had time to look through that long PDF: is there scope now for you to test security gateways (I'm thinking of, for example, ZyXel or SonicWall) to get a sense of what penalty there is for activating the UTM features? That would be a really helpful thing for those of us considering devices that can do some amount of threat protection and detection.
     
  15. System Error Message

    System Error Message Part of the Furniture

    Joined:
    Oct 14, 2014
    Messages:
    4,078
    @thiggins could you do a functional test on a mikrotik hEX lite and also test it for reliability under load? Preferably it would be better to test their MIPS with gigabit ports and PPC based ones like the RB850gx2. what i mean by reliability is that do a functional test but at much higher speeds such as how well would it cope with many upnp sessions popping up simultaneously and so on while it is under load.
     
  16. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    UTM testing is beyond the scope of what I or CDRouter can do. You need a threat/malware "zoo" and very different test processes. This is best left to the labs that focus on this work.
     
  17. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    Not likely anytime soon. There are too many other process and site changes I am working on now.
     
  18. sfx2000

    sfx2000 Part of the Furniture

    Joined:
    Aug 11, 2011
    Messages:
    13,990
    Location:
    San Diego, CA
    I think the test cases are more than enough - and to SEM's question/comment regarding the uTik - my best guess would be that the test equipment vendors used it, along with others to validate the test scripts.

    Would be interesting to see the results, but I expect NDA's would perhaps prevent disclosure in case anything went wrong with the device during test case/test plan development.

    This is all good stuff...
     
  19. System Error Message

    System Error Message Part of the Furniture

    Joined:
    Oct 14, 2014
    Messages:
    4,078
    But i wanted to buy one, or an x86 software version i could use.
     
  20. thiggins

    thiggins Mr. Easy Staff Member

    Joined:
    May 18, 2008
    Messages:
    13,792
    I'm sure if you contact the QACafe folks, they'll be happy to quote you one. But expect at least a five figure price. As has been said, this is test equipment for manufacturers and labs and is priced accordingly.