1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

10Gbe to 1Gb VERY slow transfer rate

Discussion in 'Switches, NICs and cabling' started by MaxLTV, May 19, 2019.

  1. MaxLTV

    MaxLTV New Around Here

    Joined:
    Jan 8, 2016
    Messages:
    7
    I'm pulling my hear out with this one. Hope someone can point me in the right direction. Luckily, I isolated the issue really well (had to bring equipment physically into the same room and connect to the same router).

    Components in question - 2 different Asus 10 Gbe NICs (same model) and 2 different Netgear GS110EMX switches and 3 Win 10 PCs. The problem is the same regardless of what specific hardware is used, so it's probably not faulty devices (moving around switches and NICs changes nothing). All firmware, drivers etc. updated to the latest, cables certified etc.

    In a nutshell, when a 10Gbe (copper) NIC is connected to a 10Gbe port on a switch and is sending data in such a way that it passes through anything that's not 10Gbe or any form of multigig, the speed drops to 100-500Mbit (fluctuating up and down between these two boundaries). Receiving data from 1Gb devices in the same set up is fine (full 1Gb).

    In more details, the simplest chain:
    10Gbe NIC -> 10Gbe port on a switch -> 10Gbe or 1Gb port (does not matter) on the same switch -> 1Gb NIC - results in transfer speed floating between 100 and 500Mbit. The speed in reverse direction (From 1Gbe to 10Gbe) with the same components is normal maxing out 1Gbit NIC.

    Similarly, 10Gbe NIC -> 10Gbe Switch ->1Gb switch -> 10Gbe switch -> 10Gbe NIC results in slowness in both directions (way sub 1Gbe speeds). Removing 1GB thing in the middle, makes it all work at full 10Gbe transfer speed. Forcing 10Gbe components to 1Gb mode results in normal 1Gb speed even with the same 1Gb switch in the middle.

    When same 10Gbe NICs are forced to 1Gb connection speed or connected to 1Gb ports, the speed is stable maxed-out 1Gb (so no problem)

    10Gbe components connected to each other (10 Gbe Nic -> 10Gbe switch -> 10Gbe Nic) results in stable full-speed 10Gbe. So it's not faulty cables or components.

    So the problem is 99% definitively the 10Gbe -> 1Gb transfer.

    I think that 10Gbe NIC may be sending packets or something else of some sort that trips up 1Gb equipment. I disabled Jumbo packets, just in case. It seemed to make things a little faster but hard to say.

    Any ideas? Please help....
     
  2. MaxLTV

    MaxLTV New Around Here

    Joined:
    Jan 8, 2016
    Messages:
    7
    Plot thickens - iPerf-reported speed is normal in all cases. The slow speed affects file transfers though.
     
  3. coxhaus

    coxhaus Part of the Furniture

    Joined:
    Oct 7, 2010
    Messages:
    2,688
    Location:
    texas
    Could 1 of your machines be the limiting factor, not the network. Obviously iPerf is testing also but is there something different, maybe hard drive access?
     
  4. MaxLTV

    MaxLTV New Around Here

    Joined:
    Jan 8, 2016
    Messages:
    7
    Does not look like it. When I plug two machines into 10Gbe sockets on a switch, file transfers are 650MB/s both ways. So the machines are capable. When one is plugged into 1Gb slot, transfers are 120MB/s from the slow socket machine (normal) but 20-60MB/s from the fast socket one, fluctuating up and down gradually. When both in slow slots, I get expected 120MB/s in both directions.
     
  5. MichaelCG

    MichaelCG Very Senior Member

    Joined:
    Jan 4, 2017
    Messages:
    518
    Location:
    Central US
    So iPerf worked at the expected speeds? And that is all UDP traffic? Just TCP traffic flows are causing issues....so is there an MTU and/or TCP window size problem haunting you?
     
  6. MaxLTV

    MaxLTV New Around Here

    Joined:
    Jan 8, 2016
    Messages:
    7
    TCP in iPerf is normal too - 6.5Gbit between two 10Gbe clients and 960Mbit in any other combination, including the one that has file transfer speed issues. So IPerf is always normal, but file throughput is not. I tried manually setting TCP windows just in case, but I'm not sure if I did it right because it changed nothing. The symptom is kind of similar to duplex mismatch but I do not think duplex mismatch is possible in this set up.
     
  7. MichaelCG

    MichaelCG Very Senior Member

    Joined:
    Jan 4, 2017
    Messages:
    518
    Location:
    Central US
    This sounds like a switch issue then. You have proven both sides of the transaction can handle 10Gbps speeds. You have proven when on the same side of the switching fabric, 10Gbps or 1Gbps, the clients behave. It is only when crossing the 10Gbps to 1Gbps on the switch itself you see the issue.

    Do you have another switch to test with?
     
    L&LD likes this.
  8. ColinTaylor

    ColinTaylor Part of the Furniture

    Joined:
    Mar 31, 2014
    Messages:
    8,548
    Location:
    UK
  9. MaxLTV

    MaxLTV New Around Here

    Joined:
    Jan 8, 2016
    Messages:
    7
    I fixed it by accident. I disabled Receive Side Scaling in driver settings of the NIC and now everything works fine, with all the other advanced features and speeds in line with what's expected. I do not understand how/why it worked but the solution worked for two different PCs and for both Aquantia generic and Asus-branded drivers. Stumbling upon this fix involved a bunch of other steps, where I tried another switch and was able to almost fix it with Global Flow Control mode enabled on that switch (speed got to 95% of expected) but then disabling RSS fixed everything even without Flow Control (which is not a good thing to have set globally). Another thing that seems buggy is the Receive Side Coalescing Threshold - the default "adaptive" really slows things down on 10Gb and 5Gb connections but setting it to Off, Low or Moderate returns everything to normal.
     
    Samir and L&LD like this.