A very strange problem with GT-AX6000 and QNAP QSW-1105-5T switch

[Moan]

Hi guys. I got a new Asus ROG Rapture GT-AX6000 router. My network is the following:

GT-AX6000:

• WAN: 1Gb/s download / 100Mb/s upload
• eth2: Old NAS (1Gb/s)
• eth5: Unmanaged QNAP QSW-1105-5T switch (2.5Gb/s)

QNAP QSW-1105-5T:

• eth1: GT-AX6000 (2.5Gb/s)
• eth2: New NAS (2.5Gb/s)
• eth3: My PC (2.5Gb/s)

I noticed, that I can't get the full speed of 2.5Gb/s on My PC when I'm copying files from the Old NAS, New NAS and doing a speedtest at the same time. It maxes at ~1.6Gb/s, and either slows down the transfer of one of the NASes, or doesn't give the full speed on speedtest.
After trying dozens of different things, I noticed that the problem starts when I power cycle the QNAP QSW-1105-5T switch (unplug / plug the power). Then, data transfer from the New NAS to My PC goes by the GT-AX6000 router, instead of directly from the Switch.
Router restarts, PC restarts, multiple switch restarts, or executing ethctl eth5 phy-power down in Router SSH doesn't fix this issue.

I only managed to fix it by rebooting the New NAS. So after I reboot the new NAS, I am getting the full speed of 2.5Gb/s on my PC. Unplugging the New NAS ethernet for 1 minute doesn't fix this issue - only the full reboot.
Then, after the switch power cycle, after some GT-AX6000 reboot, or after some time - the bug starts occurring again, and the transfer from NAS 2 always goes by the router instead of Switch -> PC.

This was not happening on my old (bad) router: Xiaomi AX6000 with 2.5Gb/s LAN port.
I'm really confused and I don't know how to test it. Is it possible that it's some bug in the firmware? I'm using the newest Merlin: 388.2_2_rog.

I would very appreciate any help.

 

[L&LD]

Welcome to the forums @Moan.

Put the Old NAS on the QNAP QSW-1105-5T switch instead of on the router.

 

[Moan]

Welcome to the forums @Moan.

Put the Old NAS on the QNAP QSW-1105-5T switch instead of on the router.

Thank you for the answer, but it's not a solution. I use two more ports of the QNAP switch for other devices. Since the old NAS is only 1Gb/s it would be a slot waste, and such an expensive router like GT-AX6000 should theoretically route from it with no issues. Putting the old NAS to the QNAP switch could workaround the speed issues, only in this rare scenario when I would be copying files from both NAS devices at the same time, but I still wouldn't be able to copy from the new NAS and download something from Internet with the full 1Gb/s speed at the same time.

Anyway, I found a workaround for this issue in my new NAS. The new NAS is a self-built Odroid device with an external USB3.0 2.5Gb ethernet. I noticed, that I can workaround this issue by re-building the realtek-r8152 drivers. Since ip link set eth1 down; sleep 5; ip link set eth1 up and other adapter restart commands didn't work, I noticed that when the realtek-r8152 drivers are being reinstalled, it unbinds the USB device. After hours of testing, I came up with a solution of monitoring the adapter status, and restarting it when the link is down:

#!/bin/bash
LOOP_SLEEP_TIME=10
PRIMARY_INTERFACE="eth1"

b_ScheduledUSBReset=false
while true
do
    upSeconds="$(cat /proc/uptime | grep -o '^[0-9]\+')"
    #Reset USB device on disconnect to fix switch problems:
    ETH1UP=$(cat /sys/class/net/$PRIMARY_INTERFACE/operstate)
    if [ $ETH1UP == "up" ]; then
        if [ "$b_ScheduledUSBReset" = true ]; then
            b_ScheduledUSBReset=false
            echo "$PRIMARY_INTERFACE reconnected. Resetting the USB device to fix 2.5G switch routing...";
            USB_Device=$(ls /sys/class/net/eth1/device/driver | head -n 1)
            Module=$(readlink -f /sys/bus/usb/devices/$USB_Device/driver/module)
            Final_Link=$Module/drivers/usb:r8152
            echo $USB_Device > $Final_Link/unbind; sleep 5; echo $USB_Device > $Final_Link/bind
        fi
   
    else
    if [ $ETH1UP == "down" ]; then
            if [ "${upSeconds}" -gt "120" ]; then
            echo "$PRIMARY_INTERFACE is down. Scheduling USB reset on next connect.";
            b_ScheduledUSBReset=true
            else
            echo "$PRIMARY_INTERFACE is down. Uptime < 120 seconds, so won't schedule USB reset yet.";
            fi
    fi
    fi
   
# Other monitoring functions...

sleep $LOOP_SLEEP_TIME
done

So this way, when my switch (and perhaps the router) ever restarts, the 2.5Gb USB adapter on my NAS will be unbinded and binded again. The hard part was calling the unbind / bind to the correct location of the driver, not locations that disappear after device is unbinded and then can't be binded again.

I still would like to know why I even need to use a workaround of this type. I thought that routers and switches are smarter than that...

 

[L&LD]

I believe it would be a solution. In your case, a second QNAP QSW-1105-5T switch may be needed to provide you with (all of) the faster ports you require.

I have two QNAP QSW-1105-5T switches too, without any of the issues you describe.

I'm not using USB adaptors though.

Glad you've found a workaround for your specific issue and constraints.

 

[Moan]

I believe it would be a solution. In your case, a second QNAP QSW-1105-5T switch may be needed to provide you with (all of) the faster ports you require.

It could be a workaround for the speed issue when copying from both NAS devices at the same time, but definitely not a solution.

The problem is, after the switch restart, the traffic from my new NAS goes in the wrong way:
New NAS -> QNAP Switch -> Asus Router -> QNAP Switch -> My PC.
When I start copying files, I get max. 1.6Gb/s, because the 2.5Gb port in Asus gets overloaded (incoming and outgoing traffic on that port at the same time). So besides that, there is not even throughput for the Internet traffic, because the 2.5Gb router's port is in full-use. This is why it's not a solution.

When I "fix" the bug by rebinding the USB ethernet, the transfer from the New NAS goes with the full speed of ~2Gb/s, so the transfer way is:
New Nas -> QNAP Switch -> My PC.

And then, there is even a throughput for the Internet traffic, since the 2.5Gb/s port in Asus is not longer being overloaded.

I'm just wondering why I didn't notice this problem with my old router. Maybe there's some option on ASUS routers that optimizes how switches work? It's a really strange issue. I'm glad I workarounded it with the script, but I think it would still be good to find the root cause of this problem.

 

[L&LD]

There is no way for it to go in the 'wrong way'. Are you using two ports on the router to the switch?

Can you give us a diagram of your network connections?

 

[MDM]

I suppose that the Asus Router is doing the DHCP and routing, and all must pass trough him.
So maybe a solution to you would be a Managed switch..?

 

[L&LD]

LAN doesn't need routing. The switch is capable of 12.5 GB/s non-blocking throughput. With a 16K MAC address table. Managed switch is not needed here.

IMO, the equipment is set up incorrectly.

 

[Moan]

There is no way for it to go in the 'wrong way'. Are you using two ports on the router to the switch?

Can you give us a diagram of your network connections?

The switch is only connected with 1 eth (2.5Gb) port of the router.

What diagram exactly should I give? The list of what devices are connected to which port, as in my first post?

The bad news is, my "workaround" script stopped working, or rather works randomly when I'm testing it. It's a very confusing issue to be honest

 

[L&LD]

A diagram of your network layout would be very helpful.

Worth more than 1000 words.

 

[Moan]

A diagram of your network layout would be very helpful.

Worth more than 1000 words.

There's really nothing special:

Theoretically, My PC should get transfer directly from the New NAS. Unfortunately, for some reasons it goes by the GT-AX6000 router until I unbing/bind ethernet adapter and it fixes itself in like 1 time per 4 attempts of binding/unbinding. Disconnecting the physical ethernet cable from switch also doesn't fix it. I don't know if there's any possibility to track how these packets are routed

Does anybody know what's responsible for announcing device's presence on the network? Like how does the switch know the shortest route to the client? Yesterday I thought that it's "avahi-daemon" responsible for it, but restarts / reconfiguration of it didn't give any effect.

EDIT: And I forgot to mention that the same issue is possible to reproduce on my laptop (also using Ethernet 2.5G adapter). The New NAS simply routes all data using the Asus router, instead of the Switch, until I manage to fix it by physical USB disconnects / unblindings, etc.

EDIT2: I recorded a video to show the issue: https://mega.nz/file/gM9BCRga#Xqj2DkdeVaitGvGRXYg-_b8ix54bvfHrR5kG3saGfnc
In this video, the Ethernet connection disappears when I turn off / turn on the QNAP switch. The thing to observe is the second transfer. It's from the Old NAS (connected to the Router directly). When the New NAS transfers using the router, the speed of the other one drastically drops. It's because all the traffic from the New NAS goes to the Router, and then is being sent back with the same 2.5G port, which then has no throughput for the Old NAS transfer nor Internet connection.
In the video I was typing the commands to reconnect the Ethernet device on the New NAS. As you can see, compiling the realtek-r8152 drivers seems to make a trick, but only when the Switch is running for a bit longer. This is why my automated watchdog doesn't always seem to workaround this issue. Either way, I shouldn't even need to use these workarounds. The switch and the router should automatically establish the best routes...

 

[ColinTaylor]

Does anybody know what's responsible for announcing device's presence on the network? Like how does the switch know the shortest route to the client?

Hosts use ARP to announce or discover the MAC addresses of devices on the LAN. A switch learns the MAC addresses of devices connected to it's ports by examining the ethernet frames and uses that to populate it's MAC table (which maps MAC addresses to ports). If a switch needs to send a frame to a device it doesn't know about it sends that frame out of all ports and then updates it's MAC table when it sees the reply.

How are you determining that when this problem occurs the traffic is going via the router? Are you just assuming that based on the transfer speeds or are you seeing something on the router?

 

[Moan]

How are you determining that when this problem occurs the traffic is going via the router? Are you just assuming that based on the transfer speeds or are you seeing something on the router?

I'm assuming it based on the reduced speed from the old NAS and the Internet speed (speedtest reaches very low values then). For me it looks like the Asus router's 2.5G LAN port is accepting the transfer from the New NAS, and then sending to My PC using the same port. This causes that this port gets overloaded in fullduplex, and there is not much speed left for the Old NAS, and for Internet from WAN. Other devices connected to the switch, which are attempting to speedtest, etc. also notice reduced speed (but the router makes it more "fair").

I'm not sure how else this could be tested?

 

[ColinTaylor]

I suggest you test the raw network transfer rate using iperf3. Run iperf3 in server mode on each of your NAS's with iperf3 -s -i 10. Then open two windows on your PC and start two continuous, multi-stream transfers from the servers to the PC.

iperf3 -c oldnas -t 0 -R -P 8 -i 10

iperf3 -c newnas -t 0 -R -P 8 -i 10

Observe the combined [SUM] transfer rates of each window.

 

[sfx2000]

I noticed, that I can't get the full speed of 2.5Gb/s on My PC when I'm copying files from the Old NAS, New NAS and doing a speedtest at the same time. It maxes at ~1.6Gb/s, and either slows down the transfer of one of the NASes, or doesn't give the full speed on speedtest.

Can you be specific on the following - vendors and model numbers - I have not seen this mentions on the thread...

1) old NAS
2) new NAS

This can be helpful for folks trying to debug...

 

[sfx2000]

I suggest you test the raw network transfer rate using iperf3. Run iperf3 in server mode on each of your NAS's with iperf3 -s -i 10. Then open two windows on your PC and start two continuous, multi-stream transfers from the servers to the PC.

This assumes OP can install iperf3 on old/new NAS boxes...

But there, I do agree - throwing multiple parallel threads should drive traffic to max...

 

[Moan]

I suggest you test the raw network transfer rate using iperf3. Run iperf3 in server mode on each of your NAS's with iperf3 -s -i 10. Then open two windows on your PC and start two continuous, multi-stream transfers from the servers to the PC.

iperf3 -c oldnas -t 0 -R -P 8 -i 10

iperf3 -c newnas -t 0 -R -P 8 -i 10

Observe the combined [SUM] transfer rates of each window.

Unfortunately, I'm unable to install iperf3 on the old NAS. I enabled the iperf3 daemon on the Asus router itself. It was giving me max. 1Gb/s, but I then noticed that QOS was accidentally running. After disabling QOS I am getting max. 1.7Gb/s from the router with iperf.

Iperf obviously gives a bit better speeds than SMB, however the bug is still reproducible:


Here, I am copying a file from the New NAS only, and performing iperf3 with the router (192.168.1.1). As you can see, it maxes at ~1.8Gb/s, while it should give full 2.5Gb/s in this case:

Here, I'm performing iperf test from the router only, and as you can see it gives 1.7Gb/s:

And here, I'm performing iperfs from the New NAS (left CMD window) and from the router (right CMD window) at the same time:

I'm 99.9% sure, that the transfer goes by the Asus router, instead of directly to My PC.

Obviously, when I fix it with the "workaround" USB reset (that not always works), I'm easily getting full 2.5Gb/s.

Can you be specific on the following - vendors and model numbers - I have not seen this mentions on the thread...

1) old NAS
2) new NAS

This can be helpful for folks trying to debug...

The Old NAS is an old WD My Cloud device. But at this point it doesn't matter, since I noticed that the transfers from the router directly (and Internet) are also affected by it. So I'm using the Old NAS only to make the testing process easier, but the problem is something else with the New NAS.

The New NAS is an Ondroid N2 device. I have kernel 6.3.11-arm64 on it, and it's "Debian GNU/Linux 11 (bullseye)". To get the 2.5Gb/s port, I use external USB RTL8156B device. I own two of these (they're from different manufacturers, and they look different, but RTL8156B is the same) and both of them are affected by the same bug.


I was actually assuming that my New NAS might be an issue (especially because my "workaround" fixes scripts are able to sometimes fix it. However, it's so weird... The switch and the router should do all the job themselves... It makes no sense to physically remove the Ethernet USB adapter, and it makes no sense that even unplugging/plugging the ethernet cable doesn't fix this bug.

 

[ColinTaylor]

I can't see how the traffic could go "by the Asus router", that makes no sense. The switch only understands ethernet frames and MAC addresses. The switch should know what port your PC is connected to. If it forgot that mapping (because you power cycled the switch) as soon as it came back online it would flood all the ports, your PC would answer, and it would have re-learnt that mapping. The only way traffic would go by the router is if there were some other network connection to the router that I'm not aware of. For example, a Wi-Fi device that's also connected by ethernet.

 

[ColinTaylor]

I'm a bit confused by your screen shots. Are all of then taken when the problem is occurring?

What happens if you just run iperf from the router to the PC and nothing else? Do you get 2.5Gbps or only 1.8Gbps?

 

[Moan]

Yes, all screenshots are when the problem is occurring.

When I run iperf3 from the router to PC, I only get 1.8Gb/s max. I suppose iperf is not working so well on the Asus itself? Is anyone actually getting the full 2.5Gb/s from the router with iperf only?

I can't see how the traffic could go "by the Asus router", that makes no sense. The switch only understands ethernet frames and MAC addresses. The switch should know what port your PC is connected to. If it forgot that mapping (because you power cycled the switch) as soon as it came back online it would flood all the ports, your PC would answer, and it would have re-learnt that mapping. The only way traffic would go by the router is if there were some other network connection to the router that I'm not aware of. For example, a Wi-Fi device that's also connected by ethernet.

Exactly. It makes no sense. I thought that such issues would never occur, no matter how bad the switch is. And this one isn't so cheap (due to 2.5G ports).

So the only thing I can think of is that on the switch restart, Asus router somehow answers "yes, it's me!" and the switch adds it as the mapping? And then, after USB Ethernet disconnect it detects the adapter reset and it's trying to remap device to the correct port... And sometimes it works, and sometimes it doesn't (means Asus again responds that NAS 2 is on its own port?).

On the New NAS I'm also using Pihole, Nextcloud, Wireguard (obviously I also tested with disabled wireguard after fresh reboot). The original Ethernet port (1Gb/s) from the New NAS is unplugged, and this device has no Wi-Fi at all.
I have some other devices, such as the Repeater router in the other room (connected by wire to the Switch), but for the last few days I'm testing with all of them disconnected / turned off, to make sure something else is not confusing the connections. So the devices I listed in my network diagram are the all used when this problem is reproducible.

EDIT:
And here's a screenshot when the Switch bug is not present:

So easily getting 2.5Gb/s then.