Baby ping spikes with XT8 router

[TiimJiim]

While investigating a packet loss issues (now resolved) it came to my attention that my XT8 quite frequently adds up to about 20ms to ping of the ISP gateway (using ethernet). This does not occur with another router, a Mikrotik Hap AC2. Originally I noticed this with the stock firmware, but I've since installed the XT8 compatible Merlin fork to see if that was any better (it wasn't).

Taking the fibre-ISP link out of the equation, if I connect to the XT8 to the Mikrotik (instead of the ONT) and ping the Mikrotik:
1000 packets transmitted, 1000 received, 0% packet loss, time 202620ms
rtt min/avg/max/mdev = 0.356/1.253/24.806/3.092 ms

Replacing the XT8 with the Mikrotik and pinging the ISP gateway produces substantially better results, and this also includes the fibre link:
rtt min/avg/max/mdev = 0.327/1.085/8.231/0.316 ms

I have all the Trend Micro stuff disabled (so no QoS, etc) so isn't the packet routing handled using HW acceleration independent of CPU (not that the CPU is particularly loaded)? Even if there's CPU involvement, 20ms is long delay.

Obviously it's not the end of the world, but does anyone else think this is a bit rubbish?

Thanks in advance to anyone who has any insight.

 

[L&LD]

Welcome to the forums @TiimJiim.

After flashing RMerlin firmware, did you do a full reset to factory defaults?

XT8: Need advice regarding hard reset following firmware upgrade

I'm hoping an expert out there can help. I'm currently running 2 XT8's, one set up as router and the 2nd as a node, via wireless backhaul. The system is running essentially flawlessly for my purposes on firmware 42095. However, because this firmware is now somewhat out of date, I am considering...

www.snbforums.com

 

[TiimJiim]

Welcome to the forums @TiimJiim.

After flashing RMerlin firmware, did you do a full reset to factory defaults?

XT8: Need advice regarding hard reset following firmware upgrade

I'm hoping an expert out there can help. I'm currently running 2 XT8's, one set up as router and the 2nd as a node, via wireless backhaul. The system is running essentially flawlessly for my purposes on firmware 42095. However, because this firmware is now somewhat out of date, I am considering...

www.snbforums.com

Thanks for the reply. I did not factory reset since coming from the latest stock firmware that didn't seem to be strictly necessary* going from stock to RMerlin and it's a faff to reconfigure from scratch.

To be clear, it was doing this with the stock firmware, I speculatively installed RMerlin in the faint hope:
- it fixed the latency outright.
- it exposed some settings that might help.
- it helped to isolate the cause.

Other than the jitter, everything works flawlessly pre and post RMerlin.

I take it you don't think it's reasonable for a router to inject 20ms of latency to some packets then?

I guess I can save my settings, factory reset, run a test and restore if it didn't help.

* From the wiki: "While it is generally not necessary to restore to factory defaults, it's not a bad idea, especially if there is a big jump in version number (e.g. from 112 to 178). "

 

[tgl]

I take it you don't think it's reasonable for a router to inject 20ms of latency to some packets then?

There are various reasons for a wireless connection to suffer latency spikes. Some of them you can cure by adjusting settings. Start by reading this excellent article:

Wi-Fi Ping Spikes: Causes and Fixes

The next thing I'd wonder about is whether the XT8 and Mikrotik are using the same channel, and if not whether there's more interference on the one the XT8 is using. If there's any nearby WiFi nets sharing your channel, that is going to cause all sorts of latency problems for both networks, because only one can transmit at a time.

 

[TiimJiim]

There are various reasons for a wireless connection to suffer latency spikes. Some of them you can cure by adjusting settings. Start by reading this excellent article:

Wi-Fi Ping Spikes: Causes and Fixes

The next thing I'd wonder about is whether the XT8 and Mikrotik are using the same channel, and if not whether there's more interference on the one the XT8 is using. If there's any nearby WiFi nets sharing your channel, that is going to cause all sorts of latency problems for both networks, because only one can transmit at a time.

Thanks for the reply. Yes, I'd expect WiFi to introduce some jitter but unfortunately this problem occurs using ethernet.

 

[tgl]

Thanks for the reply. Yes, I'd expect WiFi to introduce some jitter but unfortunately this problem occurs using ethernet.

Oh! You were not clear about that. Yeah, 20ms seems pretty far out of line if the router is just relaying between LAN ports. Do you have any features enabled that'd require the router to inspect packets' contents?

 

[TiimJiim]

Oh! You were not clear about that. Yeah, 20ms seems pretty far out of line if the router is just relaying between LAN ports. Do you have any features enabled that'd require the router to inspect packets' contents?

Sorry for the confusion! I have the obvious packet inspection features disabled, and have tried fiddling with what's left. There's no option to disable the real-time traffic monitor, but the following are disabled:
- AiProtection
- Parental Controls
- QoS
- AiCloud
- USB Application/*

I also tried disabling the firewall temporarily, but that didn't help. I also tried IPv6.

Disabling all the Wi-Fi radios did improve things (not fully) but it's also not really a very useful solution. I set all the Wi-fi control channels manually FWIW.

 

[tgl]

I did some experimentation with my own XT8, and got consistent ping times as long as the machine isn't too heavily loaded. Pinging from a machine connected to LAN1 to one connected to LAN3:

1000 packets transmitted, 1000 received, 0% packet loss, time 300619ms
rtt min/avg/max/mdev = 0.630/1.114/2.718/0.329 ms

From WAN to LAN3:

1000 packets transmitted, 1000 received, 0% packet loss, time 301008ms
rtt min/avg/max/mdev = 0.443/1.000/1.440/0.106 ms

But ... if I try this concurrently with a full-speed iperf3 run between the same two machines, I see ping spikes of up to 20 ms in the LAN1-LAN3 case (only when iperf3 is sending to LAN3) and up to 40 ms in the WAN-LAN3 case (only when iperf3 is receiving from LAN3). This has to be blamed on the XT8 and not any other network component, because nothing else would know whether that patch cable went to the LAN1 or WAN port. The iperf3 run itself shows a pretty nasty startup transient (note the retry column):

Accepted connection from 192.168.1.8, port 55710
[ 5] local 192.168.1.11 port 5201 connected to 192.168.1.8 port 55712
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 26.7 MBytes 224 Mbits/sec 541 269 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 268 368 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 382 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 382 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 0 383 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 383 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 383 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 383 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 383 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 383 KBytes
[ 5] 10.00-10.04 sec 3.75 MBytes 850 Mbits/sec 0 383 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 1.01 GBytes 863 Mbits/sec 809 sender

The ping latency spike only lasts about as long as that transient, too; it's fine for the rest of the run.

I don't see these behaviors when testing between the same two machines with only a couple of Netgear switches between them, which is additional evidence that the XT8 is to blame.

So for me, it's okay as long as I'm not maxing out the available bandwidth of the LAN port, but it seems like ASUS could stand to do some tuning of their packet-forwarding logic to deal better with that case.

This is with a 2021-vintage XT8 running firmware 42095 in Access Point mode. Getting back to your original question, I wonder if you are running the unit in router or AP mode.

 

[TiimJiim]

Thanks for the experimentation. I also noticed odd behaviour when running a broadband speed test one one machine while running ping from another.

Laptop ->100Mbps->} XT8 ->1Gbps-> Mikrotik ->1Gbps-> ONT
Server ->1Gbps----->}

The laptop was limited to 100mbps due to the cable and ran the speed test. While the download portion of the test was running, pings from the server to the Mikrotik shot up to about 60ms. Using a 1Gbps cable for the laptop only causes the ping to rise to 18ms despite the load from the test being 4x higher at 400Mbps. Obviously this is all quite unscientific and I can't really assign blame here, but slightly surprising.

To answer your question, I'm using my XT8 in router mode (as my primary router) and I have another as a mesh node using wireless backhaul. I turned the second one off for testing. I bought the XT8 about a year ago, now running 386.5_2-gnuton0. The Mikrotik was supplied by my ISP, I have been using it for comparison.

 

[tgl]

The laptop was limited to 100mbps due to the cable and ran the speed test. While the download portion of the test was running, pings from the server to the Mikrotik shot up to about 60ms. Using a 1Gbps cable for the laptop only causes the ping to rise to 18ms despite the load from the test being 4x higher at 400Mbps. Obviously this is all quite unscientific and I can't really assign blame here, but slightly surprising.

A possible explanation is that download traffic was stacking up at the server->Mikrotik link and the ping packets got stuck behind that queued data. Slower laptop link means more queued data = more ping delay. (My ping spikes might be explained by something like that too, perhaps.) But it's not very clear why things bottlenecked at that link and not somewhere else, nor indeed why there was a bottleneck at all. Usually the TCP algorithms try to ensure that the data source provides data at roughly the speed of the slowest link.

Can you take the XT8 out of the equation by repeating the same tests with the laptop plugged directly into the Mikrotik?

 

[TiimJiim]

"Can you take the XT8 out of the equation by repeating the same tests with the laptop plugged directly into the Mikrotik?"

The Mikrotik is worse with 100mbps and better with 1gpbs. I now think this is a distraction anyway, the ping only gets really high using my ISP speedtest, regular downloads and test sites do not flood the router with packets.

Going back to my original problem. The most likely cause of the ping spiking is that packets are being handled by the CPU rather than forwarded in hardware? Is this caused by traffic monitor or does HW acceleration provide this data? When I get longer spare moment I'll factory reset and see if the problem persists.