OpenVPN Client UDP connection issues (solved)

[Val D.]

Provider: NordVPN
ISP Speed: 250/20Mbps (246/19Mbps real)
OpenVPN TCP Speed: ~80/18Mbps
OpenVPN UDP Speed: ~220/18Mbps

Everything is configured properly and runs as expected except one thing:

220Mbps UDP connection king of chokes up with multiple open connections (torrents, for example). Once connections reach 700+ all downloads slow down to 1MB/sec total, YouTube doesn't want to play 1080p videos anymore, Web pages open with visible delay, etc. 80Mbps TCP connection on the other hand has no issues even with 2000+ connections and everything runs smooth as butter with download speeds reaching >8MB/sec with YouTube playing 1080p video in background. What is going on here?

What I have tried already:

- playing with sndbuf and rcvbuf values in OpenVPN
- attempt to change the cipher settings (server always reverts to AES-256-GCM)
- Inbound Firewall Allow/Block in OpenVPN
- changing the VPN server to a different one
- playing with QoS settings, enable/disable
- playing with DNS servers, DNS-over-TLS enable/disable
- router firewall enable/disable (for testing purposes only)
- wired/wireless connection to router (all AC clients 433-866Mbps)
- running VPN software on a PC (Intel i5 4-core CPU)

None of the above makes any difference.

The only suspicious thing is System Log when using UDP is the following:

Aug 14 20:18:28 ovpn-client1[32655]: AEAD Decrypt error: bad packet ID (may be a replay): [ #304436 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

But not too many records, every now and then. No errors whatsoever on TCP connection.
Knowing how TCP and UDP works, not surprised.

ISP Router: Hitron CGN3 in Bridge Mode (Puma 6, unfortunately)
Hardware: ASUS RT-AC86U with active cooling (CPU at 55C under max load)
Firmware: Asuswrt-Merlin 384.13
Custom scripts: None
AiProtection: Disabled
AiMesh: No
Clients: Windows 10 PCs, Android Phones/Tablets, iOS Phones/iPads, Xbox One, VoIP ATA

I know I'm tired and I'm missing something, but some fresh ideas will be greatly appreciated.
Please, no Reset/Erase/Nuke/Smash the router suggestions, this won't help. Thank You!

@RMerlin, @Xentrk, @skeal, @ColinTaylor, @thiggins, @sfx2000 , @Grisu

 

[Val D.]

More testing, I realized it could be MTU settings:

- tun-MTU size increase/decrease
- mssfix size increase/decrease

Unfortunately, same errors as above.
I found about 50 consecutive at one moment. The bigger the load on the line is, the more errors. No errors (or very few) in light Web browsing or watching YouTube on 2-3 computers. Streaming doesn't affect the line. Looks like torrent clients are literally shutting down the UDP connection. No indication of port filtering from ISP (port 1194). Investigation continues.

 

[Val D.]

OK, fixed!
Took some time though. I was looking in a wrong place.

µTP protocol in torrents client is causing the issue.
Select TCP only connection in torrents client and all goes back to normal.

Downloads are flying with >20Mbps, the rest of the Internet traffic is all good, no errors in System Log.