Bug in CTF?

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

sbsnb

Very Senior Member
Through much trial and error I found what seems like a bug involving CTF. The symptoms appear to suggest that timestamps on some packets are incorrect when CTF is enabled. This has side effects for software that use network time stamps to get the time, like NTP and Chrony. It may also be affecting my dnscrypt-proxy2, but I'm not positive of that yet.

The bug can be verified by enabling CTF and running tcpdump on the router. What you capture isn't important, just look at the timestamps. Some of the packets captured will have borked timestamps like this (see the packets with 00:00:00.xxxxxx timestamps):

Code:
00:00:00.857443 IP (tos 0x0, ttl 128, id 34890, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.127934 IP (tos 0x10, ttl 64, id 52196, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.137884 IP (tos 0x10, ttl 64, id 52197, offset 0, flags [DF], proto TCP (6), length 360)
00:00:00.857436 IP (tos 0x0, ttl 128, id 34891, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.147923 IP (tos 0x10, ttl 64, id 52198, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.157886 IP (tos 0x10, ttl 64, id 52199, offset 0, flags [DF], proto TCP (6), length 360)
00:00:00.857501 IP (tos 0x0, ttl 128, id 34892, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.167924 IP (tos 0x10, ttl 64, id 52200, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.177882 IP (tos 0x10, ttl 64, id 52201, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.180166 IP (tos 0x0, ttl 128, id 34893, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.187929 IP (tos 0x10, ttl 64, id 52202, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.197880 IP (tos 0x10, ttl 64, id 52203, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.200431 IP (tos 0x0, ttl 128, id 34894, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.207930 IP (tos 0x10, ttl 64, id 52204, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.210278 IP (tos 0x0, ttl 128, id 34895, offset 0, flags [DF], proto TCP (6), length 200)
14:59:42.210728 IP (tos 0x10, ttl 64, id 52205, offset 0, flags [DF], proto TCP (6), length 664)
14:59:42.217892 IP (tos 0x10, ttl 64, id 52206, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.219777 IP (tos 0x0, ttl 128, id 34896, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.227916 IP (tos 0x10, ttl 64, id 52207, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.237886 IP (tos 0x10, ttl 64, id 52208, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.240920 IP (tos 0x0, ttl 128, id 34897, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.247936 IP (tos 0x10, ttl 64, id 52209, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.257884 IP (tos 0x10, ttl 64, id 52210, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.259790 IP (tos 0x0, ttl 128, id 34898, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.267924 IP (tos 0x10, ttl 64, id 52211, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.277880 IP (tos 0x10, ttl 64, id 52212, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.280594 IP (tos 0x0, ttl 128, id 34899, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.287922 IP (tos 0x10, ttl 64, id 52213, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.297886 IP (tos 0x10, ttl 64, id 52214, offset 0, flags [DF], proto TCP (6), length 360)
00:00:00.857287 IP (tos 0x0, ttl 128, id 34900, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.307916 IP (tos 0x10, ttl 64, id 52215, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.317909 IP (tos 0x10, ttl 64, id 52216, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.320444 IP (tos 0x0, ttl 128, id 34901, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.327941 IP (tos 0x10, ttl 64, id 52217, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.337885 IP (tos 0x10, ttl 64, id 52218, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.339820 IP (tos 0x0, ttl 128, id 34902, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.347925 IP (tos 0x10, ttl 64, id 52219, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.357888 IP (tos 0x10, ttl 64, id 52220, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.361047 IP (tos 0x0, ttl 128, id 34903, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.367929 IP (tos 0x10, ttl 64, id 52221, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.377881 IP (tos 0x10, ttl 64, id 52222, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.379746 IP (tos 0x0, ttl 128, id 34904, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.387939 IP (tos 0x10, ttl 64, id 52223, offset 0, flags [DF], proto TCP (6), length 616)
14:59:42.397892 IP (tos 0x10, ttl 64, id 52224, offset 0, flags [DF], proto TCP (6), length 360)
14:59:42.400544 IP (tos 0x0, ttl 128, id 34905, offset 0, flags [DF], proto TCP (6), length 40)
14:59:42.406566 IP (tos 0x0, ttl 128, id 34906, offset 0, flags [DF], proto TCP (6), length 104)
14:59:42.407020 IP (tos 0x10, ttl 64, id 52225, offset 0, flags [DF], proto TCP (6), length 872)

These packets impair the functioning of NTP and/or Chrony so that clients cannot sync if one of the packets in the transaction is borked. Disabling CTF eliminates the problem.

Anyway, I would like to get this reported to Asus so I can enable CTF and have those applications functioning without issue on my router, but I'm not sure how to check for the issue on stock firmware.
 

ColinTaylor

Part of the Furniture
What tcpdump command are you using the demonstrate this problem. It's perfectly normal for some incoming packets to have a 00:00:00 timestamp.
 

sbsnb

Very Senior Member
The one where I first noticed it was tcpdump -i br0 port 123 and host 192.168.1.2 -vvv to capture NTP traffic between a specific client and the router. However, even dropping the port specification and monitoring shows lots of packets with those timestamps. When CTF is disabled I don't see those timestamps on any packets.
 

RMerlin

Asuswrt-Merlin dev
CTF is a blackbox. Even Asus does not know what goes inside it, only Broadcom does.
 

sbsnb

Very Senior Member
Great :( I can't tell for sure if the bug is with CTF or if something interacts with CTF, but it's definitely involved. If it's CTF that means it will never get fixed. At least I know to test any future routers during the return period. It seems like someone would have noticed millions of devices with errant packet timestamps, but then again I would have said the same thing about the Intel bug that kills their Avoton C2*50 CPUs dead before I lost three embedded motherboards to that bug.
 

gotjazz

Occasional Visitor
Yeah but if it is reproducible it can be reported to Asus and it stands to reason Asus would report it to broadcom, right?
 

sbsnb

Very Senior Member
Maybe. There are a number of potential pitfalls along that path:

1. It has to be reproducible on stock Asus firmware. I'm not sure I know how to test for it on stock firmware. Maybe I can try to capture packets from the router on the other end of the exchange, but that introduces the potential for Asus to dismiss it as a network issue on my end.
2. Asus would then have to prioritize it enough that it would receive more attention than simply cataloging it as a known issue.
3. Asus would have to demonstrate to Broadcom that it was not an issue with their equipment.
4. Broadcom would have to prioritize it highly enough that it would receive more attention than simply cataloging it as a known issue.
5. Any potential fix would have to be made for devices as old as mine.
6. Asus would have to incorporate Broadcom's fix in a GPL release before it made it to Merlin.

I'm not confident of even getting past #1.
 

dave14305

Part of the Furniture
It seems like someone would have noticed millions of devices with errant packet timestamps
You only observe this with third-party utilities. So it’s not likely to be noticed by most Asus users. And I agree wholeheartedly that this will never get fixed because all you really know is that it doesn’t happen when CTF is disabled. But other things like Traditional QoS or some Parental Controls also don’t work when CTF is enabled, but that is by design or known limitations of CTF.

It’s possibly a bug in Entware’s libc or whatever library they use to interact with the OS.

Safe bet is to use Busybox NTP and enjoy life.
 

sbsnb

Very Senior Member
Safe bet is to use Busybox NTP and enjoy life.
My main concern is if this is also what's been plaguing my dnscrypt-proxy ever since I started using it. When I first started there were no OpenNIC servers using DOH or DOT, but now there are so I can conceivably ditch that too.

I just hate to have to make the tradeoff between Entware utilities and getting good network speeds. I guess next time around it makes sense to get a very lightweight AP and build myself a proper router with decent hardware.
 

ColinTaylor

Part of the Furniture
I just hate to have to make the tradeoff between Entware utilities and getting good network speeds. I guess next time around it makes sense to get a very lightweight AP and build myself a proper router with decent hardware.
Or just use the wireless router as a wireless router and put proxy and ntp servers on... well, a server maybe?
 

sbsnb

Very Senior Member
Or just use the wireless router as a wireless router and put proxy and ntp servers on... well, a server maybe?
Yeah. That may be the better option. I wonder if there's any money in building integrated solutions with enough horsepower to be a lightweight server and a router. Just switching from ARM to just about anything from Intel or AMD will massively boost horsepower without boosting power requirements much. You can get 12 cores at 2 GHz with a TDP of 25W in an Intel C3858, or 8 cores at up to 4.15 GHz with a TDP of 25W in a Ryzen V2718. Those are peak power requirements at 100% CPU, which is rare.
 

sfx2000

Part of the Furniture
This has side effects for software that use network time stamps to get the time, like NTP and Chrony. It may also be affecting my dnscrypt-proxy2, but I'm not positive of that yet.

No... I really doubt it.

NTP has the time information inside the packet itself, doesn't use the timestamp of the packet...


Chrony is designed to handle odd things quite well...

As far as CTF goes - that's all on Broadcom...
 

John Davis

Regular Contributor
Yeah. That may be the better option. I wonder if there's any money in building integrated solutions with enough horsepower to be a lightweight server and a router. Just switching from ARM to just about anything from Intel or AMD will massively boost horsepower without boosting power requirements much. You can get 12 cores at 2 GHz with a TDP of 25W in an Intel C3858, or 8 cores at up to 4.15 GHz with a TDP of 25W in a Ryzen V2718. Those are peak power requirements at 100% CPU, which is rare.

the new dual 2.5gbe lan 11th gen nucs look perfect for this - small, low tdp but plenty of horespower
 

sbsnb

Very Senior Member
No... I really doubt it.

NTP has the time information inside the packet itself, doesn't use the timestamp of the packet...


Chrony is designed to handle odd things quite well...

As far as CTF goes - that's all on Broadcom...
If they're not being thrown off by the timestamps, I wonder why they're both returning packets with received timestamps that happen to match the packet timestamps. AFAIK the NTP epoch is NOT the same as the Unix epoch. Since the timestamps have the beginning of the Unix epoch I'm assuming the error is originating outside of their timekeeping code.
 

RMerlin

Asuswrt-Merlin dev
See if you can detect it by running Wireshark on a LAN client. Otherwise, you can go with Optware-NG, which is used on stock firmware for Download Master.

I'm not sure if BCM still actively maintain any SDK that uses CTF, CTF was replaced with Flow Cache starting with the HND architecture introduced back in 2017.
 

sbsnb

Very Senior Member
I may try that. I also have access to a family member's RT-AX88U I will try and see if it has the same issue.

EDIT - FWIW it doesn't appear that the HND models are affected.
 
Last edited:

Tech9

Very Senior Member
You can get 12 cores at 2 GHz with a TDP of 25W in an Intel C3858

This thing will process 10G traffic no problem. My firewall is running on 4 cores Intel C3558, Gigabit ports. The entire unit is around 25W max.
 

dave14305

Part of the Furniture
@sbsnb, what happens if you try the nvram variable discussed in this thread? It seems to somehow let udp packets (like NTP) sidestep CTF. But only for LAN-originated packets, not router packets (i.e. FORWARD table, not OUTPUT table).

Just a thought...but as I type it, I doubt it would help the router's NTP daemon. But I'm hoping you're desperate enough to try it. :rolleyes:

Code:
nvram set ctf_pt_udp=1
nvram commit
service restart_firewall
conntrack -F

 

ColinTaylor

Part of the Furniture
Just a thought...but as I type it, I doubt it would help the router's NTP daemon. But I'm hoping you're desperate enough to try it. :rolleyes:
I don't know whether it's relevant to what you're discussing but the original thread about UDP was regarding loopback traffic. The solution there additionally involved moving that rule to the PREROUTING chain (which John then implemented in his firmware).

http://www.snbforums.com/threads/potential-bug-with-udp-nat-loopback-hairpinning.70892/post-670283
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top