TAILMON Tailmon on RT-AX88u, can't ping any other tailnet machine

[sfatula]

So, I was running an AT88U and it got fried by lightning. So, I bought a used one, now have merlin 3004.388.9_2 and installed tailmon (kust like it was before on the fried router) via amtm. Did basic install so subnet routing defaulted to enabled. Enabled subnet routing as well on tailscale site. The tailnet shows the router is connected, advertising subnets. All looks good. From an iPhone, I can access any machine on my home network via Tailscale. So, subnet routers are working inbound, which is great as I am behind CGNAT. Running tailscale 1.84.0 via the update function of tailmon.

What I can not do is ping any machine connected to the tailnet from the router. I don't understand why. Other machines can reach the router just fine using the tailnet address. But I cannot reach out from the router. I turned on packet debugging as a last ditch effort and saw this (just one example, many drops):

May 28 00:00:26 kernel: DROP IN=eth0 OUT= MAC=a8:5e:45:af:dc:78:2c:c8:1b:7a:0b:a8:08:00 SRC=17.138.175.254 DST=100.94.227.36 LEN=52 TOS=0x04 PREC=0x00 TTL=49 ID=63680 DF PROTO=TCP SPT=443 DPT=57097 SEQ=2221355152 ACK=173534718 WINDOW=169 RES=0x00 ACK URGP=0 OPT (0101080A2D1B7D28BC8DE6AC) MARK=0x8000000

That is traffic destined for the tailnet. That might explain why I can't reach anything on the tailnet from the router perhaps? But I don't understand why that might be dropped. ip route shows this:

default via 100.94.224.1 dev eth0
8.8.4.4 via 100.94.224.1 dev eth0 metric 1
8.8.8.8 via 100.94.224.1 dev eth0 metric 1
100.94.224.0/22 dev eth0 proto kernel scope link src 100.94.227.36
100.94.224.1 dev eth0 proto kernel scope link
127.0.0.0/8 dev lo scope link
192.168.3.0/24 dev br0 proto kernel scope link src 192.168.3.1
192.168.4.0/24 via 192.168.3.2 dev br0 metric 1
192.168.5.0/24 via 192.168.3.2 dev br0 metric 1
192.168.8.0/24 via 192.168.3.2 dev br0 metric 1
192.168.101.0/24 dev br1 proto kernel scope link src 192.168.101.1
239.0.0.0/8 dev br0 scope link

And that is in fact correct from my ISP and a few static routes I have.

Why can I not ping out to a tailnet ip from the ASUS router, any ideas??

On another machine on my LAN, I also have tailscale installed, and it can ping machines on the tailnet just fine, and of course is behind the router that cannot. So, it appears possible at least.

Additionally, I find it stops running constantly. And it doesn't restart, even though it is configured to. Here's the cfg file for tailmon:

keepalive=1
timerloop=60
logsize=2000
autostart=1
amtmemailsuccess=0
amtmemailfailure=0
tsoperatingmode="Userspace"
persistentsettings=1
exitnode=0
advroutes=1
accroutes=0
precmd=""
args="--tun=userspace-networking --state=/opt/var/tailscaled.state --statedir=/opt/var/lib/tailscale"
preargs="nohup"
routes="192.168.3.0/24"
customcmdline=""

I will add, I do not see any screen process, or tailmon.sh running. And yes the command is in post-mount

 

[Viktor Jaep]

So, I was running an AT88U and it got fried by lightning. So, I bought a used one, now have merlin 3004.388.9_2 and installed tailmon (kust like it was before on the fried router) via amtm. Did basic install so subnet routing defaulted to enabled. Enabled subnet routing as well on tailscale site. The tailnet shows the router is connected, advertising subnets. All looks good. From an iPhone, I can access any machine on my home network via Tailscale. So, subnet routers are working inbound, which is great as I am behind CGNAT. Running tailscale 1.84.0 via the update function of tailmon.

What I can not do is ping any machine connected to the tailnet from the router. I don't understand why. Other machines can reach the router just fine using the tailnet address. But I cannot reach out from the router. I turned on packet debugging as a last ditch effort and saw this (just one example, many drops):

May 28 00:00:26 kernel: DROP IN=eth0 OUT= MAC=a8:5e:45:af:dc:78:2c:c8:1b:7a:0b:a8:08:00 SRC=17.138.175.254 DST=100.94.227.36 LEN=52 TOS=0x04 PREC=0x00 TTL=49 ID=63680 DF PROTO=TCP SPT=443 DPT=57097 SEQ=2221355152 ACK=173534718 WINDOW=169 RES=0x00 ACK URGP=0 OPT (0101080A2D1B7D28BC8DE6AC) MARK=0x8000000

That is traffic destined for the tailnet. That might explain why I can't reach anything on the tailnet from the router perhaps? But I don't understand why that might be dropped. ip route shows this:

default via 100.94.224.1 dev eth0
8.8.4.4 via 100.94.224.1 dev eth0 metric 1
8.8.8.8 via 100.94.224.1 dev eth0 metric 1
100.94.224.0/22 dev eth0 proto kernel scope link src 100.94.227.36
100.94.224.1 dev eth0 proto kernel scope link
127.0.0.0/8 dev lo scope link
192.168.3.0/24 dev br0 proto kernel scope link src 192.168.3.1
192.168.4.0/24 via 192.168.3.2 dev br0 metric 1
192.168.5.0/24 via 192.168.3.2 dev br0 metric 1
192.168.8.0/24 via 192.168.3.2 dev br0 metric 1
192.168.101.0/24 dev br1 proto kernel scope link src 192.168.101.1
239.0.0.0/8 dev br0 scope link

And that is in fact correct from my ISP and a few static routes I have.

Why can I not ping out to a tailnet ip from the ASUS router, any ideas??

On another machine on my LAN, I also have tailscale installed, and it can ping machines on the tailnet just fine, and of course is behind the router that cannot. So, it appears possible at least.

Additionally, I find it stops running constantly. And it doesn't restart, even though it is configured to. Here's the cfg file for tailmon:

keepalive=1
timerloop=60
logsize=2000
autostart=1
amtmemailsuccess=0
amtmemailfailure=0
tsoperatingmode="Userspace"
persistentsettings=1
exitnode=0
advroutes=1
accroutes=0
precmd=""
args="--tun=userspace-networking --state=/opt/var/tailscaled.state --statedir=/opt/var/lib/tailscale"
preargs="nohup"
routes="192.168.3.0/24"
customcmdline=""

I will add, I do not see any screen process, or tailmon.sh running. And yes the command is in post-mount

I just tried myself, and can't ping any other tailnet client from the router itself. I even tried using the br0 interface, as eth0 might be a bit more locked down. Interesting. I bet @ColinTaylor may have an answer for that.

Can you please verify that "screen" is installed? Also, what happens if you type "tailmon -screen"?

 

[XIII]

What I can not do is ping any machine connected to the tailnet from the router

I can ping my Raspberry Pi from my router (both in the same Tailscale network).

 

[sfatula]

Screen is installed. Running tailmon -screen does what you think, it runs it under screen.

 

[Viktor Jaep]

Screen is installed. Running tailmon -screen does what you think, it runs it under screen.

To exit screen without terminating tailmon -- CTRL A+ D

 

[sfatula]

To exit screen without terminating tailmon -- CTRL A+ D

Yes, I can do that, but the problem is reboots. I don't want to have to do that every single reboot. Wanting to solve why the startup process is not doing that as it should.

 

[Viktor Jaep]

Yes, I can do that, but the problem is reboots. I don't want to have to do that every single reboot. Wanting to solve why the startup process is not doing that as it should.

If things are working right, the post-mount should be starting it up under screen after a reboot. Once it's up, check by using "tailmon -screen", and should be running. Then CTRL A + D to exit. You should be able to see it under top/htop as well. Are you seeing anything in the syslogs that indicate a failure after a reboot?

 

[sfatula]

Yes, I know it should be running via post-mount. No errors in the log, I see the message for starting entware in the logs, but nothing about tailmon. Here's the post-mount script built for me:

#!/bin/sh
swapon /tmp/mnt/stick/myswap.swp # Added by amtm
. /jffs/addons/amtm/mount-entware.mod # Added by amtm

(sleep 30 && /jffs/scripts/tailmon.sh -screen) & # Added by tailmon

I thought of trying to adjust the sleep to see if that matters. Router is busy most of the day so hate to interrupt things.

 

[Viktor Jaep]

Yes, I know it should be running via post-mount. No errors in the log, I see the message for starting entware in the logs, but nothing about tailmon. Here's the post-mount script built for me:

#!/bin/sh
swapon /tmp/mnt/stick/myswap.swp # Added by amtm
. /jffs/addons/amtm/mount-entware.mod # Added by amtm

(sleep 30 && /jffs/scripts/tailmon.sh -screen) & # Added by tailmon

I thought of trying to adjust the sleep to see if that matters. Router is busy most of the day so hate to interrupt things.

Does your post-mount have execution permissions?

 

[sfatula]

Does your post-mount have execution permissions?

Yes, otherwise I wouldn't see the entware startup from post-mount. It has them.

 

[Viktor Jaep]

Yes, I know it should be running via post-mount. No errors in the log, I see the message for starting entware in the logs, but nothing about tailmon. Here's the post-mount script built for me:

#!/bin/sh
swapon /tmp/mnt/stick/myswap.swp # Added by amtm
. /jffs/addons/amtm/mount-entware.mod # Added by amtm

(sleep 30 && /jffs/scripts/tailmon.sh -screen) & # Added by tailmon

I thought of trying to adjust the sleep to see if that matters. Router is busy most of the day so hate to interrupt things.

This might be your problem... your swap and entware seem to be reversed compared to mine. Here's a copy of my post-mount:

#!/bin/sh
. /jffs/addons/amtm/mount-entware.mod # Added by amtm
swapon /tmp/mnt/ASUS-SSD/myswap.swp # Added by amtm
(sleep 30 && /jffs/scripts/vpnmon-r3.sh -screen) & # Added by vpnmon-r3
(sleep 30 && /jffs/scripts/tailmon.sh -screen) & # Added by tailmon

Not sure if it'll make a difference, but worth a try. Perhaps it's taking a long time to do one or the other, and doesn't give enough time to get things in order for when tailmon tries to start.

 

[sfatula]

So, for fun, reversed the order (I didn't write the script amtm did) and increased sleep to 45 seconds anyway on top of it. Rebooted, and...

# ps w | grep tail
1841 sfatula 1223m S tailscaled --tun=userspace-networking --state=/opt/var/tailscaled.state --statedir=/opt/var/lib/tailscale
2748 sfatula 3540 S {screen} SCREEN -dmS tailmon /jffs/scripts/tailmon.sh -noswitch
2750 sfatula 3564 S {tailmon.sh} /bin/sh /jffs/scripts/tailmon.sh -noswitch
3162 sfatula 5976 S grep tail

So, shows up now, either the order or the sleep delay or both seems to get the monitor going. Of course, why does it stop running, who knows, can't find any logs with any error. Or, maybe it won't stop now.

Still am curious as top why you and I cannot ping any other tailnet device.

 

[ColinTaylor]

Still am curious as top why you and I cannot ping any other tailnet device.

Try changing from userspace mode to kernel mode.

See the note on using ping in userspace mode:

The standard ping command doesn't work for tailnet destinations when Tailscale is running in userspace networking mode. Use tailscale ping instead.

 

[Viktor Jaep]

So, shows up now, either the order or the sleep delay or both seems to get the monitor going. Of course, why does it stop running, who knows, can't find any logs with any error. Or, maybe it won't stop now.

I've never seen tailscale crash, so that is pretty unusual. Keep an eye on it and see if you continue to see crashes now that it's running in the screen environment.

Still am curious as top why you and I cannot ping any other tailnet device.

So yeah, that would be why. Thank you @ColinTaylor! I'm running mine under userspace mode. For some reason running under kernel mode, I could just never get certain things to behave quite right.

 

[sfatula]

So yeah, that would be why. Thank you @ColinTaylor! I'm running mine under userspace mode. For some reason running under kernel mode, I could just never get certain things to behave quite right.

Yeah, I had the same experience (on my non fried router), kernel mode broke a few things and I went back to usermode. Might try again, if the same happens, I guess the answer is it won't work for that (outbound). I presume that also means it won't function as an exit node either.

 

[ColinTaylor]

@sfatula It is not clear to me exactly what your problem is.

Are you saying this used to work on your old router but doesn't on your new one?

Is your issue specifically with the ping command? If so the solution is in the link I provided (i.e. use tailscale ping).

 

[Viktor Jaep]

Yeah, I had the same experience (on my non fried router), kernel mode broke a few things and I went back to usermode. Might try again, if the same happens, I guess the answer is it won't work for that (outbound). I presume that also means it won't function as an exit node either.

Anymore crashing behavior?

 

[sfatula]

Anymore crashing behavior?

Have not seen any, but been repairing things so have not used it either.

 

[sfatula]

@sfatula It is not clear to me exactly what your problem is.

Are you saying this used to work on your old router but doesn't on your new one?

Is your issue specifically with the ping command? If so the solution is in the link I provided (i.e. use tailscale ping).

Ok, so, iPad as an example. Running tailscale on IOS, I can ping from command line, I can ssh, I can use programs that connect to different ports at home on various devices, etc. From the router, I can do none of that. I can't ssh out to a tailnet device (yes there appears to be a tailscale command for that but just comparing to IOS), iperf3 fails, etc. So, is that just the way it is with the router then? I actually don't know if it worked on the old one as it got fried before I moved to the next task. So, does tailmon just not support connections out from the router like it does on other devices? Or maybe doing that just requires kernel mode perhaps? I might even need to do site to site with subnet nodes but not sure yet if that will end up being a requirement. I guess I was just surprised those cli commands just don't work.

 

[Viktor Jaep]

Have not seen any, but been repairing things so have not used it either.

Glad the issue is resolved. Will be sending you an invoice in the mail. Thanks and please fill out our customer survey.