Tailscale crash after upgrade

[ragnaroknroll]

I'm running AsusWRT Merlin v3004.388.11 on an RT-AX86U router and just updated tailscale to v1.96.1 on Entware. After the upgrade, tailscale seems to crash each time I try to turn it on, with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x885080]

goroutine 427 [running]:
github.com/coreos/go-iptables/iptables.(*IPTables).runWithOutput(0x0, {0x7f2a24fbc0, 0x4, 0xb5fb00?}, {0xe5ff68, 0x7f2a754c90})
    github.com/coreos/[email protected]/iptables/iptables.go:560 +0x50
github.com/coreos/go-iptables/iptables.(*IPTables).executeList(0x0, {0x7f2a24fbc0, 0x4, 0x4})
    github.com/coreos/[email protected]/iptables/iptables.go:462 +0x58
github.com/coreos/go-iptables/iptables.(*IPTables).List(0x0, {0xdb8805, 0x6}, {0xdbdf67, 0xa})
    github.com/coreos/[email protected]/iptables/iptables.go:283 +0xc0
tailscale.com/util/linuxfw.(*iptablesRunner).AddConnmarkSaveRule(0x7f328b0600)
    tailscale.com/util/linuxfw/iptables_runner.go:537 +0x90
tailscale.com/wgengine/router/osrouter.(*linuxRouter).Set(0x7f32c5c000, 0x26?)
    tailscale.com/wgengine/router/osrouter/router_linux.go:498 +0x9e4
tailscale.com/wgengine.(*userspaceEngine).Reconfig(0x7f32a5ea88, 0x7f2a1ba240, 0x7f2a4181b0, 0x7f2a077500)
    tailscale.com/wgengine/userspace.go:1157 +0x10f0
tailscale.com/wgengine.(*watchdogEngine).Reconfig.func1()
    tailscale.com/wgengine/watchdog.go:176 +0x2c
tailscale.com/wgengine.(*watchdogEngine).watchdogErr.func2()
    tailscale.com/wgengine/watchdog.go:121 +0x28
created by tailscale.com/wgengine.(*watchdogEngine).watchdogErr in goroutine 467
    tailscale.com/wgengine/watchdog.go:120 +0x204

Doing a bit of troubleshooting with Gemini, it appears that turning tailscale on with the --netfilter-mode=off flag works and doesn't cause it to crash. Anybody else facing the same issue? Any suggestions?

 

[ColinTaylor]

Yes I see the same thing. It appears to be crashing when trying to add or delete iptables rules in the mangle table. It only effects tailscale's kernel mode and not the userspace mode (--tun=userspace-networking).

I suspect it's caused by this commit: https://github.com/tailscale/tailscale/commit/26ef46bf8196f5ab36e94aeeda458dcf65868fcf

It'll take me a while to work through it to find a solution, or report it as a bug.

router: enabling connmark-based rp_filter workaround
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x885080]

 

[bbunge]

May be related to an AMTM update? AMTM was updated again this morning. May help?

 

[ColinTaylor]

A workaround for the time being is to update tailscale with the current version on Tailscale's servers (1.96.4).

# tailscale update
This will update Tailscale from 1.96 to 1.96.4. Continue? [Y/n] y
Downloading "https://pkgs.tailscale.com/stable/tailscale_1.96.4_arm64.tgz"
Download size: 33777482
Downloaded 8192/33777482 (0.0%)
Downloaded 33777482/33777482 (100.0%)
Downloading "https://pkgs.tailscale.com/stable/tailscale_1.96.4_arm64.tgz.sig"
Signature OK
Extracting "/root/.cache/tailscale-update/tailscale_1.96.4_arm64.tgz"
Updated /tmp/mnt/TOSHIBA1/entware/bin/tailscale
Updated /tmp/mnt/TOSHIBA1/entware/bin/tailscaled
Tailscale binaries updated successfully, but failed to restart tailscaled: tailscaled is not running under systemd or init.d.
Please restart tailscaled to finish the update.

I haven't determined yet whether the problem was with the original Tailscale code or just with the Entware version.

EDIT: It looks like Entware's tailscale may have been based on a pre-release version. So rather than pursue this further I think I'll just wait to see what happens when Entware next updates tailscale.

 

[JGrana]

Just a note - if you are using TAILMON, it will update to 1.96.4.

 

[dave14305]

It looks like Entware's tailscale may have been based on a pre-release version. So rather than pursue this further I think I'll just wait to see what happens when Entware next updates tailscale.

It feels like it's been a rough release for Entware this go-round, at least for this router platform.

 

[ragnaroknroll]

@ColinTaylor - Thanks a lot for the prompt response! Will update Tailscale when back home this evening and report.

@JGrana - Had not heard of Tailmon before. Looks very handy indeed. Will add that on as well. Thanks for the note.

 

[ragnaroknroll]

I can confirm that updating Tailscale to v1.96.4 definitely fixes the problem. Have done this using Tailmon, which looks fantastic BTW! Kudos to the author for creating that utility. Loving it!

 

[jksmurf]

I can confirm that updating Tailscale to v1.96.4 definitely fixes the problem. Have done this using Tailmon, which looks fantastic BTW! Kudos to the author for creating that utility. Loving it!

Great to hear.

Yup, I am probably a bit biased because I was the annoying little squeaky wheel that pushed for it to see the light of day, but @Viktor Jaep really did a fantastic job with it, with unparalleled technical input from @ColinTaylor.

 

[ragnaroknroll]

Brilliant! I owe my thanks to @Viktor Jaep and @ColinTaylor then.

On a separate note, installing Tailmon required me to update amtm from some ancient years-old version to the latest as well. Looks like I can now even install AdGuard Home on the router. Who knew!? I've been running AdGuard Home on the home server until now and it was a bit clunky to have it running on a computer in the network rather than the router. Might give it a shot running it directly on the router to see how it goes. Cheers!

 

[jksmurf]

required me to update amtm from some ancient years-old version to the latest as well.

Well then, the floodgates have well and truly opened now...

 

[ragnaroknroll]

AdGuard Home is installed and working like a charm! Needed to switch Tailscale from the default userspace mode to kernel mode for it to play nice with AdGuard and allow AdGuard to correctly identify the various Tailscale devices using the router as an exit node. Looking pretty slick! Just wish there was a neat way to automatically keep these applications installed via AMTM like Tailscale and AdGuard (and frankly even AMTM itself) up to date.

 

[jksmurf]

Just wish there was a neat way to automatically keep these applications installed via AMTM like Tailscale and AdGuard (and frankly even AMTM itself) up to date.

1. I don’t use AG, but TAILMON has an option to set a cron to autoupdate either the script or tailscale binary files or both. You may have discovered this by now.
2. The current 6.7.2 version of amtm has a neat little ‘au’ utility to automatically update the TAILMON script only, when you check for updates via ‘u’, which also updates amtm. You can enable or disable various addons that support this au function. The TAILMON Addon cron can run simultaneously if you wish. You still need the cron enabled for the automatic tailscale binary update.
3. As far as I am aware, amtm does not have a cron for automatic updates and remains a manual process.

[EDIT] Updated to clarify the Addon script is actually called TAILMON, as correctly pointed out by @Viktor Jaep below.

 

[Viktor Jaep]

1. I don’t use AG, but Tailscale TAILMON has an option to set a cron to autoupdate either the script or tailscale binary files or both. You may have discovered this by now.
2. The current 6.7.2 version of amtm has a neat little ‘au’ utility to automatically update the tailscale TAILMON script only, when you check for updates via ‘u’, which also updates amtm. You can enable or disable various addons that support this au function. The tailscale TAILMON Addon cron can run simultaneously if you wish. You still need the cron enabled for the automatic binary update.
3. As far as I am aware, amtm does not have a cron for automatic updates and remains a manual process.

Just a couple of minor clarifications in case there's confusion... lol

 

[ragnaroknroll]

Thanks for pointing that out. Found the auto-update options for Tailmon and Tailscale in the configuration menus. Don't see anything for AdGuard though. Guess doing that manually every once in a while along with AMTM may not be the worst idea. Cheers!