spdMerlin spdMerlin causes router to reboot

[Jeffrey Young]

In a sad way, this is becoming a fairly normal (and unfortunate) cycle of FOSS…

A good developer sees a problem or wants some new features. Takes up the challenge using various free and open source parts and creates something new and used by many.

But, as time goes on, new challenges arise. New hardware platforms, new firmware updates, new updated apps (i.e. Entwares ware). Then, updates and a never ending cycle of support starts. What started with good intention, a way to help others, a way to express ones creativity - becomes a “job”.

Usually without pay or any substantial compensation (other then ”dude, you’re the man!!!”)

Then it’s no longer fun, no longer a hobby.

I kind of expected this years ago with Asuswrt-merlin. That’s why I lobbied hard for the sticky

Asuswrt-Merlin AddOn Developers Donation Links

AddOn developers create many useful scripts, WebUI addon pages, system extensions and useful utilities to enhance Asuswrt-Merlin supported routers. These "devs", like RMerlin, do this all voluntarily. This requires lots of time, effort, support of users, upkeep and maintenance. They do it for...

www.snbforums.com

Hoping that enough users would see enough value and provide some, maybe a little, compensation for the developers work and hopes that they would continue.

BTW, this is not just an Asuswrt-merlin thing. Many times I find a useful code/widget/applicatioin on github. Only to see that the last time it was updated was a few years in the past.

But, at least since it is FOSS, others, new developers might take up the torch and fix/enhance. That’s kind of why FOSS is so magical.

Speaking for myself (and hopefully others) I want to thank all the developers for what they have brought to our routers. I wish people like @Jack Yaz , @dave14305, @john9527 and many others all the best in the future and thanks for the work.

As Douglas Adam’s wrote “So long and thanks for the fish”

Im also glad to see many of the “active” developers like @thelonelycoder , @Martineau , @Viktor Jaep ,@GNUton ,@SomeWhereOverTheRainBow and the many others for keeping the spirit of FOSS alive and well.

Last, thanks to @RMerlin and Asus for providing the base for all this. Aso @ryzhov_al and his team for Entware.

Users, remember in Free and Open Source - the free is like the free in free speech, not the free as in free beer…

yep. Ain't that the truth.

 

[nzwayne]

@Jeffrey Young
1. GT-AXE16000 on rmerlin's 388.2_2. Performed a GUI "Factory Default restore + initalize selection". Setup Wireless, Admin, LAN, WAN, QoS, AiProtection panels.
2. Installed AMTM, DiskCheck (DC), FD on 512MB Patriot SSD in USB 3.0, SW of 2GB created. Debug #7 log-level set, with commit.
3. Install of spdmerlin, AXE1600 reboots during install when the initial test is run on install (see screen shot what occurs and where reboot occurs)
4. After reboot, spdmerlin is installed ok. Uninstall spdmerlin and re-install. Then as per step #3 reboot occurs. Attached the Debug file here for you (and other interested parties)
5. Reboots will usually occur twice per hour, at the scheduled default run times of the 12 & 42mins in the hour. Earlier testing showed if you disabled the auto test runs, no reboots occurred over two days I had this disabled setup.
5. Same install reboot occurs on later firmware level 388.4 beta2 aka steps 3 & 4 with my usual add-ons. Not tried yet on the minimum add-ons as per step 2 (pending).

 

[thelonelycoder]

I’ve seen his ambitions and wish him well. But good luck figuring out how to make it real for users. Either convince Merlin to merge custom changes to the httpd daemon, or maintain a fork of the Merlin code which would be slow on the uptake.

The GUI is rarely the showstopper (even if it’s missing with some addons); it’s usually butting up against the Asus/Trend/Broadcom black box code. Unexplained reboots, flowcache issues, etc.

A fully integrated GUI with respective privileges would be nice to have. Do I want one? Hell no.

Our scripts are mainly shell script based and are coded that way - as their only interface for the users. Adding a GUI with limited capabilities on top of that is just an extra burden that I am not really willing to support any longer.
Imagine a WebUI based amtm. It would be a nightmare to code.
Coding the limited WebUI for Diversion took a lot of the fun away for me. I nearly quit because of it.

 

[Jeffrey Young]

A fully integrated GUI with respective privileges would be nice to have. Do I want one? Hell no.

Our scripts are mainly shell script based and are coded that way - as their only interface for the users. Adding a GUI with limited capabilities on top of that is just an extra burden that I am not really willing to support any longer.
Imagine a WebUI based amtm. It would be a nightmare to code.
Coding the limited WebUI for Diversion took a lot of the fun away for me. I nearly quit because of it.

Agreed. 100%. Then again, like I said I grew up in the era that GUI did not exist.

A CLI environment will always be much more flexible than a GUI.

 

[Jeffrey Young]

@Jeffrey Young
1. GT-AXE16000 on rmerlin's 388.2_2. Performed a GUI "Factory Default restore + initalize selection". Setup Wireless, Admin, LAN, WAN, QoS, AiProtection panels.
2. Installed AMTM, DiskCheck (DC), FD on 512MB Patriot SSD in USB 3.0, SW of 2GB created. Debug #7 log-level set, with commit.
3. Install of spdmerlin, AXE1600 reboots during install when the initial test is run on install (see screen shot what occurs and where reboot occurs)
4. After reboot, spdmerlin is installed ok. Uninstall spdmerlin and re-install. Then as per step #3 reboot occurs. Attached the Debug file here for you (and other interested parties)
5. Reboots will usually occur twice per hour, at the scheduled default run times of the 12 & 42mins in the hour. Earlier testing showed if you disabled the auto test runs, no reboots occurred over two days I had this disabled setup.
5. Same install reboot occurs on later firmware level 388.4 beta2 aka steps 3 & 4 with my usual add-ons. Not tried yet on the minimum add-ons as per step 2 (pending).
View attachment 52293

Thanks. This using the internal speed test binary? If so, can you switch to the external binary and try again.

Have you run a manual speed test from the Asus GUI page?

From your logs, the crash is happening with the execution of the speed test binary, no no clues are given. From your screen shot, the binary make it mostly through the process.

Try the external binary and see what happens.

 

[dave14305]

If you run this manually, does it reboot? 3 variations on the script’s command.

SPEEDTEST_BINARY=/usr/sbin/ookla
CONFIG_STRING="-c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config"
IFACE="$(nvram get wan0_ifname)"
LICENSE_STRING=""
tmpfile=/tmp/spd-stats.txt

echo Test 1
"$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING
echo Test 2
"$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile"
echo Test 3
"$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile" &

 

[Jeffrey Young]

Thanks Dave. I'm at camp for a few weeks now. All I got is my Galaxy A8 to work with.

Appreciate the group fix/help

EDIT @dave14305 , tried to PM you to let you know of private chat @nzwayne started with me and additional asks for troubleshooting. But, I can't PM you.

@nzwayne , post your findings here. Dave is likely better able to get to the root of this issue then me at the moment.

 

[Jeffrey Young]

@nzwayne , can you also get the version number of the internal ookla binary. Just want to know if something changed with that router.

/usr/sbin/ookla -V

 

[nzwayne]

@nzwayne , can you also get the version number of the internal ookla binary. Just want to know if something changed with that router.

/usr/sbin/ookla -V

My summary collate of answers to the requests from @Jeffrey Young and @dave14305 No problem to action more if required. Thanks for the support.

A). Dave's #14 append. Manually running speedtest(s). Outcome: No Reboot(s).
CanaKW@GT-AXE16000:/tmp/home/root# ookla -c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config -f human-readable
Server: Rogers - Milton, ON (id: 47540)
ISP: Cogeco Connexion Inc.
Idle Latency: 9.26 ms (jitter: 1.50ms, low: 8.49ms, high: 11.99ms)
Download: 937.47 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.49 Mbps (data used: 39.4 MB)
Packet Loss: 0.0%

B. Jeffrey's #48 append. Version number of the internal ookla binary.
CanaKW@GT-AXE16000:/tmp/home/root# /usr/sbin/ookla -V
ookla 3.8.0.56 (73c537b5bf) Linux/arm-linux-musleabi 4.19.183 arm/aarch64 (asus-AXHND-A7)

C. Dave's #46 append. 3 variations on the script’s command. Outcome: No reboots.
CanaKW@GT-AXE16000:/tmp/home/root# CONFIG_STRING="-c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config"
CanaKW@GT-AXE16000:/tmp/home/root# SPEEDTEST_BINARY=/usr/sbin/ookla
CanaKW@GT-AXE16000:/tmp/home/root# CONFIG_STRING="-c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config"
CanaKW@GT-AXE16000:/tmp/home/root# IFACE="$(nvram get wan0_ifname)"
CanaKW@GT-AXE16000:/tmp/home/root# LICENSE_STRING=""
CanaKW@GT-AXE16000:/tmp/home/root# tmpfile=/tmp/spd-stats.txt
CanaKW@GT-AXE16000:/tmp/home/root# echo Test 1
Test 1
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING
Server: Rogers Wireless - Hamilton, ON (id: 47326)
ISP: Cogeco Connexion Inc.
Idle Latency: 10.78 ms (jitter: 1.38ms, low: 9.47ms, high: 12.00ms)
Download: 941.19 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.55 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

CanaKW@GT-AXE16000:/tmp/home/root# echo Test 2
Test 2
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile"
Server: Rogers Wireless - Milton, ON (id: 47330)
ISP: Cogeco Connexion Inc.
Idle Latency: 9.79 ms (jitter: 1.01ms, low: 9.44ms, high: 10.97ms)
Download: 935.68 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.50 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

CanaKW@GT-AXE16000:/tmp/home/root# echo Test 3
Test 3
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile" &
CanaKW@GT-AXE16000:/tmp/home/root#
CanaKW@GT-AXE16000:/tmp/home/root# Server: Rogers - Hamilton, ON (id: 46432)
ISP: Cogeco Connexion Inc.
Idle Latency: 7.99 ms (jitter: 1.26ms, low: 7.42ms, high: 9.99ms)
Download: 939.58 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.49 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

Added note: After running the three requests above (labelled as A, B, C) then uninstalled spdmerlin and re-installed. This time (unlike last night on 382.2_2) the install completed without reboot. Setup spdmerlin to run automatic speed tests every 10mins.

 

[nzwayne]

Update to

My summary collate of answers to the requests from @Jeffrey Young and @dave14305 No problem to action more if required. Thanks for the support.

A). Dave's #14 append. Manually running speedtest(s). Outcome: No Reboot(s).
CanaKW@GT-AXE16000:/tmp/home/root# ookla -c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config -f human-readable
Server: Rogers - Milton, ON (id: 47540)
ISP: Cogeco Connexion Inc.
Idle Latency: 9.26 ms (jitter: 1.50ms, low: 8.49ms, high: 11.99ms)
Download: 937.47 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.49 Mbps (data used: 39.4 MB)
Packet Loss: 0.0%

B. Jeffrey's #48 append. Version number of the internal ookla binary.
CanaKW@GT-AXE16000:/tmp/home/root# /usr/sbin/ookla -V
ookla 3.8.0.56 (73c537b5bf) Linux/arm-linux-musleabi 4.19.183 arm/aarch64 (asus-AXHND-A7)

C. Dave's #46 append. 3 variations on the script’s command. Outcome: No reboots.
CanaKW@GT-AXE16000:/tmp/home/root# CONFIG_STRING="-c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config"
CanaKW@GT-AXE16000:/tmp/home/root# SPEEDTEST_BINARY=/usr/sbin/ookla
CanaKW@GT-AXE16000:/tmp/home/root# CONFIG_STRING="-c http://www.speedtest.net/api/embed/vz0azjarf5enop8a/config"
CanaKW@GT-AXE16000:/tmp/home/root# IFACE="$(nvram get wan0_ifname)"
CanaKW@GT-AXE16000:/tmp/home/root# LICENSE_STRING=""
CanaKW@GT-AXE16000:/tmp/home/root# tmpfile=/tmp/spd-stats.txt
CanaKW@GT-AXE16000:/tmp/home/root# echo Test 1
Test 1
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING
Server: Rogers Wireless - Hamilton, ON (id: 47326)
ISP: Cogeco Connexion Inc.
Idle Latency: 10.78 ms (jitter: 1.38ms, low: 9.47ms, high: 12.00ms)
Download: 941.19 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.55 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

CanaKW@GT-AXE16000:/tmp/home/root# echo Test 2
Test 2
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile"
Server: Rogers Wireless - Milton, ON (id: 47330)
ISP: Cogeco Connexion Inc.
Idle Latency: 9.79 ms (jitter: 1.01ms, low: 9.44ms, high: 10.97ms)
Download: 935.68 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.50 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

CanaKW@GT-AXE16000:/tmp/home/root# echo Test 3
Test 3
CanaKW@GT-AXE16000:/tmp/home/root# "$SPEEDTEST_BINARY" $CONFIG_STRING --interface="$IFACE" --format="human-readable" --unit="Mbps" -p $LICENSE_STRING | tee "$tmpfile" &
CanaKW@GT-AXE16000:/tmp/home/root#
CanaKW@GT-AXE16000:/tmp/home/root# Server: Rogers - Hamilton, ON (id: 46432)
ISP: Cogeco Connexion Inc.
Idle Latency: 7.99 ms (jitter: 1.26ms, low: 7.42ms, high: 9.99ms)
Download: 939.58 Mbps (data used: 1.2 GB, additional servers: 3)
Upload: 31.49 Mbps (data used: 39.3 MB)
Packet Loss: 0.0%

Added note: After running the three requests above (labelled as A, B, C) then uninstalled spdmerlin and re-installed. This time (unlike last night on 382.2_2) the install completed without reboot. Setup spdmerlin to run automatic speed tests every 10mins.

Update 10:50am: Running speedtest cases manually at SSH level or GUI, this speedtest run from GUI panel start caused a re-boot at 10:47am (rmerlin firmware code 388.2_2). The GUI panel view of where it hung. Debug system log attached.

 

[Jeffrey Young]

Have you tried switching to the external ookla binary and running over a couple of hours?

 

[dave14305]

That’s not encouraging that it sometimes works and sometimes reboots. I wonder if it’s related to the CPU load from the test based on your gigabit bandwidth.

 

[ColinTaylor]

That’s not encouraging that it sometimes works and sometimes reboots. I wonder if it’s related to the CPU load from the test based on your gigabit bandwidth.

Maybe it's having a problem writing to the SQL database?

 

[Ripshod]

I don't know if it's related but I'll just drop this nugget here. Results from using the external speedtest:

Aug 12 16:30:04 ripshod kernel: potentially unexpected fatal signal 4.
Aug 12 16:30:04 ripshod kernel: CPU: 1 PID: 27763 Comm: speedtest Tainted: P           O    4.1.51 #2
Aug 12 16:30:04 ripshod kernel: Hardware name: Broadcom-v8A (DT)
Aug 12 16:30:04 ripshod kernel: task: ffffffc029adc180 ti: ffffffc03e874000 task.ti: ffffffc03e874000
Aug 12 16:30:04 ripshod kernel: PC is at 0x4b4810
Aug 12 16:30:04 ripshod kernel: LR is at 0x4b55b0
Aug 12 16:30:04 ripshod kernel: pc : [<00000000004b4810>] lr : [<00000000004b55b0>] pstate: 60000000
Aug 12 16:30:04 ripshod kernel: sp : 0000007f8df16720
Aug 12 16:30:04 ripshod kernel: x29: 0000007f8df16720 x28: 0000000000672838
Aug 12 16:30:04 ripshod kernel: x27: 0000007f8def4000 x26: 0000007f8df16884
Aug 12 16:30:04 ripshod kernel: x25: 0000007f8df16778 x24: 0000007f8df16888
Aug 12 16:30:04 ripshod kernel: x23: 0000007f8df16880 x22: 0000007f8df16788
Aug 12 16:30:04 ripshod kernel: x21: 000000002767e7a0 x20: 0000000000000000
Aug 12 16:30:04 ripshod kernel: x19: 000000002767fc60 x18: 0000000000000000
Aug 12 16:30:04 ripshod kernel: x17: 0000000000000000 x16: 0000000000000074
Aug 12 16:30:04 ripshod kernel: x15: 003b9aca00000000 x14: 002b81c04c000000
Aug 12 16:30:04 ripshod kernel: x13: ffffffff9b28a091 x12: 0000000000000018
Aug 12 16:30:04 ripshod kernel: x11: 0000000018883ad0 x10: 000000000000460e
Aug 12 16:30:04 ripshod kernel: x9 : 0000000001a67c06 x8 : 0000000001a67c06
Aug 12 16:30:04 ripshod kernel: x7 : 0000000000000000 x6 : 0000007f8df17000
Aug 12 16:30:04 ripshod kernel: x5 : 0000007f8df164f0 x4 : 0000007f8df16500
Aug 12 16:30:04 ripshod kernel: x3 : b13a61ba15650987 x2 : 000000002767e7a0
Aug 12 16:30:04 ripshod kernel: x1 : 0000007f8df16778 x0 : 000000002767fc60

Or do I have to install the external speedtest seperately?

 

[dave14305]

Maybe it's having a problem writing to the SQL database?

We’d need some extra syslogging inserted to see if it ever finishes the test before updating the display. I think all the results-processing happens after the test.

 

[Jeffrey Young]

We’d need some extra syslogging inserted to see if it ever finishes the test before updating the display. I think all the results-processing happens after the test.

If I remember correctly when I fixed the output last, the output of ookla is sent to a text file. Then the data is derived from the text file.

 

[nzwayne]

Have you tried switching to the external ookla binary and running over a couple of hours?

To make sure I understand, do you mean download "ookla-speedtest-1.2.0-win64" from ookla website and then run it's "speedtest.exe" on a device attached to the AXE16000?

 

[Jeffrey Young]

No, there should be an option in SpdMerlin where you tell it to use the built-in or ookla-cli version. If you SSH into the router and run spdmerlin, it is option 9.

Not sure where it is in the GUI.

 

[nzwayne]

Few developers migrated to other platforms. Home routers have limitations. For the price of your 4x home routers you could have x86 router with PoE switch and VLAN capable APs. All features you are looking for are there. You don't need to wait for a developer, but just read the documentation. Not as user friendly, but much more processing power, better reliability and upgradability.

Appreciate the idea, thanks. My story on the quantity of home units, off topic a bit on this thread. When the COVID work lockdown began our work business was geared up on Cisco's AnyConnect & Webex for home links. The guys at now home based many had routers unaltered for 5-15yrs, no config setups other then just get wireless & WAN active. Some bought new Routers, did just the minimum to make it work. I encouraged my team if purhcased ASUS routers I could help them with setups, an in addition if they liked the many rmerlin enhancements would help them with that too. I wrote up a document for them (and me) to define settings and procedures from the many snippets gleamed from this forum and just google searches in general.
Steered them towards AX58U (V1), AX86U, AX6000, AXE1000, AXE16000 (rolls-royce) from single person minimal device config to a busy family with several working from home, plus those "gamers". Made such a difference to the audio & video sessions, became transparent of sorts. Didn't waste time on bandwidth challenges with video, callers dropping out or who couldn't connect, video in general was encouraged.
By having Aimesh units on what they had purchased, if problems (was rare), I could take out one of my AiMesh units create a development setup and we'd work through panels & structures. When finished just put unit back into my AiMesh setup to keep it active and current.

 

[nzwayne]

No, there should be an option in SpdMerlin where you tell it to use the built-in or ookla-cli version. If you SSH into the router and run spdmerlin, it is option 9.

Not sure where it is in the GUI.

Firmware 388.2_2- AXE16000: Found the setting in GUI and SSH you were after. Changing from Built-in to External it crashed on the first speed test (timestamp 8:10pm in attached debug log). Ran ok on the second.

What I now see in spdmerlin functions, after a spdmerlin crash/reboot you loose the ability to change these two parameters in SSH & GUI, others you can though. To get these two functions back you have to uninstall spdmerlin and re-install. I did three re-installs in last 30mins, two of the three crashed/rebooted during install, as per earlier append screen shot. These two the time stamps 7:58pm & 8:11pm in Debug log.

Still playing to find the 100% always fail scenario, this is being elusive though