What's new
By:

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Stuck commands

Some details. My AC86u has been running stuck processing monitoring for ~19 months, and the stuck processes appear to happen on average once a month:

Code: 
# ls -lta /opt/var/log/Trace
drwxr-xr-x    6 admin    root          4096 Aug 11 10:27 ..
drwxrwxrwx    2 admin    root          4096 Aug 10 12:03 .
-rw-rw-rw-    1 admin    root            42 Aug 10 12:03 CheckStuckProcCmds.INDX.txt
-rw-rw-rw-    1 admin    root           907 Aug 10 12:03 StuckProcCmds_00024_15738.TRC.txt
-rw-rw-rw-    1 admin    root           371 Jul 28 03:15 StuckProcCmds_00023_08040.TRC.txt
-rw-rw-rw-    1 admin    root           305 Jul 18 02:12 StuckProcCmds_00022_07668.TRC.txt
-rw-rw-rw-    1 admin    root           491 Apr  8 04:15 StuckProcCmds_00021_35510.TRC.txt
-rw-rw-rw-    1 admin    root           739 Mar 20 04:03 StuckProcCmds_00020_05351.TRC.txt
-rw-rw-rw-    1 admin    root           326 Feb 15 04:15 StuckProcCmds_00019_36011.TRC.txt
-rw-rw-rw-    1 admin    root           711 Jan 26  2025 StuckProcCmds_00018_02088.TRC.txt
-rw-rw-rw-    1 admin    root           907 Sep  6  2024 StuckProcCmds_00017_10610.TRC.txt
-rw-rw-rw-    1 admin    root           577 Jul 20  2024 StuckProcCmds_00016_04360.TRC.txt
-rw-rw-rw-    1 admin    root           961 Jun  4  2024 StuckProcCmds_00015_50753.TRC.txt
-rw-rw-rw-    1 admin    root           907 Jun  4  2024 StuckProcCmds_00014_10465.TRC.txt
-rw-rw-rw-    1 admin    root           491 Apr 29  2024 StuckProcCmds_00013_34595.TRC.txt
-rw-rw-rw-    1 admin    root          7198 Apr 26  2024 StuckProcCmds_00012_3187491.TRC.txt
-rw-rw-rw-    1 admin    root           485 Apr 22  2024 StuckProcCmds_00011_33150.TRC.txt
-rw-rw-rw-    1 admin    root           571 Apr  1  2024 StuckProcCmds_00010_34580.TRC.txt
-rw-rw-rw-    1 admin    root           762 Feb 22  2024 StuckProcCmds_00009_35078.TRC.txt
-rw-rw-rw-    1 admin    root           458 Feb 13  2024 StuckProcCmds_00008_34886.TRC.txt
-rw-rw-rw-    1 admin    root          4419 Feb  9  2024 StuckProcCmds_00007_2328773.TRC.txt
-rw-rw-rw-    1 admin    root          7409 Jan 26  2024 StuckProcCmds_00006_05344.TRC.txt
-rw-rw-rw-    1 admin    root           449 Jan 26  2024 StuckProcCmds_00005_03983.TRC.txt
-rw-rw-rw-    1 admin    root           318 Jan 25  2024 StuckProcCmds_00004_06143.TRC.txt
-rw-rw-rw-    1 admin    root           711 Jan 25  2024 StuckProcCmds_00003_05066.TRC.txt
-rw-rw-rw-    1 admin    root           493 Jan 25  2024 StuckProcCmds_00002_04205.TRC.txt
-rw-rw-rw-    1 admin    root           478 Jan 25  2024 StuckProcCmds_00001_06276.TRC.txt

In the recent entries about half the time the stuck thing happens because of VPN server (???), which I am not even running in any capacity. I do run VPN clients, does it need to ask for VPN server info?

Code: 
# grep KILLED StuckProcCmds_000*
StuckProcCmds_00013_34595.TRC.txt:2024-04-29 04:27:19 32771 32755 admin    S     3104  0.7   0  0.0 nvram get ntp_ready [KILLED]
StuckProcCmds_00014_10465.TRC.txt:2024-06-04 18:11:06  9936  3578 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00014_10465.TRC.txt:2024-06-04 18:10:35  8231  3578 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00015_50753.TRC.txt:2024-06-04 19:03:49 50185 41934 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00015_50753.TRC.txt:2024-06-04 19:03:18 42640 41934 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00016_04360.TRC.txt:2024-07-20 01:06:18  1130  1123 admin    S     3104  0.7   0  0.0 nvram get vpn_server1_sn [KILLED]
StuckProcCmds_00017_10610.TRC.txt:2024-09-06 20:34:27  9976  1430 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00017_10610.TRC.txt:2024-09-06 20:33:56  8976  1430 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00018_02088.TRC.txt:2025-01-26 02:15:19  1376  1374 admin    S     3116  0.7   0  0.0 nvram get http_username [KILLED]
StuckProcCmds_00019_36011.TRC.txt:2025-02-15 04:15:18 32771 32753 admin    S     3116  0.7   0  0.0 nvram get productid [KILLED]
StuckProcCmds_00020_05351.TRC.txt:2025-03-20 04:03:35  1126  1124 admin    S     3536  0.8   0  0.0 /usr/sbin/wl -i eth6 nrate [KILLED]
StuckProcCmds_00020_05351.TRC.txt:2025-03-20 04:03:19  2461  2429 admin    S     3116  0.7   1  0.0 nvram get apps_sq [KILLED]
StuckProcCmds_00021_35510.TRC.txt:2025-04-08 04:15:18 32771 32763 admin    S     3116  0.7   0  0.0 nvram get productid [KILLED]
StuckProcCmds_00022_07668.TRC.txt:2025-07-18 02:12:18  2509  2507 admin    S N   3116  0.7   0  0.0 nvram get lan_ipaddr [KILLED]
StuckProcCmds_00023_08040.TRC.txt:2025-07-28 03:15:18  1864  1863 admin    S     3536  0.8   0  0.0 /usr/sbin/wl -i eth5 nrate [KILLED]
StuckProcCmds_00024_15738.TRC.txt:2025-08-10 12:03:49 14459  1649 admin    S     2984  0.7   0  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00024_15738.TRC.txt:2025-08-10 12:03:18  7188  1649 admin    S     2984  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
 
bibikalka said:
Some details. My AC86u has been running stuck processing monitoring for ~19 months, and the stuck processes appear to happen on average once a month:

Code: 
# ls -lta /opt/var/log/Trace
drwxr-xr-x    6 admin    root          4096 Aug 11 10:27 ..
drwxrwxrwx    2 admin    root          4096 Aug 10 12:03 .
-rw-rw-rw-    1 admin    root            42 Aug 10 12:03 CheckStuckProcCmds.INDX.txt
-rw-rw-rw-    1 admin    root           907 Aug 10 12:03 StuckProcCmds_00024_15738.TRC.txt
-rw-rw-rw-    1 admin    root           371 Jul 28 03:15 StuckProcCmds_00023_08040.TRC.txt
-rw-rw-rw-    1 admin    root           305 Jul 18 02:12 StuckProcCmds_00022_07668.TRC.txt
-rw-rw-rw-    1 admin    root           491 Apr  8 04:15 StuckProcCmds_00021_35510.TRC.txt
-rw-rw-rw-    1 admin    root           739 Mar 20 04:03 StuckProcCmds_00020_05351.TRC.txt
-rw-rw-rw-    1 admin    root           326 Feb 15 04:15 StuckProcCmds_00019_36011.TRC.txt
-rw-rw-rw-    1 admin    root           711 Jan 26  2025 StuckProcCmds_00018_02088.TRC.txt
-rw-rw-rw-    1 admin    root           907 Sep  6  2024 StuckProcCmds_00017_10610.TRC.txt
-rw-rw-rw-    1 admin    root           577 Jul 20  2024 StuckProcCmds_00016_04360.TRC.txt
-rw-rw-rw-    1 admin    root           961 Jun  4  2024 StuckProcCmds_00015_50753.TRC.txt
-rw-rw-rw-    1 admin    root           907 Jun  4  2024 StuckProcCmds_00014_10465.TRC.txt
-rw-rw-rw-    1 admin    root           491 Apr 29  2024 StuckProcCmds_00013_34595.TRC.txt
-rw-rw-rw-    1 admin    root          7198 Apr 26  2024 StuckProcCmds_00012_3187491.TRC.txt
-rw-rw-rw-    1 admin    root           485 Apr 22  2024 StuckProcCmds_00011_33150.TRC.txt
-rw-rw-rw-    1 admin    root           571 Apr  1  2024 StuckProcCmds_00010_34580.TRC.txt
-rw-rw-rw-    1 admin    root           762 Feb 22  2024 StuckProcCmds_00009_35078.TRC.txt
-rw-rw-rw-    1 admin    root           458 Feb 13  2024 StuckProcCmds_00008_34886.TRC.txt
-rw-rw-rw-    1 admin    root          4419 Feb  9  2024 StuckProcCmds_00007_2328773.TRC.txt
-rw-rw-rw-    1 admin    root          7409 Jan 26  2024 StuckProcCmds_00006_05344.TRC.txt
-rw-rw-rw-    1 admin    root           449 Jan 26  2024 StuckProcCmds_00005_03983.TRC.txt
-rw-rw-rw-    1 admin    root           318 Jan 25  2024 StuckProcCmds_00004_06143.TRC.txt
-rw-rw-rw-    1 admin    root           711 Jan 25  2024 StuckProcCmds_00003_05066.TRC.txt
-rw-rw-rw-    1 admin    root           493 Jan 25  2024 StuckProcCmds_00002_04205.TRC.txt
-rw-rw-rw-    1 admin    root           478 Jan 25  2024 StuckProcCmds_00001_06276.TRC.txt

In the recent entries about half the time the stuck thing happens because of VPN server (???), which I am not even running in any capacity. I do run VPN clients, does it need to ask for VPN server info?

Code: 
# grep KILLED StuckProcCmds_000*
StuckProcCmds_00013_34595.TRC.txt:2024-04-29 04:27:19 32771 32755 admin    S     3104  0.7   0  0.0 nvram get ntp_ready [KILLED]
StuckProcCmds_00014_10465.TRC.txt:2024-06-04 18:11:06  9936  3578 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00014_10465.TRC.txt:2024-06-04 18:10:35  8231  3578 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00015_50753.TRC.txt:2024-06-04 19:03:49 50185 41934 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00015_50753.TRC.txt:2024-06-04 19:03:18 42640 41934 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00016_04360.TRC.txt:2024-07-20 01:06:18  1130  1123 admin    S     3104  0.7   0  0.0 nvram get vpn_server1_sn [KILLED]
StuckProcCmds_00017_10610.TRC.txt:2024-09-06 20:34:27  9976  1430 admin    S     2972  0.7   1  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00017_10610.TRC.txt:2024-09-06 20:33:56  8976  1430 admin    S     2972  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
StuckProcCmds_00018_02088.TRC.txt:2025-01-26 02:15:19  1376  1374 admin    S     3116  0.7   0  0.0 nvram get http_username [KILLED]
StuckProcCmds_00019_36011.TRC.txt:2025-02-15 04:15:18 32771 32753 admin    S     3116  0.7   0  0.0 nvram get productid [KILLED]
StuckProcCmds_00020_05351.TRC.txt:2025-03-20 04:03:35  1126  1124 admin    S     3536  0.8   0  0.0 /usr/sbin/wl -i eth6 nrate [KILLED]
StuckProcCmds_00020_05351.TRC.txt:2025-03-20 04:03:19  2461  2429 admin    S     3116  0.7   1  0.0 nvram get apps_sq [KILLED]
StuckProcCmds_00021_35510.TRC.txt:2025-04-08 04:15:18 32771 32763 admin    S     3116  0.7   0  0.0 nvram get productid [KILLED]
StuckProcCmds_00022_07668.TRC.txt:2025-07-18 02:12:18  2509  2507 admin    S N   3116  0.7   0  0.0 nvram get lan_ipaddr [KILLED]
StuckProcCmds_00023_08040.TRC.txt:2025-07-28 03:15:18  1864  1863 admin    S     3536  0.8   0  0.0 /usr/sbin/wl -i eth5 nrate [KILLED]
StuckProcCmds_00024_15738.TRC.txt:2025-08-10 12:03:49 14459  1649 admin    S     2984  0.7   0  0.0 nvram get vpn_server1_custom [KILLED]
StuckProcCmds_00024_15738.TRC.txt:2025-08-10 12:03:18  7188  1649 admin    S     2984  0.7   1  0.0 nvram get vpn_server_custom [KILLED]
Click to expand...

Not an expert, but as far as I understood, 'nvram get' will hang when the nvram issue occurs...
The script then kills those hung 'nvram get' calls.
Thus, it is not OpenVPN but getting the OpenVPN status (e.g. when browsing the admin UI).

Unfortunately, as it looks in your case, the script does not prevent the nvram issue to occur in the first place.
And it does not really fix it after it occurred, it just kills processes which got stuck due to the issue.
Same on my side.

We used an AC86U as central AiMesh router with several nodes (partly also AC86U).
The more I used the admin ui to check the AiMesh status, the more often the router started to hang.
Also, in general, the AiMesh feature itself seems to be very active regarding nvram access.

In our case, the result was that Wifi got massively unstable, partly resulting clients not being able to connect and/or Wifi-AiMesh-nodes to permanently lose connection to the network. Also after hard-wiring all nodes via cable, the Wifi issues never stopped occurring once in a while, even with a nightly scheduled reboot... another issue was the hanging admin UI.

For us, the issue was so massive, that we incrementally started to replace this router model whenever we found a cheap replacement.
The kill-script unfortunately did not help prevent the router to get into the issue state in the first place.

IMHO, as long as you don't experience any real life issues with your router, because the hung processes get killed regularly, I think, you might be lucky. :)
 
Last edited:
MarkusI said:
Not an expert, but as far as I understood, 'nvram get' is the issue which can result in hung processes...
Thus, it is not OpenVPN but getting the OpenVPN status (e.g. when browsing the admin UI).

I experienced that myself... we used an AC86U as central AiMesh router with several nodes.
The more I used the admin ui to check the AiMesh status, the more often the router started to hang.
Click to expand...
It sadly still happens all the time... even on my GT-AX6000. I'm running 'CheckStuckProcCmds.sh' every 30 mins to clear out the gunk. As shown below, nvram get extendno got hung.

Code: 
Aug 07 2025 01:00:04 -- START_09171: [CheckStuckProcCmds.sh]
Aug 07 2025 01:00:04 -- FOUND_09171: [1]
Aug 07 2025 01:00:04 --  7923  7919 ViktorJp S     4136  0.4   1  0.0 nvram get extendno
Aug 07 2025 01:00:04 --  7919  7855 ViktorJp S     3996  0.3   2  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 --  7855 25548 ViktorJp S     3996  0.3   1  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 -- 25548 25547 ViktorJp S     3996  0.3   0  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 -- 25547  2613 ViktorJp S     3324  0.3   2  0.0 /bin/sh -c sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 --  2613     1 ViktorJp S     3452  0.3   2  0.0 crond -l 9
Aug 07 2025 01:00:05 -- FOUND_09171: [0][ 7923  7919 ViktorJp S     4136  0.4   1  0.0 nvram get extendno]
Aug 07 2025 01:00:05 -- FOUND_09171: [1]
Aug 07 2025 01:00:05 -- EXIT_09171: OK.
 
MarkusI said:
Not an expert, but as far as I understood, 'nvram get' is the issue which can result in hung processes...
Thus, it is not OpenVPN but getting the OpenVPN status (e.g. when browsing the admin UI).

I experienced that myself... we used an AC86U as central AiMesh router with several nodes.
The more I used the admin ui to check the AiMesh status, the more often the router started to hang.
Click to expand...
Right, but I am not browsing anything! It just gets stuck on its own by doing its automatic things.

My AiMesh node is AC68U, and I do get some issues with it that are separate from stuck commands - link .

Btw, I do reboot the AiMesh node nightly via cron, so you may want to do that too. This way you'll know things are fresh daily.
 
Viktor Jaep said:
It sadly still happens all the time... even on my GT-AX6000. I'm running 'CheckStuckProcCmds.sh' every 30 mins to clear out the gunk. As shown below, nvram get extendno got hung.

Code: 
Aug 07 2025 01:00:04 -- START_09171: [CheckStuckProcCmds.sh]
Aug 07 2025 01:00:04 -- FOUND_09171: [1]
Aug 07 2025 01:00:04 --  7923  7919 ViktorJp S     4136  0.4   1  0.0 nvram get extendno
Aug 07 2025 01:00:04 --  7919  7855 ViktorJp S     3996  0.3   2  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 --  7855 25548 ViktorJp S     3996  0.3   1  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 -- 25548 25547 ViktorJp S     3996  0.3   0  0.0 sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 -- 25547  2613 ViktorJp S     3324  0.3   2  0.0 /bin/sh -c sh /jffs/scripts/firewall debug genstats
Aug 07 2025 01:00:05 --  2613     1 ViktorJp S     3452  0.3   2  0.0 crond -l 9
Aug 07 2025 01:00:05 -- FOUND_09171: [0][ 7923  7919 ViktorJp S     4136  0.4   1  0.0 nvram get extendno]
Aug 07 2025 01:00:05 -- FOUND_09171: [1]
Aug 07 2025 01:00:05 -- EXIT_09171: OK.
Click to expand...
Do you experience stability issues on your router, too?
 
bibikalka said:
Right, but I am not browsing anything! It just gets stuck on its own by doing its automatic things.

My AiMesh node is AC68U, and I do get some issues with it that are separate from stuck commands - link .

Btw, I do reboot the AiMesh node nightly via cron, so you may want to do that too. This way you'll know things are fresh daily.
Click to expand...

We have a relatively huge AiMesh setup (1 AX88U-Pro now as central node, 3 AC86U nodes, 3 AC68U nodes) on a remote location.
Generally, the AC68U nodes run very stably in our setup, but they are screwed, if they need to connect to an AC86U via Wifi, since AC86U sometimes runs into Wifi stability issues due to the nvram problem - even if rebooted daily. If the central AiMesh router is struggling, the nodes are, too.

The issue also seems to depend on how many and how often clients connect and/or roam from AiMesh node to node.
This seems to tigger the nvram issue more likely.
I monitored the connection every 5 seconds for all nodes over a period of multiple months and when the people on the location were on vacation for a week, there was no disconnect at all which I was able to detect via pinging the nodes.

Long story short - I tested and experimented for several months but with no real success.
The AC86U simply is no stable router, especially in an AiMesh setup.

By the way: it seems to be recommended to run the nodes on stock firmware and only the central router on Merlin, if needed.
But in case of an AC86U it will not help you, I fear.

In other words - while you have an AC86U as AiMesh router, I would not invest any more time in analyzing connection issues on the AC68U.
They might just be a side-effects of the nvram issue on the AC86U.
At least it makes it very difficult to sort that out.
 
MarkusI said:
...
By the way: it seems to be recommended to run the nodes on stock firmware and only the central router on Merlin, if needed.
But in case of an AC86U it will not help you, I fear.

In other words - while you have an AC86U as AiMesh router, I would not invest any more time in analyzing connection issues on the AC68U.
They might just be a side-effects of the nvram issue on the AC86U.
At least it makes it very difficult to sort that out.
Click to expand...
I mean, I am monitoring stuck commands. And they don't occur as often as the node disconnects and reconnects.

I monitored the connection every 5 seconds for all nodes over a period of multiple months and when the people on the location were on vacation for a week, there was no disconnect at all which I was able to detect via pinging the nodes.

My AC68u node disconnects often do occur when a device just connected. So if there is no activity, it's usually stable doing nothing. I think it's AiMesh that's messy, plus having a WiFi backbone contributes to the problem greatly.
 
bibikalka said:
I mean, I am monitoring stuck commands. And they don't occur as often as the node disconnects and reconnects.

I monitored the connection every 5 seconds for all nodes over a period of multiple months and when the people on the location were on vacation for a week, there was no disconnect at all which I was able to detect via pinging the nodes.

My AC68u node disconnects often do occur when a device just connected. So if there is no activity, it's usually stable doing nothing. I think it's AiMesh that's messy, plus having a WiFi backbone contributes to the problem greatly.
Click to expand...
Strange... Do you also regularly reboot your central router?
Do you know if the disconnect rate increases depending on how long since the last reboot of central and node router?
If you have the possibility, you might want try if the stock firmware provides more stability.
 
MarkusI said:
Strange... Do you also regularly reboot your central router?
Do you know if the disconnect rate increases depending on how long since the last reboot of central and node router?
If you have the possibility, you might want try if the stock firmware provides more stability.
Click to expand...
Yes, the head node is reboot daily around 4am, such the reserve of "echo 4194304 > /proc/sys/kernel/pid_max" does not run out.

I had to trouble shoot a bit since a warm reboot does not follow the same sequence as the cold one (one of Asus quirks), but after a while I sorted out most problematic items (mainly Diversion/dnsmasq initialization sequence), and so now the nightly reboots go smooth. I don't recall an auto issue within the last few months.

The AiMesh node is slower, so if I start rebooting it at the same time it'll come back later, when the head node is already up. You can enable basic scripts on the AiMesh nodes, so they time sync, and do auto reboots.
 
You must log in or register to reply here.

Similar threads

Viktor Jaep
General CRU/SSH weirdness and questions
Replies
18
Views
3K
Viktor Jaep
Viktor Jaep
cmkelley
Scribe scribe 3.x_y - syslog-ng and logrotate installer
11 12 13
Replies
250
Views
28K
Tarek Yag
Tarek Yag
C
N00bs need simple page with outdated stuff & working stuff. Also simple commonly used phrases only people following Merlin for years understand :D
Replies
16
Views
2K
gattaca
G
PeterV
[Solved] AC86U - possible memory leaks. AiProtection(dcd) crashing is guilty
2
Replies
33
Views
14K
jksmurf
J
T
RT-AC86U Policy Based Routing through VPN conundrum
Replies
6
Views
4K
Martineau
Martineau

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 795 (members: 17, guests: 778)
Back
Top
✖