What's new
By:

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Release [Fork] Asuswrt-Merlin 374 LTS release 49E4

Status
Not open for further replies.
There's a 50D9 build for the AC66U in the Development/Beta folder. Haven't had a chance to test it, but it should work.

You can set an nvram to prevent the reboot when the watchdog crash is detected.

nvram set wd_norestart=1
nvram commit

assuming ssh isn't crashed as well, we may be able to see what is causing tmpfs to run out of space.
 
Running the command above yields:

Code: 
ASUSWRT-Merlin LTS RT-AC66U 50E8 Wed Jul 21 23:30:35 UTC 2021
admin@RT-AC66U:/tmp/home/root# while true; do du -ah /tmp | grep -v "mnt" | sort
 -n -r | head -n 10; sleep 60; done;
268.0K  /tmp/syslog.log-1
224.0K  /tmp/var
112.0K  /tmp/var/lib/misc
112.0K  /tmp/var/lib
104.0K  /tmp/var/run
100.0K  /tmp/var/lib/misc/rstats-speed.gz
84.0K   /tmp/etc
60.0K   /tmp/syslog.log
16.0K   /tmp/etc/dropbear
8.0K    /tmp/Beceem_firmware/Server_CA.pem.yota

My router last rebooted ~38h ago (scheduled)... its typically rebooting due to this issue every ~48-72h... when I get a chance I will try the above build
 
Reboot on 50E8 after ~57h due to Watchdog Crash. Log on pastebin. SSH output:

Code: 
268.0K  /tmp/var
268.0K  /tmp/syslog.log-1
156.0K  /tmp/var/lib/misc
156.0K  /tmp/var/lib
144.0K  /tmp/var/lib/misc/rstats-speed
120.0K  /tmp/syslog.log
104.0K  /tmp/var/run
84.0K   /tmp/etc
16.0K   /tmp/etc/dropbear
8.0K    /tmp/Beceem_firmware/Server_CA.pem.yota

Installing 50D9 and setting nvram config as above... On had for any additional testing.
 
Last edited:
frazor77 said:
Added pastebin link above
Click to expand...
What kind of device is 192.168.1.61 OPTIMUS? Doing anything special? It looks like it's doing bursts of logins to the router that trigger the httpd check fails. In the latest log, the router actually recovered a couple of times, but then finally couldn't handle it and gave up. Then 16hrs later, watchdog crashes.
 
john9527 said:
Bingo...../var is a link to /tmp/var which resides in the tmpfs space. So something is eating up the space.
Let me think a bit.....may need to do a debug build that removes the restart to we can see what it is.

Another option might be to open an ssh session and run the following
Code: 
while true; do du -ah /tmp | grep -v "mnt" | sort -n -r | head -n 10; sleep 60; done;

This will write the top ten largest files in /tmp to the console every minute.
Click to expand...
I use version 49E4. I don't use any add-ons.
I'm using DoT, DHCP reservations, DNS filtering, Parental Controls, https and that's all. Now uptime of the router is 2 days 6 hours. As I understand, everything looks normal yet:
Code: 
mark@RT-AC66U-1900:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        239524     140000      99524          0       9000      29340
-/+ buffers/cache:     101660     137864
Swap:            0          0          0


mark@RT-AC66U-1900:/tmp/home/root# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                20480     20480         0 100% /
tmpfs                   119760      4108    115652   3% /tmp
devfs                   119760         0    119760   0% /dev
/dev/mtdblock5           32768      1076     31692   3% /jffs


mark@RT-AC66U-1900:/tmp/home/root# top
Mem: 140116K used, 99408K free, 0K shrd, 9000K buff, 29344K cached
CPU:  0.0% usr  0.0% sys  0.0% nic  100% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.04 0.02 0.00 3/50 7551
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  314     1 mark     S    34340 14.2   0  0.0 /sbin/wanduck
  361     1 mark     S    27660 11.4   0  0.0 watchdog
  355     1 mark     S     8600  3.5   0  0.0 httpd -s -p 8443
  381     1 mark     S     7280  3.0   0  0.0 /usr/sbin/acsd
  321     1 mark     S     6552  2.7   0  0.0 /usr/sbin/haveged -r0 -w1024
  365     1 mark     S     5024  2.0   0  0.0 rstats
  568     1 mark     S     4916  2.0   0  0.0 stubby -g -v 4 -C /etc/stubby.yml -F /var/tmp/stubby/s
    1     0 mark     S     2940  1.2   0  0.0 /sbin/init
  549     1 mark     S     2844  1.1   0  0.0 ntp
  362     1 mark     S     2840  1.1   0  0.0 watchdog02
  335     1 mark     S     2840  1.1   0  0.0 wpsaide
  364   361 mark     S     2840  1.1   0  0.0 ots
  555     1 mark     S     2840  1.1   0  0.0 disk_monitor
 
Last edited:
john9527 said:
What kind of device is 192.168.1.61 OPTIMUS? Doing anything special? It looks like it's doing bursts of logins to the router that trigger the httpd check fails. In the latest log, the router actually recovered a couple of times, but then finally couldn't handle it and gave up. Then 16hrs later, watchdog crashes.
Click to expand...

That's the Windows 10 Desktop machine that I use as my daily driver - its also where I'm SSH'ing in from and opening the router admin webview from... I may have a few rogue tabs open in Chrome... :rolleyes:... but looking at the log it appears as multiples of ~5's and I deffo don't have 5 webview tabs open! (on checking, I in fact have 2 open)

If I logout, then go to the router IP (expecting to log in) it takes me straight back in without prompting to log in... so is the logout action not invalidating the session credentials?
 
Last edited:
frazor77 said:
If I logout, then go to the router IP (expecting to log in) it takes me straight back in without prompting to log in... so is the logout action not invalidating the session credentials?
Click to expand...
It takes 15sec (or maybe 30, don't remember off the top of my head) to actually log you out

Are you running any custom anti-virus suites? Some will attempt net scans trying common logins.
 
frazor77 said:
Reboot on 50E8 after ~57h due to Watchdog Crash. Log on pastebin. SSH output:

Code: 
268.0K  /tmp/var
268.0K  /tmp/syslog.log-1
156.0K  /tmp/var/lib/misc
156.0K  /tmp/var/lib
144.0K  /tmp/var/lib/misc/rstats-speed
120.0K  /tmp/syslog.log
104.0K  /tmp/var/run
84.0K   /tmp/etc
16.0K   /tmp/etc/dropbear
8.0K    /tmp/Beceem_firmware/Server_CA.pem.yota

Installing 50D9 and setting nvram config as above... On had for any additional testing.
Click to expand...
Interesting, I don’t see
“failed to write …….: No space left on device (retry in 60 s)”
 
If you logout, close the browser (all instances) and try to log in again, it shouldn't let you.
 
So the watchdog has died and that's now being reported (a lot) in the syslog:

https://pastebin.com/B02MLx68

Here's the SSH output:

Code: 
ASUSWRT-Merlin LTS RT-AC66U 50D9 Sun Aug  1 16:09:24 UTC 2021
admin@RT-AC66U:/tmp/home/root# while true; do du -ah /tmp | grep -v "mnt" | sort
-n -r | head -n 10; sleep 60; done;
268.0K  /tmp/syslog.log-1
252.0K  /tmp/syslog.log
232.0K  /tmp/var
116.0K  /tmp/var/lib/misc
116.0K  /tmp/var/lib
104.0K  /tmp/var/run
100.0K  /tmp/var/lib/misc/rstats-speed.gz
84.0K   /tmp/etc
16.0K   /tmp/etc/dropbear
8.0K    /tmp/Beceem_firmware/Server_CA.pem.yota

Let me know what you want me to do next ;)
 
Free:
Code: 
admin@RT-AC66U:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        239524     152528      86996          0       9292      30500
-/+ buffers/cache:     112736     126788
Swap:            0          0          0

Top:
Code: 
Mem: 152536K used, 86988K free, 0K shrd, 9292K buff, 30500K cached
CPU:  0.0% usr  0.0% sys  0.0% nic 90.9% idle  0.0% io  0.0% irq  9.0% sirq
Load average: 0.39 0.09 0.03 2/52 16453
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  319     1 admin    S    49192 20.4   0  0.0 /sbin/wanduck
14296     1 admin    S    23676  9.8   0  0.0 httpd
  372     1 admin    S     9936  4.1   0  0.0 /usr/sbin/acsd
  361     1 admin    S     9560  3.9   0  0.0 rstats
  417     1 admin    S     7176  2.9   0  0.0 usbled
  325     1 admin    S     6552  2.7   0  0.0 /usr/sbin/haveged -r0 -w1024
14297     1 admin    S     5012  2.0   0  0.0 httpd -s -p 8443
  358     1 admin    S     3136  1.3   0  0.0 watchdog02
    1     0 admin    S     2936  1.2   0  0.0 /sbin/init
  518     1 admin    S     2872  1.1   0  0.0 ntp
  357     1 admin    S     2840  1.1   0  0.0 ots
  579     1 admin    S     2840  1.1   0  0.0 disk_monitor
  339     1 admin    S     2840  1.1   0  0.0 wpsaide
  160     1 admin    S     2836  1.1   0  0.0 console
  336     1 admin    S     2572  1.0   0  0.0 /bin/eapd
  429   422 admin    S     2172  0.9   0  0.0 u2ec
  422     1 admin    S     2172  0.9   0  0.0 u2ec
  430   429 admin    S     2172  0.9   0  0.0 u2ec
  342     1 admin    S     1732  0.7   0  0.0 nas
  379     1 admin    S     1496  0.6   0  0.0 /usr/sbin/pppd file /tmp/ppp/options.wan0
16442 16441 admin    S     1492  0.6   0  0.0 -sh
  377     1 admin    S     1484  0.6   0  0.0 udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160 -b -O33 -O249
  162   160 admin    S     1484  0.6   0  0.0 /bin/sh
16453 16442 admin    R     1484  0.6   0  0.0 top
  348     1 admin    S     1480  0.6   0  0.0 crond -l 8
  182     1 admin    S     1476  0.6   0  0.0 syslogd -m 0 -O /tmp/syslog.log -S -s 256 -l 7
  184     1 admin    S     1476  0.6   0  0.0 /sbin/klogd
  326     1 admin    S     1384  0.5   0  0.0 protect_srv
  330   326 admin    S     1384  0.5   0  0.0 protect_srv
  331   330 admin    S     1384  0.5   0  0.0 protect_srv
16240     1 nobody   S     1364  0.5   0  0.0 dnsmasq --log-async
16441   333 admin    S     1336  0.5   0  0.0 dropbear -p 192.168.1.99:22 -I 1200 -j -k
  333     1 admin    S     1316  0.5   0  0.0 dropbear -p 192.168.1.99:22 -I 1200 -j -k
  350     1 admin    S     1300  0.5   0  0.0 networkmap --bootwait
  424     1 admin    S     1248  0.5   0  0.0 lpd
  369     1 admin    S     1192  0.4   0  0.0 lld2d br0
  349     1 admin    S     1156  0.4   0  0.0 /usr/sbin/infosvr br0
  578     1 admin    S      880  0.3   0  0.0 miniupnpd -f /etc/upnp/config
  125     1 admin    S      644  0.2   0  0.0 hotplug2 --persistent --no-coldplug
    3     2 admin    SWN      0  0.0   0  0.0 [ksoftirqd/0]
    4     2 admin    SW<      0  0.0   0  0.0 [events/0]
   52     2 admin    SW       0  0.0   0  0.0 [pdflush]
   98     2 admin    SW<      0  0.0   0  0.0 [mtdblockd]
  167     2 admin    SWN      0  0.0   0  0.0 [jffs2_gcd_mtd5]
    2     0 admin    SW<      0  0.0   0  0.0 [kthreadd]
    5     2 admin    SW<      0  0.0   0  0.0 [khelper]
   20     2 admin    SW<      0  0.0   0  0.0 [kblockd/0]
  186     2 admin    SW<      0  0.0   0  0.0 [khubd]
   53     2 admin    SW<      0  0.0   0  0.0 [kswapd0]
   54     2 admin    SW<      0  0.0   0  0.0 [aio/0]
   51     2 admin    SW       0  0.0   0  0.0 [pdflush]
  121     2 admin    SW<      0  0.0   0  0.0 [kmmcd]
 
frazor77 said:
Let me know what you want me to do next ;)
Click to expand...
Have to admit I'm really stumped on this one. Out of nowhere, watchdog just stops....

Last guess I have since it's always preceded by repeated httpd check fails (hours earlier), is that there is something with curl that's being left in a bad state on MIPS when it fails that finally catches up with us. So, let's try ths....

nvram set httpd_check_disable=3
nvram commit

(That var has been in the code since the check was added).
 
john9527 said:
Have to admit I'm really stumped on this one. Out of nowhere, watchdog just stops....

Last guess I have since it's always preceded by repeated httpd check fails (hours earlier), is that there is something with curl that's being left in a bad state on MIPS when it fails that finally catches up with us. So, let's try ths....

nvram set httpd_check_disable=3
nvram commit

(That var has been in the code since the check was added).
Click to expand...
Should I reboot the router first? (I'd assume so)
 
john9527 said:
Have to admit I'm really stumped on this one. Out of nowhere, watchdog just stops....

Last guess I have since it's always preceded by repeated httpd check fails (hours earlier), is that there is something with curl that's being left in a bad state on MIPS when it fails that finally catches up with us. So, let's try ths....

nvram set httpd_check_disable=3
nvram commit

(That var has been in the code since the check was added).
Click to expand...
Rebooted, applied the above.

Code: 
admin@RT-AC66U:/tmp/home/root# nvram set httpd_check_disable=3
admin@RT-AC66U:/tmp/home/root# nvram commit
admin@RT-AC66U:/tmp/home/root# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                20544     20544         0 100% /
tmpfs                   119760      4336    115424   4% /tmp
devfs                   119760         0    119760   0% /dev
/dev/mtdblock5           32768      1140     31628   3% /jffs
admin@RT-AC66U:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        239524      62484     177040          0       8844      28968
-/+ buffers/cache:      24672     214852
Swap:            0          0          0

Mem: 62552K used, 176972K free, 0K shrd, 8844K buff, 28976K cached
CPU:  0.0% usr  0.0% sys  0.0% nic 91.6% idle  0.0% io  0.0% irq  8.3% sirq
Load average: 0.46 0.51 0.23 3/53 602
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  325     1 admin    S     6552  2.7   0  0.0 /usr/sbin/haveged -r0 -w1024
  345     1 admin    S     5464  2.2   0  0.0 httpd
  346     1 admin    S     5096  2.1   0  0.0 httpd -s -p 8443
    1     0 admin    S     2908  1.2   0  0.0 /sbin/init
  356     1 admin    S     2896  1.2   0  0.0 watchdog
  319     1 admin    S     2892  1.2   0  0.0 /sbin/wanduck
  406     1 admin    S     2840  1.1   0  0.0 usbled
  358     1 admin    S     2840  1.1   0  0.0 watchdog02
  567     1 admin    S     2840  1.1   0  0.0 ntp
  574     1 admin    S     2840  1.1   0  0.0 disk_monitor
  339     1 admin    S     2840  1.1   0  0.0 wpsaide
  357   356 admin    S     2840  1.1   0  0.0 ots
  160     1 admin    S     2836  1.1   0  0.0 console
  413   412 admin    S     2172  0.9   0  0.0 u2ec
  412     1 admin    S     2172  0.9   0  0.0 u2ec
  414   413 admin    S     2172  0.9   0  0.0 u2ec
  342     1 admin    S     1732  0.7   0  0.0 nas
  379     1 admin    S     1496  0.6   0  0.0 /usr/sbin/pppd file /tmp/ppp/options.wan0
  581   580 admin    S     1488  0.6   0  0.0 -sh
  162   160 admin    S     1484  0.6   0  0.0 /bin/sh
  602   581 admin    R     1484  0.6   0  0.0 top
  377     1 admin    S     1484  0.6   0  0.0 udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc -t2 -T5 -A160
  348     1 admin    S     1480  0.6   0  0.0 crond -l 8
  184     1 admin    S     1476  0.6   0  0.0 /sbin/klogd
  182     1 admin    S     1476  0.6   0  0.0 syslogd -m 0 -O /tmp/syslog.log -S -s 256 -l 7
  326     1 admin    R     1384  0.5   0  0.0 protect_srv
  330   326 admin    S     1384  0.5   0  0.0 protect_srv
  331   330 admin    S     1384  0.5   0  0.0 protect_srv
  361     1 admin    S     1368  0.5   0  0.0 rstats
  344     1 nobody   S     1360  0.5   0  0.0 dnsmasq --log-async
  580   333 admin    S     1336  0.5   0  0.0 dropbear -p 192.168.1.99:22 -I 1200 -j -k
  333     1 admin    S     1316  0.5   0  0.0 dropbear -p 192.168.1.99:22 -I 1200 -j -k
  372     1 admin    S     1272  0.5   0  0.0 /usr/sbin/acsd
  416     1 admin    S     1248  0.5   0  0.0 lpd
  350     1 admin    S     1244  0.5   0  0.0 networkmap --bootwait
  336     1 admin    S     1196  0.5   0  0.0 /bin/eapd
  369     1 admin    S     1192  0.4   0  0.0 lld2d br0
  349     1 admin    S     1152  0.4   0  0.0 /usr/sbin/infosvr br0
  576     1 admin    S      872  0.3   0  0.0 miniupnpd -f /etc/upnp/config
  125     1 admin    S      644  0.2   0  0.0 hotplug2 --persistent --no-coldplug
   98     2 admin    SW<      0  0.0   0  0.0 [mtdblockd]
    4     2 admin    SW<      0  0.0   0  0.0 [events/0]
    3     2 admin    SWN      0  0.0   0  0.0 [ksoftirqd/0]
   52     2 admin    SW       0  0.0   0  0.0 [pdflush]
  167     2 admin    SWN      0  0.0   0  0.0 [jffs2_gcd_mtd5]
    5     2 admin    SW<      0  0.0   0  0.0 [khelper]
    2     0 admin    SW<      0  0.0   0  0.0 [kthreadd]
   20     2 admin    SW<      0  0.0   0  0.0 [kblockd/0]
  186     2 admin    SW<      0  0.0   0  0.0 [khubd]
   54     2 admin    SW<      0  0.0   0  0.0 [aio/0]
   51     2 admin    SW       0  0.0   0  0.0 [pdflush]
  121     2 admin    SW<      0  0.0   0  0.0 [kmmcd]
   53     2 admin    SW<      0  0.0   0  0.0 [kswapd0]

Back in a few days ;)
 
Will do and no probs, I really appreciate the effort you put in to maintaining this firmware for all of us and for all the assistance you've given me!

I've spotted some different entries in the log:

Code: 
dnsmasq-dhcp[16240]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device

Full Log: https://pastebin.com/wi10duUd

This would have been occurring between when I pasted the last log, but before you'd given me further instructions on next steps. Sorry I don't have the full log. I guess it could just be further deterioration?, but thought I'd share just in case. Sadly I rebooted since then when you suggested the additional steps.
 
Status
Not open for further replies.

Similar threads

RMerlin
  • Locked
Release Asuswrt-Merlin 3006.102.5 is now available
11 12 13
Replies
248
Views
39K
RMerlin
RMerlin
Q
How to permanently disable autoconf on router (wsdd2 and avahi-daemon)
Replies
3
Views
758
iTyPsIDg
I
RMerlin
Release Asuswrt-Merlin 3004.388.9 is now available
12 13 14
Replies
268
Views
49K
johnny7556
J
RMerlin
  • Locked
Release Asuswrt-Merlin 3006.102.4 is now available
32 33 34
Replies
661
Views
85K
MDM
MDM
D
Release GNUTON- 388.7_1 release [Official FORK]
Replies
4
Views
3K
MatteoPV
M
Similar threads
Thread starter Title Forum Replies Date
RMerlin Beta Asuswrt-Merlin 3004.388.10 Beta is now available Asuswrt-Merlin 64
C Throughput/bandwidth of asuswrt-merlin supported routers Asuswrt-Merlin 18
C Is changing Iptables in Asuswrt-Merlin considered acceptable network management practice? Asuswrt-Merlin 11
H RT-BE88U - ASUSWRT-Merlin - Traffic Monitoring Asuswrt-Merlin 5
J Looking for advice: Would I benefit from installing Asuswrt-Merlin on my setup? Asuswrt-Merlin 4
dionhouston ListenPort for ASUSWRT-Merlin WireGuard -Clients-? Asuswrt-Merlin 1
K vlan on RT-BE92U asuswrt-merlin 3006.102.5 Asuswrt-Merlin 10
RMerlin Release Asuswrt-Merlin 3006.102.5 is now available Asuswrt-Merlin 248
Zakalwe Disallow OVPN clients on asuswrt-merlin from accessing ANY LAN devices, but allow WAN? Asuswrt-Merlin 3
garycnew Asuswrt-Merlin - Traffic Analyzer - Statistics: Hostname NVRAM Variable Asuswrt-Merlin 9

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 858 (members: 13, guests: 845)
Back
Top
✖