RT-AX88U 386.3_2 goes about 3-4 weeks before RAM fills up.

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

theirongiant

Regular Contributor
I started sending all my RT-AX88U logs to a syslog server running in a docker container. Everything was fine for about a month, until I noticed one day when I logged into the router page the RAM and CPU bars weren't animating anymore. When I tried to ssh into the router, it would say 'connection reset by peer.'

So I rebooted the router and checked the logs that were captured on the syslog server.

For about a week, the logs indicated "no space left on device."
 
Last edited:

theirongiant

Regular Contributor
Really frustrating... I keep trying to add a comment and it says: "Oops! We ran into some problems. Please try again later. More error details may be in the browser console."
 

theirongiant

Regular Contributor
I'm going to add a cron job to my router to send the "top" command once an hour to the syslog server. Unfortunately, I can't seem to show this here because the forum keeps throwing a 403 Forbidden (I had to check the javascript console to figure this out). Gonna send a message to the admins.
 

L&LD

Part of the Furniture
What client devices are throwing these errors? No issues here from desktop and laptop computers.
 

Jeffrey Young

Senior Member
Really frustrating... I keep trying to add a comment and it says: "Oops! We ran into some problems. Please try again later. More error details may be in the browser console."

The website filters key words that are related to administration commands of Linux/Windows and key file references. If you are using mentioning key commands or listing a full pathname to certain directories, insert spaces in the filename path and/or commands.

e.g. "/ e t c /filename" to get the point across

I don't know of any single document that explains all the "hit" words/phrases.
 

theirongiant

Regular Contributor
The website filters key words that are related to administration commands of Linux/Windows and key file references. If you are using mentioning key commands or listing a full pathname to certain directories, insert spaces in the filename path and/or commands.

e.g. "/ e t c /filename" to get the point across

I don't know of any single document that explains all the "hit" words/phrases.
Don’t you think it’s a bit strict for a forum dedicated to troubleshooting small business network equipment, some of which encourages or even requires remote access and scripting via ssh? How are people supposed to share tips on how to automate things?
 

john9527

Part of the Furniture
Here's a little script you can try that can handle shrinking an open log file in most cases.....two parameters....the name of the logfile and the number of lines to keep in the log...

Code:
#!/bin/sh

#check-log

scr_name="$(basename $0)[$$]"

# Shrink logfile to last 100 lines

LOGFILE="$1"
if [ "$2" == "" ]; then
        LINES=100
else
        LINES=$2
fi

if [ -f "$LOGFILE" ]; then
        lines=$(wc -l < $LOGFILE)
        if [ $lines -gt $LINES ]; then
                echo "$(tail -n $LINES $LOGFILE)" > $LOGFILE 2>&1
                logger -s -t $scr_name "log: $LOGFILE $lines total lines $((lines-LINES)) lines pruned"
#       else
#               logger -s -t $scr_name "log: $LOGFILE $lines total lines"
        fi
else
        logger -s -t $scr_name "log: $LOGFILE not found"
fi

exit 0
 

ccleeland

New Around Here
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`

Safest is to do something like:

Bash:
tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename

The important thing is to NOT remove $LOGFILE, because the logger is holding that file open. You have to ensure that you overwrite the existing file (which the original script does) with the content you wish to persist.
 

john9527

Part of the Furniture
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`
May be due to the busybox implementations and the way the do/not do buffering, but I've never run into any problems. The problem with your solution, is that you are putting the intermediate file in /tmp, which is most likely where he is already running out space.
 

theirongiant

Regular Contributor
Sorry, I didn't mean "what router" but what storage device as indicated in the full error message, e.g. /var.

Thanks for the clarification. Here is a sample of the 'no space left' messages. It would seem that they are all in the /var/ mount point. The filesystem is tmpfs which I assume means a RAM drive. And the culprit does appear to be the strongswan.charon.log file.

Code:
2021-09-19T23:30:02+00:00 RT-AX88U-27B8-464C3A0-C hostapd: eth7: STA 92:52:4b:14:b6:f2 WPA: group key handshake completed (RSN)
2021-09-19T23:30:03+00:00 RT-AX88U-27B8-464C3A0-C hostapd: eth7: STA 38:f9:d3:1c:e0:1c WPA: group key handshake completed (RSN)
2021-09-19T23:38:40+00:00 RT-AX88U-27B8-464C3A0-C kernel: br0: port 7(eth7) neighbor 8000.2c:30:33:d3:04:9d lost
2021-09-19T23:38:40+00:00 RT-AX88U-27B8-464C3A0-C kernel: br0: topology change detected, propagating
2021-09-20T00:00:21+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T00:00:21+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-20T01:00:23+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:23+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/clientlist.lock: 28 No space left on device
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:33+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:33+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
[...]
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/wlcevent.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/tg_roaming.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/roaming.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/tg_bsd_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/networkmap.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/clientlist.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
 
Last edited:

RMerlin

Asuswrt-Merlin dev
Asus already fixed the strongswan log issue upstream a few months ago, the fix will be implemented downstream once they resume releasing GPL updates.
 

theirongiant

Regular Contributor
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`

Safest is to do something like:

Bash:
tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename

The important thing is to NOT remove $LOGFILE, because the logger is holding that file open. You have to ensure that you overwrite the existing file (which the original script does) with the content you wish to persist.

Thanks for the suggestion. I expanded on it a bit for a crontab and placed the script in the /jffs folder. Unfortunately, this did *not* shrink the log file as expected. I tried 100 lines, but it's still nearly 20MB.

Bash:
#!/bin/sh

LINES="$1"
LOGFILE="$2"

if [[ $# -lt 2 ]]  # if the script did not receive the two required arguments...
then
        echo "Error: you must provide two arguments to run this script.  Example: logtrimmer.sh 999 /path/to/file.log"
        exit 0
fi

tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename
echo "Log file $LOGFILE has been trimmed to the last $LINES lines."

And the actual cron job:
Code:
0 0 * * * /jffs/scripts/logtrimmer.sh 1000 /var/log/strongswan.charon.log
 
Last edited:

theirongiant

Regular Contributor
WTF...

The strongswan charon process appears to be holding the ENTIRE log file in memory and writing it back again to disk after it's been overwritten.

Observe...

Code:
[email protected]:/jffs/scripts# ./logtrimmer.sh 100 "/var/log/strongswan.charon.log" ; ls -la /var/log/
+ LINES=100
+ LOGFILE=/var/log/strongswan.charon.log
+ [[ 2 -lt 2 ]]
+ tmpfilename=/tmp/mktemp.17559
+ tail -n 100 /var/log/strongswan.charon.log
+ cat /tmp/mktemp.17559
+ echo Log file /var/log/strongswan.charon.log has been trimmed to the last 100 lines.
Log file /var/log/strongswan.charon.log has been trimmed to the last 100 lines.
drwxr-xr-x    3 admin    root           180 Sep 27 17:13 .
drwxrwxrwt   19 admin    root           520 Sep 28 16:52 ..
-rw-rw-rw-    1 admin    root            11 Sep 27 18:40 commit_ret
drwx------    4 admin    root            80 May  4  2018 cores
srw-rw-rw-    1 admin    root             0 May  4  2018 log
-rw-r--r--    1 admin    root         76963 Sep 28 16:53 log.nmbd
-rw-r--r--    1 admin    root           288 Sep 27 17:13 log.smbd
-rw-r--r--    1 admin    root             0 Sep 27 17:13 samba.log
-rw-rw-rw-    1 admin    root          7724 Sep 28 16:55 strongswan.charon.log

...Spongebob Card: "a few moments later..."

[email protected]:/jffs/scripts# ls -la /var/log
drwxr-xr-x    3 admin    root           180 Sep 27 17:13 .
drwxrwxrwt   19 admin    root           520 Sep 28 16:52 ..
-rw-rw-rw-    1 admin    root            11 Sep 27 18:40 commit_ret
drwx------    4 admin    root            80 May  4  2018 cores
srw-rw-rw-    1 admin    root             0 May  4  2018 log
-rw-r--r--    1 admin    root         76963 Sep 28 16:53 log.nmbd
-rw-r--r--    1 admin    root           288 Sep 27 17:13 log.smbd
-rw-r--r--    1 admin    root             0 Sep 27 17:13 samba.log
-rw-rw-rw-    1 admin    root      18795658 Sep 28 16:55 strongswan.charon.log

I just can't win...
 
Last edited:

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top