Out of Memory Errors 388.1 / No space left on device [SOLUTIONS]

[mmacedo]

8.0K    ./dm
880.0K  ./bwdpi
8.0K    ./err_rules
4.0K    ./asusdebuglog
4.0K    ./ppp
332.0K  ./.diag
0       ./asusfbsvcs
24.0K   ./db
8.0K    ./avahi
12.0K   ./asdfile
0       ./netool
16.0K   ./nc
0       ./inadyn.cache
0       ./share
0       ./notify
0       ./mnt
4.0K    ./home
0       ./var
332.0K  ./etc
0       ./confmtd
1.9M    .

I'll try it again after a few days.

After 14 hours, increase in ./diag from 332K to 1.5M and in . from 1.9M to 3.1M

12.0K   ./asdfile
8.0K    ./dm
880.0K  ./bwdpi
8.0K    ./err_rules
4.0K    ./asusdebuglog
4.0K    ./ppp
1.5M    ./.diag
0       ./asusfbsvcs
24.0K   ./db
8.0K    ./avahi
0       ./netool
16.0K   ./nc
0       ./inadyn.cache
0       ./share
0       ./notify
0       ./mnt
4.0K    ./home
0       ./var
332.0K  ./etc
0       ./confmtd
3.1M    .

From now on, Ill be using the format suggested by @Martinski. Then . becomes /tmp and ./.diag becomes /tmp/.diag. Outoput is in kilobytes.

du -xk -d 1 /tmp | sort -nrt ' ' -k 1 | grep -v "^0"

Tue Jan 24 08:09:57 EST 2023

3224    /tmp
1516    /tmp/.diag
880     /tmp/bwdpi
332     /tmp/etc
24      /tmp/db
16      /tmp/nc
12      /tmp/asdfile
8       /tmp/err_rules
8       /tmp/dm
8       /tmp/avahi
4       /tmp/ppp
4       /tmp/home
4       /tmp/asusdebuglog

 

[mmacedo]

What does that mean?

Wait some time, run the command again and compare the results to find out what is increasing. If you compare yours with mine (which has been recently rebooted), you'll notice the largest discrepancies are /tmp and /tmp.diag.

Your /tmp is using close to 55MB and your /tmp/.diag close to 52MB. These two together are using more than 100MB of you router's memory. Something seems to be wrong.

Run this command as well, to check the networkmap process:

cat /proc/$(pidof networkmap)/status | grep -E -w 'Pid|Name|VmRSS'

 

[ColinTaylor]

Your /tmp is using close to 55MB and your /tmp/.diag close to 52MB. These two together are using more than 100MB of you router's memory. Something seems to be wrong.

This is incorrect. Bear in mind that the number for /tmp includes all the subdirectories under /tmp as well, not just the files in the root directory. So the total size of everything is 55MB, not 100MB.

 

[mmacedo]

This is incorrect. Bear in mind that the number for /tmp includes all the subdirectories under /tmp as well, not just the files in the root directory. So the total size of everything is 55MB, not 100MB.

Yes, that's right, my bad. So, what is increasing in my router is basically /tmp/.diag

 

[mmacedo]

Trying to find out what's the issue with /tmp/.diag:

ls -1lhS /tmp/.diag

Tue Jan 24 08:34:12 EST 2023

-rw-rw-rw-    1  776.0K Jan 24 08:32 wifi_detect.db
-rw-rw-rw-    1  452.0K Jan 24 08:32 eth_detect.db
-rw-rw-rw-    1  136.0K Jan 24 08:32 net_detect.db
-rw-rw-rw-    1   96.0K Jan 24 08:32 sys_detect.db
-rw-rw-rw-    1   20.0K Jan 24 00:32 port_status_change.db
-rw-rw-rw-    1   20.0K Jan 23 16:19 wifi_setting.db
-rw-rw-rw-    1   20.0K Jan 23 16:19 sys_setting.db
-rw-rw-rw-    1   20.0K Jan 23 15:42 port_status_usb_change.db
-rw-rw-rw-    1   20.0K Jan 23 15:42 channel_change.db

 

[mmacedo]

I don't know if this is related, but was checking the log and it happened last night:

Jan 23 23:21:24 kernel: potentially unexpected fatal signal 11.
Jan 23 23:21:24 kernel: CPU: 1 PID: 1072 Comm: asd Tainted: P           O    4.1.52 #2
Jan 23 23:21:24 kernel: Hardware name: Broadcom-v8A (DT)
Jan 23 23:21:24 kernel: task: ffffffc01505e1c0 ti: ffffffc013f70000 task.ti: ffffffc013f70000
Jan 23 23:21:24 kernel: PC is at 0xf7001a20
Jan 23 23:21:24 kernel: LR is at 0xf7002ab8
Jan 23 23:21:24 kernel: pc : [<00000000f7001a20>] lr : [<00000000f7002ab8>] pstate: 60010010
Jan 23 23:21:24 kernel: sp : 00000000ffb90298
Jan 23 23:21:24 kernel: x12: 00000000f7016154
Jan 23 23:21:24 kernel: x11: 00000000f70051fc x10: 00000000f7738ab8
Jan 23 23:21:24 kernel: x9 : 00000000007d0b58 x8 : 0000000000000000
Jan 23 23:21:24 kernel: x7 : 00000000007d09d8 x6 : 00000000ffb902bc
Jan 23 23:21:24 kernel: x5 : 00000000ffb902c0 x4 : 00000000007d4790
Jan 23 23:21:24 kernel: x3 : 00000000f6c00ea8 x2 : 0000000000006e6f
Jan 23 23:21:24 kernel: x1 : 0000000000000080 x0 : 0000000000000000

 

[Elmer]

AX88/388.1 FW: So, I'm going to jump on this bandwagon. I had upgraded to 388.1, and a few days later I couldn't hold an ISP connection. First I blamed the ISP, but their diagnostics showed nothing, so I looked inward. I decided that a maybe a dirty upgrade to 388.1 was the cause, so I bit the bullet and did a factory reset (painful with aimesh in play). That seemed to solve the problem - for about 3 days. I watched as memory closed in on 1G and figured it was the way 388 allocated memory (oddly never attempted to use swap), then boom, no ISP connection. Did a reboot, memory went back to 488M, and no problems again - for awhile. When things work, they work well, so for now I'm rebooting every night at 3 in the morning, and waiting for an Asus fix. I strongly believe there is a fundamental FW problem.

 

[ColinTaylor]

AX88/388.1 FW: So, I'm going to jump on this bandwagon. I had upgraded to 388.1, and a few days later I couldn't hold an ISP connection. First I blamed the ISP, but their diagnostics showed nothing, so I looked inward. I decided that a maybe a dirty upgrade to 388.1 was the cause, so I bit the bullet and did a factory reset (painful with aimesh in play). That seemed to solve the problem - for about 3 days. I watched as memory closed in on 1G and figured it was the way 388 allocated memory (oddly never attempted to use swap), then boom, no ISP connection. Did a reboot, memory went back to 488M, and no problems again - for awhile. When things work, they work well, so for now I'm rebooting every night at 3 in the morning, and waiting for an Asus fix. I strongly believe there is a fundamental FW problem.

The issue discussed in the thread seems specific to the RT-AX68U. So unless you're seeing the same behaviour regarding the networkmap process (and possibly /tmp/.diag) I'd say your problem is different.

 

[TF1470]

Here are my logs 11 hours later: (10:15 PM CET)

58064 /tmp
55948 /tmp/.diag
876 /tmp/bwdpi
376 /tmp/etc
24 /tmp/nc
24 /tmp/db
16 /tmp/dm
12 /tmp/asdfile
8 /tmp/avahi
4 /tmp/home
4 /tmp/asusdebuglog

And for the details of the file /tmp/.diag :
-rw-rw-rw- 1 38.6M Jan 24 22:13 stainfo.db
-rw-rw-rw- 1 9.6M Jan 24 22:13 wifi_detect.db
-rw-rw-rw- 1 3.8M Jan 24 22:13 eth_detect.db
-rw-rw-rw- 1 1.3M Jan 24 22:13 net_detect.db
-rw-rw-rw- 1 916.0K Jan 24 22:13 sys_detect.db
-rw-rw-rw- 1 312.0K Jan 24 22:05 port_status_change.db
-rw-rw-rw- 1 20.0K Jan 16 10:47 channel_change.db
-rw-rw-rw- 1 20.0K Jan 16 10:47 port_status_usb_change.db
-rw-rw-rw- 1 20.0K Jan 24 16:52 stainfo_stable.db
-rw-rw-rw- 1 20.0K Jan 16 10:47 sys_setting.db
-rw-rw-rw- 1 20.0K Jan 16 10:47 wifi_dfs.db
-rw-rw-rw- 1 20.0K Jan 16 10:47 wifi_setting.db

 

[RMerlin]

Here are my logs 11 hours later: (10:15 PM CET)

Those databases are created by the closed source conn_diag process.

I suspect stainfo might be a list of either associated clients, or visible clients. I wonder if that retarded MAC randomnizer some clients implement might be causing it to get filled by all these bogus MACs. Try to have a look at its content, see if you can recognize anything:

sqlite3 /tmp/.diag/ stainfo.db .dump

 

[mmacedo]

I don't have stainfo, but using sqlite3 I was able to play with wifi_detect.

This is the table schema:

CREATE TABLE DATA_INFO (data_id INTEGER PRIMARY KEY AUTOINCREMENT, node_type TEXT DEFAULT '' NOT NULL,node_ip TEXT DEFAULT '' NOT NULL,node_mac TEXT DEFAULT '' NOT NULL,band TEXT DEFAULT '' NOT NULL,ifname TEXT DEFAULT '' NOT NULL,mac TEXT DEFAULT '' NOT NULL,noise INT DEFAULT 0 NOT NULL,mcs TEXT DEFAULT '' NOT NULL,capability TEXT DEFAULT '' NOT NULL,subif_count INT DEFAULT 0 NOT NULL,subif_ssid TEXT DEFAULT '' NOT NULL,chanim TEXT DEFAULT '' NOT NULL,tx_rate DOUBLE DEFAULT 0 NOT NULL,rx_rate DOUBLE DEFAULT 0 NOT NULL,tx_byte UNSIGNED BIGINT DEFAULT 0 NOT NULL,rx_byte UNSIGNED BIGINT DEFAULT 0 NOT NULL,tx_diff UNSIGNED BIGINT DEFAULT 0 NOT NULL,rx_diff UNSIGNED BIGINT DEFAULT 0 NOT NULL,txop INT DEFAULT 0 NOT NULL, data_time TIMESTAMP)

Every one minute sharp, it inserts 6 records in this table, being:

• node_type='C' node_ip=<ROUTER IP> band='2G'
• node_type='C' node_ip=<ROUTER IP> band='5G'
• node_type='R' node_ip=<NODE A IP> band='2G'
• node_type='R' node_ip=<NODE A IP> band='5G'
• node_type='R' node_ip=<NODE B IP> band='2G'
• node_type='R' node_ip=<NODE B IP> band='5G'

I don't know if there's any mechanism to delete older records after a while, but what I can see this db is always increasing in size.

I'll play with the other ones now.

 

[mmacedo]

BTW, would it be safe to just kill conn_diag? Would it fix the issue?

 

[mmacedo]

Every one minute sharp, it inserts 6 records in this table, being:

Same thing for eth_detect, 6 new records every minute.

CREATE TABLE DATA_INFO (data_id INTEGER PRIMARY KEY AUTOINCREMENT, node_type TEXT DEFAULT '' NOT NULL,node_ip TEXT DEFAULT '' NOT NULL,node_mac TEXT DEFAULT '' NOT NULL,type TEXT DEFAULT '' NOT NULL,tx_rate DOUBLE DEFAULT 0 NOT NULL,rx_rate DOUBLE DEFAULT 0 NOT NULL,tx_byte UNSIGNED BIGINT DEFAULT 0 NOT NULL,rx_byte UNSIGNED BIGINT DEFAULT 0 NOT NULL,tx_diff UNSIGNED BIGINT DEFAULT 0 NOT NULL,rx_diff UNSIGNED BIGINT DEFAULT 0 NOT NULL, data_time TIMESTAMP)

• node_type='C' node_ip=<ROUTER IP> type='FH'
• node_type='C' node_ip=<ROUTER IP> type='BH'
• node_type='R' node_ip=<NODE A IP> type='FH'
• node_type='R' node_ip=<NODE A IP> type='BH'
• node_type='R' node_ip=<NODE B IP> type='FH'
• node_type='R' node_ip=<NODE B IP> type='BH'

 

[mmacedo]

net_detect: 1 new record every minute

CREATE TABLE DATA_INFO (data_id INTEGER PRIMARY KEY AUTOINCREMENT, node_type TEXT DEFAULT '' NOT NULL,node_ip TEXT DEFAULT '' NOT NULL,node_mac TEXT DEFAULT '' NOT NULL,wan_unit TEXT DEFAULT '' NOT NULL,link_wan TEXT DEFAULT '' NOT NULL,runIP TEXT DEFAULT '' NOT NULL,runMASK TEXT DEFAULT '' NOT NULL,runGATE TEXT DEFAULT '' NOT NULL,got_Route TEXT DEFAULT '' NOT NULL,got_Redirect TEXT DEFAULT '' NOT NULL,ret_dns TEXT DEFAULT '' NOT NULL,ret_ping TEXT DEFAULT '' NOT NULL, data_time TIMESTAMP)

• node_type='C' node_ip=<ROUTER IP> wan_unit='0' link_wan='1' runIP=<EXTERNAL IP>

 

[mmacedo]

sys_detect: 1 new record every minute

CREATE TABLE DATA_INFO (data_id INTEGER PRIMARY KEY AUTOINCREMENT, node_type TEXT DEFAULT '' NOT NULL,node_ip TEXT DEFAULT '' NOT NULL,node_mac TEXT DEFAULT '' NOT NULL,cpu_usage INT DEFAULT 0 NOT NULL,mem_usage INT DEFAULT 0 NOT NULL,cpu_temp INT DEFAULT 0 NOT NULL,temp_2g INT DEFAULT 0 NOT NULL,temp_5g1 INT DEFAULT 0 NOT NULL,temp_5g2 INT DEFAULT 0 NOT NULL,temp_6g1 INT DEFAULT 0 NOT NULL,temp_6g2 INT DEFAULT 0 NOT NULL,mem_committed_as INT DEFAULT 0 NOT NULL, data_time TIMESTAMP)

• node_type='C' node_ip=<ROUTER IP>

That's all growing up in /tmp/.diag

 

[dave14305]

BTW, would it be safe to just kill conn_diag? Would it fix the issue?

run service stop_conn_diag and see how it goes.

 

[Martinski]

Trying to find out what's the issue with /tmp/.diag:

ls -1lhS /tmp/.diag

Tue Jan 24 08:34:12 EST 2023

-rw-rw-rw-    1  776.0K Jan 24 08:32 wifi_detect.db
-rw-rw-rw-    1  452.0K Jan 24 08:32 eth_detect.db
-rw-rw-rw-    1  136.0K Jan 24 08:32 net_detect.db
-rw-rw-rw-    1   96.0K Jan 24 08:32 sys_detect.db
-rw-rw-rw-    1   20.0K Jan 24 00:32 port_status_change.db
-rw-rw-rw-    1   20.0K Jan 23 16:19 wifi_setting.db
-rw-rw-rw-    1   20.0K Jan 23 16:19 sys_setting.db
-rw-rw-rw-    1   20.0K Jan 23 15:42 port_status_usb_change.db
-rw-rw-rw-    1   20.0K Jan 23 15:42 channel_change.db

To list files/dirs in "/tmp" subdirectories whose size is 100KB or larger, you can use the following command:

du -axk /tmp | sort -nrt ' ' -k 1 | grep "^[0-9]\{3,\}"

To list only those files/dirs (>= 100KB) that are in the "/tmp/.diag" subdirectory:

du -axk /tmp/.diag | sort -nrt ' ' -k 1 | grep "^[0-9]\{3,\}"

HTH

 

[TF1470]

run service stop_conn_diag and see how it goes.

I activated this command, we will see in a few hours if there is any change.

 

[mmacedo]

run service stop_conn_diag and see how it goes.

I'll let them alone for a few days to see if ASUS set a size limit on them and they stop increasing.

@router:/tmp/home/root# sqlite3 /tmp/.diag/wifi_detect.db
SQLite version 3.25.3 2018-11-05 20:37:38
Enter ".help" for usage hints.
sqlite> select count () from DATA_INFO;
14496

14,496 / 6 (records/minute) = 2,416 minutes = 1 day, 16 hours, 16 minutes. It's pretty close to router's uptime:

Hopefully, in a few days it'll start dropping older records as it adds new ones to the table.

 

[TF1470]

I activated this command, we will see in a few hours if there is any change.

That's it, two days after activating this command, I no longer notice an increase in the ram, it remains on 70%. I don't have a stability problem, everything looks correct.

So, what is the purpose of these writings in this file?