384.19 Crash ... Out of RAM

[sbsnb]

Running 384.19 on an RT-AC88U. As I was nearing 100 days of uptime my system crashed. Over the past couple months I watched the RAM graphic slowly grow from about 45% at start to very near 100% just the other day (the last time I looked).

This morning I had a non-functional router. No devices had internet access but I could access the GUI. I downloaded the log and then the GUI and WiFi went dead. No devices could even connect to the WiFi. I had to power cycle the router.

Here's what was in the log:

Jan  9 06:33:44 kernel: nt_monitor invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jan  9 06:33:44 kernel: [<8005701c>] (unwind_backtrace+0x0/0xf8) from [<800b12fc>] (dump_header.clone.7+0x6c/0x184)
Jan  9 06:33:44 kernel: [<800b12fc>] (dump_header.clone.7+0x6c/0x184) from [<800b15bc>] (oom_kill_process.clone.9+0x68/0x178)
Jan  9 06:33:44 kernel: [<800b15bc>] (oom_kill_process.clone.9+0x68/0x178) from [<800b1990>] (out_of_memory+0x154/0x2bc)
Jan  9 06:33:44 kernel: [<800b1990>] (out_of_memory+0x154/0x2bc) from [<800b4cb0>] (__alloc_pages_nodemask+0x564/0x58c)
Jan  9 06:33:44 kernel: [<800b4cb0>] (__alloc_pages_nodemask+0x564/0x58c) from [<800d2a80>] (read_swap_cache_async+0x104/0x1b4)
Jan  9 06:33:44 kernel: [<800d2a80>] (read_swap_cache_async+0x104/0x1b4) from [<800d2bb8>] (swapin_readahead+0x88/0x90)
Jan  9 06:33:44 kernel: [<800d2bb8>] (swapin_readahead+0x88/0x90) from [<800c675c>] (handle_mm_fault+0x5d8/0x844)
Jan  9 06:33:44 kernel: [<800c675c>] (handle_mm_fault+0x5d8/0x844) from [<8005864c>] (do_page_fault+0x178/0x1ec)
Jan  9 06:33:44 kernel: [<8005864c>] (do_page_fault+0x178/0x1ec) from [<800503a4>] (do_DataAbort+0x30/0x9c)
Jan  9 06:33:44 kernel: [<800503a4>] (do_DataAbort+0x30/0x9c) from [<803eb040>] (ret_from_exception+0x0/0x10)
Jan  9 06:33:44 kernel: Exception stack(0x99643fb0 to 0x99643ff8)
Jan  9 06:33:44 kernel: 3fa0:                                     00000011 00000000 7ebe1d7c 00010000
Jan  9 06:33:44 kernel: 3fc0: 00000000 7ebe1dac 7ebe1de4 7ebe1f56 0000869c 00009068 7ebe1df8 00000000
Jan  9 06:33:44 kernel: 3fe0: 2b02be84 7ebe1d78 2b017974 2afd1e84 20000010 ffffffff
Jan  9 06:33:44 kernel: Mem-info:
Jan  9 06:33:44 kernel: DMA per-cpu:
Jan  9 06:33:44 kernel: CPU    0: hi:  186, btch:  31 usd: 183
Jan  9 06:33:44 kernel: CPU    1: hi:  186, btch:  31 usd:  80
Jan  9 06:33:44 kernel: active_anon:13881 inactive_anon:13907 isolated_anon:0
Jan  9 06:33:44 kernel:  active_file:288 inactive_file:2026 isolated_file:0
Jan  9 06:33:44 kernel:  unevictable:0 dirty:0 writeback:32 unstable:0
Jan  9 06:33:44 kernel:  free:5592 slab_reclaimable:513 slab_unreclaimable:66351
Jan  9 06:33:44 kernel:  mapped:322 shmem:0 pagetables:378 bounce:0
Jan  9 06:33:44 kernel: DMA free:22368kB min:20480kB low:25600kB high:30720kB active_anon:55524kB inactive_anon:55628kB active_file:1152kB inactive_file:8104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:520192kB mlocked:0kB dirty:0kB writeback:128kB mapped:1288kB shmem:0kB slab_reclaimable:2052kB slab_unreclaimable:265404kB kernel_stack:1080kB pagetables:1512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no
Jan  9 06:33:44 kernel: lowmem_reserve[]: 0 0 0
Jan  9 06:33:44 kernel: DMA: 842*4kB 263*8kB 268*16kB 236*32kB 7*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 22368kB
Jan  9 06:33:44 kernel: 27842 total pagecache pages
Jan  9 06:33:44 kernel: 25517 pages in swap cache
Jan  9 06:33:44 kernel: Swap cache stats: add 32144405, delete 32118888, find 25881340/35561275
Jan  9 06:33:44 kernel: Free swap  = 1919380kB
Jan  9 06:33:44 kernel: Total swap = 2097148kB
Jan  9 06:33:44 kernel: 131072 pages of RAM
Jan  9 06:33:44 kernel: 6177 free pages
Jan  9 06:33:44 kernel: 2276 reserved pages
Jan  9 06:33:44 kernel: 52986 slab pages
Jan  9 06:33:44 kernel: 1807 pages shared
Jan  9 06:33:44 kernel: 25517 pages swap cached

I have been running Merlin builds for a long time and have never experienced an actual crash before, even with uptimes over 200 days. I'm curious what could have caused this.

 

[bbunge]

Running 384.19 on an RT-AC88U. As I was nearing 100 days of uptime my system crashed. Over the past couple months I watched the RAM graphic slowly grow from about 45% at start to very near 100% just the other day (the last time I looked).

View attachment 29276

This morning I had a non-functional router. No devices had internet access but I could access the GUI. I downloaded the log and then the GUI and WiFi went dead. No devices could even connect to the WiFi. I had to power cycle the router.

Here's what was in the log:

Jan  9 06:33:44 kernel: nt_monitor invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jan  9 06:33:44 kernel: [<8005701c>] (unwind_backtrace+0x0/0xf8) from [<800b12fc>] (dump_header.clone.7+0x6c/0x184)
Jan  9 06:33:44 kernel: [<800b12fc>] (dump_header.clone.7+0x6c/0x184) from [<800b15bc>] (oom_kill_process.clone.9+0x68/0x178)
Jan  9 06:33:44 kernel: [<800b15bc>] (oom_kill_process.clone.9+0x68/0x178) from [<800b1990>] (out_of_memory+0x154/0x2bc)
Jan  9 06:33:44 kernel: [<800b1990>] (out_of_memory+0x154/0x2bc) from [<800b4cb0>] (__alloc_pages_nodemask+0x564/0x58c)
Jan  9 06:33:44 kernel: [<800b4cb0>] (__alloc_pages_nodemask+0x564/0x58c) from [<800d2a80>] (read_swap_cache_async+0x104/0x1b4)
Jan  9 06:33:44 kernel: [<800d2a80>] (read_swap_cache_async+0x104/0x1b4) from [<800d2bb8>] (swapin_readahead+0x88/0x90)
Jan  9 06:33:44 kernel: [<800d2bb8>] (swapin_readahead+0x88/0x90) from [<800c675c>] (handle_mm_fault+0x5d8/0x844)
Jan  9 06:33:44 kernel: [<800c675c>] (handle_mm_fault+0x5d8/0x844) from [<8005864c>] (do_page_fault+0x178/0x1ec)
Jan  9 06:33:44 kernel: [<8005864c>] (do_page_fault+0x178/0x1ec) from [<800503a4>] (do_DataAbort+0x30/0x9c)
Jan  9 06:33:44 kernel: [<800503a4>] (do_DataAbort+0x30/0x9c) from [<803eb040>] (ret_from_exception+0x0/0x10)
Jan  9 06:33:44 kernel: Exception stack(0x99643fb0 to 0x99643ff8)
Jan  9 06:33:44 kernel: 3fa0:                                     00000011 00000000 7ebe1d7c 00010000
Jan  9 06:33:44 kernel: 3fc0: 00000000 7ebe1dac 7ebe1de4 7ebe1f56 0000869c 00009068 7ebe1df8 00000000
Jan  9 06:33:44 kernel: 3fe0: 2b02be84 7ebe1d78 2b017974 2afd1e84 20000010 ffffffff
Jan  9 06:33:44 kernel: Mem-info:
Jan  9 06:33:44 kernel: DMA per-cpu:
Jan  9 06:33:44 kernel: CPU    0: hi:  186, btch:  31 usd: 183
Jan  9 06:33:44 kernel: CPU    1: hi:  186, btch:  31 usd:  80
Jan  9 06:33:44 kernel: active_anon:13881 inactive_anon:13907 isolated_anon:0
Jan  9 06:33:44 kernel:  active_file:288 inactive_file:2026 isolated_file:0
Jan  9 06:33:44 kernel:  unevictable:0 dirty:0 writeback:32 unstable:0
Jan  9 06:33:44 kernel:  free:5592 slab_reclaimable:513 slab_unreclaimable:66351
Jan  9 06:33:44 kernel:  mapped:322 shmem:0 pagetables:378 bounce:0
Jan  9 06:33:44 kernel: DMA free:22368kB min:20480kB low:25600kB high:30720kB active_anon:55524kB inactive_anon:55628kB active_file:1152kB inactive_file:8104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:520192kB mlocked:0kB dirty:0kB writeback:128kB mapped:1288kB shmem:0kB slab_reclaimable:2052kB slab_unreclaimable:265404kB kernel_stack:1080kB pagetables:1512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no
Jan  9 06:33:44 kernel: lowmem_reserve[]: 0 0 0
Jan  9 06:33:44 kernel: DMA: 842*4kB 263*8kB 268*16kB 236*32kB 7*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 22368kB
Jan  9 06:33:44 kernel: 27842 total pagecache pages
Jan  9 06:33:44 kernel: 25517 pages in swap cache
Jan  9 06:33:44 kernel: Swap cache stats: add 32144405, delete 32118888, find 25881340/35561275
Jan  9 06:33:44 kernel: Free swap  = 1919380kB
Jan  9 06:33:44 kernel: Total swap = 2097148kB
Jan  9 06:33:44 kernel: 131072 pages of RAM
Jan  9 06:33:44 kernel: 6177 free pages
Jan  9 06:33:44 kernel: 2276 reserved pages
Jan  9 06:33:44 kernel: 52986 slab pages
Jan  9 06:33:44 kernel: 1807 pages shared
Jan  9 06:33:44 kernel: 25517 pages swap cached

I have been running Merlin builds for a long time and have never experienced an actual crash before, even with uptimes over 200 days. I'm curious what could have caused this.

Seems like your swap file filled up. Likely Skynet is the reason. See: https://www.snbforums.com/threads/skynet-router-firewall-security-enhancements.16798/page-308

 

[L&LD]

If that wasn't a full reset (M&M Config) after flashing to RMerlin 384.19_0, then it could be anything and hard to pinpoint the actual 'cause'.

 

[bennychan18]

Yeah! The Speedtest is available on my AX56U finally! Thanks Merlin!

 

[sbsnb]

Seems like your swap file filled up. Likely Skynet is the reason. See: https://www.snbforums.com/threads/skynet-router-firewall-security-enhancements.16798/page-308

That's not good. I have a 2GB swap file.

 

[sbsnb]

Since I rebooted I've been watching the RAM usage slowly climbing again. It started at 45% on fresh reboot and is now at 56%. If it crashes once it reaches 100% like before, I will crash again in around 40 days. I have tried using ps, top, and cat /proc/pid/status on various processes to find a culprit, but I don't see any. Stopping Skynet has no effect.

Any advice on finding where that RAM is allocated?

This is the output of free:

             total       used       free     shared    buffers     cached
Mem:        515184     286916     228268          0       1204      13444
-/+ buffers/cache:     272268     242916
Swap:      2097148          0    2097148

 

[sbsnb]

A large chunk of it appears to be dnscrypt-proxy. Restarting the service dropped RAM usage by 5% immediately, even though cat /proc/pid/status didn't show it using any more memory than when it started.

admin@RT-AC88U-B1E8:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        515184     282896     232288          0        824      12472
-/+ buffers/cache:     269600     245584
Swap:      2097148          0    2097148
admin@RT-AC88U-B1E8:/tmp/home/root# /opt/etc/init.d/S09dnscrypt-proxy2 restart
 Shutting down dnscrypt-proxy...              done.
 Starting dnscrypt-proxy...              done.
admin@RT-AC88U-B1E8:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        515184     257380     257804          0        836      15056
-/+ buffers/cache:     241488     273696
Swap:      2097148          0    2097148