Memory leak in Asuswrt-Merlin

[Adrian Tundrea]

Hello. First of all thanks for this wonderful firmware. I really appreciate the focus on stability.

However I would like to report and issue. I have noticed on my RT-AC68U what I consider to be a terrible memory leak in both the official firmware and in Asuswrt-Merlin one. In fact I have switched to Merlin in hope it would be gone, but it is not the case. The leak is pretty important considering I loose about 15-20 MB / 24h uptime. The version I currently have is 378.56_2. As for the official ones I have tried 3.0.0.4.380.1842, 3.0.0.4.378.9313 and 3.0.0.4.378.8258. They all have the same issue in roughly the same degree.

As for specifics... I have everything disabled... FTP, Samba, nothing else installed, nothing attached to USB, just pure router. However I have a OpenVPN server on it... always connected to a client and also a PC connected to it that permanently upload and downloads data, hosts a Plex server and a FTP one.

Let me know if I can help you with more information on this issue.

Thanks,

 

[RMerlin]

A 15-20 MB memory leak would imply the router would run out of memory after 10 days. Since quite a lot of people are managing multiple months of uptime, it's unlikely to be a leak. It's more likely to be simply Linux buffering.

 

[Adrian Tundrea]

A 15-20 MB memory leak would imply the router would run out of memory after 10 days. Since quite a lot of people are managing multiple months of uptime, it's unlikely to be a leak. It's more likely to be simply Linux buffering.

I have noticed the decrease about 10 days ago on 3.0.0.4.380.1842. I checked daily and after about almoust 5 days the free RAM was around 140 MB (down from around 190 MB). Last night I installed yours (btw the Stealth Mode is great... I cannot understand why Asus didn't implement that directly) and from over 200 MB free I am now down to 188 RAM. Maybe it is connected to the specific way I use it.

 

[BWR]

I gather you're a Windows user. There is no leak, it's just how the Linux kernel (very effectively) manages and uses system memory. As soon as a userspace application requests allocation, the kernel flushes as much as it's needed from its own buffers/caches/etc and allocates it.

There's even a website dedicated to this "problem": http://www.linuxatemyram.com
Login into the router (telnet/ssh) and run "free -m".

 

[RMerlin]

I have noticed the decrease about 10 days ago on 3.0.0.4.380.1842. I checked daily and after about almoust 5 days the free RAM was around 140 MB (down from around 190 MB). Last night I installed yours (btw the Stealth Mode is great... I cannot understand why Asus didn't implement that directly) and from over 200 MB free I am now down to 188 RAM. Maybe it is connected to the specific way I use it.

Also keep in mind that there are logfiles that are stored in RAM, which will cause the available RAM to drop more quickly during the first few days, as those logfiles are initially generated.

 

[Adrian Tundrea]

I gather you're a Windows user. There is no leak, it's just how the Linux kernel (very effectively) manages and uses system memory. As soon as a userspace application requests allocation, the kernel flushes as much as it's needed from its own buffers/caches/etc and allocates it.

There's even a website dedicated to this "problem": http://www.linuxatemyram.com
Login into the router (telnet/ssh) and run "free -m".

I am a Windows user, but I am aware of how Linux works. I program in Java a lot and the way the garbage collector works generates the same behavior... freeing memory only when needed.

This is what PuTTy gives me:
42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 65788 189932 0 2176
-/+ buffers: 63612 192108
Swap: 0 0 0
42@ASUS_RT-AC68U:/tmp/home/root#

As I see it only a small part of the RAM is given to buffer. As you can see since I initially opened the thread I lost ~3MB... I have 185MB not including the buffers (for 24h it is 18-20MB)

Also keep in mind that there are logfiles that are stored in RAM, which will cause the available RAM to drop more quickly during the first few days, as those logfiles are initially generated.

I assume there are more logs then the one in System Logs, right? I will come back in a day or two to see the progress. RMerlin it is normal to store logs in RAM? I mean it seems such a bad design decision on ASUS... a waste of RAM. Granted I am not a Linux specialist...

 

[RMerlin]

I am a Windows user, but I am aware of how Linux works. I program in Java a lot and the way the garbage collector works generates the same behavior... freeing memory only when needed.

This is what PuTTy gives me:
42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 65788 189932 0 2176
-/+ buffers: 63612 192108
Swap: 0 0 0
42@ASUS_RT-AC68U:/tmp/home/root#

As I see it only a small part of the RAM is given to buffer. As you can see since I initially opened the thread I lost ~3MB... I have 185MB not including the buffers (for 24h it is 18-20MB)



I assume there are more logs then the one in System Logs, right? I will come back in a day or two to see the progress. RMerlin it is normal to store logs in RAM? I mean it seems such a bad design decision on ASUS... a waste of RAM. Granted I am not a Linux specialist...

Logs are stored in RAM because most of them don't need to survive resets, and also because most of their routers have no writable storage space. Last year they started keeping a backup of the main system log in the JFFS partition, but other temporary logs are still kept in /tmp as there's no point in keeping them over reboots.

 

[eddiez]

I am a Windows user, but I am aware of how Linux works. I program in Java a lot and the way the garbage collector works generates the same behavior... freeing memory only when needed.

This is what PuTTy gives me:
42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 65788 189932 0 2176
-/+ buffers: 63612 192108
Swap: 0 0 0
42@ASUS_RT-AC68U:/tmp/home/root#

As I see it only a small part of the RAM is given to buffer. As you can see since I initially opened the thread I lost ~3MB... I have 185MB not including the buffers (for 24h it is 18-20MB)



I assume there are more logs then the one in System Logs, right? I will come back in a day or two to see the progress. RMerlin it is normal to store logs in RAM? I mean it seems such a bad design decision on ASUS... a waste of RAM. Granted I am not a Linux specialist...

A decrease of memory does not 'prove' that a memory leak is filling up your memory. A router 'works'...ie uses CPU and memory. What have you done to make visible that the memory used is not supposed to be used?

 

[Adrian Tundrea]

A decrease of memory does not 'prove' that a memory leak is filling up your memory. A router 'works'...ie uses CPU and memory. What have you done to make visible that the memory used is not supposed to be used?

Well a continuous decrease in memory might prove just that. Now it is at 178 free RAM, with some 2,5 MB for buffers. Nothing has changed in its functionality since I booted it or even from last night, so nothing extra was enabled, only network traffic. Either it is the logs, some form of caching or a leak. Here is the more output:

ASUSWRT-Merlin RT-AC68U_3.0.0.4 Mon Nov 2 14:58:44 UTC 2015
42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 74084 181636 0 2460
-/+ buffers: 71624 184096
Swap: 0 0 0
42@ASUS_RT-AC68U:/tmp/home/root# cat /proc/meminfo
MemTotal: 255720 kB
MemFree: 185052 kB
Buffers: 916 kB
Cached: 6560 kB
SwapCached: 0 kB
Active: 26296 kB
Inactive: 4800 kB
Active(anon): 23960 kB
Inactive(anon): 320 kB
Active(file): 2336 kB
Inactive(file): 4480 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 23620 kB
Mapped: 4852 kB
Shmem: 660 kB
Slab: 24880 kB
SReclaimable: 1716 kB
SUnreclaim: 23164 kB
KernelStack: 552 kB
PageTables: 724 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127860 kB
Committed_AS: 37512 kB
VmallocTotal: 516096 kB
VmallocUsed: 18108 kB
VmallocChunk: 455428 kB
42@ASUS_RT-AC68U:/tmp/home/root#

From that I can see the cache is only ~6MB. I'm sorry I didn't find this command yesterday, but I will come back in a couple of hours to see what happens.

 

[eddiez]

Well a continuous decrease in memory might prove just that. Now it is at 178 free RAM, with some 2,5 MB for buffers. Nothing has changed in its functionality since I booted it or even from last night, so nothing extra was enabled, only network traffic. Either it is the logs, some form of caching or a leak. Here is the more output:

ASUSWRT-Merlin RT-AC68U_3.0.0.4 Mon Nov 2 14:58:44 UTC 2015
42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 74084 181636 0 2460
-/+ buffers: 71624 184096
Swap: 0 0 0
42@ASUS_RT-AC68U:/tmp/home/root# cat /proc/meminfo
MemTotal: 255720 kB
MemFree: 185052 kB
Buffers: 916 kB
Cached: 6560 kB
SwapCached: 0 kB
Active: 26296 kB
Inactive: 4800 kB
Active(anon): 23960 kB
Inactive(anon): 320 kB
Active(file): 2336 kB
Inactive(file): 4480 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 23620 kB
Mapped: 4852 kB
Shmem: 660 kB
Slab: 24880 kB
SReclaimable: 1716 kB
SUnreclaim: 23164 kB
KernelStack: 552 kB
PageTables: 724 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127860 kB
Committed_AS: 37512 kB
VmallocTotal: 516096 kB
VmallocUsed: 18108 kB
VmallocChunk: 455428 kB
42@ASUS_RT-AC68U:/tmp/home/root#

From that I can see the cache is only ~6MB. I'm sorry I didn't find this command yesterday, but I will come back in a couple of hours to see what happens.

"Only Network traffic".... you do realize that as long the router is online and has clients connected data gets cached in the router and logs keep growing, right?

 

[RMerlin]

Check the content of /tmp, this is where the logfiles and other temporary files are located.

 

[ColinTaylor]

The "memory issue" frequently comes up. Here is the same question: http://www.snbforums.com/threads/merlin-380-57-potential-memory-issue.29928/

 

[Adrian Tundrea]

The "memory issue" frequently comes up. Here is the same question: http://www.snbforums.com/threads/merlin-380-57-potential-memory-issue.29928/

Yes that is exactly the same behavior. I had to do a restart some 6h ago and my current stats is 196 MB free.

If I do df -h /tmp:

42@ASUS_RT-AC68U:/tmp# df -h /tmp
Filesystem Size Used Available Use% Mounted on
tmpfs 124.9M 412.0K 124.5M 0% /tmp
42@ASUS_RT-AC68U:/tmp#

So the temp file is minor.

 

[RMerlin]

Try manually flushing caches:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

 

[Adrian Tundrea]

Try manually flushing caches:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

I have tried it now, but it is not clear since the used memory hasn't climbed up sufficiently to notice the difference. Here is the output:

ASUSWRT-Merlin RT-AC68U_3.0.0.4 Mon Nov 2 14:58:44 UTC 2015
42@ASUS_RT-AC68U:/tmp/home/root# free && sync && echo 3 > /proc/sys/vm/drop_cach
es && free
total used free shared buffers
Mem: 255720 58596 197124 0 2312
-/+ buffers: 56284 199436
Swap: 0 0 0
total used free shared buffers
Mem: 255720 52996 202724 0 296
-/+ buffers: 52700 203020
Swap: 0 0 0

Anyway I surely will come back with updates in a day or so when the memory usage will be more obvious.

 

[Adrian Tundrea]

Here are this morning's results. The UI showed 187 MB free (20h uptime).

42@ASUS_RT-AC68U:/tmp/home/root# free -m
total used free shared buffers
Mem: 255720 64604 191116 0 2336
-/+ buffers: 62268 193452
Swap: 0 0 0

42@ASUS_RT-AC68U:/tmp/home/root# cat /proc/meminfo
MemTotal: 255720 kB
MemFree: 194124 kB
Buffers: 936 kB
Cached: 6492 kB
SwapCached: 0 kB
Active: 20152 kB
Inactive: 4924 kB
Active(anon): 17980 kB
Inactive(anon): 300 kB
Active(file): 2172 kB
Inactive(file): 4624 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 17676 kB
Mapped: 4820 kB
Shmem: 604 kB
Slab: 21816 kB
SReclaimable: 736 kB
SUnreclaim: 21080 kB
KernelStack: 552 kB
PageTables: 688 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127860 kB
Committed_AS: 30920 kB
VmallocTotal: 516096 kB
VmallocUsed: 18108 kB
VmallocChunk: 485236 kB

42@ASUS_RT-AC68U:/tmp/home/root# free && sync && echo 3 > /proc/sys/vm/drop_cach
es && free
total used free shared buffers
Mem: 255720 61636 194084 0 936
-/+ buffers: 60700 195020
Swap: 0 0 0
total used free shared buffers
Mem: 255720 59896 195824 0 268
-/+ buffers: 59628 196092
Swap: 0 0 0

42@ASUS_RT-AC68U:/tmp/home/root# df -h /tmp
Filesystem Size Used Available Use% Mounted on
tmpfs 124.9M 492.0K 124.4M 0% /tmp

To the untrained eye it doesn't seem to be a cache issue or a logging one. Do you have any other commands I could run that could help provide more information?

 

[RMerlin]

To the untrained eye it doesn't seem to be a cache issue or a logging one. Do you have any other commands I could run that could help provide more information?

Not at this stage. Having memory usage increase over time is expected, as the router is running a number of services. dnsmasq has its own cache, the DPI engine will also generate internal databases, the networkmap will increase its internal list of known clients, etc... Any further debugging would require a lot of work, something not worth doing at this stage unless routers actually DO run out of memory, confirming that there is really a leak worth investigating, and not just some normal increased memory usage.

What makes me doubt that there is really a memory leak is the fact that you experienced the same thing with numerous past firmware releases, some of which have shown to be stable enough for people to reach multiple weeks of uptime. I myself reached three weeks uptime during the Holidays, and my router's free memory was still normal.

 

[Adrian Tundrea]

Not at this stage. Having memory usage increase over time is expected, as the router is running a number of services. dnsmasq has its own cache, the DPI engine will also generate internal databases, the networkmap will increase its internal list of known clients, etc... Any further debugging would require a lot of work, something not worth doing at this stage unless routers actually DO run out of memory, confirming that there is really a leak worth investigating, and not just some normal increased memory usage.

Totally understand. I will update this thread on a daily basis with the latest stats. Hopefully I am mistaking and the increase will stop at some point.

 

[ColinTaylor]

Daily updates are much too frequent, we'll just get swamped with pages and pages of stats. Try once or twice a week.

If you want to collect daily stats I'd suggest you put them into a single spreadsheet and give us a link to it.

 

[highwire]

A 15-20 MB memory leak would imply the router would run out of memory after 10 days. Since quite a lot of people are managing multiple months of uptime, it's unlikely to be a leak. It's more likely to be simply Linux buffering.

I'm at 79 days uptime with 380.57 on a 66U and 87U. No memory issues, but I'm not using QoS or other "advanced" features.