kmalloc-96 - Memory leak in Kernel

[kvic]

Asus ARM routers. 380 series FW. Memory leak in kernel. Inspect by command line:

$ cat /proc/slabinfo | grep kmalloc-96

kmalloc-96 14311 15288 96 42 1 : tunables 0 0 0 : slabdata 364 364 0

Each slot in this pool is 96 bytes. The first number is the active number of slots. In the above example 14311 slots are active which translate to ~ 1341KB. On fresh reboot, ~ 900 active slots.

Didn't pay attention when I was on 378.55. So not sure if it's 380 series FW specific..

Anyone else see the leak on ARM routers with a 380 series FW?

 

[kvic]

A plot of # active slots against time at 5-min interval.

The router already up for a day when the sampling started. Hence first number is about 9000 active slots.

 

[sfx2000]

This one probably needs to be addressed sooner than later...

There was a couple of security issues wrapped around similar items in the 2.6.35/36 kernels - credential leak and a possible exploit.

 

[KevTech]

Router up time is one month.

Firmware 380_3264

kmalloc-96 963 2058 96 42 1 : tunables 0 0 0 : slabdata 49 49 0

 

[kvic]

Router up time is one month.

Firmware 380_3264

kmalloc-96 963 2058 96 42 1 : tunables 0 0 0 : slabdata 49 49 0

Thanks for the feedback. Do you have DPI (AiProtection, Adaptive QoS) and CTF turned on?

 

[KevTech]

Thanks for the feedback. Do you have DPI (AiProtection, Adaptive QoS) and CTF turned on?

Only CTF is on.

 

[appleseed]

uptime: 10 days 12 hours 14 minute(s) 20 seconds

kmalloc-96 627 756 96 42 1 : tunables 0 0 0 : slabdata 18 18 0

USB, Guest, TA, AiProtection, Adaptive QoS, all off, CTF on

AC68U/Merlin 380.59

 

[sfx2000]

Asus ARM routers. 380 series FW. Memory leak in kernel. Inspect by command line:

$ cat /proc/slabinfo | grep kmalloc-96

kmalloc-96 14311 15288 96 42 1 : tunables 0 0 0 : slabdata 364 364 0

Each slot in this pool is 96 bytes. The first number is the active number of slots. In the above example 14311 slots are active which translate to ~ 1341KB. On fresh reboot, ~ 900 active slots.

Didn't pay attention when I was on 378.55. So not sure if it's 380 series FW specific..

Anyone else see the leak on ARM routers with a 380 series FW?

Probably would help to be more specific about how you got here - e.g. features enabled and in use - is there a USB drive attached, is the WebGUI up and running in a browser, etc...

Just saying there's a leak - could be anything... any good bug report will include the steps to reproduce in a consistently repeatable function - esp. with mem leaks like what you're mentioning...

 

[kvic]

Probably would help to be more specific about how you got here - e.g. features enabled and in use - is there a USB drive attached, is the WebGUI up and running in a browser, etc...

One day I was bored like you and looked at /proc/slabinfo. Saw kmalloc-96 is 'extraordinarily' high. Wrote a script to capture its value at 5-min interval.

USB storage on ext4. BWDPI on (AiProtection, Adaptive Qos, Traffic Monitor with App Analysis, but no web history). CTF on. RT-AC56U.

 

[kvic]

uptime: 10 days 12 hours 14 minute(s) 20 seconds

kmalloc-96 627 756 96 42 1 : tunables 0 0 0 : slabdata 18 18 0

USB, Guest, TA, AiProtection, Adaptive QoS, all off, CTF on

AC68U/Merlin 380.59

Thanks. Both of you have very good usage of kmalloc-96. Interestingly both with BWDPI off..

 

[RMerlin]

I recently rebooted my router after 17 days of uptime. There was only 20 MB of usage difference after the reboot, and a good portion of it is probably from data accumulated in /tmp over these 17 days, such as logfiles or temporary ASP files I bind mount on top of the ROM for development purposes.

 

[sfx2000]

kvic is on a bug hunt - and it's a good effort...

 

[kvic]

I recently rebooted my router after 17 days of uptime. There was only 20 MB of usage difference after the reboot, and a good portion of it is probably from data accumulated in /tmp over these 17 days, such as logfiles or temporary ASP files I bind mount on top of the ROM for development purposes.

After moving to 380 series, my router never pass two weeks of continuous run..

I think this leak is proportional to browsing activities (anecdotal evidence from the plot), and might have to do with BWDPI.

 

[sfx2000]

might have to do with BWDPI

That's my thought as well..

The old kern bugs are still a concern - this is somewhat of a known issue if Broadcom/Asus didn't bring those patches in... stack smashing is always a nice lever to get deeper into a system.. e.g. exploit

 

[kvic]

That's my thought as well..

The old kern bugs are still a concern - this is somewhat of a known issue if Broadcom/Asus didn't bring those patches in... stack smashing is always a nice lever to get deeper into a system.. e.g. exploit

You could be right about the old kernel bugs.

I think I saw the ones you mentioned but I'm not exactly sure without going back and checking. There are a few major ones and performance enhancements. Most of them are corner cases though.

 

[RMerlin]

After moving to 380 series, my router never pass two weeks of continuous run..

I think this leak is proportional to browsing activities (anecdotal evidence from the plot), and might have to do with BWDPI.

I use Adaptive QoS, Apps Analysis, and Malicious site blocking. I got Web History and the other security features of BWDPI disabled.

 

[RMerlin]

I see the OpenVPN devs are still plugging holes:

https://github.com/OpenVPN/openvpn/commit/cd538f2c7a128395284d23f9c30741217075503f

 

[kvic]

I see the OpenVPN devs are still plugging holes:

https://github.com/OpenVPN/openvpn/commit/cd538f2c7a128395284d23f9c30741217075503f

This looks like glueing code for PolarSSL. It shall not impact asuswrt-merlin (?!)

It may be slowly killing my iPhone as their iOS app uses PolarSSL. I just upgraded to their new version (1.0.7 released 6 days ago...) this morning.

 

[RMerlin]

This looks like glueing code for PolarSSL. It shall not impact asuswrt-merlin (?!)

It may be slowly killing my iPhone as their iOS app uses PolarSSL. I just upgraded to their new version (1.0.7 released 6 days ago...) this morning.

I was also browsing through recent OpenSSL commits today (was a slow morning at work), I saw a bunch of memory holes plugged there recently as well (most of these looked really silly to me).

At least it's still better to have memory leaks than security holes...

 

[sfx2000]

At least it's still better to have memory leaks than security holes...

It's good that folks are going in and doing maint work - it's a grind, but I think as a whole, it's gotta be done.

Mem Leaks can be security holes...