What's using cache?

[Ripshod]

Recently something seems to be building a lot of cache - so much that this morning I only had 96MBram left. After a reboot everything settled down with a minimal cache. Now, less than 10hrs later cache has grown to 555MB, and still growing. I don't transfer any files to/from the ssd, it's only for swap and entware. Before you say it I know cache is released if something needs more RAM, but to me this is a new behaviour that I want to understand.
I just want to know if there's an easy way, over ssh, to find out what is using cache and how much.

 

[ExtremeFiretop]

Hi @Ripshod

You could try something like this:

cat /proc/slabinfo | \
  awk 'NR>2 {mem=$2*$4/1024; printf "%10.1f KiB %s\n", mem, $1}' | \
  sort -nr | head

If you see something huge like nf_conntrack or dentry, that tells you the kernel is responsible. and not user processes.

You can see user processes grow with top using the this below:

while true; do
  (
    echo "  RSS-KiB  PID  Command"
    for pid in /proc/[0-9]*; do
      [ -r "$pid/status" ] || continue
      rss=$(grep -m1 '^VmRSS:' "$pid/status" | awk '{print $2}')   # KiB
      cmd=$(tr -d '\0' < "$pid/comm")
      printf "%9s  %5s  %s\n" "${rss:-0}" "${pid##*/}" "$cmd"
    done | sort -nr | head
  )
  sleep 2
  echo
done

I know you can clear some of the cache by doing:

sync; echo 1 > /proc/sys/vm/drop_caches

which is something MerlinAU does before flashing firmware. That might help without rebooting.

You can check the general cache usage over SSH anytime using:

free -h

Something like this might be more detailed:

cat /proc/meminfo | grep -E '^(MemTotal|MemFree|Buffers|Cached|SReclaimable|Shmem)'

 

[JGrana]

Do you have samba enabled and possibly doing lot of io?

 

[Ripshod]

I don't use samba on the router at all @JGrana. Everything AI/Asus/USB is disabled.
@ExtremeFiretop that last command shows what I'm looking at;

/tmp/home/root# cat /proc/meminfo | grep -E '^(MemTotal|MemFree|Bu
ffers|Cached|SReclaimable|Shmem)'
MemTotal:        2047964 kB
MemFree:          405164 kB
Buffers:          397940 kB
Cached:           580032 kB
Shmem:              4328 kB
SReclaimable:      25552 kB

500MB in less than 14 hrs after a reboot, and still growing.
Yet this indicates to me a possible problem

/tmp/home/root# cat /proc/slabinfo | \
>   awk 'NR>2 {mem=$2*$4/1024; printf "%10.1f KiB %s\n", mem, $1}' | \
>   sort -nr | head
   62911.9 KiB bpmbuf
   13273.6 KiB kmalloc-128
   12286.8 KiB buffer_head
    7636.0 KiB kmalloc-4096
    4551.1 KiB radix_tree_node
    3816.0 KiB skbuff_head_cache
    3028.1 KiB kernfs_node_cache
    2720.0 KiB kmalloc-8192
    1857.4 KiB ext4_inode_cache
    1774.7 KiB dentry

The top user is nowhere near where I'd expect it to be, considering what the UI is telling me:

Not much to see here, but I know that by the morning if I rerun this code then maybe I'll catch the little RAM hog.

 

[ColinTaylor]

Everything AI/Asus/USB is disabled.

So are you saying you don't have a USB drive plugged in? What about all those addon scripts in your signature? Some of those would be constantly writing to a USB drive.

 

[Ripshod]

Obviously I have a usb ssd attached as it's stated in my signature. But I don't use that drive for file or media storage. All usb apps are disabled, as they have been since many years ago. Three scripts running right now - BACKUPMON, MerlinAU and ntpMerlin. Though they will all run fine together I don't. From the driver's led I see 3 really short bursts every minute. USB backups have only grown around 1MB in two weeks.

*edit* Correcting myself - scribe and scMerlin are also running.

 

[RMerlin]

Any I/O will use cache. That includes reading the scripts from the disk, reading the Entware files, reading files from the flash, etc...

Nothing to see there, this is perfectly normal. Cache gets flushed if other processess need that memory for allocations.

 

[Ripshod]

Any I/O will use cache. That includes reading the scripts from the disk, reading the Entware files, reading files from the flash, etc...

Nothing to see there, this is perfectly normal. Cache gets flushed if other processess need that memory for allocations.

The point is nothing has changed. Cache has never grown so fast before. I never really saw it go over 65MB before.
If I have to reset and rebuild so be it. I just wanted to understand the what and why. I am not saying anything is broken here.
There is less than 4GB total usage on the ssd including the swap.

 

[bennor]

The point is nothing has changed.

Have you updated any of the addon scripts recently?

 

[Ripshod]

Have you updated any of the addon scripts recently?

iirc the lastst change I made was the move from the develop branch of YaxDHCP to the release of 3.2.1 about a week ago. But that's not running right now.

 

[bennor]

iirc the lastst change I made was the move from the develop branch of YaxDHCP to the release of 3.2.1 about a week ago. But that's not running right now.

Your sig line indicates spdMerlin 4.4.10, which means you likely updated that recently as well since that was released on June 8th, a couple of days ago.

 

[Ripshod]

You are correct @bennor. But that's disabled and has been since the update. There's been a lot of activity lately, particularly around the OSR and age is also starting to show on my memory.
I may just put this thread to bed and reset tomorrow. Obviously no one else is seeing this on their setups.