AdGuardHome AdGuardHome Memory Usage Analysis

ERamseth2

Occasional Visitor
Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:
  • using oisd dbl blocklist
  • mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
  • a few extra filter entries to allow a few apps to work
  • Upstream DoT and DoQ servers
  • I think everything else is basically stock
  • Service restarted 7 minutes before first data point in this set
System:
  • Asus RT-AX86U
  • Running Asuswrt-merlin 386.5_2
  • Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
  • 32GB USB storage
  • 2GB swap
  • swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
  • UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

AGH_Mem-2022-04-08..PNG


Raw data is attached in a text file (comma separated)

Thoughts/notes:
  • Going to use this as the baseline to compare changes against
  • Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
  • Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
  • I guess I'll need to open the web interface during each 24-hour run and document when that happened.
  • "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later
 

Attachments

  • AGH_Mem-2022-04-08.txt
    14.8 KB · Views: 58
Last edited:

SomeWhereOverTheRainBow

Part of the Furniture
Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:
  • using oisd dbl blocklist
  • mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
  • a few extra filter entries to allow a few apps to work
  • Upstream DoT and DoQ servers
  • I think everything else is basically stock
  • Service restarted 7 minutes before first data point in this set
System:
  • Asus RT-AX86U
  • Running Asuswrt-merlin 386.5_2
  • Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
  • 32GB USB storage
  • 2GB swap
  • swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
  • UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

View attachment 40684

Raw data is attached in a text file (comma separated)

Thoughts/notes:
  • Going to use this as the baseline to compare changes against
  • Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
  • Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
  • I guess I'll need to open the web interface during each 24-hour run and document when that happened.

The big jump might be when the filter list update happens. I notice a huge increase in memory when ever one loads a new list which leads me to think it is whenever the 24 hour list update happens. To test this theory, maybe change the list update interval to 3 days. See if after 3 days, the memory usage goes up, or if it still happens after 24 hours. If this is the list update, then it sounds like they have a bad memory release garbage cleanup policy upon list updates and loading.

My theory is that whenever it updates the list, that it makes a huge jump in memory usage upon download, and another jump once it loads the new entries. From there, the memory usage doesn't get freed. Keep in mind this is all conjecture, but the only thing that actually happens 24 hours after starting adguardhome is the list update process.
 
Last edited:

ERamseth2

Occasional Visitor
The big jump might be when the filter list update happens. I notice a huge increase in memory when ever one loads a new list which leads me to think it is whenever the 24 hour list update happens. To test this theory, maybe change the list update interval to 3 days. See if after 3 days, the memory usage goes up, or if it still happens after 24 hours. If this is the list update, then it sounds like they have a bad memory release garbage cleanup policy upon list updates and loading.

My theory is that whenever it updates the list, that it makes a huge jump in memory usage upon download, and another jump once it loads the new entries. From there, the memory usage doesn't get freed. Keep in mind this is all conjecture, but the only thing that actually happens 24 hours after starting adguardhome is the list update process.
I had a similar thought. After getting data for --no-mem-optimization I will start trying to get data for filter list reloads by tweaking the reload interval.
 

ERamseth2

Occasional Visitor
Here is a similar graph using --no-mem-optimization

AGH_Mem-2022-04-11..PNG


Some notes:
  • "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later
  • "checkpoints" where the mem usage plateaus are about the same, but come at different times
 

Attachments

  • AGH_Mem-2022-04-11.txt
    14.6 KB · Views: 26

SomeWhereOverTheRainBow

Part of the Furniture
@SomeWhereOverTheRainBow See above. Not sure --no-mem-optimization is helping (at least on a 24hr view)
I think the only thing it shows is that at some point there is a consistency again in the amount of memory usage for a period of time before the new entries get loaded, the memory optimization built in seemed to hinder us seeing that process alittle. By using the --no-mem-optimization flag, you get a more accurate understanding of what is going on with the memory.

See here W/o --no-mem-optimization

1649705936320.png


see here with --no-mem-optimization

1649706175629.png


At least now we get a somewhat better understanding of what we are seeing.
 

SomeWhereOverTheRainBow

Part of the Furniture
Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:
  • using oisd dbl blocklist
  • mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
  • a few extra filter entries to allow a few apps to work
  • Upstream DoT and DoQ servers
  • I think everything else is basically stock
  • Service restarted 7 minutes before first data point in this set
System:
  • Asus RT-AX86U
  • Running Asuswrt-merlin 386.5_2
  • Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
  • 32GB USB storage
  • 2GB swap
  • swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
  • UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

View attachment 40684

Raw data is attached in a text file (comma separated)

Thoughts/notes:
  • Going to use this as the baseline to compare changes against
  • Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
  • Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
  • I guess I'll need to open the web interface during each 24-hour run and document when that happened.
  • "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later
@ERamseth2 wouldn't using

Code:
swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)

tell your router to use your swap less than the router default of 60? Which means your router would move less of the stale process's that it deemed swappable to swap in preference of leaving it stored in the ram. (a.k.a. your router would use more ram than intended under normal use than if you were to leave it at the default 60.)

1649728015855.png


So would setting your swappiness to 45 assure that enough stale process's have cleared out of regular ram so there is enough ram necessary to accommodate the regularly active use of AdGuardHome?
 
Last edited:

ERamseth2

Occasional Visitor
@ERamseth2 wouldn't using

Code:
swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)

tell your router to use your swap less than the router default of 60? Which means your router would move less of the stale process's that it deemed swappable to swap in preference of leaving it stored in the ram. (a.k.a. your router would use more ram than intended under normal use than if you were to leave it at the default 60.)

View attachment 40773

So would setting your swappiness to 45 assure that enough stale process's have cleared out of regular ram so there is enough ram necessary to accommodate the regularly active use of AdGuardHome?
When I checked the swappiness it was defaulted to zero which I believe means use swap as an absolute last resort when we're about to completely run out of memory.

Maybe this makes sense with swap on something like a usb stick or flash memory where in theory there are limited writes. for what its worth I had a full default ubuntu desktop (installed, not live) running on the same brand of usb flash memory I am using in my router now (Samsung FIT USB or something... the little tiny samsung one) for 2 years, swap and default swap settings included, and had no problems whatsoever.

If you have a swap set up through amtm maybe you can check it too.
 

SomeWhereOverTheRainBow

Part of the Furniture
When I checked the swappiness it was defaulted to zero which I believe means use swap as an absolute last resort when we're about to completely run out of memory.

Maybe this makes sense with swap on something like a usb stick or flash memory where in theory there are limited writes. for what its worth I had a full default ubuntu desktop (installed, not live) running on the same brand of usb flash memory I am using in my router now (Samsung FIT USB or something... the little tiny samsung one) for 2 years, swap and default swap settings included, and had no problems whatsoever.

If you have a swap set up through amtm maybe you can check it too.
There may be an issue with your kernel settings ( i.e. how they are setup at build time, obviously not your fault though), but my router (the RT-AX88U) appears to default to the standard linux default for swappiness, which is 60. I have no clue why yours is zero (without your intervention), unless that: it is the new asuswrt standard, it is not correctly set for the firmware build time for your model router maybe (@RMerlin can check), or somehow you have a script running that defaults it to zero. But setting it to zero would mean it is pointless to even have a swap.

this is without my own intervention

Code:
RT-AX88U-C7C0:/tmp/home/root# cat /proc/sys/vm/swappiness
60

At either rate, having this set to a really low number is a rather bad idea if you are expecting your swap to at somepoint alleviate you ram resources of stale pages which would hopefully free up extra ram resources for your thriving AdGuardHome and other active resources.
 
Last edited:

ERamseth2

Occasional Visitor
Yeah I mean I didnt think 45 vs. 60 was terribly low. I guess I can change back to 60. Probably won't hurt anything.
 

SomeWhereOverTheRainBow

Part of the Furniture
Yeah I mean I didnt think 45 vs. 60 was terribly low. I guess I can change back to 60. Probably won't hurt anything.
I am curious to see what your ram usage looks like with adguardhome after two to three days of running with not restart, obviously if you have to, you can pop in and check what the logging looks like between each day. The reason why I bring this up is because some users had reported adguardhome needed to be restarted after a couple of days of straight running. I am wondering if it is linked to this memory chokehold. I have yet to experience this on my RTAX88U, but it could be only happening on certain models.
 

ERamseth2

Occasional Visitor
I have some logs to analyze for the 1hr filter refresh with non mem optimization. Just havn't gotten to it yet. After that I will extend my logging and restart period to 2 days and see what happens.
 

ERamseth2

Occasional Visitor
This is with --no-mem-optimization and hourly filter updates

AGH_Mem-2022-04-13.PNG


The big jumps in memory usage match up with the timing of SOME filter updates in the log, but not all filter updates logged have a jump associated with it.

Oddly enough, the log (talking about messages.log here) doesn't show hourly updates. Maybe it only logs when something changes and doesn't log when the pulled file is the same?
 

Attachments

  • AGH_Mem-2022-04-13.txt
    14.8 KB · Views: 20
Last edited:

null_

New Around Here
Yep, I agree. Setting that up now. Might have some unhappy folks around if the internet goes out.
Did you arrive at any conclusions with this? Reason I ask is that I'm seeing similar behaviour on a GT-AX6000. I'm not sure what my "swappiness" is set to (as the file shows blank in WinSCP and I have zero confidence in my CLI skills to access that info in another way) but if there's any insight into how to avoid AGH from consuming all of the available system memory I'd be glad to hear it. Thanks!
 

SomeWhereOverTheRainBow

Part of the Furniture
Did you arrive at any conclusions with this? Reason I ask is that I'm seeing similar behaviour on a GT-AX6000. I'm not sure what my "swappiness" is set to (as the file shows blank in WinSCP and I have zero confidence in my CLI skills to access that info in another way) but if there's any insight into how to avoid AGH from consuming all of the available system memory I'd be glad to hear it. Thanks!
Have a good swap. that should be it. adguardhome should relinquish memory and use it as it needs it, but on the routers it operates in a tight amount of space. Having a good swap will atleast cushin that space.
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top