1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Memory leak issue

Discussion in 'Asuswrt-Merlin' started by hyelton, Jul 22, 2019.

  1. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    I've had this issue with several different models over the years and it still occurs today all running merlin. It seems mostly related to the DPI inpsection/Statstics being enabled.

    After 15-25 days the ram will fill up to the point where nothing is stable and the GUI is no longer accessable.

    Currently happening on latest merlin and on my AC86U also occured last with my 66U-B1

    No QoS or anything, just most default, wifi disabled, statstics turned on. Its all I want, but its not stable.
     
    Grisu likes this.
  2. FTC

    FTC Senior Member

    Joined:
    Mar 2, 2016
    Messages:
    232
    Location:
    Barcelona
    Hi, do you have swapping enabled ? could be a bypass to at least prolong that period much longer. Other option would be to program a periodic router reboot every 2 weeks or so.. Note that merlin 384.12 fixed a leak for european models which could cause your sympthoms, but as you say you are already running the latest merlin this should not be your case..
     
  3. L&LD

    L&LD Part of the Furniture

    Joined:
    Dec 9, 2013
    Messages:
    9,624
    After flashing to the latest firmware, did you happen to do a full reset to factory defaults and then do a minimal and manual configuration to secure the router and connect to your ISP?

    Or, have you been doing dirty upgrades to each new firmware?

    I have customers with similar usage and 384.12_0 is running stable with no issues for them for almost 4 weeks now (26 days 16 hours).

    As FTC suggests, I suggest trying using the amtm Step-by-Step guide to setup a swap file for your router on a spare USB drive.

    Please see the link in my signature for both the M&M Config to fully reset your router and also the amtm guide too. :)
     
    dasus likes this.
  4. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    I have an OCD when it comes to the dirty flash/clean install so yeah 2 factory resets usually before setup.

    Yeah I've tried the swap file thing before and it still happens.
     
    L&LD likes this.
  5. FTC

    FTC Senior Member

    Joined:
    Mar 2, 2016
    Messages:
    232
    Location:
    Barcelona
    If the problem (unstability) happened with swapping enabled, then probably the leak is caused by something that causes unstability instead of unstability being the consequence of the leak.

    Several versions ago and in my previous router I had a similar case where the connections table was leaking.. the number of connections was increasing with time until a moment where it caused unstability besides using much memory.. are you looking at how many connections do you have opened? dose it increase ? If this is the case you can also bypass this by reserving more space for the connections table.. i.e :

    To set the connections table to 8192 :
    nvram get ct_hashsize < to see current value (default used to be just under 4000)
    nvram set ct_hashsize=8192
    nvram commit
    service restart_conntrack

    or low level (without persistance) :

    cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets < to see current value
    echo "8192" > /sys/module/nf_conntrack/parameters/hashsize

    .. but of course this is just a guess.. To see where the leak is you may have to compare the output of a 'top' command right after reboot with the same several days after.
     
  6. brummygit

    brummygit Regular Contributor

    Joined:
    Jan 19, 2013
    Messages:
    176
    Are you using any of the Tend Micro functionality. I have had the same for many releases and routinely reboot my router at least once per week to resolve.

    There was a thread recently that implicate the closed source Trend Micro components and I believe it's been reported but no fix yet. Merlin can't resolve as it's in the closed source code.

    [Edit] Found the thread https://www.snbforums.com/threads/s...ks-aiprotection-dcd-crashing-is-guilty.55416/
     
  7. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    How does one make the statistics use the swap file? As since doing the swap file, it never gets used at all
     
  8. Vexira

    Vexira Very Senior Member

    Joined:
    Jan 20, 2017
    Messages:
    1,768
    Location:
    Australia
    I just set the reboot schedule to reboot every day, you could set it to once a week.
     
  9. PeterV

    PeterV Regular Contributor

    Joined:
    Nov 5, 2013
    Messages:
    67
    Location:
    Ukraine, Kyiv
    Do you use a Diversion with Pixelserv-tls ?
    Currently it is an interoperability issue between Trendmicro engine(dcd) and Diversion/Pixelserv-tls. It's due to diversion creating a sub-interface for pixelserv-tls. And Trenmicro can't work correctly with this sub-interface, and crashed every 15-30 minutes, without memory releasing.
    This behavior currently identiied only on ac86u model ...
    So, you have three option:
    • disable trendmicro
    • not use pixelserv-tls
    • schedule a periodical reboot
     
    Grisu likes this.
  10. FTC

    FTC Senior Member

    Joined:
    Mar 2, 2016
    Messages:
    232
    Location:
    Barcelona
    Hi, how to check for this ? My router is an ax88 but I can see some cases of unexplainable memory consumption, and I am using Diversion with Pixelserv-tls.. Any specific periodic message to look for in the logs or.. processes in the 'TOP' command listing ? Thanks in advance...
     
  11. PeterV

    PeterV Regular Contributor

    Joined:
    Nov 5, 2013
    Messages:
    67
    Location:
    Ukraine, Kyiv
    Do you have in the syslog a lot of dcd process crashing messages?
    In parallel you can check
    Code:
    cat /proc/meminfo
    
    If Slab, and especially SUnreclaim is growing - there is a result of memory leaks
     
    FTC likes this.
  12. FTC

    FTC Senior Member

    Joined:
    Mar 2, 2016
    Messages:
    232
    Location:
    Barcelona
    Thanks for this.
    I don't see dcd process crashing messages in my syslog, although slab and sureclaim are quite big.. will keep an eye on those.
     
  13. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    Nope.

    And trendmicro is only enabled for statistics to work.




    Sent from my iPhone using Tapatalk
     
  14. Elmer

    Elmer Regular Contributor

    Joined:
    Jun 27, 2017
    Messages:
    81
    Recently upgraded from an AC88 to and AX88, and it works wonderfully - except - memory is gobbled up over the course of a day. It typically starts out at 600M and climbs slowly to max over a 30 hour cycle, I, too, run diversion and skynet. If this is a leak, it was certainly unsustainable on the AC88, I had constant crashes. With the AX88 a once a day reboot is handling things so far. I do have a huge number of "smart home" devices but it's not like they are bandwidth bandits. Eventually someone will figure this out. For now the AX88 and a daily reboot are working, and that was not the case with the AC88.
     
  15. Elmer

    Elmer Regular Contributor

    Joined:
    Jun 27, 2017
    Messages:
    81
    Would also be nice if someone would write a script to monitor memory and do a reboot at some percentage. Be a nice bandaid till things get sorted.
     
  16. RMerlin

    RMerlin Super Moderator

    Joined:
    Apr 14, 2012
    Messages:
    30,853
    Location:
    Canada
    No, because the way Linux manages memory, "free memory" is meaningless. Memory gets used by buffers until it is required by processes, at which point it will release those buffers.
     
    joegreat and Makaveli like this.
  17. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    Well it doesn’t lol. When it fills up the routers gui is no longer accessible and packets randomly start dropping


    Sent from my iPhone using Tapatalk
     
  18. RMerlin

    RMerlin Super Moderator

    Joined:
    Apr 14, 2012
    Messages:
    30,853
    Location:
    Canada
    There's a difference between memory used by processes, and memory used by buffers. What I'm telling you is you cannot just monitor the free memory amount and reboot because it gets low - just copying a large file from a shared USB disk would cause your router to reboot then, as that free memory would get close to zero due to buffering.
     
    Makaveli, L&LD and hyelton like this.
  19. Elmer

    Elmer Regular Contributor

    Joined:
    Jun 27, 2017
    Messages:
    81
    Thanks for the feedback.

    My AX88 runs throughout the day at about 970 MB, that is, 30 MB or less free. It's working (my AC88 was constantly crashing). It's odd. I reboot once a day.

    There is one thing I'm doing that might be out of the ordinary, and that's link aggregation. I suppose for the sake of science I should run without and see if there's a difference. Haven't seen any posts about link aggregation problems, but maybe that's because hardly anyone is using it.
     
  20. hyelton

    hyelton Regular Contributor

    Joined:
    Jan 2, 2013
    Messages:
    175
    It’s the traffic statistics specifically that’s causing my ram usage.


    Sent from my iPhone using Tapatalk