Need help tracking down recurring AC68U space issue

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

fr33z0n3r

Regular Contributor
I am getting an ongoing issue with space on my router. I'd like to know how I can track down the cause of this. I'm not using this for anything but standard router/wifi. I've already reset my config, to see if that helps.

If anyone has ideas on commands to run regularly, as well as anything I could do with alerting to be able to get in before space is too low to access it. Or logging to capture the proof of it would be appreciated.

How I see the problem appear in syslog (remote):

1. It seems like there are always topology change events occurring, then a few days before the device stop working well they stop coming in.

2. A few days after that this event appears:

Code:
dnsmasq-dhcp 250 - - dnsmasq-dhcp[250]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
 

ColinTaylor

Part of the Furniture
You appear to have run out of memory.

Are you running any third-party scripts, e.g. ad-blocking? What firmware version are you using?

When the problem occurs SSH into the router and run ps w and free to try and see what is consuming the memory.
 

fr33z0n3r

Regular Contributor
the issue is its unreachable when I find this. I am running the latest firmware. Again, not doing anything except wifi and firewall.
 

fr33z0n3r

Regular Contributor
I could open an ssh session and run top perpetually. But there doesn't seem to be any memory usage stats in there. Is there any way to tie memory usage to a process o nthis?
 

bbunge

Very Senior Member
the issue is its unreachable when I find this. I am running the latest firmware. Again, not doing anything except wifi and firewall.
Latest Firmware tells us nothing. There are several betas that folks consider latest. Merlin 384.19 is also latest. Just which firmware do you have and what add-ons?
 

john9527

Part of the Furniture
It's not really process memory usage. You are running out of space on the /tmp partition in memory.
First thing is to try and find what file is using up the space. Running the following command will show the top 10 largest files in that partition.
Code:
du -ahx /tmp | sort -n -r | head -n 10
 

fr33z0n3r

Regular Contributor
Latest always means non beta latest - 384.19.

I will try to run that command. How can I run that command set with 'watch'?
 

fr33z0n3r

Regular Contributor
Thanks John! Not sure if the SSH session will die, but I'm going to use its session log to track the data.
 

john9527

Part of the Furniture
I suspect you'll be able to spot a file that is growing before it actually gets to the point of causing a problem.
 

fr33z0n3r

Regular Contributor
So, I this watch command running for days. The top entry was always /tmp, and it got up to 1004K. I stopped the command to check some other stuff, but when I reran the same watch command the /tmp entry is not visible on the list. It should be the top entry still.

Any ideas why?

Still no issues with the router.
 

ColinTaylor

Part of the Furniture
I think the command you were using was wrong, or perhaps your version of du formats its output differently than mine.

Try this instead (shows sizes in kB):
Code:
du -akx /tmp | sort -n -r | head -n 10

It's probably worth including the top command as well if your router supports the necessary options:
Code:
watch -n 30 "du -akx /tmp | sort -n -r | head -n 10; echo; top -b -m -n 1 | head -n 15"
 
Last edited:

fr33z0n3r

Regular Contributor
Both show /tmp for me, thanks. for the longer command, there is an error saying "top: no process info in proc"
 

ColinTaylor

Part of the Furniture
there is an error saying "top: no process info in proc"
OK, it looks like your version of top doesn't support the memory option. John's firmware had the same problem until he patched it last year.

Oh well, you could use this:
Code:
watch -n 30 "du -akx /tmp | sort -n -r | head -n 10; echo; /bin/ps w | sort -n -r -k 3 | head -n 15"
 

KsWoodsMan

Regular Contributor
So far only the syslog.log file is growing.
What about the directories under /tmp ?
Are any of the directory sizes also growing ?
My (/tmp)/var/log direcory would grow to over 18MB. It did, untill I moved /var/log/dnsmasq.log to mounted storage at /opt/var/log/dnsmasq.log.

/tmp/syslog.log should be clamped in size to keep it and the backup from getting to big.
On mine the size is clamped to 256KB. Once it exceeds this size it gets rotated.
Unless it changed, you can check it with nvram get log_size .

Relocating /var/log/dnsmasq.log to /opt solved my problem like you are having.
 
Last edited:

fr33z0n3r

Regular Contributor
Yeah, nothing is growing very large. I'm just gonna keep this running for weeks and see if the issue reoccurs. Thanks for all the input everyone!
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top