RT-AC-86U - Help with No Free Space Left for GC (Solved for now)

[Chipster]

Hello SNB land. Long a Merling fan - but I have this 86U that's still on Asus code - version 384.32799 - and then to 384.45149.

Last few days I'm getting this in the logs - and Google hasn't helped much. Via telnet - nothing unusual about JFFS - everything seems OK. But I'm not a good Linux nerd.

Can anyone assist with these errors? They repeat forever..

Apr 19 09:16:10 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 09:16:10 kernel: jffs2: Error garbage collecting node at 01bd5ca0!
Apr 19 09:16:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
​

Thank you for any help!
Chipster

 

[OzarkEdge]

Hello SNB land. Long a Merling fan - but I have this 86U that's still on Asus code - version 384.32799 - and then to 384.45149.

Last few days I'm getting this in the logs - and Google hasn't helped much. Via telnet - nothing unusual about JFFS - everything seems OK. But I'm not a good Linux nerd.

Can anyone assist with these errors? They repeat forever..

Apr 19 09:16:10 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 09:16:10 kernel: jffs2: Error garbage collecting node at 01bd5ca0!
Apr 19 09:16:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
​

Thank you for any help!
Chipster

Sorry, I can't help with the log entries. I would upgrade to the new release 3.0.0.4.384.45713, reset to defaults, re-configure from scratch (no restore)... and hopefully leave the issue behind.

Maybe you are using Trend Micro features in 45149 and its crashing is messing with the logging storage.

OE

 

[RMerlin]

What is the size of your Traffic Analyzer database?

ls -l /jffs/.sys/TrafficAnalyzer/ -ah

If it's larger than 30 MB, then for some reason maintenance isn't being done on it. Try running it manually:

TrafficAnalyzer -d 30720
TrafficAnalyzer -e

Wait 5-10 minutes, then check again the size.

 

[Chipster]

What is the size of your Traffic Analyzer database?

ls -l /jffs/.sys/TrafficAnalyzer/ -ah

If it's larger than 30 MB, then for some reason maintenance isn't being done on it. Try running it manually:

TrafficAnalyzer -d 30720
TrafficAnalyzer -e

Wait 5-10 minutes, then check again the size.

Thanks for the tricks Mr. Merlin!

But it appears to be 30M based on that command:

admin@RT-AC86U:/tmp/home/root# ls -l /jffs/.sys/TrafficAnalyzer/ -ah
drw-rw-rw-    2 admin    root           0 Apr 18 03:30 .
drwxrwxrwx    4 admin    root           0 Nov 28 21:55 ..
-rw-rw-rw-    1 admin    root       30.0M Apr 19 20:00 TrafficAnalyzer.db
-rw-rw-rw-    1 admin    root       15.0M Apr 18 03:30 TrafficAnalyzer.db-journal

And the other command:

admin@RT-AC86U:/tmp/home/root# TrafficAnalyzer -d 30720
table_main-1: over size 30720, timestamp=1554098400
start to delete some rules from /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db because of over size
SQL error: disk I/O error

While the last command gave no feedback at the command line..

admin@RT-AC86U:/tmp/home/root# TrafficAnalyzer -e
admin@RT-AC86U:/tmp/home/root#

And the system log continues to pump out the OP errors by the 100's..

 

[ColinTaylor]

What do you get from this command?

df -h

 

[Chipster]

Sorry, I can't help with the log entries. I would upgrade to the new release 3.0.0.4.384.45713, reset to defaults, re-configure from scratch (no restore)... and hopefully leave the issue behind.

Maybe you are using Trend Micro features in 45149 and its crashing is messing with the logging storage.

OE

Hi OE.. Just downloaded v45713. Reconfigure from scratch sounds like a good Saturday pastime - good thing I keep scrupulous notes.

I don't have any Trend Micro stuff switched on..

But also hoping to learn some things as I muck about with Telnet... and some of RM suggestions.

The issue isn't completely killing the router... at least not yet.

Thanks!

 

[Chipster]

What do you get from this command?

df -h

Oh yeah.. found that before asking here - nothing weird (just a minute ago). Could I have dead flash?:

admin@RT-AC86U:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.3M     55.0M     22.3M  71% /
devtmpfs                215.0M         0    215.0M   0% /dev
tmpfs                   215.1M    232.0K    214.9M   0% /var
mtd:bootfs                4.3M      3.2M      1.0M  76% /bootfs
mtd:data                  8.0M    580.0K      7.4M   7% /data
tmpfs                   215.1M      2.5M    212.5M   1% /tmp
/dev/mtdblock8           48.0M     36.5M     11.5M  76% /jffs
admin@RT-AC86U:/tmp/home/root#

Edit:

And the error logs continue:

Apr 19 21:21:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Apr 19 21:21:32 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 21:21:32 kernel: jffs2: Error garbage collecting node at 01bd5ca0!
Apr 19 21:21:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Apr 19 21:21:32 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 21:21:32 kernel: jffs2: Error garbage collecting node at 01bd5ca0!
Apr 19 21:22:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Apr 19 21:22:32 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 21:22:32 kernel: jffs2: Error garbage collecting node at 01bd5ca0!
Apr 19 21:22:32 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Apr 19 21:22:32 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Apr 19 21:22:32 kernel: jffs2: Error garbage collecting node at 01bd5ca0!

Chipster

 

[RMerlin]

Delete that 15 MB journal file, that shouldn't be there (and it's from yesterday anyway).

 

[Chipster]

Delete that 15 MB journal file, that shouldn't be there (and it's from yesterday anyway).

Well... I tried to... Wrong command? ¯\_(ツ)_/¯

admin@RT-AC86U:/jffs/.sys/TrafficAnalyzer# rm TrafficAnalyzer.db-journal
rm: can't remove 'TrafficAnalyzer.db-journal': No space left on device
admin@RT-AC86U:/jffs/.sys/TrafficAnalyzer#

Edit: After a reboot - the file dates went back to yesterday:

admin@RT-AC86U:/jffs/.sys/TrafficAnalyzer# ls -l
-rw-rw-rw-    1 admin    root      31466496 Apr 18 03:30 TrafficAnalyzer.db
-rw-rw-rw-    1 admin    root      15752624 Apr 18 03:30 TrafficAnalyzer.db-journal

 

[ColinTaylor]

Your jffs partition is only 48MB in size. Those two files alone take up 45MB, then there's the other files that are stored there. So you'd only need a couple of uncommitted write operations to use up all the free space.

If as Merlin suggested the TrafficAnalyzer.db-journal is not needed I'd suggest you try and remove it as soon as possible. You might need to disable the Traffic Analyzer before it will allow you to do that. Hopefully it won't reappear when you re-enable the Traffic Analyzer.

EDIT: The output of your df command only showed 36.5MB being used, not >45MB. So that suggests that it was trying to write that journal file but failing due to lack of free space.

EDIT 2: It beggars belief that a modern router like the RT-AC86U with 256MB of flash memory only has 48MB available for jffs. When an old router like the RT-AC68U with 128MB of flash has 63MB! /RANT OFF

 

[Chipster]

Tried deleting the -journal - but got an error.

In the meantime - I loaded Merlin 384.10.2 - set it to reformat JFFS - and now all is working. (I didn't even reset it - 384.10 picked everything up).

The errors no longer show in the system log.

I may flash back to Asus code out of curiosity - now that the JFFS is cleared. Seems there should be a way to format JFFS from a terminal without flashing Eric's code and using his option to format on boot.

Thanks for everyone's help..

 

[Ronald Schwerer]

Another way to free the space (without actually deleting the file) would be:
cat /dev/null > /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db

 

[Dave Parker]

Tried deleting the -journal - but got an error.

In the meantime - I loaded Merlin 384.10.2 - set it to reformat JFFS - and now all is working. (I didn't even reset it - 384.10 picked everything up).

The errors no longer show in the system log.

I may flash back to Asus code out of curiosity - now that the JFFS is cleared. Seems there should be a way to format JFFS from a terminal without flashing Eric's code and using his option to format on boot.

Thanks for everyone's help..

It's my understanding that when flashing from stock Asus firmware to Merlins firmware a factory default reset is not optional but necessary. Someone correct me if I'm wrong.

 

[ColinTaylor]

It's my understanding that when flashing from stock Asus firmware to Merlins firmware a factory default reset is not optional but necessary. Someone correct me if I'm wrong.

No that's not always the case.

While it is generally not necessary to restore to factory defaults, it's not a bad idea, especially if there is a big jump in version number (e.g. from 112 to 178).

But on the other hand:

If something looks weird, don't waste too much time: save your settings, reset to factory default, reconfigure the basics, and see if the issue is resolved. If not, you can always restore your saved settings and do some more advanced troubleshooting.

In the case of the OP the firmware versions of stock and Merlin's were probably very close.

 

[L&LD]

Tried deleting the -journal - but got an error.

In the meantime - I loaded Merlin 384.10.2 - set it to reformat JFFS - and now all is working. (I didn't even reset it - 384.10 picked everything up).

The errors no longer show in the system log.

I may flash back to Asus code out of curiosity - now that the JFFS is cleared. Seems there should be a way to format JFFS from a terminal without flashing Eric's code and using his option to format on boot.

Thanks for everyone's help..

Be very careful about formatting anything within the router via cli commands. A mistake may prove very costly.

Unless you need AiMesh support and also can't wait until RMerlin has access to the updated GPL that fixes the network devices pages, there is very little reason to stay with stock Asus firmware. In the long run, much happier user experience with fewer bugs popping up, more features (as you've seen already) and more performance too.

All that is true, of course, if you do a full reset to factory defaults first.

Like the readme states in the RMerlin files, you downloaded.

https://www.snbforums.com/threads/faq-nvram-and-factory-default-reset.22822/

The above link is a great read on why you should be doing one. If you want the experience to be as positive as possible.

Have a look below at my signature for further details on how to get your router fully reset using the M&M Config guide.

 

[RMerlin]

Going from Asus to Merlin is fine. Going from Merlin to Asus will leave you with potentially corrupted DHCP reservations (I added a third field for the hostname), and VPN settings (different OpenVPN versions and settings supported). You have to either be ready to handle these issues, or else do a factory default reset.

 

[Chipster]

Going from Asus to Merlin is fine. Going from Merlin to Asus will leave you with potentially corrupted DHCP reservations (I added a third field for the hostname), and VPN settings (different OpenVPN versions and settings supported). You have to either be ready to handle these issues, or else do a factory default reset.

I'll be doing a factory reset soon - and type it all back in. No need for AIMesh.. so I'll keep with Merlin.
Sooo many DHCP reservations, WOL, port forwards.. ..

 

[L&LD]

I'll be doing a factory reset soon - and type it all back in. No need for AIMesh.. so I'll keep with Merlin.
Sooo many DHCP reservations, WOL, port forwards.. ..

Screenshots, text files, spreadsheets!

 

[RMerlin]

nvram get dhcp_staticlist

nvram set dhcp_staticlist="blah"
nvram commit

Edit afterward through the webui to add hostnames if you want to.

 

[Chipster]

Well... the router was locked up this morning. I have it to reboot at 330AM every day - but it never completed.

So.. it did the factory reset, keyed in most things - and did a reboot (from the interface). Booted good. Then I finished some more settings - and reboot again.. and it hung on "Completed"... had to power cycle.

So now I'm confused why it won't reliably boot from a "soft" reboot..