RT-AX86U - VPN Director Rules are not saving. Logs flooded with kernel: jffs2: Argh. No free space left for GC

[thunder2k]

Hi,
I'm on RT-AX86U with latest Merling firmware 3004.388.9_2 and I'm having an issue.

As per the title, any new VPN Director Rules don't save. I have around 20 active ones that are working fine. If I try to add a new one, it's added to the table after I create it, but then when I click apply, it disappears.

My logs are flooded with the following, but no mention of the new rule creation failure:

Jun  6 19:00:01 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Jun  6 19:00:01 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Jun  6 19:00:01 kernel: jffs2: Error garbage collecting node at 001fc298!

I've done some research, but my jfss partition seems to be healthy:

XXXX@RT-AX86U-9E48:/tmp/home/root# ls -l /jffs/.sys/TrafficAnalyzer/ -ah
drw-rw-rw-    2 XXXX root           0 Apr  7 01:00 .
drw-rw-rw-    8 XXXX root           0 Mar  1  2023 ..
-rw-rw-rw-    1 XXXX root        4.4M Apr  7 01:00 TrafficAnalyzer.db
XXXX@RT-AX86U-9E48:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.7M     71.5M      6.2M  92% /
devtmpfs                455.9M         0    455.9M   0% /dev
tmpfs                   456.0M    488.0K    455.6M   0% /var
tmpfs                   456.0M      4.6M    451.5M   1% /tmp/mnt
ubi1:data                 4.5M     80.0K      4.1M   2% /data
tmpfs                   456.0M      4.6M    451.5M   1% /tmp/mnt
tmpfs                   456.0M      4.6M    451.5M   1% /tmp
/dev/mtdblock9           47.0M     34.4M     12.6M  73% /jffs
/dev/sda1               931.5G    831.4G    100.2G  89% /tmp/mnt/Seagate_Expansion_Drive
/dev/sdb2                 5.5T      4.8T    647.8G  88% /tmp/mnt/Seagate_Backup_Plus_Drive
XXXX@RT-AX86U-9E48:/tmp/home/root# ls -l /jffs/.sys/nc/ -ah
drwxrwxrwx    2 XXXX root           0 Apr  6 14:22 .
drw-rw-rw-    8 XXXX root           0 Mar  1  2023 ..
-rw-r--r--    1 XXXX root       36.0K Apr  6 14:22 nt_db.d

Any help is appreciated

Thanks!

 

[JGrana]

The kernel is reporting out of any erase blocks. This often occurs when there are many bad blocks on the mtd/nvram device. If that’s the case - unfortunately the nvram has reached it’s end of life.

You can try this application and see how many bad blocks you have. If it’s a large number - Im afraid it’s approaching door stop state…

mtd_check

 

[thunder2k]

Hi,
Thanks for your reply.

Unfortunately, installation didn't work:

XXXX@RT-AX86U-9E48:/tmp/home/root# uname -m
aarch64
XXXX@RT-AX86U-9E48:/tmp/home/root# /usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/JGrana01/mtd_check/m
aster/mtd_check_install" -o "/jffs/scripts/mtd_check_install" && chmod 0755 /jffs/scripts/mtd_check_install && /jffs/scr
ipts/mtd_check_install
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file /jffs/scripts/mtd_check_install: No space
Warning: left on device
100  1342  100  1342    0     0   3962      0 --:--:-- --:--:-- --:--:--  4246
curl: (23) Failure writing output to destination

Anything I can do about it? Can I install the script somewhere else?

 

[JGrana]

Unfortunately no, it needs access through the AX86U Linux kernel. From your earlier df command, the jffs partition should have plenty of space.

Maybe try a reboot. It might not reboot though - I believe the kernel does some nvram garbage collection during the boot up phase.

 

[octopus]

Hi,
Thanks for your reply.

Unfortunately, installation didn't work:

XXXX@RT-AX86U-9E48:/tmp/home/root# uname -m
aarch64
XXXX@RT-AX86U-9E48:/tmp/home/root# /usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/JGrana01/mtd_check/m
aster/mtd_check_install" -o "/jffs/scripts/mtd_check_install" && chmod 0755 /jffs/scripts/mtd_check_install && /jffs/scr
ipts/mtd_check_install
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file /jffs/scripts/mtd_check_install: No space
Warning: left on device
100  1342  100  1342    0     0   3962      0 --:--:-- --:--:-- --:--:--  4246
curl: (23) Failure writing output to destination

Anything I can do about it? Can I install the script somewhere else?

Have you turned ON?

 

[thunder2k]

Unfortunately no, it needs access through the AX86U Linux kernel. From your earlier df command, the jffs partition should have plenty of space.

Maybe try a reboot. It might not reboot though - I believe the kernel does some nvram garbage collection during the boot up phase.

I tried rebooting, and it rebooted smoothly, but still no luck with script installation.

If I do a backup and then a factory reset, can I restore the backup, or will it "restore" the issue?

Have you turned ON?

View attachment 66144

Yes, that's enabled

 

[JGrana]

If it’s some kind of nvram hardware issue it will be there no matter what ;-)

A backup - factory reset - restore is worth a shot!

 

[thunder2k]

If it’s some kind of nvram hardware issue it will be there no matter what ;-)

A backup - factory reset - restore is worth a shot!

I'm just scared of doing that, since now everything that's already been configured is working fine, I can't change configurations as if it went into a "read-only" state. So I'm scared that it won't allow me to restore the settings, and I'll just need to bin it. Any other less invasive tests you can think of?

 

[Jeffrey Young]

@JGrana, on the assumption that the nvram is toast on @thunder2k router, can your utility be installed and run from the USB drive? Just to verify the that the nvram is indeed done.

 

[JGrana]

I don’t blame you.
I can’t think of any non invasive tests off hand.

Asus does put some mtd commands on the router. You might want to try /bin/mtdinfo.

Try the -M flag on /dev/mtd9.
You are looking for number of bad blocks.

Im on an AX88U Pro runnning the latest firmware - everything it ubi based now (rather than mtd based on the older devices/firmewares).

Maybe @ColinTaylor has some ideas. He implemented the format and fsck functions in AMTM.
Not sure about running an fsck on a live file system.

 

[JGrana]

Hmm, good question on running it from a USB device. The application shouldn’t care where it runs from.

Take the curl line and replace the /jffs location with your mounted USB partition.
Or, maybe just download the mtd_check64 binary from github onto the USB stick. Then plug into the router and run it off the stick - good idea!

 

[JGrana]

I compiled both a 64bit and 32bit binary. The install script tries to get clever and determine if your running 32 or 64 bit.

 

[thunder2k]

I don’t blame you.
I can’t think of any non invasive tests off hand.

Asus does put some mtd commands on the router. You might want to try /bin/mtdinfo.

Try the -M flag on /dev/mtd9.
You are looking for number of bad blocks.

Im on an AX88U Pro runnning the latest firmware - everything it ubi based now (rather than mtd based on the older devices/firmewares).

Maybe @ColinTaylor has some ideas. He implemented the format and fsck functions in AMTM.
Not sure about running an fsck on a live file system.

This is the output. Only 1 BAD sector, is it good, right? Can that single BAD sector cause the issues?

XXXX@RT-AX86U-9E48:/tmp/home/root# /bin/mtdinfo -M /dev/mtd9
mtd9
Name:                           misc2
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          376 (49283072 bytes, 47.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  2048 bytes
OOB size:                       64 bytes
Character device major/minor:   90:18
Bad blocks are allowed:         true
Device is writable:             true
Eraseblock map:
   0: 00000000           1: 00020000           2: 00040000           3: 00060000
   4: 00080000           5: 000a0000           6: 000c0000           7: 000e0000
   8: 00100000           9: 00120000          10: 00140000          11: 00160000
  12: 00180000          13: 001a0000          14: 001c0000          15: 001e0000
  16: 00200000          17: 00220000          18: 00240000          19: 00260000
  20: 00280000          21: 002a0000          22: 002c0000          23: 002e0000
  24: 00300000          25: 00320000          26: 00340000          27: 00360000
  28: 00380000          29: 003a0000          30: 003c0000          31: 003e0000
  32: 00400000          33: 00420000          34: 00440000          35: 00460000
  36: 00480000          37: 004a0000          38: 004c0000          39: 004e0000
  40: 00500000          41: 00520000          42: 00540000          43: 00560000
  44: 00580000          45: 005a0000          46: 005c0000          47: 005e0000
  48: 00600000          49: 00620000          50: 00640000          51: 00660000
  52: 00680000          53: 006a0000          54: 006c0000          55: 006e0000
  56: 00700000          57: 00720000          58: 00740000          59: 00760000
  60: 00780000          61: 007a0000          62: 007c0000          63: 007e0000
  64: 00800000          65: 00820000          66: 00840000          67: 00860000
  68: 00880000          69: 008a0000          70: 008c0000          71: 008e0000
  72: 00900000          73: 00920000          74: 00940000          75: 00960000
  76: 00980000          77: 009a0000          78: 009c0000          79: 009e0000
  80: 00a00000          81: 00a20000          82: 00a40000          83: 00a60000
  84: 00a80000          85: 00aa0000          86: 00ac0000          87: 00ae0000
  88: 00b00000          89: 00b20000          90: 00b40000          91: 00b60000
  92: 00b80000          93: 00ba0000          94: 00bc0000          95: 00be0000
  96: 00c00000          97: 00c20000          98: 00c40000          99: 00c60000
 100: 00c80000         101: 00ca0000         102: 00cc0000         103: 00ce0000
 104: 00d00000         105: 00d20000         106: 00d40000         107: 00d60000
 108: 00d80000         109: 00da0000         110: 00dc0000         111: 00de0000
 112: 00e00000         113: 00e20000         114: 00e40000         115: 00e60000
 116: 00e80000         117: 00ea0000    BAD  118: 00ec0000         119: 00ee0000
 120: 00f00000         121: 00f20000         122: 00f40000         123: 00f60000
 124: 00f80000         125: 00fa0000         126: 00fc0000         127: 00fe0000
 128: 01000000         129: 01020000         130: 01040000         131: 01060000
 132: 01080000         133: 010a0000         134: 010c0000         135: 010e0000
 136: 01100000         137: 01120000         138: 01140000         139: 01160000
 140: 01180000         141: 011a0000         142: 011c0000         143: 011e0000
 144: 01200000         145: 01220000         146: 01240000         147: 01260000
 148: 01280000         149: 012a0000         150: 012c0000         151: 012e0000
 152: 01300000         153: 01320000         154: 01340000         155: 01360000
 156: 01380000         157: 013a0000         158: 013c0000         159: 013e0000
 160: 01400000         161: 01420000         162: 01440000         163: 01460000
 164: 01480000         165: 014a0000         166: 014c0000         167: 014e0000
 168: 01500000         169: 01520000         170: 01540000         171: 01560000
 172: 01580000         173: 015a0000         174: 015c0000         175: 015e0000
 176: 01600000         177: 01620000         178: 01640000         179: 01660000
 180: 01680000         181: 016a0000         182: 016c0000         183: 016e0000
 184: 01700000         185: 01720000         186: 01740000         187: 01760000
 188: 01780000         189: 017a0000         190: 017c0000         191: 017e0000
 192: 01800000         193: 01820000         194: 01840000         195: 01860000
 196: 01880000         197: 018a0000         198: 018c0000         199: 018e0000
 200: 01900000         201: 01920000         202: 01940000         203: 01960000
 204: 01980000         205: 019a0000         206: 019c0000         207: 019e0000
 208: 01a00000         209: 01a20000         210: 01a40000         211: 01a60000
 212: 01a80000         213: 01aa0000         214: 01ac0000         215: 01ae0000
 216: 01b00000         217: 01b20000         218: 01b40000         219: 01b60000
 220: 01b80000         221: 01ba0000         222: 01bc0000         223: 01be0000
 224: 01c00000         225: 01c20000         226: 01c40000         227: 01c60000
 228: 01c80000         229: 01ca0000         230: 01cc0000         231: 01ce0000
 232: 01d00000         233: 01d20000         234: 01d40000         235: 01d60000
 236: 01d80000         237: 01da0000         238: 01dc0000         239: 01de0000
 240: 01e00000         241: 01e20000         242: 01e40000         243: 01e60000
 244: 01e80000         245: 01ea0000         246: 01ec0000         247: 01ee0000
 248: 01f00000         249: 01f20000         250: 01f40000         251: 01f60000
 252: 01f80000         253: 01fa0000         254: 01fc0000         255: 01fe0000
 256: 02000000         257: 02020000         258: 02040000         259: 02060000
 260: 02080000         261: 020a0000         262: 020c0000         263: 020e0000
 264: 02100000         265: 02120000         266: 02140000         267: 02160000
 268: 02180000         269: 021a0000         270: 021c0000         271: 021e0000
 272: 02200000         273: 02220000         274: 02240000         275: 02260000
 276: 02280000         277: 022a0000         278: 022c0000         279: 022e0000
 280: 02300000         281: 02320000         282: 02340000         283: 02360000
 284: 02380000         285: 023a0000         286: 023c0000         287: 023e0000
 288: 02400000         289: 02420000         290: 02440000         291: 02460000
 292: 02480000         293: 024a0000         294: 024c0000         295: 024e0000
 296: 02500000         297: 02520000         298: 02540000         299: 02560000
 300: 02580000         301: 025a0000         302: 025c0000         303: 025e0000
 304: 02600000         305: 02620000         306: 02640000         307: 02660000
 308: 02680000         309: 026a0000         310: 026c0000         311: 026e0000
 312: 02700000         313: 02720000         314: 02740000         315: 02760000
 316: 02780000         317: 027a0000         318: 027c0000         319: 027e0000
 320: 02800000         321: 02820000         322: 02840000         323: 02860000
 324: 02880000         325: 028a0000         326: 028c0000         327: 028e0000
 328: 02900000         329: 02920000         330: 02940000         331: 02960000
 332: 02980000         333: 029a0000         334: 029c0000         335: 029e0000
 336: 02a00000         337: 02a20000         338: 02a40000         339: 02a60000
 340: 02a80000         341: 02aa0000         342: 02ac0000         343: 02ae0000
 344: 02b00000         345: 02b20000         346: 02b40000         347: 02b60000
 348: 02b80000         349: 02ba0000         350: 02bc0000         351: 02be0000
 352: 02c00000         353: 02c20000         354: 02c40000         355: 02c60000
 356: 02c80000         357: 02ca0000         358: 02cc0000         359: 02ce0000
 360: 02d00000         361: 02d20000         362: 02d40000         363: 02d60000
 364: 02d80000         365: 02da0000         366: 02dc0000         367: 02de0000
 368: 02e00000         369: 02e20000         370: 02e40000         371: 02e60000
 372: 02e80000         373: 02ea0000         374: 02ec0000         375: 02ee0000

 

[JGrana]

1 or 2 is typical. Interesting - run the same command on all the mtd devices. Maybe it’s not the jffs partition.
Look for any partition with more than say 10.

If it’s run out of erase blocks the system is either confused/corrupted or there really are lots of bad blocks and it has exhausted available replacement blocks.

 

[thunder2k]

Hmm, good question on running it from a USB device. The application shouldn’t care where it runs from.

Take the curl line and replace the /jffs location with your mounted USB partition.
Or, maybe just download the mtd_check64 binary from github onto the USB stick. Then plug into the router and run it off the stick - good idea!

I've downloaded the raw file from https://github.com/JGrana01/mtd_check/raw/refs/heads/main/mtd_check64, but when I try to execute it, it says not found:

XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ls
mtd_check64        mtd_check_install
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# /tmp/mnt/Seagate_Expansion_Drive/scripts/mtd_check64
-sh: /tmp/mnt/Seagate_Expansion_Drive/scripts/mtd_check64: not found
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check64
-sh: ./mtd_check64: not found
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ls -ltrh
-rwxr-xr-x    1 XXXX root        1.3K Jun  7 15:45 mtd_check_install
-rwxrwxrwx    1 XXXX root       23.5K Jun  7 15:52 mtd_check64
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check64
-sh: ./mtd_check64: not found

If I try to install it with mtd_check_install, I get the following error:

XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check_install


mtd_check installer
This will install the mtd_check application to /opt/bin
To continue, enter Y Y

Checking for EntwareEntware not installed
Can't install mtd_check...

Any suggestions?

 

[thunder2k]

1 or 2 is typical. Interesting - run the same command on all the mtd devices. Maybe it’s not the jffs partition.
Look for any partition with more than say 10.

If it’s run out of erase blocks the system is either confused/corrupted or there really are lots of bad blocks and it has exhausted available replacement blocks.

All of the following?

mtd0        mtd10ro     mtd12ro     mtd14ro     mtd2        mtd4        mtd6        mtd8        mtdblock0   mtdblock12  mtdblock2   mtdblock6
mtd0ro      mtd11       mtd13       mtd15       mtd2ro      mtd4ro      mtd6ro      mtd8ro      mtdblock1   mtdblock13  mtdblock3   mtdblock7
mtd1        mtd11ro     mtd13ro     mtd15ro     mtd3        mtd5        mtd7        mtd9        mtdblock10  mtdblock14  mtdblock4   mtdblock8
mtd10       mtd12       mtd14       mtd1ro      mtd3ro      mtd5ro      mtd7ro      mtd9ro      mtdblock11  mtdblock15  mtdblock5   mtdblock9

Any suggestions to run a single command for all?

 

[JGrana]

Not sure. Try mtdinfo —help

I believe you can ignore any ending in “ro”

 

[JGrana]

I've downloaded the raw file from https://github.com/JGrana01/mtd_check/raw/refs/heads/main/mtd_check64, but when I try to execute it, it says not found:

XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ls
mtd_check64        mtd_check_install
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# /tmp/mnt/Seagate_Expansion_Drive/scripts/mtd_check64
-sh: /tmp/mnt/Seagate_Expansion_Drive/scripts/mtd_check64: not found
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# pwd
/tmp/mnt/Seagate_Expansion_Drive/scripts
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check64
-sh: ./mtd_check64: not found
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ls -ltrh
-rwxr-xr-x    1 XXXX root        1.3K Jun  7 15:45 mtd_check_install
-rwxrwxrwx    1 XXXX root       23.5K Jun  7 15:52 mtd_check64
XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check64
-sh: ./mtd_check64: not found

If I try to install it with mtd_check_install, I get the following error:

XXXX@RT-AX86U-9E48:/tmp/mnt/Seagate_Expansion_Drive/scripts# ./mtd_check_install


mtd_check installer
This will install the mtd_check application to /opt/bin
To continue, enter Y Y

Checking for EntwareEntware not installed
Can't install mtd_check...

Any suggestions?

Yes, the install script checks for entware.
The actual binary (mtd_check64) should run fine. Just download that.

i’m going out for a few hours. i’ll check in later

Good luck!

 

[thunder2k]

I figured this out, this is the output:

Not sure. Try mtdinfo —help

I believe you can ignore any ending in “ro”

Hi,
here the results:

XXXX@RT-AX86U-9E48:/dev# /bin/mtdinfo -M /dev/mtd9mtd9Name: - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

Yes, the install script checks for entware.
The actual binary (mtd_check64) should run fine. Just download that.

i’m going out for a few hours. i’ll check in later

Good luck!

I tried to run mtd_check64, but it says not found (please check my previous comment).

Thanks for your help now, talk later!

 

[JGrana]

Sorry, I was out longer than I expected.

I looked at the mtdinfo dump on pastebin. Nothing looks out of the ordinary.
You have a few Bad Blocks - but that is normal.
I looked at mtdinfo.c (on github) and I can’t see where it colors the block numbers white and blue.
In looking at some of the mtd devices, it looks like the blue colored numbers are erase blocks in use.
You have plenty of available erase blocks and just a normal handful of Bad Blocks.

It looks like to me, either the mtd driver in the kernel has a corrupted table or there really is a hardware problem with the nand device.

I would first do an nvram and jffs backup. Both are stored in the nand device and it appears reads work fine. It’s when a write occurs.

Then, hold breath and do a reboot. If it’s a somehow corrupted table in the kernel it should fix it.

In any case, you can’t run long term without the ability to write to nand/flash…

BTW, erase block is somewhat a misleading name. Think of an erase block as a “sector” on a hard disk.