ext4 journaled doesn't seem to work with 3.8.5-2 on AX86u

[jorhett]

I'm using the latest 3.8.5 merlin image for the RT-AX86U and it works fine. I finally bought an USB3 SSD so that I could install Entware... and spent 2 days circling around trying to figure out why it didn't work. I used amtm every time. After each reboot I saw this error message in the log

May 21 22:24:24 kernel: EXT4-fs error (device sda1): ext4_find_extent:908: inode #8: comm hotplug: pblk 127959039 bad header/extent: invalid magic - magic ffff, entries 65535, max 65535(0), depth 65535(0)
May 21 22:24:24 kernel: jbd2_journal_bmap: journal block not found at offset 0 on sda1-8
May 21 22:24:24 kernel: jbd2_journal_init_inode: Cannot locate journal superblock
May 21 22:24:24 kernel: EXT4-fs (sda1): Could not load journal inode
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext3 due to feature incompatibilities
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext2 due to feature incompatibilities
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext3 due to feature incompatibilities

Fsck on the device didn't work, complaining about the lack of fsck.auto:

# fsck /dev/sda1
fsck (busybox 1.25.1, 2022-03-25 10:23:25 EDT)
fsck: fsck.auto: No such file or directory

Oddly, the firmware doesn't contain fsck.ext4

# find / -name fsck.ext\*
/usr/sbin/fsck.ext2
/usr/sbin/fsck.ext3

Poking around in the forum I found e2fsck options, but it always reported the same lack of a journal every time:

# e2fsck -p /dev/sda1
entware: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

entware: Journal inode is not in use, but contains data.  CLEARED.
entware: Recreate journal.
Creating journal (262144 blocks):  Done.

*** journal has been regenerated ***
entware: 11/64004096 files (0.0% non-contiguous), 4297508/255999783 blocks

# e2fsck -p /dev/sda1
entware: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

entware: Journal inode is not in use, but contains data.  CLEARED.
entware: Recreate journal.
Creating journal (262144 blocks):

Given these outputs, I recreated it without a journal and the problem disappeared. I guess since I'm only using it for entware packages there won't be enough changes to worry about the journal ...

 

[bbunge]

Could be the USB device. Also could be the USB3. I use USB2 and yesterday used AMTM to format the thumb drive to EXT4 with journal, added a swap file and installed Diversion. My AX86U is on Merlin 386.5_2 as well .

 

[John Fitzgerald]

I'm using the latest 3.8.5 merlin image for the RT-AX86U and it works fine. I finally bought an USB3 SSD so that I could install Entware... and spent 2 days circling around trying to figure out why it didn't work. I used amtm every time. After each reboot I saw this error message in the log

May 21 22:24:24 kernel: EXT4-fs error (device sda1): ext4_find_extent:908: inode #8: comm hotplug: pblk 127959039 bad header/extent: invalid magic - magic ffff, entries 65535, max 65535(0), depth 65535(0)
May 21 22:24:24 kernel: jbd2_journal_bmap: journal block not found at offset 0 on sda1-8
May 21 22:24:24 kernel: jbd2_journal_init_inode: Cannot locate journal superblock
May 21 22:24:24 kernel: EXT4-fs (sda1): Could not load journal inode
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext3 due to feature incompatibilities
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext2 due to feature incompatibilities
May 21 22:24:24 kernel: EXT4-fs (sda1): couldn't mount as ext3 due to feature incompatibilities

Fsck on the device didn't work, complaining about the lack of fsck.auto:

# fsck /dev/sda1
fsck (busybox 1.25.1, 2022-03-25 10:23:25 EDT)
fsck: fsck.auto: No such file or directory

Oddly, the firmware doesn't contain fsck.ext4

# find / -name fsck.ext\*
/usr/sbin/fsck.ext2
/usr/sbin/fsck.ext3

Poking around in the forum I found e2fsck options, but it always reported the same lack of a journal every time:

# e2fsck -p /dev/sda1
entware: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

entware: Journal inode is not in use, but contains data.  CLEARED.
entware: Recreate journal.
Creating journal (262144 blocks):  Done.

*** journal has been regenerated ***
entware: 11/64004096 files (0.0% non-contiguous), 4297508/255999783 blocks

# e2fsck -p /dev/sda1
entware: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

entware: Journal inode is not in use, but contains data.  CLEARED.
entware: Recreate journal.
Creating journal (262144 blocks):

Given these outputs, I recreated it without a journal and the problem disappeared. I guess since I'm only using it for entware packages there won't be enough changes to worry about the journal ...

The way I do it is to format the USB drive via a PC. Windows, NTFS, the disk is initialized and I can check its overall function.

Then I move it to the router, selecting the USB 3.0 port I want. After that, I use AMTM script to "fd" format disk, this is where you select EXT4 with Journaling and select the size of the SWAP File you want to use.

It should now show as mounted in the router GUI. From there (AMTM) you can select "ep" to install Entware.

From AMTM you can also install "dc" disk check. This will show the status of the drive on each router reboot, using "dcl" disk check log.

It sounds like you are not using AMTM for your install and I think that's your problem. The router does not have EXT natively, it's done through the built in script AMTM/Rmerlin Firmware. Stock ASUS firmware does not have it.

 

[ColinTaylor]

The router does not have EXT natively, it's done through the built in script AMTM/Rmerlin Firmware. Stock ASUS firmware does not have it.

This is incorrect/misleading. ext2/3/4 is "native" to the the router as it part of the Linux kernel and the preferred filesystem type. What you probably meant to say was that there is no option in the router's GUI to format a USB device to those filesystem types.

 

[John Fitzgerald]

This is incorrect/misleading. ext2/3/4 is "native" to the the router as it part of the Linux kernel and the preferred filesystem type. What you probably meant to say was that there is no option in the router's GUI to format a USB device to those filesystem types.

Yes I understand, but why is it not available in the GUI, if it's native to the Linux Kernel.

Unavailable is just that. Lack of an option for it in the GUI appears to be an omission, on purpose, by ASUS.

And thanks for adding your comment.

 

[ColinTaylor]

Yes I understand, but why is it not available in the GUI, if it's native to the Linux Kernel.

You'd have to ask Asus. My guess is that they expect their customers to be Windows or Mac users. Therefore they have chosen filesystems that are compatible with their PCs (i.e. FAT, NTFS and HFS) should the user decide to move the USB device back and forth between the router and PC.

 

[John Fitzgerald]

You'd have to ask Asus. My guess is that they expect their customers to be Windows or Mac users. Therefore they have chosen filesystems that are compatible with their PCs (i.e. FAT, NTFS and HFS) should the user decide to move the USB device back and forth between the router and PC.

I could have sworn that exFAT was an option at one time. Was it? If it was, ASUS has eliminated that option as well. How's my memory???

I tried writing my reply in a way that if others were doing a search for the same issue and they were on Stock Firmware, it would indicate that they too would have problems. I fell a bit short.

 

[BreakingDad]

I use https://www.partitionwizard.com/free-partition-manager.html in windows to set up my usb drives. Then plug them into router.

NTFS for data (movies music etc), and EXT4 for swap (router stuff) . works perfectly on ax86u with 2tb mypassport.

Router Stuff is a technical term.

 

[ColinTaylor]

I could have sworn that exFAT was an option at one time. Was it? If it was, ASUS has eliminated that option as well. How's my memory???

exFAT has never been supported as far as I can remember.

N.B. John's firmware did briefly try and add support for it many years ago but it didn't work properly and was quickly reverted.

Plug-n-Share Disks Compatibility List

 

[jorhett]

John, thank you for replying but every assumption you made defies what I said in my post.

The way I do it is to format the USB drive via a PC. Windows, NTFS, the disk is initialized and I can check its overall function.

I did set it up on a Mac with FAT and confirmed the disk worked fine before adding to the Asus.

It sounds like you are not using AMTM for your install and I think that's your problem.

As I said, I *only* used amtm to do the formatting, I never tried any other method. And yes, I know how to use "ep" to install entware, but you can't do this when the disk won't mount.

The router does not have EXT natively, it's done through the built in script AMTM/Rmerlin Firmware. Stock ASUS firmware does not have it.

This statement is just so wrong that I can't even unpack it. The amtm script does not provide the ext filesystem support that is part of the firmware: /usr/sbin/fsck.ext3, /usr/sbin/fsck.ext, /sbin/fsck, etc -- are part of the Linux firmware. The amtm script just runs a series of commands that are already part of the firmware.

I should probably clarify I've been using Linux for 29 years now -- back in the 0.2 kernel days, and rebuilding the entire tree from scratch. My question is not "how does ext4 work" -- it was a statement that the script doesn't appear to handle something correctly as regards this drive.

 

[jorhett]

To further add to the situation, about 20 minutes after my original post I went back to continue working with Entware and found that every command I typed failed with a screen full of followed by File name too long

# opkg update �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������: File name too long

The drive itself appeared to be working in so far as directory listings worked, but that was likely coming from the file cache. I unmounted the drive and ran fsck and got a bunch of junk:

# e2fsck -p /dev/sda1
space contains a file system with errors, check forced.
space: Inode 23855149 has the casefold flag set but is not a directory.

space: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
    (i.e., without -a or -p options)

# e2fsck -y /dev/sda1
e2fsck 1.45.6 (20-Mar-2020)
space contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 23855149 has the casefold flag set but is not a directory.  Clear flag? yes

Inode 23855149 has inline data and extent flags set but i_block contains junk.
Clear inode? yes

Inode 23855150 has the casefold flag set but is not a directory.  Clear flag? yes

... snip about 60 other inodes ...

Pass 2: Checking directory structure
Entry 'entware' in / (2) has deleted/unused inode 23855105.  Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Inode 2 ref count is 4, should be 3.  Fix? yes

Pass 5: Checking group summary information
Block bitmap differences:  -(33796--33999) -(34304--34306) -34308 -34320 -(34326--34327) -(34329--34815) -(15736864--15761407) -(16261152--16285695) -(16785440--16809983) -(17309728--17334271) -(17834016--17858559) -(18358304--18382847) -(18882592-
   ...
-251691007) -(252190752--252215295) -(252715040--252739583) -(253239328--253263871) -(253763616--253788159) -(254287904--254312447) -(254812192--254836735) -(255336480--255361023) -(255855114--255885311)
Fix? yes

Free blocks count wrong for group #1 (31042, counted=31740).
Fix? yes

Free blocks count wrong for group #2913 (29578, counted=32768).
Fix? yes

Free blocks count wrong (251960529, counted=251964417).
Fix? yes

Inode bitmap differences:  -(23855105--23863296) -(55050241--55058432)
Fix? yes

Free inodes count wrong for group #2912 (7776, counted=8192).
Fix? yes

Directories count wrong for group #2912 (44, counted=0).
Fix? yes

Free inodes count wrong (64003663, counted=64004079).
Fix? yes

space: ***** FILE SYSTEM WAS MODIFIED *****
space: 17/64004096 files (0.0% non-contiguous), 4035366/255999783 blocks

# e2fsck /dev/sda1
e2fsck 1.45.6 (20-Mar-2020)
space: clean, 17/64004096 files, 4035366/255999783 blocks

Re-running fsck ran clean. The drive successfully remounted, but all the files/directories were now orphaned:

# mount /dev/sda1 /tmp/mnt/space
# ls -la /tmp/mnt/space/
drwxrwxrwx    3 jorhett  root          4096 May 21 23:20 .
drwxrwxrwx    3 jorhett  root            60 May 21 23:18 ..
-rw-rw-rw-    1 jorhett  root             6 May 21 23:18 .___var.txt
-rw-rw-rw-    1 jorhett  root             0 May 21 23:18 .___var.txt.6
-rw-rw-rw-    1 jorhett  root             9 May 21 23:18 .__folder_list.txt
-rw-rw-rw-    1 jorhett  root             0 May 21 23:18 .__folder_list.txt.9
-rw-rw-rw-    1 jorhett  root             6 May 21 23:18 .__jorhett_var.txt
-rw-rw-rw-    1 jorhett  root             0 May 21 23:18 .__jorhett_var.txt.6
drwx------    2 jorhett  root         16384 May 21 23:16 lost+found

Either

* something about amtm is misreading the drive / setting it up incorrectly
* something about fsck is inconsistent about when it expects things to be stored

I dislike that `fsck` returns the error it does. Is there something wrong with ext4 support on this firmware?

 

[ColinTaylor]

Either

* something about amtm is misreading the drive / setting it up incorrectly
* something about fsck is inconsistent about when it expects things to be stored

If a drive's partition is formatted as ext4 on a different machine it's possible to create a filesystem with features that the router doesn't support. This is a known problem, but it wouldn't create the hard errors that you're seeing with the drive. You've also now stated that you're formatting the drive using amtm so that shouldn't be an issue.

Everything you've posted so far suggests some sort of hardware issue or incompatibility. I've seen similar errors when a USB device has gone into read-only mode. Are you seeing other error messages in the router's syslog related to scsi, sd or usb? If the SSD is in an enclosure perhaps it has a power saving feature that is turning off the drive and corrupting the filesystem.

I dislike that `fsck` returns the error it does. Is there something wrong with ext4 support on this firmware?

fsck.auto is a busybox helper applet that's entirely optional. It's just a lazy way of running fsck without specifying the filesystem type.

 

[RMerlin]

ext4 support is whatever is present in the firmware's kernel. It's not something added to the firmware, it's part of the Linux kernel. And I highly doubt that ext4 support would be broken in kernel 4.1.xx.

 

[SomeWhereOverTheRainBow]

ext4 support is whatever is present in the firmware's kernel. It's not something added to the firmware, it's part of the Linux kernel. And I highly doubt that ext4 support would be broken in kernel 4.1.xx.

Can confirm it is not!

 

[John Fitzgerald]

It would be nice to know the Brand, Model and Size of the SSD, so if it's a bad pairing others can avoid it. If it's in a third party enclosure, that info as well.

 

[jorhett]

I can say that the following thread is very informative and relevant specifically to Asus drives, and it helped me select a better drive https://www.snbforums.com/threads/best-usb-3-hdd-for-ac68u.17379

 

[elorimer]

So the fix was a different SSD.