Attached USB Flash Drive Storage Not Reporting Accurate Available Space

[ColinTaylor]

You're using a very old version of e2fsck. I suggest you plug the drive into a Linux PC and run e2fsck on it from there.

 

[garycnew]

You're using a very old version of e2fsck. I suggest you plug the drive into a Linux PC and run e2fsck on it from there.

@ColinTaylor

At this point, running e2fsck from my Debian Live DVD is probably the best option.

I'll attempt it and let you know the results, tomorrow.

Thanks, again.


Gary

 

[garycnew]

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           384M  1.6M  382M   1% /run
/dev/sr0        3.2G  3.2G     0 100% /run/live/medium
/dev/loop0      2.8G  2.8G     0 100% /run/live/rootfs/filesystem.squashfs
tmpfs           1.9G  224M  1.7G  12% /run/live/overlay
overlay         1.9G  224M  1.7G  12% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G  8.0K  1.9G   1% /tmp
tmpfs           384M  172K  384M   1% /run/user/1000
/dev/sdb2       437G  382G   56G  88% /media/user/Time Capsule
/dev/sdb1        29G   15G   13G  55% /media/user/HitachiHDD

# e2fsck /dev/sdb1
e2fsck 1.46.2 (28-Feb-2021)
HitachiHDD: clean, 1917597/1917600 files, 4002871/7669021 blocks

# e2fsck -f /dev/sdb1
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Inode 1783538 extent tree (at level 1) could be narrower.  Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Entry 'prxylist' in /tmp/home/root/###/###/###/114/112/66 (1837664) has deleted/unused inode 1837665.  Clear<y>? yes
Entry '1761' in /tmp/home/root/###/###/### (785471) has deleted/unused inode 1766638.  Clear<y>? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -6867810 -6867826 -(6868100--6868101) -6868395 -6868549 -6869618 -7375423
Fix<y>? yes
Free blocks count wrong for group #209 (6910, counted=6917).
Fix<y>? yes
Free blocks count wrong for group #225 (4737, counted=4738).
Fix<y>? yes
Free blocks count wrong (3666151, counted=3666159).
Fix<y>? yes
Inode bitmap differences:  -1766638 -1837665
Fix<y>? yes
Free inodes count wrong for group #216 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #225 (0, counted=1).
Fix ('a' enables 'yes' to all) <y>? yes
Free inodes count wrong (3, counted=5).
Fix ('a' enables 'yes' to all) <y>? yes

HitachiHDD: ***** FILE SYSTEM WAS MODIFIED *****
HitachiHDD: 1917595/1917600 files (1.2% non-contiguous), 4002862/7669021 blocks

# opkg install psmisc
Installing psmisc (23.4-2a) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/psmisc_23.4-2a_armv7-2.6.ipk
Collected errors:
 * pkg_get_installed_files: Failed to make temp file /opt/tmp/opkg-px9LoN/psmisc.list.pimlAw.: No space left on device.
 * pkg_get_installed_files: Failed to make temp file /opt/tmp/opkg-px9LoN/psmisc.list.OWXCxA.: No space left on device.
 * wfopen: /opt/bin/prtstat: No space left on device.
 * wfopen: /opt/bin/pslog: No space left on device.
 * wfopen: /opt/bin/pstree: No space left on device.
 * wfopen: /opt/libexec/killall: No space left on device.
 * pkg_write_filelist: Failed to open //opt/lib/opkg/info/psmisc.list: No space left on device.
 * opkg_install_pkg: Failed to extract data files for psmisc. Package debris may remain!
 * opkg_install_cmd: Cannot install package psmisc.
 * opkg_prep_intercepts: Failed to make temp dir /opt/tmp/opkg-px9LoN/opkg-intercept-M4Gc9G: No space left on device.

 

[ColinTaylor]

So has that fixed the problem when you plug it back into your router?

 

[garycnew]

So has that fixed the problem when you plug it back into your router?

@ColinTaylor

Negative... It's still stating "No space left on device."

# opkg install psmisc
Installing psmisc (23.4-2a) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/psmisc_23.4-2a_armv7-2.6.ipk
Collected errors:
 * pkg_get_installed_files: Failed to make temp file /opt/tmp/opkg-px9LoN/psmisc.list.pimlAw.: No space left on device.
 * pkg_get_installed_files: Failed to make temp file /opt/tmp/opkg-px9LoN/psmisc.list.OWXCxA.: No space left on device.
 * wfopen: /opt/bin/prtstat: No space left on device.
 * wfopen: /opt/bin/pslog: No space left on device.
 * wfopen: /opt/bin/pstree: No space left on device.
 * wfopen: /opt/libexec/killall: No space left on device.
 * pkg_write_filelist: Failed to open //opt/lib/opkg/info/psmisc.list: No space left on device.
 * opkg_install_pkg: Failed to extract data files for psmisc. Package debris may remain!
 * opkg_install_cmd: Cannot install package psmisc.
 * opkg_prep_intercepts: Failed to make temp dir /opt/tmp/opkg-px9LoN/opkg-intercept-M4Gc9G: No space left on device.

Although, it shows that the HitachiHDD has 54% used space.

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                29.1M     29.1M         0 100% /
devtmpfs                124.7M         0    124.7M   0% /dev
tmpfs                   124.8M      2.2M    122.7M   2% /tmp
/dev/mtdblock4           62.8M     11.0M     51.8M  18% /jffs
/dev/sda1                28.8G     14.8G     12.5G  54% /tmp/mnt/HitachiHDD
/dev/mtdblock4           62.8M     11.0M     51.8M  18% /usr/sbin/dnsapi
/dev/mtdblock4           62.8M     11.0M     51.8M  18% /usr/sbin/acme.sh
/dev/sda2               436.5G    381.5G     55.0G  87% /tmp/mnt/Time_Capsule
/dev/mtdblock4           62.8M     11.0M     51.8M  18% /www/Main_LogStatus_Content.asp

I just don't get it.

Respectfully,


Gary

P.S. Would thousands of symlinks on the HitachiHDD not be reported correctly?

 

[ColinTaylor]

Can you do a sanity check on these two symlinks:

# ls -l /opt
lrwxrwxrwx    1 admin    root             7 Dec  2 18:52 /opt -> tmp/opt
# ls -l /tmp/opt
lrwxrwxrwx    1 admin    root            25 Mar 26 20:40 /tmp/opt -> /tmp/mnt/TOSHIBA1/entware

P.S. Would thousands of symlinks on the HitachiHDD not be reported correctly?

I would have thought it unlikely, but I suppose anything is possible. Hard or soft links?

 

[sfx2000]

I have a USB Flash Drive attached to my Asuswrt-Merlin Router that shows only 57% of its space used (using df -h), but I've started receiving "No space left on device" messages; when, trying to write to it. I've even rebooted the router, which still reports the same usage and "No space left on device."

If I remove a few unnecessary files, from the USB Flash Drive, it still reports the same available space, but I can then write to it, until it is full, again.

Thanks.

Ya know, at the end of the day - something odd is going on here...

That being said, let the router be a router, and not a NAS - disk sharing has always been a minefield with AsusWRT, since basically it's a checkbox feature that doesn't get a lot of love...

USB drive to support entware, that's one thing, extending the OS itself...

File shares to support SMB and TimeMachine - you're having to deal with Samba (and perhaps Netatalk) on what is a pretty old kernel and samba versions.

Might be better to get a dedicated machine - Intel N100 small form factor boxes aren't that expensive, and runnning an LTS version of Ubuntu isn't a bad thing - and it will be much more performant...

 

[sfx2000]

That being said, let the router be a router, and not a NAS - disk sharing has always been a minefield with AsusWRT, since basically it's a checkbox feature that doesn't get a lot of love...

remember - it's your data - make the appropriate choices here - when I say minefield - it's just that - works for some, and with the wrong step, it can blow up on you...

Again, let the router be just that - a router - it's good at that...

 

[garycnew]

Can you do a sanity check on these two symlinks:

# ls -l /opt
lrwxrwxrwx    1 admin    root             7 Dec  2 18:52 /opt -> tmp/opt
# ls -l /tmp/opt
lrwxrwxrwx    1 admin    root            25 Mar 26 20:40 /tmp/opt -> /tmp/mnt/TOSHIBA1/entware

The primary symlinks look fine.

# ls -l /opt
lrwxrwxrwx    1 admin    root             7 Aug 14  2020 /opt -> tmp/opt
admin@gnutech-wap01:/tmp/home/root# ls -l /tmp/opt
lrwxrwxrwx    1 admin    root            27 Apr  9 02:49 /tmp/opt -> /tmp/mnt/HitachiHDD/entware

I would have thought it unlikely, but I suppose anything is possible. Hard or soft links?

Soft Links

 

[ColinTaylor]

P.S. Would thousands of symlinks on the HitachiHDD not be reported correctly?

Soft Links

I think you may have nailed it.

I was so concentrated on the number of blocks I missed this in your earlier output from tune2fs:

Free inodes:              0

I also missed this from e2fsck :

HitachiHDD: clean, 1917594/1917600 files, 4003673/7669021 blocks

The solution unfortunately is to recreate the filesystem with more inodes. The default inode ratio for a filesystem that size is 16384. I suggest changing that to 4096.

mke2fs -t ext4 -i 4096 /dev/sdb1

 

[garycnew]

I think you may have nailed it.

I was so concentrated on the number of blocks I missed this in your earlier output from tune2fs:

Free inodes:              0

I also missed this from e2fsck :

HitachiHDD: clean, 1917594/1917600 files, 4003673/7669021 blocks

The solution unfortunately is to recreate the filesystem with more inodes. The default inode ratio for a filesystem that size is 16384. I suggest changing that to 4096.

mke2fs -t ext4 -i 4096 /dev/sdb1

@ColinTaylor

Good Eye! This is why troubleshooting is worthwhile in understanding a problem and not perpetuating the issue.

I have a Synology920+ that I have been meaning to migrate this data to and intend to continue to grow the thousands of symlinks.

That being said... Would you still recommend a block size of 4096 or something smaller like 1024 or 2048 for the new ext4 partition?

Thanks, again, for your time and assistance!

Respectfully,


Gary

 

[ColinTaylor]

Thinking aloud... My understanding is that the default inode size is large enough to store basic metadata as well as the target of a symbolic link. So theoretically it doesn't really matter what the blocksize is for the links, only for the actual data files.

I think this is just one of those things you're going have monitor yourself and adjust accordingly. If the data files are the typical mixture of sizes I'd leave it at the default of 4096. If the majority of the data files are small use 1024. In either case I'd still use an inode ratio to 4096 unless you're planning on massively increasing the size of that partition.

 

[garycnew]

Thinking aloud... My understanding is that the default inode size is large enough to store basic metadata as well as the target of a symbolic link. So theoretically it doesn't really matter what the blocksize is for the links, only for the actual data files.

I think this is just one of those things you're going have monitor yourself and adjust accordingly. If the data files are the typical mixture of sizes I'd leave it at the default of 4096. If the majority of the data files are small use 1024. In either case I'd still use an inode ratio to 4096 unless you're planning on massively increasing the size of that partition.

@ColinTaylor

It appears that the default block size for an ext4 file system is 4096, which I have confirmed observing that the symlinks and directories on the existing file system are 4096 bytes each.

The majority of small files I am working with are close to 4096 bytes in size or less.

I'm concerned that the smaller 1024 block size might reduce performance and put additional wear on the disk, but I feel like it might be the best option in this scenario?

I appreciate your thoughts.

Respectfully,


Gary

 

[ColinTaylor]

The majority of small files I am working with are close to 4096 bytes in size or less.

Then I'd leave it at 4096 blocksize. The output of your tunefs didn't suggest the blocksize was going to be a problem.

 

[garycnew]

Then I'd leave it at 4096 blocksize. The output of your tunefs didn't suggest the blocksize was going to be a problem.

@ColinTaylor

It sounds like the 4096 block size has been optimized and is the sweet-spot for the ext4 file system.

If I leave it at the 4096 block size, it means the inode ratio will remain the same; thus, the only option is to move the data to a larger volume?

Respectfully,


Gary

 

[garycnew]

@ColinTaylor

I'm still running through data integrity tests. With the exception of having to change the user ownership for all files, the data appears to be migrated successfully to the new Synology 920+ volume.

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/md0                7.9G  2.5G  5.3G  33% /
devtmpfs                3.8G     0  3.8G   0% /dev
tmpfs                   3.9G  236K  3.9G   1% /dev/shm
tmpfs                   3.9G   14M  3.8G   1% /run
tmpfs                   3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                   3.9G  8.3M  3.8G   1% /tmp
/dev/mapper/cachedev_0  1.3T   13G  1.3T   1% /volume1

With 88670198 free inodes, I should be good for a bit.

# tune2fs -l /dev/mapper/cachedev_0
tune2fs 1.44.1 (24-Mar-2018)
Filesystem volume name:   1.44.1-69057
Last mounted on:          /volume1
Filesystem UUID:          39bea384-b329-4c02-8c3a-92c9968e6f98
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
incompat_flags:           0x2c6
ro_compat_flags:          0x7b
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              88670208
Block count:              354680832
Reserved block count:     25600
Free blocks:              348832444
Free inodes:              88670198
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Apr 17 19:50:24 2024
Last mount time:          Wed Apr 17 19:50:32 2024
Last write time:          Wed Apr 17 19:50:32 2024
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Apr 17 19:50:24 2024
Check interval:           0 (<none>)
Lifetime writes:          1031 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     36
Desired extra isize:      36
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      71eaf3ff-5f8f-4fec-81c9-b8e4ba26bbb6
Journal backup:           inode blocks
rbd_mapping_table_first_offset: 0
syno_capability_flags:    0x0
is_valid_capability_flag: 1

Thanks for helping diagnose the issue and weigh possible solutions.

I believe migrating to the Synology 920+ might resolve my NFS locking issues as well.

Respectfully,


Gary

 

[garycnew]

@ColinTaylor

I thought you might like to know... While directories are 4096 bytes in size, symlinks appear to only be 28 or 29 bytes in size.

Respectfully,


Gary

 

[ColinTaylor]

I thought you might to know... While directories are 4096 bytes in size, symlinks appear to only be 28 or 29 bytes in size.

Yes, that's correct. Remember that symbolic links are just files whose contents are the target. Think of them as text files.

symlink(7) - Linux manual page

A symbolic link is a special type of file whose contents are a
string that is the pathname of another file, the file to which
the link refers. (The contents of a symbolic link can be read
using readlink(2).) In other words, a symbolic link is a pointer
to another name, and not to an underlying object. For this
reason, symbolic links may refer to directories and may cross
filesystem boundaries.

But the point, as I mentioned above, is that because the contents of the symlink file is so small it can be included in the inode and therefore doesn't consume any data blocks.

Examine the file sizes below:

# ls -al libz.so*
lrwxrwxrwx    1 admin    root             9 Feb 26 13:40 libz.so -> libz.so.1
lrwxrwxrwx    1 admin    root            13 Feb 26 13:40 libz.so.1 -> libz.so.1.3.1
-rw-r--r--    1 admin    root         88040 Feb  7 12:59 libz.so.1.3.1

See that libz.so is 9 bytes in size because it's a file that contains "libz.so.1".

Now check to see how many 512 byte data blocks it consumes (i.e. zero)

# stat libz.so
  File: libz.so -> libz.so.1
  Size: 9               Blocks: 0          IO Block: 4096   symbolic link
Device: 8,1     Inode: 392547      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/   admin)   Gid: (    0/    root)
Access: 2024-04-20 10:55:49.943987059 +0100
Modify: 2024-02-26 13:40:07.685887767 +0000
Change: 2024-02-26 13:40:07.685887767 +0000
 Birth: -

 

[garycnew]

Yes, that's correct. Remember that symbolic links are just files whose contents are the target. Think of them as text files.

symlink(7) - Linux manual page

But the point, as I mentioned above, is that because the contents of the symlink file is so small it can be included in the inode and therefore doesn't consume any data blocks.

Examine the file sizes below:

# ls -al libz.so*
lrwxrwxrwx    1 admin    root             9 Feb 26 13:40 libz.so -> libz.so.1
lrwxrwxrwx    1 admin    root            13 Feb 26 13:40 libz.so.1 -> libz.so.1.3.1
-rw-r--r--    1 admin    root         88040 Feb  7 12:59 libz.so.1.3.1

See that libz.so is 9 bytes in size because it's a file that contains "libz.so.1".

Now check to see how many 512 byte data blocks it consumes (i.e. zero)

# stat libz.so
  File: libz.so -> libz.so.1
  Size: 9               Blocks: 0          IO Block: 4096   symbolic link
Device: 8,1     Inode: 392547      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/   admin)   Gid: (    0/    root)
Access: 2024-04-20 10:55:49.943987059 +0100
Modify: 2024-02-26 13:40:07.685887767 +0000
Change: 2024-02-26 13:40:07.685887767 +0000
 Birth: -

@ColinTaylor

When I stat the related files and symlinks on the new Synology volume, it shows that the files and symlinks report the same inode number in use and the same block size for both.

Thank you for the example and clarification.

Respectfully,


Gary