What's new

amtm ext4 partition with journaling - 15 GB size, 1150 GB of writes in 20 months

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

bibikalka

Regular Contributor
As I was slightly re-partitioning my external AMTM drive (a very low end dram-less 240GB SSD hanging on a USB2/SATA ~10 inch adapter cable), I noticed something really interesting. In 20 months this partition with Entware and a 2 GB swap file had 1152 GB of disk writes. So this is about 2GB of writes per 24 hours on average. With respect to 240GB SSD size, this is ~5x write cycles, which is very trivial given than an SSD should survive multiple hundreds depending on the flash type (TLC, QLC, etc). But, had this been a ~32GB flash cheapo flash drive, directly attached to the router thus getting all the nice heat transfer from the motherboard, it'd probably be really dead by now (1152/32=36 write cycles with possibly poor write balancing due to very basic flash drive controller, and heat, heat, and more heat).

I am thinking that perhaps a separate low overhead partition without journaling on the same SSD could be created just to hold the swap file (ext2???). I am not seeing how journaling the swap disk helps anything. A dedicated swap partition is poorly supported by AMTM, so that's not a solution either.

Anyway, if anybody wants to share their partition statistics, it could become a nice fact based conversation.


Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 15282156 4305852 10178360 30% /tmp/mnt/ac86u


Code:
admin@RT-AC86U-9988:/# tune2fs -l /dev/sda1
tune2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   ac86u
Last mounted on:          /tmp/mnt/ac86u
Filesystem UUID:          ...
Filesystem magic number:  ...
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              977280
Block count:              3907803
Reserved block count:     195390
Overhead blocks:          87264
Free blocks:              2749312
Free inodes:              970542
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      954
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Flex block group size:    16
Filesystem created:       Wed Feb  2 12:02:01 2022
Last mount time:          Fri Sep 22 16:52:48 2023
Last write time:          Fri Sep 22 16:52:48 2023
Mount count:              129
Maximum mount count:      -1
Last checked:             Wed Feb  2 12:02:01 2022
Check interval:           0 (<none>)
Lifetime writes:          1152 GB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ...
Journal backup:           inode blocks
 
What makes you suspect it's swap activity that's causing the writes? If you're using third party scripts it could be the constant writing to log/database files.
 
That USB drive plugged to my router was last formatted 11 years ago. Before being formatted to use on my brand new RT-N66U back in 2012, this thumbdrive was used as a boot drive on my WDTV to boot a custom development version of WDLXTV, so it has been in use for even longer.

They don't make them as reliable as they used to...

Code:
admin@stargate:/tmp/home/root# tune2fs -l /dev/sda1
tune2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   KT410
Last mounted on:          /tmp/mnt/KT410
Filesystem UUID:          e9552b39-2340-4004-886c-6051321c5bd1
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr filetype needs_recovery sparse_super large_file
Filesystem flags:         unsigned_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              492032
Block count:              984056
Reserved block count:     49202
Free blocks:              866319
Free inodes:              488337
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         15872
Inode blocks per group:   496
Filesystem created:       Mon Mar  5 01:39:10 2012
Last mount time:          Sat May  5 01:06:11 2018
Last write time:          Sat May  5 01:06:11 2018
Mount count:              950
Maximum mount count:      24
Last checked:             Fri Jul 31 20:00:28 2015
Check interval:           15552000 (6 months)
Next check after:         Wed Jan 27 19:00:28 2016
Lifetime writes:          9 GB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      fda0f80b-0b13-4d05-93d0-198931eb0b4a
Journal backup:           inode blocks
 
As I was slightly re-partitioning my external AMTM drive (a very low end dram-less 240GB SSD hanging on a USB2/SATA ~10 inch adapter cable), I noticed something really interesting. In 20 months this partition with Entware and a 2 GB swap file had 1152 GB of disk writes. So this is about 2GB of writes per 24 hours on average. With respect to 240GB SSD size, this is ~5x write cycles, which is very trivial given than an SSD should survive multiple hundreds depending on the flash type (TLC, QLC, etc). But, had this been a ~32GB flash cheapo flash drive, directly attached to the router thus getting all the nice heat transfer from the motherboard, it'd probably be really dead by now (1152/32=36 write cycles with possibly poor write balancing due to very basic flash drive controller, and heat, heat, and more heat).

I am thinking that perhaps a separate low overhead partition without journaling on the same SSD could be created just to hold the swap file (ext2???). I am not seeing how journaling the swap disk helps anything. A dedicated swap partition is poorly supported by AMTM, so that's not a solution either.

Anyway, if anybody wants to share their partition statistics, it could become a nice fact based conversation.


Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 15282156 4305852 10178360 30% /tmp/mnt/ac86u


Code:
admin@RT-AC86U-9988:/# tune2fs -l /dev/sda1
tune2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   ac86u
Last mounted on:          /tmp/mnt/ac86u
Filesystem UUID:          ...
Filesystem magic number:  ...
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              977280
Block count:              3907803
Reserved block count:     195390
Overhead blocks:          87264
Free blocks:              2749312
Free inodes:              970542
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      954
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Flex block group size:    16
Filesystem created:       Wed Feb  2 12:02:01 2022
Last mount time:          Fri Sep 22 16:52:48 2023
Last write time:          Fri Sep 22 16:52:48 2023
Mount count:              129
Maximum mount count:      -1
Last checked:             Wed Feb  2 12:02:01 2022
Check interval:           0 (<none>)
Lifetime writes:          1152 GB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ...
Journal backup:           inode blocks
Code:
ASUSWRT-Merlin RT-AC68U 386.12_0 Mon Sep  4 15:47:31 UTC 2023
admin******@RT-AC68U-C760:/tmp/home/root# tune2fs -l /dev/sda1
tune2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   AMTM
Last mounted on:          /tmp/mnt/AMTM
Filesystem UUID:          9f0a8fb8-b04f-40ff-8d78-bf79d40738e3
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              9773056
Block count:              39072080
Reserved block count:     1953604
Free blocks:              37293289
Free inodes:              9768539
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1014
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sun Feb 16 13:05:04 2020
Last mount time:          Sun Sep 17 10:39:10 2023
Last write time:          Sun Sep 17 10:39:10 2023
Mount count:              63
Maximum mount count:      -1
Last checked:             Sun Feb 16 13:05:04 2020
Check interval:           0 (<none>)
Lifetime writes:          2471 GB
Reserved blocks uid:      0 (user adminasus398k)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      1998c9ec-e70f-4a39-a25c-7573bb3702d6
Journal backup:           inode blocks
admin*******@RT-AC68U-C760:/tmp/home/root#
Don’t worry: mine works out at a similar 2GB per day over 3.5 years (on an RT-AC68U). I run Diversion, Skynet and Unbound Manager, uiScribe and uiDivstats. I use an SSD. [Edit: well, I thought I did; I just checked it and found I’m using an old 160GB hard drive!]. Never looked at this subject before. I followed Colin’s suggestion and saw logging enabled in Skynet, which I turned off. (I may well uninstall uiScribe and uiDivstats and see what difference it makes: I rarely look at them, anyway). Otherwise, I’ll just continue to ignore it (other than swapping for an SSD at some stage, although it’s very tempting to leave it and see just how long it’ll go on before strange things start happening).
 
Last edited:
The reason I keep a bunch of old 4GB USB drives. Came HP branded from a project many years ago, no single one failed.
I still have one or two of those old white DataTraveler in my desk drawer. The drive that I've shown above is a DataTraveler 410 if I remember correctly, which was their high performance USB 2.0 drive at the time.

I've had good results from Patriot drives. I carried one in my pocket for many years before it finally died, with all my usual IT tech tools on it. I replaced it with a pair of Patriot drives since then, I barely use them however. I also got a bunch of Sandisk which I mostly use as USB drives for my development routers (for when I need Entware debugging tools) that aren't really old yet.

Over the years I've had HP, Corsair and PNY drives that all died on me within a year of very light use. Haven't really tried Kingston's newer drives yet.
 
What makes you suspect it's swap activity that's causing the writes? If you're using third party scripts it could be the constant writing to log/database files.

I am thinking that perhaps a separate low overhead partition without journaling on the same SSD could be created just to hold the swap file (ext2???). I am not seeing how journaling the swap disk helps anything.
Just to add... Since Linux kernel 2.6 writing to a swap file bypasses the normal filesystem code (including journaling) and writes directly to the underlying disk blocks. So for solid state storage (or a HDD using contiguous storage) there is no difference between a swap file and a swap partition.
 
tune2fs tracks only the lifetime bits from the last time the fs was created...

That being said - you seem to have a lot of write action happening there - see below, which is my "edge" box running apache2 and openssh facing out to the internet - and it's also running an REST/HTTPS endpoint for some telemetry endpoints which adds about 500KB/daily...

Built the fs back last March (2023), so in the past 6 months, we have about 400GB written to the disk...

Code:
tune2fs 1.46.5 (30-Dec-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          f0026714-2db3-4877-8cd0-27843e7038fc
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              15237120
Block count:              60918272
Reserved block count:     3045913
Overhead clusters:        1234009
Free blocks:              50073289
Free inodes:              14929658
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Thu Mar  9 16:45:20 2023
Last mount time:          Tue Sep 19 11:08:45 2023
Last write time:          Tue Sep 19 11:08:44 2023
Mount count:              80
Maximum mount count:      -1
Last checked:             Thu Mar  9 16:45:20 2023
Check interval:           0 (<none>)
Lifetime writes:          404 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       14813267
Default directory hash:   half_md4
Directory Hash Seed:      147de059-1849-4245-959e-bf24f3c9feec
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xe3389b29
 
I still have one or two of those old white DataTraveler in my desk drawer. The drive that I've shown above is a DataTraveler 410 if I remember correctly, which was their high performance USB 2.0 drive at the time.

I've had good results from Patriot drives. I carried one in my pocket for many years before it finally died, with all my usual IT tech tools on it. I replaced it with a pair of Patriot drives since then, I barely use them however. I also got a bunch of Sandisk which I mostly use as USB drives for my development routers (for when I need Entware debugging tools) that aren't really old yet.

Over the years I've had HP, Corsair and PNY drives that all died on me within a year of very light use. Haven't really tried Kingston's newer drives yet.

The reason I keep a bunch of old 4GB USB drives. Came HP branded from a project many years ago, no single one failed.

If you really want to do USB drives - head over to https :// www . usbdev . ru
They have a couple of utils, ChipGenius and Flash Drive Information Extractor:

Some antivirals complain, but this site has been around for years. These utils may tell you the controller and memory chip type of your USB drive - except for Sandisk that does not work like that. Some really old pre-2010 drives have MLC chips in them (2 bits) vs post-2010 TLC (3 bits). MLC chips are way more robust, even with older controllers. The USB drive brand is usually quite uninformative. I have a couple of Kingston Traveler G2s, 16GB, same outside, one is MLC, another one is TLC.

Don’t worry: mine works out at a similar 2GB per day over 3.5 years (on an RT-AC68U). I run Diversion, Skynet and Unbound Manager, uiScribe and uiDivstats. I use an SSD. [Edit: well, I thought I did; I just checked it and found I’m using an old 160GB hard drive!]. Never looked at this subject before. I followed Colin’s suggestion and saw logging enabled in Skynet, which I turned off. (I may well uninstall uiScribe and uiDivstats and see what difference it makes: I rarely look at them, anyway). Otherwise, I’ll just continue to ignore it (other than swapping for an SSD at some stage, although it’s very tempting to leave it and see just how long it’ll go on before strange things start happening).

Skynet logging is something that writes to the general log I think. I turned off Skynet logging, and it totally messed up my Scribe. I had to re-install Scribe again so it'd be after Skynet with logging off. We probably have the same setup with scripts that create that same 2GB per day writes.

Btw, it's super easy to replace one drive with another in ~ 10 minutes if you are handy. The way EntWare works is that system mounts a partition first, and then Entware looks for /entware directory in a partition, and if it finds it, it launches all the scripts with that specific partition. If it does not find that, it won't launch anything. So here is the workflow that's fast and easy:
  • take a fresh SSD, partition it anyway you want on a Linux box (Ubuntu!) - Gparted. I like this because there I have all the easy options in the world. Don't format yet.
  • plug this SSD into a Merlin router, and format the partition you need as ext4, Ubuntu ext4 format may introduce incompatibility
    Code:
    # mke2fs -t ext4 /dev/sda1
    # tune2fs -L "yourDiskLabel" /dev/sda1
  • shut down your production Merlin router, take Merlin drive and the fresh SSD back to a linux computer
  • Copy old disk to new, for example, run tar cf - OLD_DISK_ENTWARE | ( cd NEW_DISK_ENTWARE;tar xf -) [can exclude swap file here, or even copy it!]. I like tar to keep all attributes/timestamps as they were.
  • check that your partition of choice on the new SSD looks exactly like the old one, and has /entware in it and that's it
  • plug the new SSD instead of the old disk, boot up your router
  • go to amtm and check/recreate the swap file

So basically router went to sleep with the old drive, and woke up with the new that looks exactly the same.
 
To talk more about flash drives, I once went through my collection to find MLC ones. There was exactly 1 in the sea of TLC! Here is the MLC one, and its later TLC version. Notice how Kingston totally swapped the hardware, but kept a bunch of IDs the same, and of course, the casing remained the same rounded white shape with blue stripes.

DataTraveler G3 MLC
Code:
Flash Drive Information Extractor
Volume: F:
Controller: Phison 2251-50/30 (2250)
Possible Memory Chip(s):
  Toshiba TH58NVG7D2FTAK0
  Toshiba TH58NVG7D2FTA20
Memory Type: MLC
Flash ID: 98DE9532 7A55
Chip F/W: 02.06.10
Firmware Date: 2009-12-01
MP Ver.: MPALL v3.13.00
VID: 0930
PID: 6545
Manufacturer: Kingston
Product: DataTraveler G3
Query Vendor ID: Kingston
Query Product ID: DataTraveler G3
Query Product Revision: PMAP
Physical Disk Capacity: 16001269760 Bytes
Windows Disk Capacity:  15992881152 Bytes
Internal Tags: 2Q6P-S74J
File System: FAT32
Relative Offset: 4096 KB
USB Version: 2.00
Declared Power: 200 mA
ContMeas ID: 5950-04-00
Microsoft Windows 7 SP1 Build 7601
------------------------------------
http://www.antspec.com/usbflashinfo/
Program Version: 9.4.0.645

ChipGenius
Description: [F:]USB Mass Storage Device(Kingston DataTraveler G3)
Device Type:        Mass Storage Device

Protocal Version: USB 2.00
Current Speed: High Speed
Max Current: 200mA

USB Device ID: VID = 0930 PID = 6545
Serial Number: 001CC0C83B3CEAA1D427XXXX

Device Vendor: Kingston
Device Name: DataTraveler G3
Device Revision: 0100

Manufacturer: Kingston
Product Model: DataTraveler G3
Product Revision: PMAP

Controller Vendor: Phison
Controller Part-Number: PS2251-50(PS2250) - F/W 02.06.10 [2009-12-01]
Flash ID code:      98DE95327A55 - KIOXIA TH58NVG7D2FTA20 [MLC-8K]

Tools on web:       http://dl.mydigit.net/special/up/phison.html

DataTraveler G3 TLC
Code:
Flash Drive Information Extractor
Volume: F:
Controller: Phison 2251-61 (2261)
Possible Memory Chip(s):
  Toshiba TH58NVG7T2HTA20
  Toshiba TC58NVG6T2HTA00
Memory Type: TLC
Flash ID: 98DE9892 7256
Chip F/W: 05.09.53
Firmware Date: 2011-11-02
ID_BLK Ver.: 1.1.63.0
MP Ver.: MPALL v3.20.0A
VID: 0930
PID: 6545
Manufacturer: Kingston
Product: DataTraveler G3
Query Vendor ID: Kingston
Query Product ID: DataTraveler G3
Query Product Revision: PMAP
Physical Disk Capacity: 15606349824 Bytes
Windows Disk Capacity:  15597928448 Bytes
Internal Tags: 2Q6P-S74J
File System: FAT32
Relative Offset: 4032 KB
USB Version: 2.00
Declared Power: 200 mA
ContMeas ID: 5950-34-00
Microsoft Windows 7 SP1 Build 7601
------------------------------------
http://www.antspec.com/usbflashinfo/
Program Version: 9.4.0.645

ChipGenius
Description: [F:]USB Mass Storage Device(Kingston DataTraveler G3)
Device Type:        Mass Storage Device

Protocal Version: USB 2.00
Current Speed: High Speed
Max Current: 200mA

USB Device ID: VID = 0930 PID = 6545
Serial Number: 0013729B6607EC50C5BBXXXX

Device Vendor: Kingston
Device Name: DataTraveler G3
Device Revision: 0100

Manufacturer: Kingston
Product Model: DataTraveler G3
Product Revision: PMAP

Controller Vendor: Phison
Controller Part-Number: PS2251-61(PS2261) - F/W 05.09.53 [2011-11-02]
Flash ID code:      98DE98927256 - KIOXIA TC58NVG6TDKTA00 [TLC-16K]

Tools on web:       http://dl.mydigit.net/special/up/phison.html

For comparison, here is Sandisk 128GB Ultra (this one does not talk much ...)
Code:
Flash Drive Information Extractor
Volume: F:
Controller: SanDisk
Possible Memory Chip(s): Not available
VID: 0781
PID: 5581
Manufacturer:  USB
Product:  SanDisk 3.2Gen1
Query Vendor ID:  USB
Query Product ID:  SanDisk 3.2Gen1
Query Product Revision: 1.00
Physical Disk Capacity: 123060879360 Bytes
Windows Disk Capacity:  123030798336 Bytes
Internal Tags: 3Y4X-QACJ
File System: FAT32
Relative Offset: 16 KB
USB Version: 3.00 in 2.00 port
Declared Power: 224 mA
ContMeas ID: 5950-16-00
Microsoft Windows 7 SP1 Build 7601
------------------------------------
http://www.antspec.com/usbflashinfo/
Program Version: 9.4.0.645

ChipGenius
Description: [F:]USB Mass Storage Device(USB SanDisk 3.2Gen1)
Device Type:        Mass Storage Device

Protocal Version: USB 2.10 <- Hint: This device can run faster when plugged to a USB3.0 port
Current Speed: High Speed
Max Current: 224mA

USB Device ID: VID = 0781 PID = 5581
Serial Number: 0901D6D82AE787EAC88DCF22AF244B2E75D76DC595E3719731A180427F94AB8D310F0000000000000000000052E50FF3FF15602081558107A5AA82D3

Device Vendor: USB
Device Name: SanDisk 3.2Gen1
Device Revision: 0100

Manufacturer: USB
Product Model: SanDisk 3.2Gen1
Product Revision: 1.00

Controller Part-Number: Unknown
 
What makes you suspect it's swap activity that's causing the writes? If you're using third party scripts it could be the constant writing to log/database files.
Just to add... Since Linux kernel 2.6 writing to a swap file bypasses the normal filesystem code (including journaling) and writes directly to the underlying disk blocks. So for solid state storage (or a HDD using contiguous storage) there is no difference between a swap file and a swap partition.

So you got me thinking about this. I have a cron script that's outputting a few things every 15 min, so I added the print out of the disk writes for the session.

a line in /jffs/scripts/post-mount
Code:
cru a temp "*/15 * * * * /jffs/scripts/temp_check  2>&1 | tr -d '\n' | /usr/bin/logger -t Temp"

/jffs/scripts/temp_check
Code:
# Wireless Module Temperatures with calculations performed (eth6: 2.4GHz, eth7: 5GHz)
wl -i eth5 phy_tempsense | awk '{printf "%4.1f \n", $1 / 2 + 20}'
wl -i eth6 phy_tempsense | awk '{printf "%4.1f \n", $1 / 2 + 20}'

free |grep Mem |awk '{print "; Mem free: " $4 " KB"}'
free |grep Swap |awk '{print "; Swap used: " $3 " KB"}'
echo "; sda1 wrote: $(echo "scale=2;$(cat /sys/fs/ext4/sda1/session_write_kbytes)/1024" | bc) MB"

Relevant lines in the log file that show that I get about 1MB worth of writing every minute:
Code:
admin@RT-AC86U-9988:/tmp/mnt/ac86u/entware/var/log# grep Temp messages|tail -10
Sep 25 15:45:00 RT-AC86U-9988 Temp: 54.2 41.0 49.5 ; Mem free: 27608 KB; Swap used: 32336 KB; sda1 wrote: 2274.80 MB
Sep 25 16:00:00 RT-AC86U-9988 Temp: 54.2 41.0 50.0 ; Mem free: 26876 KB; Swap used: 32336 KB; sda1 wrote: 2289.91 MB
Sep 25 16:15:01 RT-AC86U-9988 Temp: 53.7 41.0 49.5 ; Mem free: 29712 KB; Swap used: 32440 KB; sda1 wrote: 2307.98 MB
Sep 25 16:30:00 RT-AC86U-9988 Temp: 54.2 41.0 49.5 ; Mem free: 27636 KB; Swap used: 32436 KB; sda1 wrote: 2321.85 MB
Sep 25 16:45:00 RT-AC86U-9988 Temp: 53.7 41.0 49.5 ; Mem free: 38800 KB; Swap used: 32612 KB; sda1 wrote: 2335.68 MB
Sep 25 17:00:00 RT-AC86U-9988 Temp: 54.2 41.0 50.0 ; Mem free: 32680 KB; Swap used: 32612 KB; sda1 wrote: 2351.95 MB
Sep 25 17:15:00 RT-AC86U-9988 Temp: 53.7 40.5 49.5 ; Mem free: 30228 KB; Swap used: 32608 KB; sda1 wrote: 2371.10 MB
Sep 25 17:30:00 RT-AC86U-9988 Temp: 53.2 40.5 49.5 ; Mem free: 30508 KB; Swap used: 32888 KB; sda1 wrote: 2386.29 MB
Sep 25 17:45:00 RT-AC86U-9988 Temp: 53.7 40.5 49.5 ; Mem free: 29208 KB; Swap used: 32888 KB; sda1 wrote: 2398.90 MB
Sep 25 18:00:00 RT-AC86U-9988 Temp: 52.3 40.5 49.0 ; Mem free: 28908 KB; Swap used: 32872 KB; sda1 wrote: 2415.25 MB

Here is my amtm screen:
Code:
 amtm 3.8 FW               by thelonelycoder
 RT-AC86U (aarch64) FW-386.12 @ 192.168.1.1
 Operation Mode: Wireless router
    The Asuswrt-Merlin Terminal Menu

 1  open     Diversion                 v4.3.3
 2  open     Skynet                    v7.4.6

 5  open     scribe                    v3.2.0

 7  open     unbound Manager            v3.23

 j1 open     connmon                   v3.0.2
 j2 open     ntpMerlin                 v3.4.5
 j3 open     scMerlin                  v2.4.0

 j5 open     uiDivStats                v3.0.2
 j6 open     uiScribe                  v1.4.5

 vn open     vnStat                    v2.0.4
 rt open     RTRMON                     v1.57

 ep manage   Entware packages

 em setup    email settings
 fd run      Format disk         fdl show log

 sh manage   shell history
 rd manage   Router date keeper
 sw manage   Swap file /mnt/ac86u 2.0G

 i  show     all available scripts or tools
 u  check    for script updates

    amtm options
 e  exit      t  theme   r  reset   a  about
_____________________________________________

 Enter option

Which script is doing all the writing? I tried to run iotop, but something is not configured in the kernel:
Code:
admin@RT-AC86U-9988:/# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
  - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
admin@RT-AC86U-9988:/# uname -a
Linux RT-AC86U-9988 4.1.27 #2 SMP PREEMPT Mon Sep 4 12:17:28 EDT 2023 aarch64 ASUSWRT-Merlin
 
So you got me thinking about this. I have a cron script that's outputting a few things every 15 min, so I added the print out of the disk writes for the session.

a line in /jffs/scripts/post-mount
Code:
cru a temp "*/15 * * * * /jffs/scripts/temp_check  2>&1 | tr -d '\n' | /usr/bin/logger -t Temp"

/jffs/scripts/temp_check
Code:
# Wireless Module Temperatures with calculations performed (eth6: 2.4GHz, eth7: 5GHz)
wl -i eth5 phy_tempsense | awk '{printf "%4.1f \n", $1 / 2 + 20}'
wl -i eth6 phy_tempsense | awk '{printf "%4.1f \n", $1 / 2 + 20}'

free |grep Mem |awk '{print "; Mem free: " $4 " KB"}'
free |grep Swap |awk '{print "; Swap used: " $3 " KB"}'
echo "; sda1 wrote: $(echo "scale=2;$(cat /sys/fs/ext4/sda1/session_write_kbytes)/1024" | bc) MB"

Relevant lines in the log file that show that I get about 1MB worth of writing every minute:
Code:
admin@RT-AC86U-9988:/tmp/mnt/ac86u/entware/var/log# grep Temp messages|tail -10
Sep 25 15:45:00 RT-AC86U-9988 Temp: 54.2 41.0 49.5 ; Mem free: 27608 KB; Swap used: 32336 KB; sda1 wrote: 2274.80 MB
Sep 25 16:00:00 RT-AC86U-9988 Temp: 54.2 41.0 50.0 ; Mem free: 26876 KB; Swap used: 32336 KB; sda1 wrote: 2289.91 MB
Sep 25 16:15:01 RT-AC86U-9988 Temp: 53.7 41.0 49.5 ; Mem free: 29712 KB; Swap used: 32440 KB; sda1 wrote: 2307.98 MB
Sep 25 16:30:00 RT-AC86U-9988 Temp: 54.2 41.0 49.5 ; Mem free: 27636 KB; Swap used: 32436 KB; sda1 wrote: 2321.85 MB
Sep 25 16:45:00 RT-AC86U-9988 Temp: 53.7 41.0 49.5 ; Mem free: 38800 KB; Swap used: 32612 KB; sda1 wrote: 2335.68 MB
Sep 25 17:00:00 RT-AC86U-9988 Temp: 54.2 41.0 50.0 ; Mem free: 32680 KB; Swap used: 32612 KB; sda1 wrote: 2351.95 MB
Sep 25 17:15:00 RT-AC86U-9988 Temp: 53.7 40.5 49.5 ; Mem free: 30228 KB; Swap used: 32608 KB; sda1 wrote: 2371.10 MB
Sep 25 17:30:00 RT-AC86U-9988 Temp: 53.2 40.5 49.5 ; Mem free: 30508 KB; Swap used: 32888 KB; sda1 wrote: 2386.29 MB
Sep 25 17:45:00 RT-AC86U-9988 Temp: 53.7 40.5 49.5 ; Mem free: 29208 KB; Swap used: 32888 KB; sda1 wrote: 2398.90 MB
Sep 25 18:00:00 RT-AC86U-9988 Temp: 52.3 40.5 49.0 ; Mem free: 28908 KB; Swap used: 32872 KB; sda1 wrote: 2415.25 MB

Here is my amtm screen:
Code:
 amtm 3.8 FW               by thelonelycoder
 RT-AC86U (aarch64) FW-386.12 @ 192.168.1.1
 Operation Mode: Wireless router
    The Asuswrt-Merlin Terminal Menu

 1  open     Diversion                 v4.3.3
 2  open     Skynet                    v7.4.6

 5  open     scribe                    v3.2.0

 7  open     unbound Manager            v3.23

 j1 open     connmon                   v3.0.2
 j2 open     ntpMerlin                 v3.4.5
 j3 open     scMerlin                  v2.4.0

 j5 open     uiDivStats                v3.0.2
 j6 open     uiScribe                  v1.4.5

 vn open     vnStat                    v2.0.4
 rt open     RTRMON                     v1.57

 ep manage   Entware packages

 em setup    email settings
 fd run      Format disk         fdl show log

 sh manage   shell history
 rd manage   Router date keeper
 sw manage   Swap file /mnt/ac86u 2.0G

 i  show     all available scripts or tools
 u  check    for script updates

    amtm options
 e  exit      t  theme   r  reset   a  about
_____________________________________________

 Enter option

Which script is doing all the writing? I tried to run iotop, but something is not configured in the kernel:
Code:
admin@RT-AC86U-9988:/# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
  - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
admin@RT-AC86U-9988:/# uname -a
Linux RT-AC86U-9988 4.1.27 #2 SMP PREEMPT Mon Sep 4 12:17:28 EDT 2023 aarch64 ASUSWRT-Merlin
Unbound can be pretty noisy if you have logging turned on?
 
As I was slightly re-partitioning my external AMTM drive (a very low end dram-less 240GB SSD hanging on a USB2/SATA ~10 inch adapter cable), I noticed something really interesting. In 20 months this partition with Entware and a 2 GB swap file had 1152 GB of disk writes. So this is about 2GB of writes per 24 hours on average. With respect to 240GB SSD size, this is ~5x write cycles, which is very trivial given than an SSD should survive multiple hundreds depending on the flash type (TLC, QLC, etc). But, had this been a ~32GB flash cheapo flash drive, directly attached to the router thus getting all the nice heat transfer from the motherboard, it'd probably be really dead by now (1152/32=36 write cycles with possibly poor write balancing due to very basic flash drive controller, and heat, heat, and more heat).

I am thinking that perhaps a separate low overhead partition without journaling on the same SSD could be created just to hold the swap file (ext2???). I am not seeing how journaling the swap disk helps anything. A dedicated swap partition is poorly supported by AMTM, so that's not a solution either.

Anyway, if anybody wants to share their partition statistics, it could become a nice fact based conversation.


Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 15282156 4305852 10178360 30% /tmp/mnt/ac86u


Code:
admin@RT-AC86U-9988:/# tune2fs -l /dev/sda1
tune2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   ac86u
Last mounted on:          /tmp/mnt/ac86u
Filesystem UUID:          ...
Filesystem magic number:  ...
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              977280
Block count:              3907803
Reserved block count:     195390
Overhead blocks:          87264
Free blocks:              2749312
Free inodes:              970542
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      954
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Flex block group size:    16
Filesystem created:       Wed Feb  2 12:02:01 2022
Last mount time:          Fri Sep 22 16:52:48 2023
Last write time:          Fri Sep 22 16:52:48 2023
Mount count:              129
Maximum mount count:      -1
Last checked:             Wed Feb  2 12:02:01 2022
Check interval:           0 (<none>)
Lifetime writes:          1152 GB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ...
Journal backup:           inode blocks

It makes no deference how you partition the SSD as SSD Wear Leveling will take place in the background.

 
It makes no deference how you partition the SSD as SSD Wear Leveling will take place in the background.


I am very aware of this. If you notice, in my original thread I divided the written amount by the full disk size assuming perfect wear leveling.

Unbound can be pretty noisy if you have logging turned on?
Turned off unbound logging explicitly - same write rate. Will try to chase some more on the week-end.
 
What makes you suspect it's swap activity that's causing the writes? If you're using third party scripts it could be the constant writing to log/database files.

I think I got close enough to the bottom of the issue. Tried a few different commands, some don't work (iotop, fatrace). But some clues are below.
Code:
admin@RT-AC86U-9988:/tmp/mnt/ac86u# opkg install fatrace
...
admin@RT-AC86U-9988:/tmp/mnt/ac86u# fatrace
Cannot initialize fanotify: Function not implemented
admin@RT-AC86U-9988:/tmp/mnt/ac86u# iotop
Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
  - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
admin@RT-AC86U-9988:/tmp/mnt/ac86u#

Managed to get dstat (opkg install dstat), and after seeing disk activity, immediately searched for recently changed files. Seems like ./entware/share/uiDivStats.d/csv/SQLQueryLog.htm file gets replaced in its entirety a few times per minute (300k worth of file size). vnstat and dnsmasq are also logging frequently. connmon saves often too. Plus I guess there is journaling on top of it - so more writes. This write rate is OK for HDDs/SSDs, but probably very bad for cheap low quality USB drives.

Code:
admin@RT-AC86U-9988:/tmp/mnt/ac86u# dstat -tdD total,sda1
----system---- -dsk/total----dsk/sda1-
     time     | read  writ: read  writ
28-09 17:37:00|  13k   17k:  13k   17k
28-09 17:37:01|   0   296k:   0   296k
28-09 17:37:02|   0     0 :   0     0 ^C

admin@RT-AC86U-9988:/tmp/mnt/ac86u# find . -type f -mmin -0.1 -exec ls -lta {} \;
-rw-r--r--    1 admin    root         32768 Sep 28 17:37 ./entware/var/lib/vnstat/vnstat.db-shm
-rw-r--r--    1 admin    root       3168312 Sep 28 17:37 ./entware/var/lib/vnstat/vnstat.db-wal
-rw-rw-rw-    1 admin    root        300165 Sep 28 17:37 ./entware/share/uiDivStats.d/csv/SQLQueryLog.htm
admin@RT-AC86U-9988:/tmp/mnt/ac86u# find . -type f -mmin -2 -exec ls -lta {} \;
-rw-r--r--    1 admin    root         32768 Sep 28 17:39 ./entware/var/lib/vnstat/vnstat.db-shm
-rw-r--r--    1 admin    root       3225992 Sep 28 17:39 ./entware/var/lib/vnstat/vnstat.db-wal
-rw-rw----    1 nobody   root       2680634 Sep 28 17:39 ./entware/var/log/dnsmasq.log
-rw-rw-rw-    1 admin    root        300273 Sep 28 17:39 ./entware/share/uiDivStats.d/csv/SQLQueryLog.htm
-rw-r--r--    1 admin    root     165945344 Sep 28 17:39 ./entware/share/uiDivStats.d/dnsqueries.db
admin@RT-AC86U-9988:/tmp/mnt/ac86u# find . -type f -mmin -5 -exec ls -lta {} \;
-rw-r--r--    1 admin    root         32768 Sep 28 17:39 ./entware/var/lib/vnstat/vnstat.db-shm
-rw-r--r--    1 admin    root       3225992 Sep 28 17:39 ./entware/var/lib/vnstat/vnstat.db-wal
-rw-rw----    1 nobody   root       2680634 Sep 28 17:39 ./entware/var/log/dnsmasq.log
-rw-rw-rw-    1 admin    root        300273 Sep 28 17:39 ./entware/share/uiDivStats.d/csv/SQLQueryLog.htm
-rw-r--r--    1 admin    root     165945344 Sep 28 17:39 ./entware/share/uiDivStats.d/dnsqueries.db
-rw-r--r--    1 admin    root        118784 Sep 28 17:36 ./entware/share/connmon.d/connstats.db
-rw-rw-rw-    1 admin    root          2702 Sep 28 17:36 ./entware/share/connmon.d/csv/Ping_raw_daily.htm
...
-rw-rw-rw-    1 admin    root           743 Sep 28 17:36 ./entware/share/connmon.d/csv/LineQuality_hour_daily.htm
-rw-rw-rw-    1 admin    root         18433 Sep 28 17:36 ./entware/share/connmon.d/csv/Ping_raw_weekly.htm
-rw-rw-rw-    1 admin    root           136 Sep 28 17:36 ./entware/share/connmon.d/connstatstext.js
-rw-rw-rw-    1 admin    root           411 Sep 28 17:36 ./entware/share/connmon.d/lastx.csv
-rw-r--r--    1 admin    root          1006 Sep 28 17:36 ./entware/share/connmon.d/config
admin@RT-AC86U-9988:/tmp/mnt/ac86u#
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top