USB died and new USB dying again! Moving to SSD/Flash!

[aex.perez]

This is what was happening and happened before the USB stick up and died,
Jun 24 00:44:33 Router kernel: jffs2: warning: (1494) jffs2_sum_write_data: Summary too big (-32 data, -1525 pad) in eraseblock at 00380000
Jun 24 08:55:10 Router kernel: jffs2: warning: (1376) jffs2_sum_write_data: Summary too big (-32 data, -697 pad) in eraseblock at 00320000
Jun 24 09:18:30 Router kernel: jffs2: warning: (28707) jffs2_sum_write_data: Summary too big (-32 data, -697 pad) in eraseblock at 00220000
Jun 24 10:15:18 Router kernel: jffs2: warning: (1375) jffs2_sum_write_data: Summary too big (-32 data, -679 pad) in eraseblock at 00100000

Since this was happening for a while and I ignored because it wasn't impacting anything, until it was as it died and impacted everything, and my backups were corrupted (GIGO) and I've been tying to fix things as I run into them

This morning, after a week, again! And so it begins!
Jun 29 12:20:22 Router kernel: jffs2: warning: (1358) jffs2_sum_write_data: Summary too big (-32 data, -679 pad) in eraseblock at 00700000

At the EXACT same time that WiFi started going nuts and randomly changing channels and bandwidth on the 5Ghz (trypically, I'm VERY stable with 60/160)

Hooray!
The SSD Flash enclosure arrives this afternoon, so since I recovered from the 24th (mostly) have plenty of practice recovering and redoing anything that got broken, NEVER wanting to go through this mess again!

Things Like:
AMTM locking up on start
NTP not starting/synching
Not being able to start any script without a warning about different files being locked and preventing another instance of the selected script from starting
Then scribe and filters headaches (see this post and acroll up a few) still can't get the nodes log updates to show up on the router syslog again, let alone in a filtered log.
WiFi settings / would default to Auto/Auto and couldn't change it, had to trick it to get back to:

Various other scripts and their settings to many to remember
Settings on the Router, oh the hours I spent trying to get asusrouter.com to work again, before realizing what I did to break it and then the ease with which to make it work again! (Hint: don't define the Internet DNS server under the LAN/ DHCP server LAN DNS and WINS Server Setting, but under WAN DNS Setting under the WAN - Internet Connection Tab instead, this fixed other problems as is noticeably faster on Internet DNS lookups)
Getting the VPNs set back up, Wiregard / Opensense, IPSEC for the wife's Win11 laptop (native and easy) is still pending
AiMesh bindings stored in nvram/sta_binding_list got corrupted hence not being able to bind/unbind devices and devices bound incorrectly
and other annomalies that I've been chasing and resolved until this jffs error this after noon sent WiFi (acsd, and a flood of CFG80211-ERROR and other related stuff) and kept causing router to restart connectivity (not reboot) every few seconds
Setting the Router USB to USB2 even if had a USB3 stick, that caused other anomalies

So needless to say, going to be busy resetting everything AGAIN and making sure it all works and hopefully close this chapter once I get everything moved over, then rethink and implement a new backup retention schedule to use with BACKUPMON.
Had gotten complacent from being so stable for so long, and getting lazy because between BACKUPMON (@Viktor Jaep) and MerlinAU (@ExtremeFiretop), and the work by (@Martinski, @thelonelycoder) on the scripts my maintenace has been near zero.
As for the others (@Tech9 amongst MANY others, and you know who you all are) recommending to do this, get off USB and get onto SSD Flash, I know - I know, lesson learned!

Hopefully others after seeing this and the pain and anomalies it caused will make the switch as well.
(Left a LOT of trouble shooting detail out, for brevity, as it has taken 6 days to get here, and still few more to go, now )

 

[Viktor Jaep]

Hopefully others after seeing this and the pain and anomalies it caused will make the switch as well.
(Left a LOT of trouble shooting detail out, for brevity, as it has taken 6 days to get here, and still few more to go, now )

Enjoy that new SSD!

 

[ColinTaylor]

TL;DR but the jffs2 error messages are not referencing USB drives. They refer to... well, jffs.

 

[aex.perez]

Enjoy that new SSD!

SSD I have 512MB came with the wife's laptop, until I replaced it with a Samsung 990 2TB

 

[aex.perez]

TL;DR but the jffs2 error messages are not referencing USB drives. They refer to... well, jffs.

Then maybe the router is dying after all, just going with that first as all the issues have been correlated.

 

[Tech9]

get off USB and get onto SSD Flash

But your error messages are not about USB as mentioned above...

 

[Viktor Jaep]

Then maybe the router is dying after all, just going with that first as all the issues have been correlated.

You may want to install @JGrana's MTD Checker tool and see if it's detecting some bad blocks... would be indicative of a router going bad...

Beta - Asuswrt-Merlin 388.1 Beta is available for select models

Thats quite a few bad blocks. The nand device on most any router will have some bad blocks. But, as long as the mtd driver is doing it's job, it would mark them as bad and unusable. You will have less available storage on the partition(s) that those blocks are on - but as you said - everything...

www.snbforums.com

 

[bbunge]

I use a 16 GB USB2 San Disk..it just keeps rolling along...

 

[ducky124]

I had a SanDisk USB stick in my setup that’s been rock-solid for countless years—basically a tech veteran. I even migrated it from my AC86U to my AX88U Pro last year, and it didn’t throw a single tantrum the whole time. Then, just for sh*ts and giggles, I upgraded to an SSD a few months ago. So far, it’s had not one, but two little drama moments. Nothing serious, but enough to remind me why I usually live by “If it ain’t broke, don’t fix it.” Honestly, I’m still scratching my head wondering why I decided to switch in the first place!

 

[Jeffrey Young]

“If it ain’t broke, don’t fix it.”

Amen

 

[Tech9]

“If it ain’t broke, don’t fix it.”

I really hope you don't follow the same rule for your car brakes.

 

[aex.perez]

You may want to install @JGrana's MTD Checker tool and see if it's detecting some bad blocks... would be indicative of a router going bad...

Beta - Asuswrt-Merlin 388.1 Beta is available for select models

Thats quite a few bad blocks. The nand device on most any router will have some bad blocks. But, as long as the mtd driver is doing it's job, it would mark them as bad and unusable. You will have less available storage on the partition(s) that those blocks are on - but as you said - everything...

www.snbforums.com

Thanks I was looking for this!
I'm thinking that, it's either the..

A) Router RAM and/or
B) USB (USB went R/O, new USB3 stick requied the Router change the USB to USB2 mode to work with some level of reliability) and or
C) USB stick itself ( which impacting both Router RAM and anything on the USB) and or
D) Anything I haven't cleaned up, YET!

or a combination of the above, either way making the making the move to SSD Flash to tackle B) & C)
and for A) and D). I may end up replacing the AX88u (wishing the BE88u had a 6mhz radio as I consider what to do with the AX86u nodes and the move to the 3006 family from 3004), though the logs have been clean since I've been cleaning everything up, so I may be able to put that off once I switch from USB.

Just one remaing issue I can't figure out though not earth shattering urgent, more of a convience...

UPDATE:
Router looking good, A) looks fine...

Flash type of /dev/mtd0 is 4 (NANDFLASH)
Flash flags are 0x400 - WRITEABLE
Block size 131072 (128K) Page size 2048 (2K) OOB size 64
Device: 85721088 (81M) bytes, 654 blocks
MTD Stats: Bad Blocks: 0 ECC Corrected: 0 ECC Uncorrected: 0

No Region Information on device
B Bad block; . Empty; - Partially filled; * Full; R Reserved (BBT); S has a JFFS2 summary node

-------------*******************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
****************************--**********----------------------------------------
--------------
Summary /dev/mtd0:
Empty Blocks: 0, Full Blocks: 585, Partially Full: 69, Bad Blocks: 0, Reserved Blocks (BBT) 0

Update: Tested all (/dev/mtd0 through /dev/mtd10), a few times, then for giggles and as a control, tested the AX86 nodes. Both the AX88 and 1 node it through with no errors. The other node had 5 Bad Blocks, in total, across all of the mtd character devices, but again it's a node, so easier to rebuild if (or more accurately when) it goes.

Hopefully I won't have distractions tomorrow and can make the quick USB/Flash swap.

Still amazed that the 128GB Samsung FIT Plus USB dies, while the free 4GB noname USB a.k.a CatWoman (glued to the top of the drive) USB still lives

 

[aex.perez]

I really hope you don't follow the same rule for your car brakes.

Yes!!!!

To make the point more vividly . The day you lose braking, taking a tight concentric corner (hint - double apex - "Corner 10" at what was PBIR) after doing 150mph+ in the straight and braking to 30mph to make the corner, after 10 laps and your brakes rotors glowing red/orange (even with racing pads and rotors, and brake fluid). You get a new appreciation for having good brakes and have a much better understanding of the what term "hitting the wall" means.

As the licensed racecar driver, sitting in my passenger seat, all of a sudden realizes and appreciates that even having a crazy fast Cadillac (around a road track and 9 sec 1/4 mile strip), with a manual transmission (as the backup to the brakes) as you quickly downshift to slow your car 4000lb down enough to allow you both to exit the track safely and drive around slowly, to cool down the brakes.

Yes, I'll agree. This is an extreme, but it can and does happen to a "normal" car in stop and go traffic under the right conditions, and remember this simple truth, "an ounce of prevention is worth more that a pound of the cure"

Long story short, hence the move from USB to SSD

 

[Tech9]

What about your corrupted backups? There is a saying "every backup is unsuccessful unless restored". You were carrying a flat spare tire. Perhaps re-thinking the whole custom scripts galore configuration may save you downtime in the future.

 

[aex.perez]

What about your corrupted backups? There is a saying "every backup is unsuccessful unless restored". You were carrying a flat spare tire.

Too much weight, but had them on standby in the trailer, already mounted, directional 285/35 r 20 in front, 345/25 r 20 on the rears, she a big booty girl but I digress

The corrupted backups, GIGO and yes that's another reason I need to examine the retention, most of it came from the USB and some from Flash on the router. But feeling much better about the Router now that I have the results of mtd_check...

 

[aex.perez]

DONE!

So many lessons learned. Need to have a secondary backup target, preferably on the Router.

Took me way to looooooong to trouble shoot the NAS target to be able to access the backups. Had one cable that worked itself loose messing with the 8023.ad bonding, turned that bonding and jumbo frames off, accessed the NAS and completed the restore. Then re-seated the Ethernet cable that worked itself loose as I connected the SSD. Redid the settings, rebooted to make sure it would come up cleaning and accessible.

Everything working!

Minor nuisances:
A) The Node IP addresses changed (likely due to the amount spent of the NAS after the WPS reset of the router), but nothing else as far as AiMesh and the Router settings, just needed to update the SSH profiles on the client.

B) And then for whatever reason, I still can't get the nodes syslog to send to, or the router to accept log messages even with log_ipaddr set to the router with scribe on the router or without. I need to search the forum a bit more and see what else I can try.

Router USB set to USB3 mode things on the router are bit more responsive between USB3/Flash vs USB2/USB, not a race but just crisper if I had to use one word to describe it.

 

[Tech9]

Took me way to looooooong to trouble shoot

We've seen this before...

 

[aex.perez]

We've seen this before...

Made some room, replaced some cables that connect firmer, with a solid clicking sound, as well once this happened to avoid it happening again!

 

[jksmurf]

Minor nuisances:
A) The Node IP addresses changed (likely due to the amount spent of the NAS after the WPS reset of the router), but nothing else as far as AiMesh and the Router settings, just needed to update the SSH profiles on the client.

I use fixed addresses for my nodes. After the Router assigns them an address at AIMesh config time and I know the MAC (you can probably get this before then), i assign it as an addresse I want, reboot the node and can use the same Putty and WinSCP profiles every time. My nodes are wired.

 

[aex.perez]

I use fixed addresses for my nodes. After the Router assigns them an address at AIMesh config time and I know the MAC (you can probably get this before then), i assign it as an addresse I want, reboot the node and can use the same Putty and WinSCP profiles every time. My nodes are wired.

Never been an issue in the past, but I haven't left them abandoned from the router for so long either. Probablly under LAN DHCP I can find them before the router adds or gets to them as once the router has them join the mesh they sort of disappear, unless I'm missing the method you're using, mine wired as well.

Or using the nvram variable asus_device_list or cfg_device_list or lan_ipaddr and to set it