RT-AC3200 (.9135 firmware) kernel panic

[Puppa]

I've been seeing random reboots on my RT-AC3200 since flashing the latest firmware.
In the log, I see kernel panics that indicate something is wrong with the JFFS2 filesystem.
I haven't done anything special to this device, no Merlin firmware, factory resets every firmware update, no beta firmware, etc.

Can I somehow manually reformat JFFS so this stops? Are there other filesystems or settings lurking that don't get fixed by a factory reset?

 

[RMerlin]

Could be the router running out of RAM due to a known issue with 9135. Try downgrading to the previous firmware.

 

[Puppa]

It's definitely not running out of RAM, based on what the kernel reports. That it happened nearly immediately after a soft-reboot also tends to rule out the memory issue (you're talking about the DNS debugging spew, right?). Highest utilization I've ever seen on this device is 34%. I have a ton of features turned off.

No SPI, no USB storage, no IPv6, ad nauseum.
Here are the lines that make me think I have a JFFS2 problem from the syslog:
Jul 31 20:00:13 kernel: JFFS2 notice: (372) check_node_data: wrong data CRC in data node at 0x015ef000: read 0x4889d8cf, calculated 0xfd0f300c.
Jul 31 20:00:13 kernel: JFFS2 notice: (372) check_node_data: wrong data CRC in data node at 0x015ed9b0: read 0x2441f870, calculated 0xf07fcda0.

(The odd timestamp above is because I did a reboot just prior to the panic, and it hadn't synced the time yet.)

You mentioned somewhere that ASUS is now storing some system parameters in the JFFS2 partition. (Previously I think only TA used this in vanilla ASUSWRT). I suspect a firmware update does not erase this partition if errors were present previously. Is this correct?

If I were to ssh into the router and reinitialize the JFFS2 partition (assuming I have the privileges), then do a factory reset, am I in danger of bricking my device? Is there some feature in particular that uses JFFS2 that I should turn off first?

I did notice that a factory reset doesn't seem to reset everything on the device. For example, my DHCP reservations remained behind after a reset with some earlier firmware. I haven't hit any problems with this thanks to my near-default setup, but I'd expect somebody who changed their subnet addresses could get burned by this unless there are some safeguards in place such as checking that the table's addresses match the subnet well below the UI-level logic.

Just FYI, the same thing happened on my Netgear R7000, so this seems to be a common theme probably coming from the similar backgrounds of the firmware. This surprised me, though, because I expected a factory reset to reset everything, not just part of everything. I've been poring over the various subforums and haven't found this discussed, though I might have missed.

 

[RMerlin]

You mentioned somewhere that ASUS is now storing some system parameters in the JFFS2 partition. (Previously I think only TA used this in vanilla ASUSWRT). I suspect a firmware update does not erase this partition if errors were present previously. Is this correct?

That's correct. That partition does not get erased automatically. Asus will erase it if you do a factory default reset if I remember correctly (but not 100% sure).

In addition to TA data, Asus now stores OpenVPN key/certs, and custom networkmap icons in the JFFS partition. There's also some (light) Trend Micro debug logs there.

If I were to ssh into the router and reinitialize the JFFS2 partition (assuming I have the privileges), then do a factory reset, am I in danger of bricking my device? Is there some feature in particular that uses JFFS2 that I should turn off first?

It's fine. You can't brick this router unless you touch its bootloader.

I did notice that a factory reset doesn't seem to reset everything on the device. For example, my DHCP reservations remained behind after a reset with some earlier firmware.

It should erase everything, unless there was a bug in the firmware version you were running at the time.

 

[Puppa]

I put your latest Merlin firmware on it to avoid the DNS debugging memory issue and factory reset and sure enough, it cleaned up my old reservations and port forward settings as you said. Loving it so far. Noticed another potential memory leak, though: if I reconfigure on the Advanced wireless page and hit Apply, it seems to lose a big chunk of memory every time the wireless restarts. Of course, I don't expect wireless to restart on its own, and a reboot clears it up, but thought I'd let you know about it in case you happen to see something related.

Sent from my SM-N910V using Tapatalk