[RT-AX86U] Consistent Crashes on Merlin 3.88.1 and 3.22.2

[breadlord]

Hi all,

I've been running merlin 3.86.5_2 for well over a year with no issues, but decided to add a stock AC-66U-B1 as an AiMesh node, so thought it was time to upgrade.

I am now experiencing farily consistent crashes with both versions and have no real idea what the underlying root cause is - I remember when I tried to move the 386.7 I had the same symptoms, although I don't know if it was the same root cause.

From the crashlogs it appears to be caused by a kernel panic when accessing a virtual memory location during an interrupt - but I'm not familiar enough with the hardware to be able to see exactly what is crashing.

// Previous crashlog entries - all in the attached syslog
May  5 06:05:07 crashlog: <0>Call trace:
May  5 06:05:07 crashlog: <4>[<ffffffc0000d7728>] __note_gp_changes+0x68/0xd8
May  5 06:05:07 crashlog: <4>[<ffffffc0000d7800>] note_gp_changes+0x68/0xb8
May  5 06:05:07 crashlog: <4>[<ffffffc0000d83e8>] rcu_process_callbacks+0xa0/0x538
May  5 06:05:07 crashlog: <4>[<ffffffc0000963b0>] __do_softirq+0x100/0x238
May  5 06:05:07 crashlog: <4>[<ffffffc0000967ac>] irq_exit+0x9c/0xe0
May  5 06:05:07 crashlog: <4>[<ffffffc0000cd040>] __handle_domain_irq+0x60/0xb8
May  5 06:05:07 crashlog: <4>[<ffffffc000080c18>] gic_handle_irq+0x38/0x90
May  5 06:05:07 crashlog: <4>Exception stack(0xffffffc03e8cbdc0 to 0xffffffc03e8cbef0)
May  5 06:05:07 crashlog: <4>bdc0: b4ca360c 0000022b 00000000 00000080 3e8cbf10 ffffffc0 00381594 ffffffc0
May  5 06:05:07 crashlog: <4>bde0: b4ca360c 0000022b 343d9568 0016ae7d 00010803 00000000 14000b9a 00000000
May  5 06:05:07 crashlog: <4>be00: 003f2750 00000000 00000018 00000000 ec46a29a 001699dc 3ea23d98 ffffffc0
May  5 06:05:07 crashlog: <4>be20: 3e8b2630 ffffffc0 3e8cbec0 ffffffc0 0056e548 ffffffc0 000004ce 00000000
May  5 06:05:07 crashlog: <4>be40: f73b8db4 00000000 f73b8dc8 00000000 f73b9920 00000000 00000000 00000000
May  5 06:05:07 crashlog: <4>be60: 000f5a40 ffffffc0 00000000 00000000 00000000 00000000 b4ca360c 0000022b
May  5 06:05:07 crashlog: <4>be80: 3ffc5338 ffffffc0 00000001 00000000 00000001 00000000 b3b7af4c 0000022b
May  5 06:05:07 crashlog: <4>bea0: 3e8c8000 ffffffc0 0091d000 ffffffc0 0074a000 ffffffc0 3ffc5338 ffffffc0
May  5 06:05:07 crashlog: <4>bec0: 0078d580 ffffffc0 3e8cbf10 ffffffc0 0038158c ffffffc0 3e8cbf10 ffffffc0
May  5 06:05:07 crashlog: <4>bee0: 00381594 ffffffc0 60000145 00000000
May  5 06:05:07 crashlog: <4>[<ffffffc000083f00>] el1_irq+0x80/0xf8
May  5 06:05:07 crashlog: <4>[<ffffffc000381698>] cpuidle_enter+0x18/0x20
May  5 06:05:07 crashlog: <4>[<ffffffc0000c5e5c>] cpu_startup_entry+0x1ec/0x250
May  5 06:05:07 crashlog: <4>[<ffffffc00008d190>] secondary_start_kernel+0x150/0x178
May  5 06:05:07 crashlog: <0>Code: f0003381 91320021 3900627f d538d083 (f8616861)
May  5 06:05:07 crashlog: <4>---[ end trace b7cdea4242637bf9 ]---
May  5 06:05:07 crashlog: <0>Kernel panic - not syncing: Fatal exception in interrupt
May  5 06:05:07 crashlog: <2>CPU2: stopping
May  5 06:05:07 crashlog: <4>CPU: 2 PID: 0 Comm: swapper/2 Tainted: P      D    O    4.1.52 #2
May  5 06:05:07 crashlog: <4>Hardware name: Broadcom-v8A (DT)
May  5 06:05:07 crashlog: <0>Call trace:
May  5 06:05:07 crashlog: <4>[<ffffffc000087398>] dump_backtrace+0x0/0x150
May  5 06:05:07 crashlog: <4>[<ffffffc0000874fc>] show_stack+0x14/0x20
May  5 06:05:07 crashlog: <4>[<ffffffc00055bcf0>] dump_stack+0x90/0xb0
May  5 06:05:07 crashlog: <4>[<ffffffc00008d710>] handle_IPI+0x190/0x1a0
May  5 06:05:07 crashlog: <4>[<ffffffc000080c68>] gic_handle_irq+0x88/0x90
May  5 06:05:07 crashlog: <4>Exception stack(0xffffffc03e8cfdc0 to 0xffffffc03e8cfef0)
May  5 06:05:07 crashlog: <4>fdc0: fdd7d820 0000022b 00000000 00000080 3e8cff10 ffffffc0 00381594 ffffffc0
May  5 06:05:07 crashlog: <4>fde0: fdd7d820 0000022b 48bc4114 005fbc1f 0378cd6c 00000000 14000b9a 00000000
May  5 06:05:07 crashlog: <4>fe00: 003f2754 00000000 00000018 00000000 913d0a1c 001a4bea 15388d74 0000022c
May  5 06:05:07 crashlog: <4>fe20: 3e8c3a70 ffffffc0 3e8cfec0 ffffffc0 00000000 00000000 f6e5df9c 00000000
May  5 06:05:07 crashlog: <4>fe40: f6e5de04 00000000 f6e5de18 00000000 f6e5e920 00000000 00000000 00000000
May  5 06:05:07 crashlog: <4>fe60: 000f5a40 ffffffc0 00000000 00000000 00000000 00000000 fdd7d820 0000022b
May  5 06:05:07 crashlog: <4>fe80: 3ffd4338 ffffffc0 00000001 00000000 00000001 00000000 fdce49db 0000022b
May  5 06:05:07 crashlog: <4>fea0: 3e8cc000 ffffffc0 0091d000 ffffffc0 0074a000 ffffffc0 3ffd4338 ffffffc0
May  5 06:05:07 crashlog: <4>fec0: 0078d580 ffffffc0 3e8cff10 ffffffc0 0038158c ffffffc0 3e8cff10 ffffffc0
May  5 06:05:07 crashlog: <4>fee0: 00381594 ffffffc0 60000145 00000000
May  5 06:05:07 crashlog: <4>[<ffffffc000083f00>] el1_irq+0x80/0xf8
May  5 06:05:07 crashlog: <4>[<ffffffc000381698>] cpuidle_enter+0x18/0x20
May  5 06:05:07 crashlog: <4>[<ffffffc0000c5e5c>] cpu_startup_entry+0x1ec/0x250
May  5 06:05:07 crashlog: <4>[<ffffffc00008d190>] secondary_start_kernel+0x150/0x178
May  5 06:05:07 crashlog: <4>SMP: failed to stop secondary CPUs
May  5 06:05:07 crashlog:
May  5 06:05:07 crashlog:
May  5 06:05:07 crashlog:
May  5 06:05:07 crashlog:
May  5 06:05:07 crashlog:

I've attached the full log from the router, and the call trace does seem to be consistently backtracking to the call gic_handle_irq+0x88/0x90 - but I don't know what that handler is for.

Any help gratefully accepted - My security cameras are going up and down and when they come back up they're not always the best about picking up their DNS entries, which isn't ideal.

 

[Tech9]

Load stock Asuswrt 388_22525, hard reset the router to factory defaults (WPS method), do minimum configuration only (user/pass, SSID/pass) to test and observe the behavior. Don't connect your node yet. If stable and no crashlogs, load Asuswrt-Merlin 388.2 and test again. Asuswrt-Merlin 388.2 is based on Asuswrt 388_22525. If still no crashlogs - your configuration had some issues. If it crashes all the time - contact Asus for RMA.

I have the same RT-AX86U and both Asuswrt 388_22525 and Asuswrt-Merlin 388.2 work fine. There is one configuration situation requiring factory reset to recover, but it's a rare case (no one found it in beta testing thread and may be model specific) and was reported to Asus. No crashing though.

 

[breadlord]

Thank you for the reply - I have reset using WPS, which I didn't actually know about before - I was using the RESET button; I assume the WPS method does a more thorough NVRAM / JFFS scrub, or you wouldn't have specified it.

I'll stay on Merlin for the short term, as the kernel versions etc, should all be the same and see how I get along. I can't really see how much configuration damage I could have done, as this was already reset - I have just changed the LAN CIDR to 10.0.0.0/23 and added some IP allocations. No JFFS or installed applications, and the AiMesh Node has been turned off since I noticed the kernel panics.

Hopefully this thread dies here

 

[bbunge]

Thank you for the reply - I have reset using WPS, which I didn't actually know about before - I was using the RESET button; I assume the WPS method does a more thorough NVRAM / JFFS scrub, or you wouldn't have specified it.

I'll stay on Merlin for the short term, as the kernel versions etc, should all be the same and see how I get along. I can't really see how much configuration damage I could have done, as this was already reset - I have just changed the LAN CIDR to 10.0.0.0/23 and added some IP allocations. No JFFS or installed applications, and the AiMesh Node has been turned off since I noticed the kernel panics.

Hopefully this thread dies here

I have had times when the Hard Factory Reset on the AX86U , AKA WPS reset, did not fully work. A Factory Reset from the GUI does seem to work for me all the time.

 

[Tech9]

I assume the WPS method does a more thorough NVRAM / JFFS scrub

WPS hard reset does the same like Restore + Initialize in GUI.

Hopefully this thread dies here

Hopefully your router is fine and you don't have to deal with warranty.

 

[Hypothosis]

Thank you for the reply - I have reset using WPS, which I didn't actually know about before - I was using the RESET button; I assume the WPS method does a more thorough NVRAM / JFFS scrub, or you wouldn't have specified it.

I'll stay on Merlin for the short term, as the kernel versions etc, should all be the same and see how I get along. I can't really see how much configuration damage I could have done, as this was already reset - I have just changed the LAN CIDR to 10.0.0.0/23 and added some IP allocations. No JFFS or installed applications, and the AiMesh Node has been turned off since I noticed the kernel panics.

Hopefully this thread dies here

did any method resolve the problem, i also get intermittent crashes on my AXE11000