GT-AX11000 Crashing - Syslog

[meelokun]

My GT-AX11000 (running merlinwrt 386.4) has been exibiting random crashes/reboots several times a day, intermittently. Some days more than others. Some days may be reboot free. Today (1/20/22) was the most egregious at 5 reboots in total. mind you these reboots are not new to 386.4, ive had them in all previous firmware iterations.

I have two AC68Us in AP mode connected by ethernet backhaul, and are also running the same firmware.
I have approx 74 clients at a time.

Pastebin of most recent crashes https://pastebin.com/RnU3XsrM - the dates appear to be out of order

 

[Invisible Boy]

I have had my AX11000 (A1.1) for just over 2 weeks w/o any issue. I have also dirty flashed it with Merlin 386.4 and the latest 2 official FWs back & forth w/o issue. It misbehaved once because I was toying with the settings too much and a hard reset fixed it. Though not related to this router, I had PCs reboot themselves due to overheat in the past. Maybe you can observe the temperature in Merlin and see if it's related.

 

[meelokun]

its not temps - i have two fans under it, and its reporting the following

Legend: 2.4 GHz - 5 GHz - CPU
Current Temperatures: 36 °C - 36 °C - 44 °C

im gonna try switching to AiMesh mode (Ethernet backhaul), as opposed to AP Mode on both my AC68U's - and see if that makes any difference. then ill also try binding them to their respective closest AP, or adding them to the roaming block list.

 

[meelokun]

so i have AiMesh going, and i even bothered to do Manual IP address assignments on all 75 devices. still get random reboots (less than before). posted is the newest syslog for today after it crashed...

 

[meelokun]

new crashes after 386.5 and factory reset - still persisting reboots. im at a loss.

syslog ax11000 - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[meelokun]

snippet

May  5 01:05:10 crashlog: <6>nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.
May  5 01:05:10 crashlog: <3>FPM Pool 0: invalid token 0x00fe8000 freed
May  5 01:05:10 crashlog: <6>FPM Pool 0: ISR timer is enabled. There could be multiple occurrences of the reported issue
May  5 01:05:10 crashlog: <6>CFG80211-ERROR) wl_cfg80211_change_station : WLC_SCB_AUTHORIZE sta_flags_mask not set
May  5 01:05:10 crashlog: <6>CFG80211-ERROR) wl_cfg80211_change_station : WLC_SCB_AUTHORIZE sta_flags_mask not set
May  5 01:05:10 crashlog: <6>CFG80211-ERROR) wl_cfg80211_change_station : WLC_SCB_AUTHORIZE sta_flags_mask not set
May  5 01:05:10 crashlog: <6>CFG80211-ERROR) wl_cfg80211_change_station : WLC_SCB_AUTHORIZE sta_flags_mask not set
May  5 01:05:10 crashlog: <6>httpd (1659): drop_caches: 1
May  5 01:05:10 crashlog: <6>httpd (1659): drop_caches: 1
May  5 01:05:10 crashlog: <6>Init chrdev /dev/idp with major 190
May  5 01:05:10 crashlog: <6>tdts: tcp_conn_max = 8000
May  5 01:05:10 crashlog: <6>
May  5 01:05:10 crashlog: <6>tdts: tcp_conn_timeout = 300 sec
May  5 01:05:10 crashlog: <6>
May  5 01:05:10 crashlog: <4>SHN Release Version: 2.0.2 36f59aa
May  5 01:05:10 crashlog: <4>UDB Core Version: 0.2.20
May  5 01:05:10 crashlog: <6>Init chrdev /dev/idpfw with major 191
May  5 01:05:10 crashlog: <6>IDPfw: flush fc
May  5 01:05:10 crashlog: <6>
May  5 01:05:10 crashlog: <6>IDPfw: IDPfw is ready
May  5 01:05:10 crashlog: <4>sizeof forward pkt param = 280
May  5 01:05:10 crashlog: <2>Bad mode in Synchronous Abort handler detected, code 0x86000007 -- IABT (current EL)
May  5 01:05:10 crashlog: <4>CPU: 1 PID: 231 Comm: bcmsw_rx Tainted: P           O    4.1.51 #2
May  5 01:05:10 crashlog: <4>Hardware name: Broadcom-v8A (DT)
May  5 01:05:10 crashlog: <4>task: ffffffc03027eb00 ti: ffffffc02fc14000 task.ti: ffffffc02fc14000
May  5 01:05:10 crashlog: <4>PC is at 0x0
May  5 01:05:10 crashlog: <4>LR is at blog_finit_args+0xf4/0x210
May  5 01:05:10 crashlog: <4>pc : [<0000000000000000>] lr : [<ffffffc0003fdb2c>] pstate: 40000145
May  5 01:05:10 crashlog: <4>sp : ffffffc02fc17cd0
May  5 01:05:10 crashlog: <4>x29: ffffffc02fc17cd0 x28: ffffffc00084c650
May  5 01:05:10 crashlog: <4>x27: 0000000020030000
May  5 01:05:10 crashlog: <6>IDPfw: Exit IDPfw
May  5 01:05:10 crashlog: <6>mod epilog takes 0 jiffies
May  5 01:05:10 crashlog: <6>IDPfw: Exit IDPfw
May  5 01:05:10 crashlog: <4>x26: 0000000000000003
May  5 01:05:10 crashlog: <4>x25: ffffffc0302af800 x24: 0000000000000000
May  5 01:05:10 crashlog: <4>x23: ffffffc03cc81600 x22: ffffffc02fc17de0
May  5 01:05:10 crashlog: <4>x21: 0000000000000000 x20: ffffffc00084c650
May  5 01:05:10 crashlog: <4>x19: 0000000000000002 x18: 0000000000000000
May  5 01:05:10 crashlog: <4>x17: ffffffc032200000 x16: 000000000000000e
May  5 01:05:10 crashlog: <4>x15: ffffffc03c4b6462 x14: 0000000000000000
May  5 01:05:10 crashlog: <4>x13: 0000000000000000 x12: ffffffbffc3f3040
May  5 01:05:10 crashlog: <4>x11: 0000000000000001 x10: ffffffc03cc81740
May  5 01:05:10 crashlog: <4>x9 : 0000000000000000 x8 : ffffffbffc2c8618
May  5 01:05:10 crashlog: <4>x7 : 0000000000000001 x6 : 0000000000000000
May  5 01:05:10 crashlog: <4>x5 : 0000000000000001 x4 : 000000000000a888
May  5 01:05:10 crashlog: <4>x3 : 0000000000000000 x2 : ffffffc02fc17de0
May  5 01:05:10 crashlog: <4>x1 : ffffffc0302af800 x0 : ffffffc03cc81600
May  5 01:05:10 crashlog: <4>
May  5 01:05:10 crashlog: <0>Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
May  5 01:05:10 crashlog: <4>CPU: 1 PID: 231 Comm: bcmsw_rx Tainted: P           O    4.1.51 #2
May  5 01:05:10 crashlog: <4>Hardware name: Broadcom-v8A (DT)
May  5 01:05:10 crashlog: <4>task: ffffffc03027eb00 ti: ffffffc02fc14000 task.ti: ffffffc02fc14000
May  5 01:05:10 crashlog: <4>PC is at 0x0
May  5 01:05:10 crashlog: <4>LR is at blog_finit_args+0xf4/0x210
May  5 01:05:10 crashlog: <4>pc : [<0000000000000000>] lr : [<ffffffc0003fdb2c>] pstate: 40000145
May  5 01:05:10 crashlog: <4>sp : ffffffc02fc17cd0
May  5 01:05:10 crashlog: <4>x29: ffffffc02fc17cd0 x28: ffffffc00084c650
May  5 01:05:10 crashlog: <4>x27: 0000000020030000 x26: 0000000000000003
May  5 01:05:10 crashlog: <4>x25: ffffffc0302af800 x24: 0000000000000000
May  5 01:05:10 crashlog: <4>x23: ffffffc03cc81600 x22: ffffffc02fc17de0
May  5 01:05:10 crashlog: <4>x21: 0000000000000000 x20: ffffffc00084c650
May  5 01:05:10 crashlog: <4>x19: 0000000000000002 x18: 0000000000000000
May  5 01:05:10 crashlog: <4>x17: ffffffc032200000 x16: 000000000000000e
May  5 01:05:10 crashlog: <4>x15: ffffffc03c4b6462 x14: 0000000000000000
May  5 01:05:10 crashlog: <4>x13: 0000000000000000 x12: ffffffbffc3f3040
May  5 01:05:10 crashlog: <4>x11: 0000000000000001 x10: ffffffc03cc81740
May  5 01:05:10 crashlog: <4>x9 : 0000000000000000 x8 : ffffffbffc2c8618
May  5 01:05:10 crashlog: <4>x7 : 0000000000000001 x6 : 0000000000000000
May  5 01:05:10 crashlog: <4>x5 : 0000000000000001 x4 : 000000000000a888
May  5 01:05:10 crashlog: <4>x3 : 0000000000000000 x2 : ffffffc02fc17de0
May  5 01:05:10 crashlog: <4>x1 : ffffffc0302af800 x0 : ffffffc03cc81600
May  5 01:05:10 crashlog: <4>

 

[meelokun]

i was smooth sailing forawhile. then the issue reared its ugly head back. I factory reset it and my ainodes - and this error is constantly being spammed in the log

what is

kernel: CFG80211-ERROR) wl_cfg80211_change_station : WLC_SCB_AUTHORIZE sta_flags_mask not set

May 29 20:38:08 hostapd: eth7: STA d4:1b:81:12:d0:bf IEEE 802.11: disassociated - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[meelokun]

It was a rouge smart plug that for whatever reason just couldnt re-associate with the route

 

[Tech9]

@Morris shared his experience with main router crashing because of AiMesh nodes. You may want to take a look:

AiMesh Adventures including Stock v Merlin on node

Part 1: Adding a second as a node. I factory reset an old and proven RT-AX86U which had Merlin code and add it as a node. Within minutes I notice some of my IOT devices dropping and connecting again and again. I recall seeing others with similar complaints about IOT devices dropping. I turn...

www.snbforums.com

 

[meelokun]

nvm - still coming back - even though i resolved the rouge smart plug. not sure whats causing it to error out

May 29 21:10:38 hostapd: eth6: STA 60:ab:14:b3:d0:a4 IEEE 802.11: associatedMa - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[meelokun]

I guess what i HAVENT tried - is loading stock asus firmware on the main router (ax11000) and nodes(ac68u's)... :-/ i... really dont want to for obviouse reasons. but if thats what itll take... ill set it up another day. i have far to many tenants that use the network regularly, and me fiddling with it is causing alot of complaints lol.

 

[meelokun]

switched all nodes to dumb AP's. error is gone now. *sigh*

 

[Morris]

I guess what i HAVENT tried - is loading stock asus firmware on the main router (ax11000) and nodes(ac68u's)... :-/ i... really dont want to for obviouse reasons. but if thats what itll take... ill set it up another day. i have far to many tenants that use the network regularly, and me fiddling with it is causing alot of complaints lol.

Merlin seems to be fine for the router. I've had better luck with Asus firmware on my nodes.

 

[Morris]

switched all nodes to dumb AP's. error is gone now. *sigh*

I agree, way more effort than it should be.

 

[meelokun]

welp - even in dumb AP mode, i still get crashes - realized thats how i started, and moved to AiMesh thinking it would solve my issues. So now im just going to replace my AC68U's with the WAX202 ap (cause on sale for $36), and see what happens.

 

[Morris]

welp - even in dumb AP mode, i still get crashes - realized thats how i started, and moved to AiMesh thinking it would solve my issues. So now im just going to replace my AC68U's with the WAX202 ap (cause on sale for $36), and see what happens.

Is it the APs that are crashing or the router? If it is the router replacing the APs probably will not help. One has to wonder why such a new router is discounted that much. I'd ask on a Netgear forum and see what there reliability is.

My RT-AX86U ran for years and then I set up AI mesh and it started crashing. It seems to be related to the network converging when the nodes are on wireless backhaul. Particularly the RT-AX86U node. Things like this are frustrating. You need to keep a log of everything you try and learn and eventually things become clear.

 

[meelokun]

Is it the APs that are crashing or the router? If it is the router replacing the APs probably will not help. One has to wonder why such a new router is discounted that much. I'd ask on a Netgear forum and see what there reliability is.

My RT-AX86U ran for years and then I set up AI mesh and it started crashing. It seems to be related to the network converging when the nodes are on wireless backhaul. Particularly the RT-AX86U node. Things like this are frustrating. You need to keep a log of everything you try and learn and eventually things become clear.

The router is the point that crashes. There was a time when everything was fine for awhile after a factory reset. But the issue came back. I suspect it's the APs, cause of range issues (the log is constantly spammed with what looks like some device not able to decide which AP to associate with). I already bought a wax202 from microcenter to test, and it operated wonderfully. Returned it and am waiting for the 3 pack of them from Netgear. Might open one up to see what its running underneath. It'll be a tiny spec upgrade (wifi5 to wifi6) and alittle more range.

 

[jlodvo]

i have all sorts of problem since using the asus gt-ax11000

 

[Morris]

The router is the point that crashes. There was a time when everything was fine for awhile after a factory reset. But the issue came back. I suspect it's the APs, cause of range issues (the log is constantly spammed with what looks like some device not able to decide which AP to associate with). I already bought a wax202 from microcenter to test, and it operated wonderfully. Returned it and am waiting for the 3 pack of them from Netgear. Might open one up to see what its running underneath. It'll be a tiny spec upgrade (wifi5 to wifi6) and alittle more range.

"The router is the point that crashes." Why are you playing with the APs? You need to address the router. Stay away from AI Mesh as it's just complicating things. I note a few things about your environment:
- Tenants... What are they running? If they are running peer to peer, this type of application can overwhelmed even high end corporates routers. If you are in the US, you are the owner of the network and responsible if a copyright holder sues.
- Not that complicated, a router and a few APs

Recommended things to try:
- Turn off the wireless on the GT-AX11000 and if this creates a coverage hole, use one of your RT-AC68U in AP mode in that location
- Enable CAKE-QOS. I think this requires Merlin code. This is to address abusive clients and prevent them from overwhelming your router
- Replace the GT-AX11000 with RT-AC68U

Good luck,

Morris

 

[jlodvo]

ive check my temp on the ax11000 its at 72c cpu temp and when it reaches this temps its buggy like my internet will disconnect and reconnect every now and then, might this be the cause?