RT-AX3000 crashing every few days in bcmsw_rx (Tried hw/sw reset/rebuilds and even new replacement router)

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

L&LD

Part of the Furniture
Sometimes, it's not just the firmware version, it's also what you do after, too.
 

L&LD

Part of the Furniture
You're changing too many things. That is not a minimal setup to me. Follow the links I provided and see if you can methodically track down the step(s) you're doing that are causing you issues.

I just checked on a customer this morning that has been using an RT-AX58U for the last year (I called him back to verify). I updated Diversion and considered restarting the network too. This was hours earlier, but there is no need. The router/network has been working stable since Sep 17 @ 18:11.

There are no issues with 386.xx firmware at all, in any RMerlin supported router that has been properly set up.
 

seanwo

Occasional Visitor
You're changing too many things. That is not a minimal setup to me. Follow the links I provided and see if you can methodically track down the step(s) you're doing that are causing you issues.

I just checked on a customer this morning that has been using an RT-AX58U for the last year (I called him back to verify). I updated Diversion and considered restarting the network too. This was hours earlier, but there is no need. The router/network has been working stable since Sep 17 @ 18:11.

There are no issues with 386.xx firmware at all, in any RMerlin supported router that has been properly set up.
I will definitely try to to minimize my setup even more but at some point I don't see the point of using the router if I can't take advantage of basic features like DNS and Guest networks. Since all these features (and more) worked quite well in older builds I am not sure how my router is "not properly setup". I have been very careful to do hardware resets and jffs formating with every build I tested for months to narrow down the actual github commit that introduced instability in my case. My setup is a compromise between trying to find what is wrong and having a usable network in my home during that time. I have a friend that has the same router and it works well for him, so yes, I would say this is working for some people and not others. It might be related to features that are turned on. It might be related to clients and usage patterns. In the end, I would not expect a router to crash for either of those things. We will keep trying to get to the bottom of it. Thanks for the advice.

Looking at your guides, I actually have been following the one for WPS reset and initialization you wrote; I did not actually put the name to what I had printed out. I found that guide at the beginning of my journey here. Great guide in what to do to get the firmware flashed to a clean state. Thanks for that too.
 
Last edited:

L&LD

Part of the Furniture
There are many things that have changed in the 386.xx builds. Do not assume anything that was working before will work now. Everything is suspect, and everything needs testing/verifying before being considered part of the new 'defaults', for you.

Documentation | Asuswrt-Merlin
 

seanwo

Occasional Visitor
There are many things that have changed in the 386.xx builds. Do not assume anything that was working before will work now. Everything is suspect, and everything needs testing/verifying before being considered part of the new 'defaults', for you.

Documentation | Asuswrt-Merlin
Totally :) I started by focussing on what changed when the testing showed the instability started https://github.com/RMerl/asuswrt-merlin.ng/compare/386.2_4...386.2_6. Time to go further down the rabbit hole when time permits me.
 

seanwo

Occasional Visitor
The next test I will preform this weekend will basically be to use the latest stable 386 build (I don't want to introduce any problems by using the alpha) with the following configuration:

Full WPS reset as per @L&LD

lan:lan ip
ip address: 192.168.0.1

administration:system
router login name: [admin username]
new password: [new password]
retype password: [new password]
enable jffs custom scripts and configs: yes

lan:dhcp server
[add manual assignments] <-- I have to have a few in my house; no getting around that.

wireless:general
(2.4ghz)
network name: [network name]
wpa pre-shared key: [key]

(5ghz)
network name: [network name]
wpa pre-shared key: [key]

[Only enable ssh long enough to add a wan-start script to send an email to detect a reboot; then turn it off once the wan-start is created]:

create the jffs/scripts/wan-start

Anyone disagree this is a good minimal configuration to start with?
 

m0rF1uS

Occasional Visitor
I would say after you do the reset just set up WiFi password and whatever manual IP addresses you need to do and leave it to run like that. No scrips, just the default settings for the firmware. After 7days you can log in and see if the router restarted or not …. Doesn’t really matter if it did on the first day or last. That will exclude most of the things L&LD was talking about as variables.
after this is complete and depending on the results you can continue troubleshooting accordingly.
 

seanwo

Occasional Visitor
I would say after you do the reset just set up WiFi password and whatever manual IP addresses you need to do and leave it to run like that. No scrips, just the default settings for the firmware. After 7days you can log in and see if the router restarted or not …. Doesn’t really matter if it did on the first day or last. That will exclude most of the things L&LD was talking about as variables.
after this is complete and depending on the results you can continue troubleshooting accordingly.
OK. I will forgo running scripts and just check it every 24 hours to see what happened if I don't notice the reboot during daily use.
 

JWoo

Senior Member
Personally, I login once a day and look at the uptime on the General Log page. If you look after 7 days when the logfile is pretty large, it can be a science project to figure out what the uptime is and how many crashes have been experienced. I am only myself checking once a day to develop an understanding of the problem. If the router were stable, I would probably logon once a week or so. I am right now on 384.18 on my RT-AX56U and so far it is stable with a full configuration (no scripts or addons, but IPv6, 2 guest networks, 2 main networks, DoT). If it gets through the weekend with no crashes, I will probably switch back to the 386 base and start off with a reduced configuration to narrow down what function might be causing the problem. For me, I would probably take away Stubby first, then IPv6 if the router is still crashing and then Guest Networks to see which function is causing the crashes.

My opinion is that these crashes on the 386 base are widespread, but many users aren't checking frequently enough their uptime to know. As above, I login daily (which takes 5 minutes) and if my uptime on the General Log page is less than a day, then I know the router crashed in the last 24 hours and I save the crash log. If the uptime is more than a day, I know the router has not crashed.
 

JWoo

Senior Member
The other odd thing that I notice is that on the 384.18 build, I have 7 devices that pick up IPv6 addresses on the LAN side (in the IPv6 log). On any of the 386 builds (386.3-2, 386.2-2 and 386.4 alpha 1), the LAN side (IPv6 log) will have 0 devices or sometimes 1 device. And on the 386 builds, the wireless log (which shows the LAN side IP addresses including IPv6) never matches the IPv6 log. On the 384.18 build, the IPv6 addresses shown for the 7 devices on the wireless log match the 7 devices/IP addresses in the IPv6 log. So there is definitely something going on regarding IPv6 on the 386 builds.
 

seanwo

Occasional Visitor
Personally, I login once a day and look at the uptime on the General Log page. If you look after 7 days when the logfile is pretty large, it can be a science project to figure out what the uptime is and how many crashes have been experienced. I am only myself checking once a day to develop an understanding of the problem. If the router were stable, I would probably logon once a week or so. I am right now on 384.18 on my RT-AX56U and so far it is stable with a full configuration (no scripts or addons, but IPv6, 2 guest networks, 2 main networks, DoT). If it gets through the weekend with no crashes, I will probably switch back to the 386 base and start off with a reduced configuration to narrow down what function might be causing the problem. For me, I would probably take away Stubby first, then IPv6 if the router is still crashing and then Guest Networks to see which function is causing the crashes.

My opinion is that these crashes on the 386 base are widespread, but many users aren't checking frequently enough their uptime to know. As above, I login daily (which takes 5 minutes) and if my uptime on the General Log page is less than a day, then I know the router crashed in the last 24 hours and I save the crash log. If the uptime is more than a day, I know the router has not crashed.
Agreed on most of the points above. I did not notice it for a while because the crash would happen overnight a lot. But when it started crashing in the middle of business meetings I started to get more serious figuring out what was going on. I truly believe it is the Broadcom driver update in 386.2_6 (to fix the fragattack) and that 386.2_4 is the last stable build; it took weeks of effort to narrow that down.

For me using wan-start gave me a way to audit the crashes since it would send me email every time it rebooted: https://github-wiki-see.page/m/RMerl/asuswrt-merlin.ng/wiki/Sending-Email Basically, if you get this email and the uptime is 0, it crashed. With that, I could now run firmware version experiments and be notified whether if failed. If I had not heard from the router in a week, I knew that version of the firmware was stable and moved on to the next one.

At this point, I am just going to go basically go with a default setup.. configuring my SSIDs and network keys to show that this has nothing to do with my setup. My money is on clients triggering this with those Broadcom drivers or some interaction with the guest network blowing it up. If it is the guest network, I should know with the next set of tests. I need to wait till the weekend to spin up more testing since we have day jobs too and actually need to use the internet. I have been stable for 13 days on 386.2_4 to get work done. :)
 

JWoo

Senior Member
That sounds like a good plan. For me, once I get through the weekend confirming stability on 384.18 or not, I will reduce my configuration on the 386 base by taking away Stubby and IPv6. If it is still not stable, I will remove the guest networks.
 

Suzib6sw

Occasional Visitor
That sounds like a good plan. For me, once I get through the weekend confirming stability on 384.18 or not, I will reduce my configuration on the 386 base by taking away Stubby and IPv6. If it is still not stable, I will remove the guest networks.
Just to throw a wrench into that hypothesis, Im not running any scripts or IPV6. I do have Guest networks . I have been "up" for 2 days now since flashing Alpha 4 over itself on all nodes.
 

seanwo

Occasional Visitor
Just to throw a wrench into that hypothesis, Im not running any scripts or IPV6. I do have Guest networks . I have been "up" for 2 days now since flashing Alpha 4 over itself on all nodes.
If you make it to 7 days I would be interested in a list of changes to defaults and whether that same configuration fails on the latest stable build.
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top