WRT1900AC Spontaneous Reboots, lockups

[sfx2000]

Yeah, that's not happening here.

I'm putting my ISPs DSN servers in the 1st and 2nd slots. The WRT1900 simply isn't passing them on for whatever reason.

It doesn't on the client side... it'll always show the router AP as the only DNS server.

Inside the WRT - DNSMasq is intercepting the client DNS queries - and resolves them internally first, and then goes to the manual DNS servers you've configured in the router GUI config.

 

[htismaqe]

Then what's the purpose of having those manual entries available?

Normally, with any other router I've ever used, you set DNS servers on both the WAN and DHCP LAN interface.

If you configure the DHCP LAN interface with nothing (or the IP address of the router) the router proxies all DNS requests to the DNS servers listed in the WAN configuration.

That's not a very intuitive setup at all to have the manual DNS entries on the LAN/DHCP configuration page is that's not really what they are for...

 

[chadster766]

Well, it's not JUST the GUI that is causing this but it just has to be part of it.

First of all, I've noticed the router is completely ignoring my DNS settings. I have DNS1 and DNS2 hard set to my ISPs DNS servers. DNS3 is still set to 0.0.0.0. All of the devices I checked this morning have a DNS server address of the router.

So I tried changing the settings and of course the router rebooted. So I disconnected it, brought it back to my office, and connected to it via ethernet cable.

I reset it to factory defaults and completely reconfigured it from scratch. During that entire period, it didn't reboot a single time.

I took it back in and plugged everything back in and noticed still that it's ignoring my DNS settings. So I tried changing them again and the router immediately rebooted.

For addition details on how the DNS works see post below:

WRT1900AC Static DNS Behaviour

 

[RMerlin]

Find that hard to believe that it's a web client side issue, as reports on the reboots haven't been Mac specific...

Even if it were - and I'm also convinced it isn't - a web browser should never be able to cause a device to kernel panic. At best it's a serious bug. At worse it's a security issue - hellooooo DoS.

This is probably just a case of first level tech support with no technical knowledge and just a few scripted troubleshooting steps being applied to the customer's request.

 

[chadster766]

Well I finally finished making a router stress testing platform.

I tried to recreate the "Kernel Panic" reboot that I captured from the serial console.

I wrote a script that loops through commands below 100 times:

1. Connect adapter1 to the WRT Wireless
2. Run iPerf throughput test
3. Disconnect adapter1
4. Connect adapter2 to the WRT Wireless
5. Run iPerf throughput test
6. Disconnect adapter2

I ran this twice consecutively. Once with adapter's set static ip's and once DHCP release and renews.

Approx 20gig throughput not a single failure

Any suggestions on addition testing? I'm thinking put the router's 2.4 ghz in mixed mode and have the adapters change protocols each time they connect.

What are your thoughts on whether or not this is a hardware interrupt conflict in the router's hardware config?

How can hardware interrupts be changed in U-Boot?

 

[RMerlin]

Based on the log you posted, I suspect this is a case of either a buffer overrun, or reusing a freed block. In these cases, the issue will always be semi-random unfortunately. The best way to track these down is to enable certain kernel options at compile time that will allow these to be caught as soon they occur, rather than at a later point when you might be trying to access a memory structure that was corrupted by a PREVIOUS operation.

 

[sfx2000]

Was able to grab Chad's PDF trace before the post was taken down... I have a good idea, but need a bit more info - this is not the same crash that htismage is seeing - that's another issue.

@chad - Couple of things to look at:

1) Run same test, but on the LAN ports, with no STA's associated - willing to bet the crash does not happen

2) On your jPerf runs over wireless - using TCP packets

a) send from LAN client to WLAN client - observe if the client is disconnecting/reconnecting
b) send from WLAN client to LAN client - same thing
c) do a bi-directional run

Repeat with UDP

Also, what is the wireless client adapter(s) and driver/host OS?

thx

sfx

 

[sfx2000]

Based on the log you posted, I suspect this is a case of either a buffer overrun, or reusing a freed block. In these cases, the issue will always be semi-random unfortunately. The best way to track these down is to enable certain kernel options at compile time that will allow these to be caught as soon they occur, rather than at a later point when you might be trying to access a memory structure that was corrupted by a PREVIOUS operation.

I think you're on the right track...

my thoughts based on the trace - see the STA repeatedly getting associated/deassociated - if the WLAN driver is not releasing memory, the heap will get fragmented, so things get to a point where the driver is requesting memory on the association - total free memory may be there, but not within a contiguous block, and then the kernel falls down and goes boom.

So the first issue - why is the STA getting dropped, and then why do we run out of memory - sounds like WLAN driver issue to me...

sfx

 

[sfx2000]

This is what is speaking to me for the WLAN driver...

info: Handled event notification_event-device-online.
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82
AP-STA-DISCONNECTED 78:54:2e:1a:1d:82
AP-STA-CONNECTED 78:54:2e:1a:1d:82

 

[htismaqe]

I never thought anything of it previously but when I first configured this router, I had to exit completely out of the GUI and re-enter in Safari because it wouldn't let me add DHCP reservations. I changed the router address to a 10.x.x.x. address but in the DHCP reservations, the first two octets still showed 192.168. The only way to get it to show up right was to exit completely out, close the tab, and then re-login.

As I mentioned before, I also have had problems with settings not sticking and having to enter them multiple times.

But now I'm 100% convinced the web UI is buggy.

After talking with the Linksys support tech, I decided to go ahead and try out an alternate browser, on a completely different OS. I logged in via IE9 on a Windows XP VM and immediately noticed that it was slow as dirt.

I checked Task Manager and iexplore.exe is consuming 324MB of memory. I only have 1024MB allocated to this VM, so the web UI was consuming almost half of the memory available. I also notice that when I move from page to page or hit buttons, the CPU usage spikes, sometimes all the way to 99%.

Some additional strangeness, somewhat related to the idea that it's not saving my edits. By default, the UI shows 7 widgets. When I first configured the router, I removed Parental Controls. When I login via Safari, I have 6 widgets. Logged in via IE9 on WinXP, I have the original 7, including Parental Controls.

Also, I checked the DNS settings and sure enough, all 3 entries show 0.0.0.0. I checked Safari and they're set to my ISP DNS. IE9 exhibits the same behavior when changing DNS settings. I enter them, hit Apply, the router goes through its little routine and the page refreshes with the settings unchanged.

And to top it all off, when I was trying to change the DNS settings, it rebooted.

 

[sfx2000]

And when I say WLAN driver, this would be the marvell driver on the WRT, not the WLAN driver on the wireless client.

And I was able to chase down Chad's WLAN card, looks like it's a DLink DWA-171 USB mini-dongle... which is a Realtek/Mediatek chipset.

sfx

 

[htismaqe]

For addition details on how the DNS works see post below:

WRT1900AC Static DNS Behaviour

I've read this and I don't understand how that applies to me. I'm not using internal LAN DNS outside of the WRT1900.

Are the DNS entries on the Local Connectivity tab actually statically assigned on the WAN interface? Just by their placement in the GUI, one would assume that they are for statically assigning DNS server addresses via DHCP.

 

[sfx2000]

I've read this and I don't understand how that applies to me. I'm not using internal LAN DNS outside of the WRT1900.

Are the DNS entries on the Local Connectivity tab actually statically assigned on the WAN interface? Just by their placement in the GUI, one would assume that they are for statically assigning DNS server addresses via DHCP.

The WRT is intercepting DNS queries and proxying them out to the assigned DNS servers - either from the upstream operator via DHCP/LCP (for PPPoE guys), or the manually assigned ones in the WebGUI.

This is needed so that the Guest Network Captive Portal can work correctly, in addtion to being able to use smartwifi URL to login to the router, as those are local entries in the dnsmasq configuration.

So it's doing what you want it to do - e.g. when a client does the DNS query, it'll resolve to the local cache, and then send the query upstream to the configured DNS servers.

Many SOHO AP's are configured link this now days...

 

[chadster766]

Was able to grab Chad's PDF trace before the post was taken down... I have a good idea, but need a bit more info - this is not the same crash that htismage is seeing - that's another issue.

@chad - Couple of things to look at:

1) Run same test, but on the LAN ports, with no STA's associated - willing to bet the crash does not happen

2) On your jPerf runs over wireless - using TCP packets

a) send from LAN client to WLAN client - observe if the client is disconnecting/reconnecting
b) send from WLAN client to LAN client - same thing
c) do a bi-directional run

Repeat with UDP

Also, what is the wireless client adapter(s) and driver/host OS?

thx

sfx

I took the post down with the PDF log because it may not be accurate about the reported reboots

I'm running the same DHCP release\renew with WLAN to LAN throughput test but increased the loop count to 400. Just to be sure and to get us the best results.

Thanks for your test suggestions I will give those a shot.

Stress Test Platform:
HP 8100E CMT I5 3.20Ghz 2GB RAM 250GB HD
OS: Debian x64 Kernel 3.2.54-2

Adapters:
DWA-552
DWA-180

iPerf Host Server:
Window 7 x64 HP XZ974UT

 

[chadster766]

The WRT is intercepting DNS queries and proxying them out to the assigned DNS servers - either from the upstream operator via DHCP/LCP (for PPPoE guys), or the manually assigned ones in the WebGUI.

This is needed so that the Guest Network Captive Portal can work correctly, in addtion to being able to use smartwifi URL to login to the router, as those are local entries in the dnsmasq configuration.

So it's doing what you want it to do - e.g. when a client does the DNS query, it'll resolve to the local cache, and then send the query upstream to the configured DNS servers.

Many SOHO AP's are configured link this now days...

The big problem I see with this DNS proxy setup is that client computers don't communicate directly with the Local DNS server.

This breaks Active Directory DNS client registration which is a Active Directory requirement. Keep in mind that the Linksys stock firmware wasn't designed for small business networks

 

[sfx2000]

I took the post down with the PDF log because it may not be accurate about the reported reboots

that's ok, but in the future, don't post PDF's - either post the text, or use pastbin or something similar

I'm running the same DHCP release\renew with WLAN to LAN throughput test but increased the loop count to 400. Just to be sure and to get us the best results.

Thanks for your test suggestions I will give those a shot.

thx - I've been digging around inside the marvell wlan driver source...

Stress Test Platform:
HP 8100E CMT I5 3.20Ghz 2GB RAM 250GB HD
OS: Debian x64 Kernel 3.2.54-2

Adapters:
DWA-552
DWA-180

iPerf Host Server:
Window 7 x64 HP XZ974UT

which is the wireless client? the Debian box or the Windows machine?

 

[chadster766]

that's ok, but in the future, don't post PDF's - either post the text, or use pastbin or something similar



thx - I've been digging around inside the marvell wlan driver source...



which is the wireless client? the Debian box or the Windows machine?

Debian box

 

[sfx2000]

The big problem I see with this DNS proxy setup is that client computers don't communicate directly with the Local DNS server.

This breaks Active Directory DNS client registration which is a Active Directory requirement. Keep in mind that the Linksys stock firmware wasn't designed for small business networks

Yep, also makes it kind of useless for folks that run an internal local DNS for handheld client device app development - as it's not reasonable to put a /etc/hosts entry in the client handset

Also makes it tough for WebDev's that might want to use a private IP for dev before kicking things into Production...

There is a way to work around it in those cases - put the WRT1900ac in AP mode, and route WAN via a different box...

sfx

 

[htismaqe]

The WRT is intercepting DNS queries and proxying them out to the assigned DNS servers - either from the upstream operator via DHCP/LCP (for PPPoE guys), or the manually assigned ones in the WebGUI.

This is needed so that the Guest Network Captive Portal can work correctly, in addtion to being able to use smartwifi URL to login to the router, as those are local entries in the dnsmasq configuration.

So it's doing what you want it to do - e.g. when a client does the DNS query, it'll resolve to the local cache, and then send the query upstream to the configured DNS servers.

Many SOHO AP's are configured link this now days...

In my old Netgear router and Zyxel routers, there were DNS entries in the WAN (Internet) section AND the LAN/DHCP section.

If you hard set the WAN/Internet DNS entries, the router itself would use those addresses to do lookups.

If you left the LAN/DHCP settings alone, the router would act as a proxy and forward DNS queries to whatever DNS entries appeared in the WAN section. If you hard set the LAN/DHCP settings to another DNS server however, the router would assign those servers to the clients via DHCP and bypass the router DNS proxy.

I understand what the router is doing and I understand why. However, it's still a very counterintuitive place to put those settings. If those DNS entries are designed to be used by the router and all client lookups are supposed to be proxies, those DNS entries should not be in the local connectivity page. It would make more sense to put them in the Internet page, just like pretty much every other router I've ever used.

 

[sfx2000]

Debian box

Happen on both adapters?

One is Realtek based (the 180), the other is Atheros (the 552)... in kernel drivers, or via wireless-compat or direct download/build the modules directly?

Just trying to rule out the client side...

sfx