r7800 radio outages and router restarts -- help debugging

[2020aj]

Hi there, sorry for my lack of knowledge in advance. I'm a bit inexperienced mucking around in linux but I'm hopeful someone here will be able to point me in the right direction.

I'm trying to determine the cause behind occasional radio outages/reboots on my r7800. When I log in via telnet I see /var is symlinked to /tmp and all the logs get cleared after reboot. Is there a way to disable the clearing of /tmp/log/message and /tmp/log/log-message so that I can properly investigate this issue?

Running voxel V1.0.2.74.1SF, though I was experiencing this issue on stock since I updated fw (I forget which version I was on originally). Let me know if there's any other information I should include in case it would be helpful.

Thanks!

 

[microchip]

I don't think there's a way to keep the logs. Have you tried a factory reset and reconfigure from scratch?

 

[kamoj]

This is a Netgear system. They do things their own way.

For dmesg you can try something like, to continuously write to a USB-memory (at sda1):

echo "Y" >/sys/module/printk/parameters/time
while true; do dmesg -c | awk -v d="$(date +"%F %T")" '{printf d" "$0"\n"'} >>/mnt/sda1/dmesg.txt;sync;usleep 100000; done &

The system logs are in /var/log/
You can find e.g. /var/log/messages and /var/log/log-message

To continuously write a copy of the system message logs to a USB-memory (at sda1), you can use e.g.:

tail -f /var/log/messages >>/mnt/sda1/messages.txt &
tail -f /var/log/log-message >>/mnt/sda1/log-message.txt &

You can add these lines of code to e.g. /etc/rc.local, to be run at every boot.

 

[2020aj]

I don't think there's a way to keep the logs. Have you tried a factory reset and reconfigure from scratch?

As part of installing voxel I reset settings and reconfigured, though I did not try this on stock. I've only changed very basic settings though, such as SSID and pw, subnet addr, static IPs and port forwarding.

This is a Netgear system. They do things their own way.

For dmesg you can try something like, to continuously write to a USB-memory (at sda1):

echo "Y" >/sys/module/printk/parameters/time
while true; do dmesg -c | awk -v d="$(date +"%F %T")" '{printf d" "$0"\n"'} >>/mnt/sda1/dmesg.txt;sync;usleep 100000; done &

The system logs are in /var/log/
You can find e.g. /var/log/messages and /var/log/log-message

To continuously write a copy of the system message logs to a USB-memory (at sda1), you can use e.g.:

tail -f /var/log/messages >>/mnt/sda1/messages.txt &
tail -f /var/log/log-message >>/mnt/sda1/log-message.txt &

You can add these lines of code to e.g. /etc/rc.local, to be run at every boot.

Thanks Kamoj! I was googling around and saw that you replied to someone else with this on the ng board and was about to try it.

I have some more details now as well. Usually I notice high LAN pings for several minutes before the dropout and reset occur. I had logs running during this time and saw many similar entries and was not sure if they are related to my problem or not... here's an excerpt:

current version: V1.0.0.361; new version: V1.0.0.361
[wifi0] FWLOG: [457135] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1e6 )
[wifi0] FWLOG: [457136] WAL_DBGID_DEV_RESET ( 0xc01d, 0xdeaddead, 0x2 )
[wifi0] FWLOG: [457369] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1e7 )
[wifi0] FWLOG: [457369] WAL_DBGID_DEV_RESET ( 0xc01d, 0xdeaddead, 0x2 )
[wifi0] FWLOG: [457458] WAL_DBGID_TX_BA_SETUP ( 0x4491ec, 0x39780006, 0x0, 0x2, 0x0 )
[wifi0] FWLOG: [457496] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1e8 )
[wifi0] FWLOG: [457496] WAL_DBGID_DEV_RESET ( 0xc01d, 0x270c )
Switching to Tx Mode-1 Threshold 1000
[wifi0] FWLOG: [458434] WAL_DBGID_TX_BA_SETUP ( 0x4491ec, 0x39780006, 0xc1, 0x40, 0x1 )
[wifi0] FWLOG: [459228] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1e9 )
[wifi0] FWLOG: [459228] WAL_DBGID_DEV_RESET ( 0xc01d, 0xdeaddead, 0x2 )
[wifi0] FWLOG: [459886] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1ea )
[wifi0] FWLOG: [459886] WAL_DBGID_DEV_RESET ( 0xc01d, 0xdeaddead, 0x2 )
[wifi0] FWLOG: [460062] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x1eb )
[wifi0] FWLOG: [460062] WAL_DBGID_DEV_RESET ( 0xc01d, 0x26c1 )
Switching to Tx Mode-1 Threshold 0
Switching to Tx Mode-1 Threshold 1000

 

[2020aj]

Update:

I couldn't figure out what was wrong with it but I'm almost positive it's a software and/or environment issue of some kind. Since reverting to stock V1.0.2.58 I have had no issues (>4 days uptime). On latest stock, as well as latest voxel, I was experiencing the issues described in OP--- I also experienced a new issue where the router would remain connected to the modem WITH internet access (confirmed via telnet), maintain both wired/wireless connections with all devices, but the devices themselves would report no internet access. Probably some issue with the loopback address on router??? I'm going to record logs with the older/stable firmware, and compare to the full logs from the current/unstable and try to find any anomalies. Will report back if I find anything more specific.

 

[kanzerx]

Update:

I couldn't figure out what was wrong with it but I'm almost positive it's a software and/or environment issue of some kind. Since reverting to stock V1.0.2.58 I have had no issues (>4 days uptime). On latest stock, as well as latest voxel, I was experiencing the issues described in OP--- I also experienced a new issue where the router would remain connected to the modem WITH internet access (confirmed via telnet), maintain both wired/wireless connections with all devices, but the devices themselves would report no internet access. Probably some issue with the loopback address on router??? I'm going to record logs with the older/stable firmware, and compare to the full logs from the current/unstable and try to find any anomalies. Will report back if I find anything more specific.

I no longer have the router but i bet u are using custom dns, choose get auto from isp, and voila u get internet. One of many reasons I just returned that thing

 

[ThePoisonDonut]

Did you end up with a solution? I'm running Voxel 1.0.2.82.2SF (in AP mode) and my logs came out spammed with:

[wifi1] FWLOG: [1195039] WAL_DBGID_DEV_RESET ( 0xc01d, 0x479 )
[wifi1] FWLOG: [1195198] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x11a9 )

as well, after an automatic reboot. This is supposed to be a release with Voxel's fix applied, but there may be more than meets the eye with this bug.

 

[OcramSagev]

Did you end up with a solution? I'm running Voxel 1.0.2.82.2SF (in AP mode) and my logs came out spammed with:

[wifi1] FWLOG: [1195039] WAL_DBGID_DEV_RESET ( 0xc01d, 0x479 )
[wifi1] FWLOG: [1195198] WAL_DBGID_DEV_RX_TIMEOUT ( 0x5e9, 0x11a9 )

as well, after an automatic reboot. This is supposed to be a release with Voxel's fix applied, but there may be more than meets the eye with this bug.

I was having a similar problem, Voxel mentioned he released a fix based on a recommendation by Kamoj. It worked for me for about a day and then the reboot happened. Voxel's .79 is the version that has worked rock solid for me. Please share if you find more info on the errors you reported.

 

[ThePoisonDonut]

I was having a similar problem, Voxel mentioned he released a fix based on a recommendation by Kamoj. It worked for me for about a day and then the reboot happened. Voxel's .79 is the version that has worked rock solid for me. Please share if you find more info on the errors you reported.

So I ended up following Voxel's (I think) instructions on resetting the router combined with the newest release by him. I can't find them at the moment so I will relay them here.

You want to download a FW version that you don't want to stick with after this process, and the FW that you do want to stick with (I used the latest from Voxel's site).

First, install the version you don't want to use. Then when that's finished, uprade to the version you do want to use. Go to xxx.xxx.xxx.xxx/debug.htm and enable telnet.

Telnet into the router and enter these commands:

mtd erase netgear
nvram default
nvram commit
reboot

Then once the router reboots, set it up as you want it, and you should be good. This is the process that fixed the issue for me.

 

[OcramSagev]

So I ended up following Voxel's (I think) instructions on resetting the router combined with the newest release by him. I can't find them at the moment so I will relay them here.

You want to download a FW version that you don't want to stick with after this process, and the FW that you do want to stick with (I used the latest from Voxel's site).

First, install the version you don't want to use. Then when that's finished, uprade to the version you do want to use. Go to xxx.xxx.xxx.xxx/debug.htm and enable telnet.

Telnet into the router and enter these commands:

mtd erase netgear
nvram default
nvram commit
reboot

Then once the router reboots, set it up as you want it, and you should be good. This is the process that fixed the issue for me.

I've been through all this already. What I've found out that works is disabling ReadyCloud and kwilt. His fix worked for me for a while but then I got the reboots, something else is going on. I'm sticking with .79 for now because I'm using the R7800 as an AP and frankly, I am not using the router features.