Troubleshooting random reboots.

[trekker99]

Hi all,

I am trying to track down random reboots which started happening a while back. I have set up backing up of syslog to a thumb drive but there is nothing in the log which hints to what might be the cause. Are there any other logs I can retrieve? Or a verbose mode I could activate?

Setup: ac66u with a samsung s2 portable (ext3) and a 1 gb cruzer mini (fat).
Clients: 2 wired computers, 2 android devices on 802.11g, one dell xps 12 on n and 1 deskop on ac (asus adapter). One of the android devices should be always connected, the rest are on demand.

The problem started a month ago when I was still on 270.25. I noticed it rebooted whenever I logged in or access a shared drive. I finally tracked it down to the samsung s2 which stopped mounting properly. After removing it, the random rebooting stopped. I did a fsck on it on a linux machine (couldn't do it on the router for some reason) and plugged it back and it was okay. I started backing up the syslog in case it randomly rebooted again (I added in the cruzer to store the logs). It was fine.

I then decided to upgrade to the latest merlin firmware yesterday. I did a factory reset after flashing and manually re=entered all the settings. It has since rebooted 8 times since last night (almost a 24 hr period). I am rather puzzled as to why. There are no discernable pattern. There were 3 reboots within an hour but no reboot for the subsequent 5 hrs. I also believe only the anfroid was connected during the three reboots.

I plan to continue investigating this but was hoping to get some tips from you guys.

Sorry for the long post. Thanks.

Sent from my GT-P7310 using Tapatalk 2

 

[GoNz0]

Do you have the download master installed?

I finally got my 3tb Toshiba USB drive today, hooked it up, installed DM and the router rebooted while trying to open a folder during a transfer. repeated and the same again.

Uninstalled DM and all is fine. I uninstalled after finding a few forum threads saying it is buggy.

 

[RMerlin]

Syslog won't give you much information, you would need a serial cable connected to the router's board to troubleshoot that type of error.

Are you using NFS? NFS is known to be unstable.

 

[trekker99]

Do you have the download master installed?

I finally got my 3tb Toshiba USB drive today, hooked it up, installed DM and the router rebooted while trying to open a folder during a transfer. repeated and the same again.

Uninstalled DM and all is fine. I uninstalled after finding a few forum threads saying it is buggy.

Nope, not using DM. I installed entware on the samsung drive.

 

[trekker99]

Syslog won't give you much information, you would need a serial cable connected to the router's board to troubleshoot that type of error.

Are you using NFS? NFS is known to be unstable.

Okay, I don't think I want to open up the router just yet.

And nope, I am not using NFS.

 

[trekker99]

It happened 9 times today. And one of the saved logs had this at the end

Aug  6 08:14:08 kernel: Instruction bus error, epc == 7fff7000, ra == 7fff7000
Aug  6 08:14:08 kernel: Eeek! page_mapcount(page) went negative! (-1)
Aug  6 08:14:08 kernel:   page pfn = 62005
Aug  6 08:14:08 kernel:   page->flags = 80000404
Aug  6 08:14:09 kernel:   page->count = 1
Aug  6 08:14:09 kernel:   page->mapping = 00000000
Aug  6 08:14:09 kernel:   vma->vm_ops = 0x802f1278
Aug  6 08:14:09 kernel:   vma->vm_ops->nopage = special_mapping_nopage+0x0/0xe4
Aug  6 08:14:09 kernel:   vma->vm_ops->fault = _stext+0x7fffec00/0x18
Aug  6 08:14:09 kernel: Kernel bug detected[#1]:
Aug  6 08:14:09 kernel: Cpu 0
Aug  6 08:14:09 kernel: $ 4   : 802eddb0 00000000 ffffffff 00002b52
Aug  6 08:14:09 kernel: $ 8   : 00000001 ffffffff 00000004 fffffffb
Aug  6 08:14:09 kernel: $12   : 0000000f 874c3a50 8028854c 00000000
Aug  6 08:14:09 kernel: $16   : 81c400a0 80335f00 874a4470 fffffffe
Aug  6 08:14:09 kernel: $20   : 00364e3d 62005ecb ffffffff 00000000
Aug  6 08:14:09 kernel: $24   : 00000008 8001791c                  
Aug  6 08:14:09 kernel: $28   : 874c2000 874c3d00 874a4470 8006de9c
Aug  6 08:14:09 kernel: Hi    : 00000000
Aug  6 08:14:09 kernel: Lo    : ea2c8000
Aug  6 08:14:09 kernel: epc   : 8006decc page_remove_rmap+0x2b0/0x2c4     Tainted: P      
Aug  6 08:14:09 kernel: ra    : 8006de9c page_remove_rmap+0x280/0x2c4
Aug  6 08:14:09 kernel: Status: 1100fc03    KERNEL EXL IE 
Aug  6 08:14:09 kernel: Cause : 00000024
Aug  6 08:14:09 kernel: PrId  : 00019749
Aug  6 08:14:09 kernel: Modules linked in: nf_nat_sip nf_conntrack_sip nf_nat_h323 nf_conntrack_h323 nf_nat_rtsp nf_conntrack_rtsp nf_nat_ftp nf_conntrack_ftp ip6table_filter ip6table_mangle nls_cp437 sr_mod cdrom zaurus rndis_host net1080 cdc_ether asix usbnet usblp ohci_hcd ehci_hcd ufsd(P) vfat fat ext2 ext3 jbd mbcache usb_storage sg sd_mod scsi_wait_scan scsi_mod usbcore jffs2 zlib_inflate zlib_deflate nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre nf_conntrack_proto_gre wl(P) igs(P) emf(P) bcm57x
Aug  6 08:14:09 kernel: Process watchdog (pid: 361, threadinfo=874c2000, task=87fa6c00)
Aug  6 08:14:09 kernel: Stack : 874abee0 00000000 ffffffff 00002aaa 874a3fdc 7fff7000 81c400a0 7fff8000
Aug  6 08:14:09 kernel:         80065c1c 80065b4c 87fa6f3c 802eecb0 00000000 80030c38 00000000 7fff8000
Aug  6 08:14:09 kernel:         7fff7fff 8033b014 874a8a60 8749c7fc 8749c7fc 00000000 00000001 00000000
Aug  6 08:14:09 kernel:         874a42cc 874a8a60 874c3da0 0000000a 80335f00 802eecb0 874c3e80 60000efc
Aug  6 08:14:09 kernel:         874c3f30 80069e68 874c3da0 00000000 00000000 ffffffff 874c3da4 00000000
Aug  6 08:14:09 kernel:         ...
Aug  6 08:14:09 kernel: Call Trace:
Aug  6 08:14:09 kernel: [<8006decc>] page_remove_rmap+0x2b0/0x2c4
Aug  6 08:14:09 kernel: [<80065c1c>] unmap_vmas+0x2ac/0x630
Aug  6 08:14:09 kernel: [<80069e68>] exit_mmap+0x78/0x170
Aug  6 08:14:09 kernel: [<80027cd4>] mmput+0x9c/0x164
Aug  6 08:14:09 kernel: [<8002e620>] do_exit+0x164/0x8dc
Aug  6 08:14:09 kernel: [<8002ede0>] do_group_exit+0x48/0xa4
Aug  6 08:14:09 kernel: [<80038a94>] get_signal_to_deliver+0x270/0x404
Aug  6 08:14:09 kernel: [<8000e60c>] do_notify_resume+0x90/0x33c
Aug  6 08:14:09 kernel: [<8000b4f0>] work_notifysig+0xc/0x14
Aug  6 08:14:09 kernel: Code: 00b32824  0c014455  ac45002c <0200000d> 0801b734  8e030000  0801b789  8e04000c  27bdffc8 
Aug  6 08:14:09 kernel: Fatal exception: panic in 5 seconds

RMerlin, does this help in narrowing the cause of the reboots?

Thanks.

 

[trekker99]

Just wondering, is there a way to run some form of diagnostics like to check if the ram is okay? I want to at least rule out hardware problems.

Sent from my GT-P7310 using Tapatalk 2

 

[trekker99]

Well, I decided to do a dirty flash of the same firmware (redownloaded).

I also tried using memtester which states everything is okay. I tried up to 150 MB.

Hopefully, this will resolve my random reboots.

If anyone has any other advice, do share! Thanks.

 

[TonyH]

Hi,
My old habit is on a problem like that, I'd try with another AC adapter for the router.
I really never use original adapter on my routers. I have heavier rated one to replace it.
If you have a 'scope you can check the adapter. Just checking o/p voltage with a multimeter is not good enough. Specially with Hi Z dgital one. Analog multimeter still has it's value
when working on certain situation.

 

[trekker99]

Hi,
My old habit is on a problem like that, I'd try with another AC adapter for the router.
I really never use original adapter on my routers. I have heavier rated one to replace it.
If you have a 'scope you can check the adapter. Just checking o/p voltage with a multimeter is not good enough. Specially with Hi Z dgital one. Analog multimeter still has it's value
when working on certain situation.

Interesting idea and it makes sense as a possible cause of weird errors. I don't have a scope handy though. It will also be tricky to measure the output voltage under load though. The best I have is a Fluke multimeter.

Anyway another kernel panic

Aug 9 10:10:06 kernel: Data bus error, epc == 800f65d0, ra == 8005d358
Aug 9 10:10:06 kernel: Oops[#1]:
Aug 9 10:10:06 kernel: Cpu 0
Aug 9 10:10:06 kernel: $ 0 : 00000000 00000001 8700f0e0 8700f0e0
Aug 9 10:10:06 kernel: $ 4 : 8894fc2c 00000000 00000000 03008401
Aug 9 10:10:06 kernel: $ 8 : 8700f0d0 02180fff 0c905ff4 00000000
Aug 9 10:10:06 kernel: $12 : 86d31d58 ffffffff 00000000 00000000
Aug 9 10:10:06 kernel: $16 : 00000000 00000000 87054704 00000017
Aug 9 10:10:06 kernel: $20 : 00000000 00000004 87054700 86d31c58
Aug 9 10:10:06 kernel: $24 : 00000000 2aaaa904
Aug 9 10:10:06 kernel: $28 : 86d30000 86d31c48 00000000 8005d358
Aug 9 10:10:06 kernel: Lo : 000ccc9a
Aug 9 10:10:06 kernel: epc : 800f65d0 radix_tree_lookup_element+0x90/0xb4 Tainted: P
Aug 9 10:10:06 kernel: ra : 8005d358 __do_page_cache_readahead+0xbc/0x27c
Aug 9 10:10:06 kernel: Status: 1100fc03 KERNEL EXL IE
Aug 9 10:10:06 kernel: Cause : 0000001c
Aug 9 10:10:06 kernel: PrId : 00019749
Aug 9 10:10:06 kernel: Modules linked in: nf_nat_sip nf_conntrack_sip nf_nat_h323 nf_conntrack_h323 nf_nat_rtsp nf_conntrack_rtsp nf_nat_ftp nf_conntrack_ftp ip6table_filter ip6table_mangle nls_cp437 sr_mod cdrom zaurus rndis_host net1080 cdc_ether asix usbnet usblp ohci_hcd ehci_hcd ufsd(P) vfat fat ext2 ext3 jbd mbcache usb_storage sg sd_mod scsi_wait_scan scsi_mod usbcore jffs2 zlib_inflate zlib_deflate nf_nat_pptp nf_conntrack_pptp nf_nat_proto_gre nf_conntrack_proto_gre wl(P) igs(P) emf(P) bcm57x
Aug 9 10:10:06 kernel: Process lighttpd-arppin (pid: 1819, threadinfo=86d30000, task=87fe7800)
Aug 9 10:10:06 kernel: Stack : 87f7520c 86d31ea0 00000000 87ff6320 86d31c58 86d31c58 8008a140 8008a12c
Aug 9 10:10:06 kernel: 2aaaf000 821e6c80 86d31cd8 86d31d00 87f7520c 87ff6320 2aaaf000 874b2580
Aug 9 10:10:06 kernel: 2aaaf000 00000101 00000001 00000001 822a2b40 fe002000 86d31d38 87054700
Aug 9 10:10:06 kernel: 8718a400 00000000 00000004 86d31e70 00000000 8718a400 87054668 8005d5ac
Aug 9 10:10:06 kernel: 800869e0 8718a400 8700bea4 00000000 86d31d38 00000001 00000020 87054700
Aug 9 10:10:06 kernel: ...
Aug 9 10:10:06 kernel: Call Trace:
Aug 9 10:10:06 kernel: [<800f65d0>] radix_tree_lookup_element+0x90/0xb4
Aug 9 10:10:06 kernel: [<8005d358>] __do_page_cache_readahead+0xbc/0x27c
Aug 9 10:10:06 kernel: [<8005d5ac>] blockable_page_cache_readahead+0x94/0x130
Aug 9 10:10:06 kernel: [<8005d89c>] page_cache_readahead+0x17c/0x24c
Aug 9 10:10:06 kernel: [<800557c0>] do_generic_mapping_read+0x470/0x73c
Aug 9 10:10:06 kernel: [<80057980>] generic_file_aio_read+0x1a8/0x1f0
Aug 9 10:10:06 kernel: [<8007e078>] do_sync_read+0xd8/0x15c
Aug 9 10:10:06 kernel: [<8007eb10>] vfs_read+0xc0/0x15c
Aug 9 10:10:06 kernel: [<8007f084>] sys_read+0x58/0xa8
Aug 9 10:10:06 kernel: [<80013964>] stack_done+0x20/0x40
Aug 9 10:10:06 kernel: Code: 8c470000 254afffa 2529ffff <10e0fff1> 00e04021 1520fff5 00000000 14c0ffee 00000000
Aug 9 10:10:06 kernel: Fatal exception: panic in 5 seconds

 

[trekker99]

Okay. I just factory resetted and flash the latest asus official firmware (374). Let's see how this goes.

Sent from my GT-P7310 using Tapatalk 2

 

[trekker99]

It has been a day and it has not resetted so far. The primary differences between my current setup (Asus 3.0.0.4.374.130) vs the previous (Asuswrt-Merlin 3.0.0.4.372.31) are the following: I activated jffs, installed entware, sshd. I also had switched on traffic logging to usb device and enabled usb device spindown, but I did switch those off to see if it made my router more stable.

It sucks I can't use Merlin's firmware, but I will have to live with the official firmware for the time being.

 

[trekker99]

Still thinking what might be the cause. I just looked through my settings from when i was on 270, and I noticed I had ssh disabled. As the official firmware has no ssh, I am wondering if there is known instability with ssh. If I am free this weekend, I might try flashing back to merlin latest and test. I am interested in hearing any experiences in the mean time though.

Oh and the router has crossed 2 days.

Sent from my GT-P7310 using Tapatalk 2

 

[RMerlin]

Still thinking what might be the cause. I just looked through my settings from when i was on 270, and I noticed I had ssh disabled. As the official firmware has no ssh, I am wondering if there is known instability with ssh. If I am free this weekend, I might try flashing back to merlin latest and test. I am interested in hearing any experiences in the mean time though.

Oh and the router has crossed 2 days.

Sent from my GT-P7310 using Tapatalk 2

I have SSH enabled on mine without any problem (and I use it almost on a daily basis).

You could try resetting your keys in case one got corrupted:

nvram unset sshd_dsskey
nvram unset sshd_hostkey
nvram unset sshd_authkeys
nvram commit
service restart_sshd

This can be the case for instance if you restored a saved configuration at some point - older versions had trouble properly encoding the authkeys when saving them.