In some dire need of assistance (AX86U)

[ColinTaylor]

At this point, to prove concept, I may remove the pihole from the equation to see if I can get back to the initial setup working completely. This would beyond a shadow of a doubt now say the pihole is the cause of the problems; and then I can open a support thread and hope for some help there.

This really needs to be your next step.

Make sure the PiHole's DHCP server is not enabled.

 

[shadowfiber]

This really needs to be your next step.

Make sure the PiHole's DHCP server is not enabled.

Yep! Will report back after it's in that method for a day or so (usually pretty quick to tell).

DHCP is definitely off on the pihole.

 

[shadowfiber]

Well, the solution has been found. Although I do not understand why.

The pihole has two "advanced dns" configurations
Never forward non-FQDNs
Never forward reverse lookups for private IP ranges

Both of these were selected in my prior configuration. They are both deselected now and the environment is working well.

I have not yet determined if it is one or both required at this point. I have verified with the AC66U it does not care if these options are selected on the merlin 370 firmware. The devices work fine.

 

[shadowfiber]

Looks like I spoke too soon. All the google homes dropped off the network yesterday evening after running well for 2-3 days.

Back to the drawing board... Logs indicate an issue with the WPA shared key not working properly. But the key exchange was set to 60 minutes before I so figure it should have popped up sooner.

Ugh. Thought I had it...

 

[ColinTaylor]

Looks like I spoke too soon. All the google homes dropped off the network yesterday evening after running well for 2-3 days.

Back to the drawing board... Logs indicate an issue with the WPA shared key not working properly. But the key exchange was set to 60 minutes before I so figure it should have popped up sooner.

Ugh. Thought I had it...

Upload the complete syslog for us to look at. These symptoms are different than the ones you originally described.

 

[shadowfiber]

Hey Colin, here is the paste

Pastebin.com - Locked Paste

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

I will DM you the pw.

 

[ColinTaylor]

Thanks.

Quick question - The log appears to show that you have 4 different devices all with the same DNS name, "Google-Home-Mini". Whilst generally not being a good idea I didn't even think the GUI allowed you to enter duplicates. Have you set these up in LAN - DHCP Server? Maybe the GUI allows you to enter long names but dnsmasq is truncating them to 16 characters.

 

[shadowfiber]

Hey there,

Also noticed that, and often confused by the names, those names aren't coming from me. Here is a screenshot of what I have attempted to set in the UI. You'll notice in the 2nd screenshot of the client list only 2 are showing up - those are the two i power cycled this morning.

Not sure where the names are coming from.

 

[ColinTaylor]

Not sure where the names are coming from.

I think your screenshots are of the Client List in the Network Map so they will be showing the "client name" not the DNS host names. Have you got anything setup under LAN > DHCP Server > Manually Assigned IP...

 

[shadowfiber]

That 2nd screenshot is the LAN -> DHCP Server area.

Here's a bigger one.

 

[ColinTaylor]

OK. Yes, you don't have anything set in the "Hostname (Optional)" column so each Google-Home-Mini is trying to register the same DNS name.

This is probably unrelated to your problem, but to eliminate is as a possibility and the make the log entries clearer I suggest that you enter your client names into the host name field as well.

 

[shadowfiber]

Hey there,

Got it - all of them now have the name reflected from the left.

I can do a full reset this evening and then get a new log when they drop off again?

 

[ColinTaylor]

I can do a full reset this evening and then get a new log when they drop off again?

Sounds like a good plan.

I couldn't see an obvious cause of your problem in the log. The 5GHz WiFi driver did crash at 21:28 which is rather concerning. But it's not clear whether that's a cause, a symptom or unrelated. Having a new log to compare with will be useful.

 

[shadowfiber]

Hi Colin.
Upgraded to Beta1 - wiped config per merlin and did everything over again. Set the HostName off the bat for logs. Everything is working at the moment. Will report back as things progress. Thank you for your interest and persistence in helping.

 

[shadowfiber]

So, a bit over 36 hours into the test, - changes being the beta1 firmware, and setting host names initially - no drops (yet). The logs are largely unexciting however I did notice the crashes you were speaking about of the radio. Those seem to be persisting and random. Code below. Longest run we've had so far is 2-3 days, so more to come of course. Not sure what to make of the radio crashes. Seems we have a few more individuals coming out seeing the problem on their environments as well...

For those individuals - I will continue to post updates and share what we find... I also have an escalated case with Asus that is with their R&D department and made them aware of the two threads here.

More to come

@ColinTaylor Any suggestions on why the radio crash may be occurring, or is that something more for Merlin/ASUS. It didn't seem to affect the googles - at least this time.

Spoiler: crash

Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wl1: PSM microcode watchdog fired at 95822 (seconds). Resetting.
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wl1: reason = psm watchdog at 95822 seconds. corerev 129.1 ucode revision 1570.159 features 0x9106
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 psmdebug 0x01ff000d phydebug 0x1 macctl 0x4160403 maccmd 0x4
Dec 8 22:11:45 kernel: CONSOLE: psm_brc 0xa000 psm_brc_1 0x000c M_UCODE_DBGST 0x2
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 PSMR1: psmdebug 0x0000009b macctl 0x4060402 maccmd 0x0 M_UCODE_DBGST 0x2
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 brwk_0 0x0501 brwk_1 0x0010 brwk_2 0x82a0 brwk_3 0x0ff0 brwk_4 0x0006
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 ifsstat 0x9da0 ifsstat1 0xe0 txectl 0x481d txestat 0x445
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 rxestat1 0x1 rxestat2 0x500 rcv_frmcnt 0x0 rxe_rxcnt 0x68
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wepctl 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 aqmf_ready0 0x0 aqmf_ready1 0x0 aqmf_ready2 0x0 aqmf_ready3 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 aqmf_ready4 0x2 wepstat 0x0 wep_ivloc 0x1a wep_psdulen 0x157
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 rxe_errval 0x0 rxe_errmask 0x20f rxe_status3 0x108 txe_status2 0x445
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 dbg_bmc_status 0x0 psm_chk0_err 0x0 psm_chk1_err 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 PC :
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R0: 0x24 0x24 0x24 0x24
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 phydebug :
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 pktproc 0x101 pktproc 0x101 pktproc 0x101 pktproc 0x101
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 TxFifoStatus0 0x1c3 TxFifoStatus1 0x1c3
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 gpiosel 0x0 gpiolo 0x0 gpiohi 0x0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 gpiosel 0x1 gpiolo 0x0 gpiohi 0x0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 psm stack_status : 0x8000
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 psm stack_entries:
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0024
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1d99
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0629
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1f21
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x086c
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0848
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0848
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x086a
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 1) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 2) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 3) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: wlc_check_assert_type HAMMERING: reinit_reason 2
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: fatal error, reinitializing
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: fatal error, reinitializing, total count of reinit's[6]
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: 802.11 reinit reason[2], count[6]
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wl1: wlc_bmac_suspend_mac_and_wait: waited 83000 uS and MI_MACSSPNDD is still not on.
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wl1: reason = mac suspend failure at 95822 seconds. corerev 129.1 ucode revision 1570.159 features 0x9106
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 psmdebug 0x01ff000d phydebug 0xb macctl 0x4160402 maccmd 0x4
Dec 8 22:11:46 kernel: CONSOLE: psm_brc 0xa000 psm_brc_1 0x000c M_UCODE_DBGST 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 PSMR1: psmdebug 0x0000009b macctl 0x4060402 maccmd 0x0 M_UCODE_DBGST 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 brwk_0 0x0583 brwk_1 0x0095 brwk_2 0xaaa0 brwk_3 0x0ef0 brwk_4 0x0006
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wlc_dump_ucode_fatal: log_done 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 HWA4a TxSTAT: hwa_txstat_reclaim: reinit<1> stall<0>
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 HWA3b TxFIFO: reclaim fifo 1 pkts 1
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1b RxFILL: mac_counter_status sat<0> need_post<0> aval<256>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1b RxFILL: reclaim core<0> 256 rxbuffers RD<218> WR<474>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1a RxPOST: wi_cnt<126> in hwa internal memory<RD:98 WR:96>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4000400
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start (mctrl | MCTL_PSM_JMP_0)=0x4020406 0x4020406
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4020402
Dec 8 22:11:46 kernel: CONSOLE: 152722.126 wl1: CORE INIT : mode 4 pktclassify 1 rxsplit 1 hdr conversion 1 DMA_CT Enabled

 

[shadowfiber]

104 Hour Update - Longest run so far - no disconnects or issues with the environment.

Puzzling results - as the only change from the last change was hostnames on DHCP. Interesting. Still getting random watchdog crashes - they don't seem to affect anything noticeable.

For people following - could you please try adding host names to your DHCP reservations and see if that resolves your issues too?

 

[KevTech]

In the wifi settings, make sure you also disable Airtime Fairness - I've seen a lot of devices having problems with that, especially on the 2.4 GHz band. Can't recall if Asus finally changed that to be disabled by default (we've had talks about it in the past).

Default is now disabled at least on the ax86.

 

[ColinTaylor]

Puzzling results - as the only change from the last change was hostnames on DHCP.

...and you installed the beta firmware, and you did a factory reset.

 

[shadowfiber]

...and you installed the beta firmware, and you did a factory reset.

Ah yes, My initial thinking was that there were factory resets on both official 384 and official beta 386 with the same results - but yes, in this case, Merlins 386 beta and factory reset was used. Alpha 4 was also a full reset with no change - but the hostnames were not present there.

 

[cooloutac]

Alright, apologies for the delay, did some extended testing.

First, reset the router and installed a fresh alpha4 - reconfigured everything on the network - also removed the pi.hole from the equation completely. After extended testing, no google home connectivity issues - everything was working perfectly. Even the Sonos.
Second, reintroduced the pi.hole with a brand new configuration - wiped it out, redid everything. Google Homes disconnected randomly with the DHCP issues. Sonos was fine.
Third, attempted your DHCP broadcast test. No change.
Fourth, attempted random changes to pihole - unable to get back to things working. (even attempted the DNS bypass and it didn't work)

At this point, to prove concept, I may remove the pihole from the equation to see if I can get back to the initial setup working completely. This would beyond a shadow of a doubt now say the pihole is the cause of the problems; and then I can open a support thread and hope for some help there.

I had originally thought the issue was the router as the pihole was working just fine with the old router... Perhaps someone with some institutional history may have some thoughts on how DHCP/DNS changed between 370 and 384/6 firmwares? It seems something has changed enough for that it requires some additional configurations now for Google devices. I attempted to "static" the dns on the router to bypass the pihole to 8.8.8.8 and that did not work too.

I know I am missing something. I just don't know what.

are they disconnecting from router, or just not getting internet? How did you set up the pi-hole? All I do in the router is add the pi hole address to the dhcp dns entry in lan settings. I don't put a domain name or anything else. in the wan section of router I just put any public dns.