Roku Netflix problem on Merlin Firmware

[Xentrk]

I can tell you my daughter gave me the latest version AFTV for Christmas, to run with our original Amazon Echo and our latest Amazon Dots. I use a Cisco RV340 router and I block all UDP DNS traffic including 8.8.8.8 and 8.8.4.4 on my router. I only allow my ISP and QUAD9. My devices seem to work fine. I bet your problem is related to pfsense. I ran pfsense a few years ago and I had issues with Hulu commercials for a while after one of pfsense's updates. The commercial would freeze when using an AppleTV. The program would run fine and then when it ran a commercial it would freeze sometimes.

I took the defaults when I setup the AFTV on Christmas eve. We were drinking a lot.

Yep, something still mucked up. I can get the Echo working for 24 hours or so. Then, the next day I get crickets and Alexa says "sorry, I'm having understanding right now. Please try again later". I do block client DNS requests and reroute them to DNS Resolver (Unbound) so I think that is what may be causing my issue. But it works for the first 24 hours or so. Will require some more analysis. I'll have to dig into the logs to see where the issue may be. No issues with the AFTV devices though.

Update: My 5 GHz WiFi channel had stopped broadcasting when I noticed the problem yesterday when I wrote this. I rebooted the Asus RT-AC68U Imuse as the AP and it starred working again.

 

[coxhaus]

Yep, something still mucked up. I can get the Echo working for 24 hours or so. Then, the next day I get crickets and Alexa says "sorry, I'm having understanding right now. Please try again later". I do block client DNS requests and reroute them to DNS Resolver (Unbound) so I think that is what may be causing my issue. But it works for the first 24 hours or so. Will require some more analysis. I'll have to dig into the logs to see where the issue may be. No issues with the AFTV devices though.

I use a Cisco RV340 router as my DNS server as it is the one with a cache. Sometimes I test handing outside DNS server numbers but I usually go back to my router because it works better. Even my Cisco layer 3 switch does not work as well as my router for DNS as it does not cache DNS.

Finally... I know those problems with pfsense, it also had issues working well with a layer 3 switch. pfsense always seems to kind of work. I stopped using it. I don't like constant projects.

 

[SawKyrom]

SawKyrom; Xentrk is great isn't he? Very interesting information and food for though. Blocking the Avengers, clearly infers that 'Iron Man' in any of the Marvel films would of course be off-color, but by the same token, Patrick Macnee's character, 'John Steed' in the UK's 'The Avengers' series (and not on NF), would also be nixed by simple association with a horse. 'State of Jones' (a character, show or colloquial physical expression) is downright funny. The number of ordinary words that can be associated as something naughty as applied by the keyword filter, has me reconsidering our senior citizen's version keyword filters (not). It does eaves everything to the filter's imagination what any human might find offensive. Tsk, leading those poor older AFTV generations astray. On a serious note, do later version of AFTVs, tablets or other devices run into similar problems? Thanks for an uplifting moment and close to a long day; no pun intended.

Ha! Your assessment of these programs is both hilarious and accurate. I'm glad I can lift your spirits. To answer your question, albeit belatedly, it is only related to the 1st generation AFTV. The 2nd generation AFTV and every other device on the network (Fire HD, PS4, PS3, windows PCs, etc) function without issue and in part why it took me so long to diagnose. Don't know why they would respond differently since I would think the router would be the controlling entity, but they do...

And yes, Xentrk is great... Thanks to all once more!

 

[st3v3n]

SawKyrom, thanks glad we can help lift Xentrk's spirits too. Am considering latching onto a new AFTV 4K if/when Amazon discounts it for Prime day or in the holiday season. (Having their 30-day return period as an equipment trial/burn in ensures that everything performs well). Too much work vs too few hours, combined with regular maintenance duties and never-ending studies keeps one busy and out of trouble. There's always a bit of humor to be found as long as the subject can be viewed from all angles. Reading the forum once a week rounds out the schedule nicely. Cheers.

 

[sfx2000]

I bet your problem is related to pfsense. I ran pfsense a few years ago and I had issues with Hulu commercials for a while after one of pfsense's updates.

I wonder if this is related to pfSense integrating unbound as a resolver/forwarder - previously, they used dnsmasq, and upgrading could run into a race condition with both trying to do the same thing on the same port.

The fix was to disable the DNS Forwarder (dnsmasq) and enable the Resolver (unbound) - much better performance, but at a cost as the feature sets were different.

Current pfSense with unbound works pretty well - even with DNS over TLS with providers like Cloudflare and Google Public DNS....

I block all UDP DNS traffic including 8.8.8.8 and 8.8.4.4 on my router.

This is becoming a bit of a concern actually - as clients can establish a VPN profile to directly access the DNS providers, bypassing blocks at the edge router... the clients are getting smarter these days.

On the road, when I'm on travel - I can appreciate the DNS bypass - but on my network, it's a bit of a loss of control there...

 

[coxhaus]

This is becoming a bit of a concern actually - as clients can establish a VPN profile to directly access the DNS providers, bypassing blocks at the edge router... the clients are getting smarter these days.
.

Your are right about VPN which does allow outside DNS. I don't believe there is any VPN out bound from my home but I should block it any way. I wonder if it is possible?

PS
Blocking VPN may be an issue.
The basic ports are:
TCP 1723
IP 47
UDP 500
UDP 1701

This may require Untangle over a basic router to block OpenVPN.

 

[sfx2000]

Your are right about VPN which does allow outside DNS. I don't believe there is any VPN out bound from my home but I should block it any way. I wonder if it is possible?

I say VPN lightly... as it's not real VPN, just for DNS over TLS and/or DNS over HTTPS

https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/iphone/
https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/android/

This might also raise concerns for VPN users in general as this can be a path for leakage...

DNS over TLS - UDP/TCP 853 as that's a known port
DNS over HTTPS - that's a bit of a train-wreck to block, as these are common ports for other services...

Untangle - might block things at the gateway and break apps...

Gah...

 

[Xentrk]

Been experimenting with my Amazon Echo issue the past few days. I can only set it up when the Echo is assigned to the WAN interface. I then assign it to a shared VPN server. It works for awhile, then the next morning, it is offline and can't phone home. I turned off Snort to make sure it was not the cause of the issues and checked pfBlockerNG logs to make sure there are no domains listed. I suspect the order of my selective routing firewall rules may come into play. I just finished getting it back online by assigning it to the WAN interface. After successful testing, I reassigned it to my Private IP address, which is able to circumvent VPN blocks by Amazon Prime. The rule for this assignment is high up on the list as pfSense processes Firewall-LAN rules in top to bottom order. I am now beginning to think there was an issue with the order of my selective routing rules that was impacting the Echo, which is why I now have the assignment as one of the top rules in the list.

I also had rules to allow the Echo to use 8.8.8.8 and 8.8.4.4. I could tell it was matching on the 8.8.8.8 DNS but never the 8.8.4.4. rule. The rule had no impact on the issue if it was enabled or disabled. For now, I disabled it.

 

[coxhaus]

Xentrk an easy way to test would be to add a rule at the top of your ACL list for your echo to pass. IF it works then you have found the problem. I doubt blocking 8.8.8.8 is causing your problem since my echo and dots work fine with DNS 8.8.8.8 blocked.

 

[Xentrk]

Xentrk an easy way to test would be to add a rule at the top of your ACL list for your echo to pass. IF it works then you have found the problem. I doubt blocking 8.8.8.8 is causing your problem since my echo and dots work fine with DNS 8.8.8.8 blocked.

That is what I basically did. I suspect it is a similar problem I was having with Pluto TV. I had to move the rule higher on the list as a rule lower down was creating a conflict. I think I have ruled out the DNS calls to 8.8.8.8 as being an issue.

I do need to correct my post above. It appears that pfBlockerNG is blocking device-metrics-us.amazon.com. But it seems to work okay with the block. I am only seeing Echo calling these domains so far:

d3p8zr0ffa9t17.cloudfront.net
d29r7idq0wxsiz.cloudfront.net
dcape-na.amazon.com
device-metrics-us.amazon.com
pindorama.amazon.com

I was expecting more. These domains belong the Amazon AWS AS16509 farm and I already route all those to my Private IP. But the rule was in the middle of the list. Towards the end of the rules, I assign Echo to the Private IP. It will be interesting to see if moving the rule to the top will make the issue go away.

 

[coxhaus]

Maybe I should just block all the high ports. I don't game any more. It would add a level of security to my network.

 

[Xentrk]

Well, things looked good with my changes above during the day yesterday. However, this morning, Echo was not able to phone home again. As soon as I removed it from the VPN and assigned it to the WAN, it started working again. I will leave it set this way for the next 24 hours to see what happens. Appears there is something about the VPN assignment it does not like. The current test should help confirm that. Perhaps a port issue? The port of the VPN tunnel I had Echo assigned to is 1194. This post recommends it could be a timezone issue. It was getting the local time zone from my router. So I did manually set it. But the issue still occurred. Mystery continues.

The echo worked for many months before this started occurring. I have seen a few other reports on the pfsense forum. This one is of particular interest.
https://forum.netgate.com/topic/132391/amazon-echo-no-longer-working/2

But I don't like his fix, setting DNS to 8.8.8.8.

 

[Xentrk]

After assigning Echo to the WAN interface yesterday, Echo was alive and well this morning and was able to phone home. That narrows the issue to some incompatibility with the VPN tunnel. Not sure what steps to take to help determine the root cause. Perhaps a deeper look using ntopng or packet sniffing tool. The problems started about 5 months ago. The Alexa app was overhauled around the same time. Something got changed that caused the issue. I also see a few posts on other forums reporting similar issue around the same time.

 

[CaptainSTX]

I will be interested if you find the root cause of the issue and a solution. After installing my Echo six months ago I found out it wouldn't run through the VPN tunnel consistently so I had to move it back to the WAN. For now I run the Echo on its own guest network which I use for IoT devices and it is also on its own subnet.

 

[Xentrk]

I will be interested if you find the root cause of the issue and a solution. After installing my Echo six months ago I found out it wouldn't run through the VPN tunnel consistently so I had to move it back to the WAN. For now I run the Echo on its own guest network which I use for IoT devices and it is also on its own subnet.

Interesting. Your time frame appears to align when my issue started occurring. PIA users on the pfSense forum also report the issue around the same time. What the OP noted is

one of the things that it does is a DNS query for www.example.com asking for a AAAA record (an ipv6 address). If it does not get a sucessful answer back it CANNOT register with Amazon - and what the user will hear from Echo is some message that it cannot communicate to the internet. I changed my DNS configuration in PFSENSE to use google (8.8.8.8)...all 4 Echo devices immediately was able to register.

My VPN tunnel is configured for IPv4. I may need to grovel in the pfSense forum. I could fire up my Asus router and test there over the weekend to see if the problems occurs there. Maybe I can create a firewall rule for example.com or a DNS Host Override. Looks like I am going to have to fire up ntopng for more information and see if I see AAAA records being queried. Wireshark now requires a spendy price to use. But there is a free trial period I could use. Been awhile since I last used it. The above post is the best I could find on what the root cause may be.

 

[st3v3n]

Xentrk, Good idea, bet the PfSense folks will help, you won't stoop too low. Decades back, a female assistant department head accused me of groveling to the actual department head to gain a salary increase. I'd have had to grovel far over his head to corporate entities I had no wish to meet much less grovel to. (She was quite a grouse though Keep the information coming back.

 

[coxhaus]

One more piece of information is I have IPv6 turned off in my router. My layer 3 switch does not support it. So I have no IPv6 and I block 8.8.8.8. My Echo and Dots still work. I do not run VPN.

 

[Xentrk]

For debugging, I do have the checkbox checked on the System->Advanced->Networking page to allow IPv6 traffic.

Last night, I added the Cloudflare IPv6 DNS servers to Unbound in case Echo was using IPv6 DNS to phone home and reassigned it to the VPN iface. An hour or two it stopped working. Once I reassigned it back to the WAN iface, it started working again.

This morning, I fired up Wireshark and have the Echo streaming some music. I went to filter on the static IP assigned to Echo and no data appeared. I did an arp -a command and the IP address of the Echo does not show up in the list. I checked the DHCP status and the Echo is correctly detected as being online. I see DNS queries in the DNS Resolver for the IP address assigned to the Echo. This is a strange one.

 

[st3v3n]

Xentrk, Definitely strange; maybe Bezos owes you a new Echo? In jest Cheers.

 

[Xentrk]

Retrying my experiment of including the Cloudflare IPv6 servers. I forgot to add the mandatory @853 at the end. Doh!

forward-addr: 2606:4700:4700::1111@853
forward-addr: 2606:4700:4700::1001@853

Edit: Had to back out the change. Caused routing issues with streaming services.