Diversion Diversion Question- Configuring the Cache TTL

[Twiglets]

Spoke too soon []:

Final hack.

address=/xx.prod.service.minerva.devices.a2z.com/
local=/xx.prod.service.minerva.devices.a2z.com/

These entries will filter out all queries that include "xx.prod.service.minerva.devices.a2z.com" in the query.

This should match your insane client.

Repeat the pattern with anything else it is trying to query infintely.

address=/<whatever>/
local=/<whatever>/

I will really butt out now !!!!!

 

[snapieee]

Yikes! How frequent are the black holed queries in comparison?

About 10-20 min after restarting dnsmasq, and then it is non-stop for a few min - didn't count that one.

If you run top on the router, presumably it's dnsmasq that's using all the CPU?

It seems to be bouncing between these processes: {diversion} /bin/sh /opt/bin/diversion, dnsmasq --log-async, and networkmap --bootwait.

At first, I was watching CPU usage through the GUI, but I stopped doing that so I could get a more objective view using top. When I noticed Diversion climbing the list, I initially thought it might be related to me actively viewing or tailing the dnsmasq log. To test that, I paused the log (Ctrl+C), and almost immediately Diversion dropped back down and CPU usage returned to normal. As soon as I unpause it, CPU spikes again and Diversion shoots back up the list.

What’s interesting is that I don’t see any spikes at all when filtering entries via 0.0.0.0.

That said, this isn’t a constant thing. If it were happening 100% of the time, my router would’ve melted by now . It shows up roughly every 10 minutes. Sometimes it stops on its own, and other times it just keeps running nonstop for who knows how long.


On a side note, I previously had divstats installed, and the query count for that domain was around 1.5 million in just two days. That honestly caught my attention pretty fast. I ended up uninstalling divstats because I was worried that, as the database grows, it could turn into a bigger problem over time.

That’s actually what pushed me toward handling the filtering directly through dnsmasq.conf.add instead.

EDIT:
When filtering using 0.0.0.0 and/or ::1, there are no CPU spikes at all—even while I’m actively viewing or tailing the log through Diversion.

If you don't use IPv6 you can put in dnsmasq.conf.add the following:

filter-AAAA

It filters out all IPv6 addresses returned, not sure if that would include NXDOMAIN from IPv6 queries !!!???

The real problem is the device that is generating all these queries ... where are they coming from ???
Can you 'kill' the app that is responsible ???

PS. Sorry for cross posting ... I will butt out !!!

I really don't really use ipv6 but i'll consider this option.

Spoke too soon []:

Final hack.

address=/xx.prod.service.minerva.devices.a2z.com/
local=/xx.prod.service.minerva.devices.a2z.com/

These entries will filter out all queries that include "xx.prod.service.minerva.devices.a2z.com" in the query.

This should match your insane client.

Repeat the pattern with anything else it is trying to query infintely.

address=/<whatever>/
local=/<whatever>/

I will really butt out now !!!!!

That's the first thing I have tried, but i opted in doing minerva.devices.a2z.com to cover more sub-domains just in case.

 

[snapieee]

filter-AAAA

I just add it in a single line to dnsmasq.conf.add, and it is seems to be working as I don't see any aaaa queries/replies in dnsmasq.

The real problem is the device that is generating all these queries ... where are they coming from ???
Can you 'kill' the app that is responsible ???

I’m pretty sure it’s one of my Amazon smart devices — I just haven’t figured out exactly which one yet. Normally this would be easy to spot by checking the IP in dnsmasq, but after upgrading to 3006, GuestNetworkPro traffic wasn’t being filtered by Diversion at all.

I ended up using a workaround to get filtering working again. The downside is that all DNS queries from GuestNetworkPro now show up in dnsmasq as if they’re coming from the main router IP, which makes tracking the actual device a lot harder.

Here’s the workaround I’m using:

#!/bin/sh

CONFIG="$1"
. /usr/sbin/helper.sh
pc_delete "servers-file=" "$CONFIG"
pc_append "server=192.168.x.x" "$CONFIG"

 

[Twiglets]

I just add it in a single line to dnsmasq.conf.add, and it is seems to be working as I don't see any aaaa queries/replies in dnsmasq.


I’m pretty sure it’s one of my Amazon smart devices — I just haven’t figured out exactly which one yet. Normally this would be easy to spot by checking the IP in dnsmasq, but after upgrading to 3006, GuestNetworkPro traffic wasn’t being filtered by Diversion at all.

I ended up using a workaround to get filtering working again. The downside is that all DNS queries from GuestNetworkPro now show up in dnsmasq as if they’re coming from the main router IP, which makes tracking the actual device a lot harder.

Here’s the workaround I’m using:

#!/bin/sh

CONFIG="$1"
. /usr/sbin/helper.sh
pc_delete "servers-file=" "$CONFIG"
pc_append "server=192.168.x.x" "$CONFIG"
[/QUOTE]

I just add it in a single line to dnsmasq.conf.add, and it is seems to be working as I don't see any aaaa queries/replies in dnsmasq.


I’m pretty sure it’s one of my Amazon smart devices — I just haven’t figured out exactly which one yet. Normally this would be easy to spot by checking the IP in dnsmasq, but after upgrading to 3006, GuestNetworkPro traffic wasn’t being filtered by Diversion at all.

I ended up using a workaround to get filtering working again. The downside is that all DNS queries from GuestNetworkPro now show up in dnsmasq as if they’re coming from the main router IP, which makes tracking the actual device a lot harder.

Here’s the workaround I’m using:

#!/bin/sh

CONFIG="$1"
. /usr/sbin/helper.sh
pc_delete "servers-file=" "$CONFIG"
pc_append "server=192.168.x.x" "$CONFIG"

I know I said I would butt out ... BUT this is annoying me , I want to fix it if I can understand what is going on !!!

What file is this ???

I want to understand what your workaround has changed from the normal setup !!!

 

[dave14305]

What file is this ???

I want to understand what your workaround has changed from the normal setup !!!

It looks like it’s /jffs/scripts/dnsmasq-sdn.postconf reconfiguring each Guest dnsmasq instance to forward requests to the main LAN dnsmasq instance where Diversion is active.

 

[Twiglets]

@snapieee could you provide a copy of the following files:
[Not sure where they 'should be' as I have 'hacked' my setup a lot and the files are now in multiple places ]

Trying to see all the things that impact the final dnsmasq.conf in /etc ... where the changes are coming from, what addresses are predefined in the hosts file & if the resolv files are changing anything !!!

/etc/dnsmasq.conf <=== the final version of dnsmasq after being processed by various scripts
/tmp/resolv.conf ... might be in /etc
/tmp/resolv.dnsmasq ... might be in /etc
/tmp/resolv.dnsmasq.sdn? ... might be multiple files
/jffs/scripts/dnsmasq-sdn.postconf
/jffs/scripts/dnsmasq.postconf
/jffs/configs/dnsmasq.conf.add
/jffs/configs/hosts
/jffs/configs/hosts.dnsmasq

P.S. What addons have you installed on your router and versions ???

 

[snapieee]

P.S. What addons have you installed on your router and versions ???

Diversion 5.4.6, Skynet 8.0.7, FlexQoS 1.5.2, scMerlin 2.5.47, YazDHCP 1.2.4, Router date keeper (no version showing)

 

[snapieee]

@snapieee could you provide a copy of the following files:

/etc/dnsmasq.conf <=== the final version of dnsmasq after being processed by various scripts

/etc/dnsmasq.conf output: cat /etc/dnsmasq.conf
=========================

pid-file=/var/run/dnsmasq.pid
user=nobody
bind-dynamic
interface=br0
interface=pptp*
no-dhcp-interface=pptp*
no-resolv
servers-file=/tmp/resolv.dnsmasq
no-poll
no-negcache
cache-size=1500
min-port=4096
addn-hosts=/etc/ hosts
dns-forward-max=1500
bogus-priv
domain-needed
dhcp-range=lan,192.168.x.x,192.168.x.254,255.255.255.0,86400s
dhcp-option=lan,3,192.168.x.x
dhcp-option=lan,252,"\n"
dhcp-authoritative
interface=tun21
no-dhcp-interface=tun22
no-dhcp-interface=tun23
interface=wgs1
no-dhcp-interface=wgs1
address=/use-application-dns.net/
address=/_dns.resolver.arpa/
address=/mask.icloud.com/mask-h2.icloud.com/
dhcp-name-match=set:wpad-ignore,wpad
dhcp-ignore-names=tag:wpad-ignore
dhcp-script=/sbin/dhcpc_lease
script-arp
edns-packet-max=1232

address=/minerva.devices.a2z.com/0.0.0.0 #stop_dns_flood_ipv4#
address=/minerva.devices.a2z.com/::1 #stop_dns_flood_ipv6#
address=/global.telemetry.insights.video.a2z.com/0.0.0.0 #stop_dns_flood_ipv4#
address=/global.telemetry.insights.video.a2z.com/::1 #stop_dns_flood_ipv6#
local-ttl=3600 #dnsmasq_cache_ttl#
neg-ttl=7200 #NXDOMAIN_cache_ttl#
dhcp-hostsfile=/jffs/addons/YazDHCP.d/.staticlist #YazDHCP_staticlist#
addn-hosts=/jffs/addons/YazDHCP.d/.hostnames #YazDHCP_hostnames#
ipset=/0.north-america.pool.ntp.org/api.us-east-1.aiv-delivery.net/asuswrt-merlin.net/asuswrt.lostrealm.ca/big.oisd.nl/bin.entware.net/cdn.jsdelivr.net/codeload.github.com/diversion.ch/drv.ms/entware.diversion.ch/entware.net/fls-na.amazon.com/fwupdate.asuswrt-merlin.net/garycnew.github.io/localhost.localdomain/maurerr.github.io/Skynet-WhitelistDomains # Skynet
ipset=/mirrors.bfsu.edu.cn/mirrors.cernet.edu.cn/mirrors.cqupt.edu.cn/mirrors.nju.edu.cn/oisd.nl/onedrive.live.com/openstreetmap.org/pgl.yoyo.org/pkg.entware.net/pool.ntp.org/raw.githubusercontent.com/small.oisd.nl/snbforums.com/someonewhocares.org/sourceforge.net/sunrisesunset.io/urlhaus.abuse.ch/us.pool.ntp.org/Skynet-WhitelistDomains # Skynet
ipset=/ipdeny.com/ipapi.co/api.db-ip.com/api.bgpview.io/asn.ipinfo.app/speedguide.net/otx.alienvault.com/github.com/iplists.firehol.org/astrill.com/strongpath.net/nwsrv-ns1.asus.com/0.us.pool.ntp.org/1.us.pool.ntp.org/2.us.pool.ntp.org/Skynet-WhitelistDomains # Skynet

# start of Diversion directives #
conf-file=/opt/share/diversion/list/allowlist.conf
conf-file=/opt/share/diversion/list/blockinglist.conf
conf-file=/opt/share/diversion/list/denylist.conf
log-async
log-queries
log-facility=/opt/var/log/dnsmasq.log
# end of Diversion directives #

===========================================================================================================
/tmp/resolv.conf output: cat /tmp/resolv.conf
========================

nameserver 1.1.1.1
nameserver 1.0.0.1

===========================================================================================================
/tmp/resolv.dnsmasq output: cat /tmp/resolv.dnsmasq
===========================

server=127.0.1.1

===========================================================================================================
/tmp/resolv.dnsmasq.sdn? output:
================================

> cat /tmp/resolv.dnsmasq.sdn0

server=1.1.1.1
server=1.0.0.1

> cat /tmp/resolv.dnsmasq.sdn1

server=1.1.1.1
server=1.0.0.1

> cat /tmp/resolv.dnsmasq.sdn2

server=1.1.1.1
server=1.0.0.1

===========================================================================================================
/jffs/scripts/dnsmasq-sdn.postconf output:
========================================== cat /jffs/scripts/dnsmasq-sdn.postconf

#!/bin/sh

CONFIG="$1"
. /usr/sbin/helper.sh
pc_delete "servers-file=" "$CONFIG"
pc_append "server=192.168.x.x" "$CONFIG"

===========================================================================================================
/jffs/scripts/dnsmasq.postconf output:
====================================== cat /jffs/scripts/dnsmasq.postconf

#!/bin/sh

. /opt/share/diversion/file/post-conf.div # Added by Diversion

===========================================================================================================
/jffs/configs/dnsmasq.conf.add output:
====================================== cat /jffs/scripts/dnsmasq.postconf

address=/minerva.devices.a2z.com/0.0.0.0 #stop_dns_flood_ipv4#
address=/minerva.devices.a2z.com/::1 #stop_dns_flood_ipv6#
address=/global.telemetry.insights.video.a2z.com/0.0.0.0 #stop_dns_flood_ipv4#
address=/global.telemetry.insights.video.a2z.com/::1 #stop_dns_flood_ipv6#
local-ttl=3600 #dnsmasq_cache_ttl#
neg-ttl=7200 #NXDOMAIN_cache_ttl#
dhcp-hostsfile=/jffs/addons/YazDHCP.d/.staticlist #YazDHCP_staticlist#
addn-hosts=/jffs/addons/YazDHCP.d/.hostnames #YazDHCP_hostnames#
ipset=/0.north-america.pool.ntp.org/api.us-east-1.aiv-delivery.net/asuswrt-merlin.net/asuswrt.lostrealm.ca/big.oisd.nl/bin.entware.net/cdn.jsdelivr.net/codeload.github.com/diversion.ch/drv.ms/entware.diversion.ch/entware.net/fls-na.amazon.com/fwupdate.asuswrt-merlin.net/garycnew.github.io/localhost.localdomain/maurerr.github.io/Skynet-WhitelistDomains # Skynet
ipset=/mirrors.bfsu.edu.cn/mirrors.cernet.edu.cn/mirrors.cqupt.edu.cn/mirrors.nju.edu.cn/oisd.nl/onedrive.live.com/openstreetmap.org/pgl.yoyo.org/pkg.entware.net/pool.ntp.org/raw.githubusercontent.com/small.oisd.nl/snbforums.com/someonewhocares.org/sourceforge.net/sunrisesunset.io/urlhaus.abuse.ch/us.pool.ntp.org/Skynet-WhitelistDomains # Skynet
ipset=/ipdeny.com/ipapi.co/api.db-ip.com/api.bgpview.io/asn.ipinfo.app/speedguide.net/otx.alienvault.com/github.com/iplists.firehol.org/astrill.com/strongpath.net/nwsrv-ns1.asus.com/0.us.pool.ntp.org/1.us.pool.ntp.org/2.us.pool.ntp.org/Skynet-WhitelistDomains # Skynet

===========================================================================================================
/jffs/configs/hosts output:
=========================== cat /jffs/configs/hosts

cat: can't open '/jffs/configs/hosts': No such file or directory

===========================================================================================================
/jffs/configs/hosts.dnsmasq output:
=================================== cat /jffs/configs/hosts.dnsmasq

cat: can't open '/jffs/configs/hosts.dnsmasq': No such file or directory

 

[Twiglets]

Thanks.
I will look at the files soon.

It is currently 0800 and I am still not awake fully ... need coffee, food, dog walked etc.

P.S. Happy Holidays to All, hope you all got something nice from Santa !!!

 

[Twiglets]

1st thing I noticed is that in the dnsmasq.conf file you have:

addn-hosts=/etc/ hosts

Remove the space !!!

Could you send a copy of the hosts file & hosts.dnsmasq if it exists.

 

[Twiglets]

2nd thing I noticed is that all your resolve files are set to use external DNS Servers !!!

Shouldn't you be using your router ???

i.e. the resolv files should be redirecting to whatever dnsmasq is setup to listen on.

I will look properly after 'Doggie time'. i.e. his walk for a couple of hours.

To correct my locations ... all the files that are used such as resolv, hosts, dnsmasq.conf etc should be in /etc.
(They are in other locations as well when they are created/manipulated by scripts etc, hence my variation on locations. Sorry to confuse.)

 

[ColinTaylor]

1st thing I noticed is that in the dnsmasq.conf file you have:

addn-hosts=/etc/ hosts

Remove the space !!!

He had to insert the space for the forum to allow it to be posted.

Could you send a copy of the hosts file & hosts.dnsmasq if it exists.

He did. They don't exist (under /jffs/configs). hosts.dnsmasq doesn't exist under /etc either.

2nd thing I noticed is that all your resolve files are set to use external DNS Servers !!!

Shouldn't you be using your router ???

No. That is correct.

This is getting way off topic. Perhaps it's best if you continue your conversation by direct message?

 

[jrichard326]

Regarding Amazon/Minerva , this worked for me. Now minerva only tries to make contact once per hour. Note that my router is off during a portion of the night which accounts for the time gap. Thanks.

address=/minerva.devices.a2z.com/0.0.0.0 #stop_dns_flood_ipv4#
address=/minerva.devices.a2z.com/::1 #stop_dns_flood_ipv6#
local-ttl=3600 #dnsmasq_cache_ttl#
neg-ttl=7200 #NXDOMAIN_cache_ttl#