Diversion Diversion 5.1.2 - the Router Ad-Blocker, April 21, 2024

[Cam]

When I select the Small or Minimal lists, dnsmasq does not reload due to a stray quote appearing in the blocklist - notably local=/rss+xml"/

Looking at the files downloaded, I assume this has come from /opt/share/diversion/backup/hostsfile_pgl.yoyo.org-as-serverlist.php-hostformat which includes html code with the content <link rel=alternate type="application/rss+xml" etc...> that I think might be pulled in by mistake.

This is with diversion force updated after the latest whitelisting fix.

Spoiler: a 'fix'

As a temp fix in case this affects other people and in case @thelonelycoder is away, I updated the grep statement in line 487 of update-bl.div to

/opt/bin/grep -E "^(local|server)=" "${hf_inuse}" | cut -d"/" -f2 \

 

[SomeWhereOverTheRainBow]

I'm going to have to refresh my memory/understanding of dnsmasq and unbound to sort out the differences in my head, because right now it seems I've both running, and both are supposed to be caching. As I currently (and likely mistakenly) recall, while dnsmasq might be faster, it's not recursive like unbound. and there's no AND for the two - it's an either/or proposition...so back to manpages unless somebody wants to jump in and tell me to just comment something out somewhere....

If you really want to know, unbound manager has two options to run unbound.

1) Dnsmasq acting as dns at port 53 forwarding request to unbound (which uses a different port other than port 53) as it's upstream

2) dnsmasq not servicing dns, but dhcp only while unbound listens on port 53 servicing dns requests directly which is only "advertised" by dnsmasq to clients as a dhcp option. In this role, dnsmasq only services DHCP and not DNS requests.

If you chose to setup unbound using unbound managers 2nd choice, then your dns requests never make it to dnsmasq to be serviced hence why diversion would be useless.

If you chose the 1st choice dnsmasq services your dns requests before they are forwarded to be serviced by unbound. Diversion will act as the first blocking layer. You may not even see unbound adblock block anything unless your lists between the two do not over lap.

To answer the original question:- I suppose this will not work together with Unbound as DNS resolver and disabled dnsmasq?

Correct, it does bypass diversion hence why you should choose unbound adblock in this scenario and not diversion.

However, if you switch to the 1st option, I would only use diversion since it would now be pointless to adblock using unbound adblock options when it would be easier to manage using diversion.

Hopefully this clears up @Milan confusion about why it would be pointless.

@heysoundude it is not necessarily about which one is caching in this scenario, it is about which one is blocking the request first. If dnsmasq receives the dns request first and diversion is running - it is able to block first which means the blocked response is cached there and never goes to unbound. If dnsmasq dns is disabled (i.e. dns port set to 0), then it only serves dhcp options to clients which means unbound must be servicing the dns requests first at port 53 hence why diversion for adblocking would not work since diversion relies on dnsmasq to service block dns requests.

You are probably using option 1 which means dnsmasq is forwarding requests to unbound which explains why both cache are being used in your scenario.

 

[thelonelycoder]

Going to the "Canaries"... for "work"... <Ahem> Riiiiiggghhhht.

Thank you for all your hard work on this, @thelonelycoder! We'll be here when you get back! Don't worry about us... we got this!!

It‘s not for fun when your told to go

 

[Milan]

If you really want to know, unbound manager has two options to run unbound.

1) Dnsmasq acting as dns at port 53 forwarding request to unbound (which uses a different port other than port 53) as it's upstream

2) dnsmasq not servicing dns, but dhcp only while unbound listens on port 53 servicing dns requests directly which is only "advertised" by dnsmasq to clients as a dhcp option. In this role, dnsmasq only services DHCP and not DNS requests.

If you chose to setup unbound using unbound managers 2nd choice, then your dns requests never make it to dnsmasq to be serviced hence why diversion would be useless.

If you chose the 1st choice dnsmasq services your dns requests before they are forwarded to be serviced by unbound. Diversion will act as the first blocking layer. You may not even see unbound adblock block anything unless your lists between the two do not over lap.

To answer the original question:- I suppose this will not work together with Unbound as DNS resolver and disabled dnsmasq?

Correct, it does bypass diversion hence why you should choose unbound adblock in this scenario and not diversion.

However, if you switch to the 1st option, I would only use diversion since it would now be pointless to adblock using unbound adblock options when it would be easier to manage using diversion.

Hopefully this clears up @Milan confusion about why it would be pointless.

@heysoundude it is not necessarily about which one is caching in this scenario, it is about which one is blocking the request first. If dnsmasq receives the dns request first and diversion is running - it is able to block first which means the blocked response is cached there and never goes to unbound. If dnsmasq dns is disabled (i.e. dns port set to 0), then it only serves dhcp options to clients which means unbound must be servicing the dns requests first at port 53 hence why diversion for adblocking would not work since diversion relies on dnsmasq to service block dns requests.

You are probably using option 1 which means dnsmasq is forwarding requests to unbound which explains why both cache are being used in your scenario.

Yes, it answers my question. So now I think I will enable dnsmasq DNS again and enable Diversion here. Merging some other AdBlock list to unbound is a pain in the butt ...

 

[Amwjujo]

Haven't installed 5 yet but for adult content, why not use DNS filtering in the WAN section? https://cleanbrowsing.org/filters/ is available or you can enter other ones

Haven't installed 5 yet but for adult content, why not use DNS filtering in the WAN section? https://cleanbrowsing.org/filters/ is available or you can enter other ones manually.

Will this work with Diversion? Having WAN dns filter set on cleanbrowsing.com and Diversion as adblocker?

 

[martinr]

Will this work with Diversion? Having WAN dns filter set on cleanbrowsing.com and Diversion as adblocker?

Sure: just think of it as layers of defence or successive filters. If a request to visit a site you want to be protected from doesn’t get blocked before it leaves your router (by Diversion), the next port of call is the Cleanbrowsing DNS server you have set, where, hopefully, the unwanted request would get blocked.

 

[Amwjujo]

Sure: just think of it as layers of defence or successive filters. If a request to visit a site you want to be protected from doesn’t get blocked before it leaves your router (by Diversion), the next port of call is the Cleanbrowsing DNS server you have set, where, hopefully, the unwanted request would get blocked.

Great. Thank you for explaining. I'll give it a go soon.
One last question while i am here, as I've seen you guys keep debating about it: Will Unbound still fit in this scenario as a resolver?

 

[Milan]

Great. Thank you for explaining. I'll give it a go soon.
One last question while i am here, as I've seen you guys keep debating about it: Will Unbound still fit in this scenario as a resolver?

Running here this scenario.
Unbound as resolver, dnsmasq with diversion is forwarding requests to unbound ...

 

[Viktor Jaep]

Running here this scenario.
Unbound as resolver, dnsmasq with diversion is forwarding requests to unbound ...

Same here... lookup path using Diversion -> Unbound -> Across VPN Tunnel -> DNS Root Servers... all working well!

 

[kuki68ster]

Running here this scenario.
Unbound as resolver, dnsmasq with diversion is forwarding requests to unbound ...

Could you show me your config for this setup?

I use diversion and unbound in default installation... Curious to learn if there is some setting or parameter I need to change...

 

[heysoundude]

If you really want to know, unbound manager has two options to run unbound.

1) Dnsmasq acting as dns at port 53 forwarding request to unbound (which uses a different port other than port 53) as it's upstream

2) dnsmasq not servicing dns, but dhcp only while unbound listens on port 53 servicing dns requests directly which is only "advertised" by dnsmasq to clients as a dhcp option. In this role, dnsmasq only services DHCP and not DNS requests.

If you chose to setup unbound using unbound managers 2nd choice, then your dns requests never make it to dnsmasq to be serviced hence why diversion would be useless.

If you chose the 1st choice dnsmasq services your dns requests before they are forwarded to be serviced by unbound. Diversion will act as the first blocking layer. You may not even see unbound adblock block anything unless your lists between the two do not over lap.

To answer the original question:- I suppose this will not work together with Unbound as DNS resolver and disabled dnsmasq?

Correct, it does bypass diversion hence why you should choose unbound adblock in this scenario and not diversion.

However, if you switch to the 1st option, I would only use diversion since it would now be pointless to adblock using unbound adblock options when it would be easier to manage using diversion.

Hopefully this clears up @Milan confusion about why it would be pointless.

@heysoundude it is not necessarily about which one is caching in this scenario, it is about which one is blocking the request first. If dnsmasq receives the dns request first and diversion is running - it is able to block first which means the blocked response is cached there and never goes to unbound. If dnsmasq dns is disabled (i.e. dns port set to 0), then it only serves dhcp options to clients which means unbound must be servicing the dns requests first at port 53 hence why diversion for adblocking would not work since diversion relies on dnsmasq to service block dns requests.

You are probably using option 1 which means dnsmasq is forwarding requests to unbound which explains why both cache are being used in your scenario.

Thank you! I hadn't had the time to look it up/grok it. This is the perfect explanation.

 

[martinr]

Same here... lookup path using Diversion -> Unbound -> Across VPN Tunnel -> DNS Root Servers... all working well!

Other than getting an instant resolution on previously-visited sites (instead of, say, a 20ms lookup time), are there any distinct advantages to running Unbound (with Unbound Manager)?

 

[thelonelycoder]

When I select the Small or Minimal lists, dnsmasq does not reload due to a stray quote appearing in the blocklist - notably local=/rss+xml"/

Looking at the files downloaded, I assume this has come from /opt/share/diversion/backup/hostsfile_pgl.yoyo.org-as-serverlist.php-hostformat which includes html code with the content <link rel=alternate type="application/rss+xml" etc...> that I think might be pulled in by mistake.

This is with diversion force updated after the latest whitelisting fix.

Spoiler: a 'fix'

As a temp fix in case this affects other people and in case @thelonelycoder is away, I updated the grep statement in line 487 of update-bl.div to

/opt/bin/grep -E "^(local|server)=" "${hf_inuse}" | cut -d"/" -f2 \

Now I know where this is coming from. Seen it once. Thanks for reporting.

 

[martinr]

Now I know where this is coming from. Seen it once. Thanks for reporting.

How are the Canaries? Warm and sunny?

 

[Viktor Jaep]

Other than getting an instant resolution on previously-visited sites (instead of, say, a 20ms lookup time), are there any distinct advantages to running Unbound (with Unbound Manager)?

Without Unbound, all your unencrypted DNS lookup traffic can be watched by your ISP as soon as it leaves your WAN connection... and obviously can be watched by your DNS provider. If you're using DoT, your DNS query traffic will not be visible to your ISP, but can be monitored by your DNS provider.

So with Unbound:
1.) You are not reliant, or being tracked by your DNS provider since you are your own DNS Resolver, and connecting directly with DNS Root Servers.
2.) Out-of-the box, your ISP would be able to track your unencrypted Unbound traffic going to DNS Root Servers
3.) Enabling the functionality to pipe your Unbound traffic over your VPN tunnel prevents your ISP from being able to track your DNS traffic, and at that point, the only unencrypted port 53 traffic would be going from your public VPN IP directly to the DNS Root Servers. Would that get tracked by your VPN provider? Probably not depending on who you use, but that's the risk you take.

 

[thelonelycoder]

How are the Canaries? Warm and sunny?

26 degrees Celsius, blue sky, not bad when coming from -10. We call them the islands with the never ending spring.

 

[Viktor Jaep]

26 degrees Celsius, blue sky, not bad when coming from -10. We call them the islands with the never ending spring.

Hope you get to code on the beach!!

 

[kuki68ster]

Same here... lookup path using Diversion -> Unbound -> Across VPN Tunnel -> DNS Root Servers... all working well!

Could you show me your config for this setup?

I use diversion and unbound in default installation... Curious to learn if there is some setting or parameter I need to change...

I only use Diversion + Unbound, without VPN involved (even with the closest VPN server, my ping would be in the 80s)...

 

[Viktor Jaep]

Could you show me your config for this setup?

I use diversion and unbound in default installation... Curious to learn if there is some setting or parameter I need to change...

I only use Diversion + Unbound, without VPN involved (even with the closest VPN server, my ping would be in the 80s)...

Diversion and Unbound are both vanilla setups. The only changes being made to my unbound.conf file is when my VPN private IP changes after reconnecting so it knows where to send DNS Query traffic. I don't believe I've made any other changes, but I'd be happy to send that to you via PM?

Then, using a combination of (1) looking at the live raw Unbound log traffic (enabling "lo" under Unbound to show full logging), (2) following the dnsmasq log in Diversion (using "f" option), and seeing that lookups are using port #53535, the (3) always helpful dnscheck.tools website, along with (4) Eibgrad's DNS Monitor script... all of these can help you determine if things are working the way you want them to when it comes to DNS.

Example from Diversion dnsmasq log:

Jan  3 15:34:47 dnsmasq[13563]: forwarded connectivitycheck.gstatic.com to 127.0.0.1#53535

Example from Unbound showing full logs:

Jan 03 11:37:11 unbound[13297:0] query: 127.0.0.1 connectivitycheck.gstatic.com. A IN
Jan 03 11:37:11 unbound[13297:0] reply: 127.0.0.1 connectivitycheck.gstatic.com. A IN NOERROR 0.000000 1 63
Jan 03 11:37:11 unbound[13297:0] query: 127.0.0.1 connectivitycheck.gstatic.com. AAAA IN
Jan 03 11:37:11 unbound[13297:0] reply: 127.0.0.1 connectivitycheck.gstatic.com. AAAA IN NOERROR 0.000000 1 47

 

[kuki68ster]

Diversion and Unbound are both vanilla setups. The only changes being made to my unbound.conf file is when my VPN private IP changes after reconnecting so it knows where to send DNS Query traffic. I don't believe I've made any other changes, but I'd be happy to send that to you via PM?

Then, using a combination of (1) looking at the live raw Unbound log traffic (enabling "lo" under Unbound to show full logging), (2) following the dnsmasq log in Unbound (using "f" option), and seeing that lookups are using port #53535, the (3) always helpful dnscheck.tools website, along with (4) Eibgrad's DNS Monitor script... all of these can help you determine if things are working the way you want them to when it comes to DNS.

Example from Diversion dnsmasq log:

Jan  3 15:34:47 dnsmasq[13563]: forwarded connectivitycheck.gstatic.com to 127.0.0.1#53535

Example from Unbound showing full logs:

Jan 03 11:37:11 unbound[13297:0] query: 127.0.0.1 connectivitycheck.gstatic.com. A IN
Jan 03 11:37:11 unbound[13297:0] reply: 127.0.0.1 connectivitycheck.gstatic.com. A IN NOERROR 0.000000 1 63
Jan 03 11:37:11 unbound[13297:0] query: 127.0.0.1 connectivitycheck.gstatic.com. AAAA IN
Jan 03 11:37:11 unbound[13297:0] reply: 127.0.0.1 connectivitycheck.gstatic.com. AAAA IN NOERROR 0.000000 1 47

Thanks for the reply...

Diversion and Unbound are both vanilla setups. The only changes being made to my unbound.conf file is when my VPN private IP changes after reconnecting so it knows where to send DNS Query traffic. I don't believe I've made any other changes, but I'd be happy to send that to you via PM?

At this moment, I am going to use it in the vanilla mode only ...No need to go that deep yet Thanks