Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[Centrifuge]

I can confirm: no nightly drop with Diversion:

I have stat envy. Here's mine:

 

[Martineau]

Never used it. I notice that logorotate runs at the time in question.

12 4 * * * curl -o \/opt\/var\/lib\/unbound\/root\.hints https://www.internic.net/domain/named.cache #root_servers#
 0 5 * * * /opt/var/lib/unbound/adblock/gen_adblock.sh #adblock#
<snip>
 5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#

Are you sure?

 

[Centrifuge]

cru d adblock

That sucker is gone, thanks.

 

[Centrifuge]

Are you sure?

I must of hit y, when I should have hit enter duh...thanks guys.

 

[Martineau]

I had to change the syslog-ng config for unbound to be this:

filter f_unbound {                                                                                                                                                       
    program("unbound") or                                                                                                                                               
    program("gen_adblock");                                                                                                                                             
};

Now my script outputs to the same log.

I assumed you were going to override/force the "unbound" program name tag rather than use "$(basename $0)" in 'gen_adblock.sh'

NOTE: I'll ensure 'unbound_manager' adds the additional syslog-ng clause.

program("gen_adblock");

I am ready to push an update, but need a recommendation. Stay with "restart" or change to "reload"? reload is faster, but both reset caches.

I'd recommend 'reload' as this means users can boast how long unbound has been up

unbound (pid 19728) is running... uptime: 9 Days, 25:65:90 version: 1.9.6 # rgnldo Github Version=v1.07 Martineau update (Date Loaded by unbound_manager Sat Mar 14 05:00 GMT 2020)

but obviously if using Ad Block, the 'Loaded' timestamp will always be '05:00'

If I want to keep firefox_doh enabled, or CPU tweaks, do I need to remember to enter in more numbers like "i 3 5 6" .

Yes

 

[dave14305]

I must of hit y, when I should have hit enter duh...thanks guys.

It’s worth double-checking your unbound.conf too to make sure you’re not including the Adblock file which will become stale from now on without the job. And did you remove from services-start so it doesn’t recreate the cron job on the next reboot?

 

[Martineau]

I'm concerned about the complaints that the cache is not being retained.

My definition of a cache reset is that every counter is reset to zero e.g. rrset.cache.count=0

So cache stats before performing the reload

e  = Exit Script

A:Option ==> s

total.num.queries=13448             total.num.zero_ttl=1366             total.requestlist.exceeded=0            total.tcpusage=0
total.num.queries_ip_ratelimited=0  total.num.recursivereplies=527      total.requestlist.current.all=0         msg.cache.count=1267
total.num.cachehits=12921           total.requestlist.avg=0.517007      total.requestlist.current.user=0        rrset.cache.count=4477
total.num.cachemiss=527             total.requestlist.max=11            total.recursion.time.avg=0.105167       infra.cache.count=1087
total.num.prefetch=1678             total.requestlist.overwritten=0     total.recursion.time.median=0.0493009   key.cache.count=166

Summary: Cache Hits success=96.00%

So in the few seconds it took for me to issue the reload

./unbound_manager reload config=

Reloading 'unbound.conf' <<== /opt/var/lib/unbound/unbound.conf status=ok

the cache stats have indeed changed:

e  = Exit Script

A:Option ==> s

total.num.queries=77                total.num.zero_ttl=0                total.requestlist.exceeded=0            total.tcpusage=0
total.num.queries_ip_ratelimited=0  total.num.recursivereplies=10       total.requestlist.current.all=0         msg.cache.count=548
total.num.cachehits=67              total.requestlist.avg=0.0833333     total.requestlist.current.user=0        rrset.cache.count=3713
total.num.cachemiss=10              total.requestlist.max=1             total.recursion.time.avg=0.269343       infra.cache.count=1087
total.num.prefetch=2                total.requestlist.overwritten=0     total.recursion.time.median=0.065536    key.cache.count=15

Summary: Cache Hits success=87.00%

i.e. the four cache metrics

       Before                     After
msg.cache.count=1267       msg.cache.count=548             PCT change -56.75%
rrset.cache.count=4477     rrset.cache.count=3713          PCT change -17.10%
infra.cache.count=1087     infra.cache.count=1087          PCT change      0%
key.cache.count=166        key.cache.count=15              PCT change -90.97%

Am I correctly understanding the issue or am I missing something?

 

[juched]

I believe I found a way to reload the adblock list without restart or reload. Will share what I find.

 

[SomeWhereOverTheRainBow]

I believe I found a way to reload the adblock list without restart or reload. Will share what I find.

I think I know what you have found but I I won't steal your thunder. I am so thankful to have you, @Martineau, and @Jack Yaz around maintaining this project. You guys do a brilliant job.

 

[Centrifuge]

remove from services-start

Yes done. I could'nt see anything in unbound.conf, only the adblock blacklist stuff and its commented out. thanks.

 

[juched]

I know there have been too many updates, sorry.

However, I pushed v1.0.6 of gen_adblock.sh.

- it uses a new method to remove and push new zones dynamically, meaning the nightly update doesn't reload or restart unbound, no more cache hit drops!
- uses logger so there is output in the logs in the UI for when it runs with results.
- if line begins with # in allowsites or blocksites, that URL is skipped (keep old lists around without having to delete the line)

please post if you have issues with the new way of reloading. In my limited testing it works fairly well, but there may be edge cases.

To install run the "i" command and pick your options.

 

[SomeWhereOverTheRainBow]

I know there have been too many updates, sorry.

However, I pushed v1.0.6 of gen_adblock.sh.

- it uses a new method to remove and push new zones dynamically, meaning the nightly update doesn't reload or restart unbound, no more cache hit drops!
- uses logger so there is output in the logs in the UI for when it runs with results.
- if line begins with # in allowsites or blocksites, that URL is skipped (keep old lists around without having to delete the line)

please post if you have issues with the new way of reloading. In my limited testing it works fairly well, but there may be edge cases.

To install run the "i" command and pick your options.

great job man

 

[ika]

I know there have been too many updates, sorry.

However, I pushed v1.0.6 of gen_adblock.sh.

- it uses a new method to remove and push new zones dynamically, meaning the nightly update doesn't reload or restart unbound, no more cache hit drops!
- uses logger so there is output in the logs in the UI for when it runs with results.
- if line begins with # in allowsites or blocksites, that URL is skipped (keep old lists around without having to delete the line)

please post if you have issues with the new way of reloading. In my limited testing it works fairly well, but there may be edge cases.

To install run the "i" command and pick your options.

Reporting an issue: Now I can't enable addblock anymore with the script. The whole script is full of the "SSL handshake failed" errors, and if I select adblock to be enabled, the scripts always fails with an"ERROR unbound-control - failed?" message at the end (after answering the redownload/keep config question).

 

[Martineau]

Reporting an issue: Now I can't enable addblock anymore with the script. The whole script is full of the "SSL handshake failed" errors, and if I select adblock to be enabled, the scripts always fails with an"ERROR unbound-control - failed?" message at the end (after answering the redownload/keep config question).

Try restarting unbound

e  = Exit Script

A:Option ==> rs

 

[ika]

Try restarting unbound

e  = Exit Script

A:Option ==> rs

Rs restarts unbound but the adblock is not installed by the script (see pics)

edit: also already tried uninstalling unbound and reinstalling it, upgrading to .16 beta1, rebooting router.

 

[Martineau]

However, I pushed v1.0.6 of gen_adblock.sh.
- it uses a new method to remove and push new zones dynamically, meaning the nightly update doesn't reload or restart unbound, no more cache hit drops!

Kudos - very elegant solution! - you are most definitely the Ad Block SME now …. just CNAME 'cloaking' and additional GUI metrics too if you please!

- uses logger so there is output in the logs in the UI for when it runs with results.

Usually the interactive execution of 'gen_adblock.sh' is only performed once (unless you push a new version! ) so the side-effect of double spacing of the 'logger' messages to me is 'untidy'
I personally prefer retaining the 'echo' statements and only use 'logger -st' for error messages that must be displayed on both the console and in Syslog, and use 'logger -t' for messages that only need to be sent to Syslog for extraction by the GUI feature.

e.g.

Spoiler: Logger -st output vs echo output

Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                           
 _____   _ _   _         _ 
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

(gen_adblock.sh): Removing possible temporary files..

(gen_adblock.sh): Downloading list(s) from block site(s) configured...

(gen_adblock.sh): Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
(gen_adblock.sh): Downloading list(s) from allow site(s) configured...

(gen_adblock.sh): Adding user requested hosts to list...

(gen_adblock.sh): Removing user requested hosts from list...

(gen_adblock.sh): Removing required hosts from list...

(gen_adblock.sh): Removing unnecessary formatting from the domain list...

(gen_adblock.sh): Generating Unbound adservers file...

(gen_adblock.sh): Number of adblocked hosts: 52826

(gen_adblock.sh): Generating Unbound unload/load lists...

(gen_adblock.sh): Loading/Unload Unbound local-zones to take effect...

(gen_adblock.sh): Removing temporary files...

(gen_adblock.sh): Adblock update complete!

vs.

 _____   _ _   _         _ 
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 1 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
Number of adblocked hosts: 52826
Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 52826 zones
added 52826 zones
Removing temporary files...
(gen_adblock.sh): Adblock update complete!

Also to avoid unduly alarming users I suggest you only report the true number of URLs to be downloaded, rather than include the comment lines

i.e.

echo "Attempting to Download $count of $(wc -l < $sites) from $url."

change to

echo "Attempting to Download $count of $(awk 'NF && !/^[:space:]*#/' $sites | wc -l) from $url."

 

[ika]

Rs restarts unbound but the adblock is not installed by the script (see pics)

edit: also already tried uninstalling unbound and reinstalling it, upgrading to .16 beta1, rebooting router.

The only way to make it work (with the script) was to force exit the script at the last step (when it was asking for downloading or keeping the config file), and manually restarting unbound.

 

[juched]

Kudos - very elegant solution! - you are most definitely the Ad Block SME now …. just CNAME 'cloaking' and additional GUI metrics too if you please!
Usually the interactive execution of 'gen_adblock.sh' is only performed once (unless you push a new version! ) so the side-effect of double spacing of the 'logger' messages to me is 'untidy'
I personally prefer retaining the 'echo' statements and only use 'logger -st' for error messages that must be displayed on both the console and in Syslog, and use 'logger -t' for messages that only need to be sent to Syslog for extraction by the GUI feature.

e.g.

Spoiler: Logger -st output vs echo output

Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                         
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

(gen_adblock.sh): Removing possible temporary files..

(gen_adblock.sh): Downloading list(s) from block site(s) configured...

(gen_adblock.sh): Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
(gen_adblock.sh): Downloading list(s) from allow site(s) configured...

(gen_adblock.sh): Adding user requested hosts to list...

(gen_adblock.sh): Removing user requested hosts from list...

(gen_adblock.sh): Removing required hosts from list...

(gen_adblock.sh): Removing unnecessary formatting from the domain list...

(gen_adblock.sh): Generating Unbound adservers file...

(gen_adblock.sh): Number of adblocked hosts: 52826

(gen_adblock.sh): Generating Unbound unload/load lists...

(gen_adblock.sh): Loading/Unload Unbound local-zones to take effect...

(gen_adblock.sh): Removing temporary files...

(gen_adblock.sh): Adblock update complete!

vs.

 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 1 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
Number of adblocked hosts: 52826
Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 52826 zones
added 52826 zones
Removing temporary files...
(gen_adblock.sh): Adblock update complete!

Also to avoid unduly alarming users I suggest you only report the true number of URLs to be downloaded, rather than include the comment lines

i.e.

echo "Attempting to Download $count of $(wc -l < $sites) from $url."

change to

echo "Attempting to Download $count of $(awk 'NF && !/^[:space:]*#/' $sites | wc -l) from $url."

I like the tweak on file count. Will incorporate.

But I am not getting double spacing when using the Say command you shared. Perhaps it is my terminal app config.

Do you see double spacing in terminal or logs? For me having some indication that adblock ran in the logs is useful and including items like version and numbers of results is important. Perhaps I can change to just show those two lines and any errors.

--- edit ----

Pushed small hotfix for V1.0.6 which tweaks the output as suggested. Now only the version header, the adblock count and any errors are sent to the logger. Rest of the messages are only seen on local output.

 

[SomeWhereOverTheRainBow]

Kudos - very elegant solution! - you are most definitely the Ad Block SME now …. just CNAME 'cloaking' and additional GUI metrics too if you please!
Usually the interactive execution of 'gen_adblock.sh' is only performed once (unless you push a new version! ) so the side-effect of double spacing of the 'logger' messages to me is 'untidy'
I personally prefer retaining the 'echo' statements and only use 'logger -st' for error messages that must be displayed on both the console and in Syslog, and use 'logger -t' for messages that only need to be sent to Syslog for extraction by the GUI feature.

e.g.

Spoiler: Logger -st output vs echo output

Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                          
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

(gen_adblock.sh): Removing possible temporary files..

(gen_adblock.sh): Downloading list(s) from block site(s) configured...

(gen_adblock.sh): Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
(gen_adblock.sh): Downloading list(s) from allow site(s) configured...

(gen_adblock.sh): Adding user requested hosts to list...

(gen_adblock.sh): Removing user requested hosts from list...

(gen_adblock.sh): Removing required hosts from list...

(gen_adblock.sh): Removing unnecessary formatting from the domain list...

(gen_adblock.sh): Generating Unbound adservers file...

(gen_adblock.sh): Number of adblocked hosts: 52826

(gen_adblock.sh): Generating Unbound unload/load lists...

(gen_adblock.sh): Loading/Unload Unbound local-zones to take effect...

(gen_adblock.sh): Removing temporary files...

(gen_adblock.sh): Adblock update complete!

vs.

 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 1 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
Number of adblocked hosts: 52826
Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 52826 zones
added 52826 zones
Removing temporary files...
(gen_adblock.sh): Adblock update complete!

Also to avoid unduly alarming users I suggest you only report the true number of URLs to be downloaded, rather than include the comment lines

i.e.

echo "Attempting to Download $count of $(wc -l < $sites) from $url."

change to

echo "Attempting to Download $count of $(awk 'NF && !/^[:space:]*#/' $sites | wc -l) from $url."

this is a good suggestion especially, since you added the option for commenting out urls. I can't wait to see what other tricks you have been hiding.

 

[Martineau]

@juched Did you see @ika's reported issue here?

I updated to v384.16 Beta this morning.....

Spoiler: Fresh unbound install 'z';amtm;i,7 then Ad Block 'i 3' and all h*ll breaks loose....

Restarting dnsmasq.....
Done.
Option Auto Reply 'y' Installing Ads and Tracker Blocking.....
 adblock/gen_adblock.sh downloaded successfully
 adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Adding Ad and Tracker 'include: /opt/var/lib/unbound/adblock/adservers'
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                           
 _____   _ _   _         _ 
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 17451 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 1 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 17451 Number of adblocked hosts: 52826
Generating Unbound unload/load lists...
[1584288202] unbound-control[17678:0] error: connect: Connection refused for 127.0.0.1 port 953
Loading/Unload Unbound local-zones to take effect...
(gen_adblock.sh): 17451 Warning unbound NOT running!
Removing temporary files...
Adblock update complete!

Auto install unbound Customisation complete 1 minutes and 47 seconds elapsed - Please wait for up to 10 seconds for status.....

 ***ERROR unbound went AWOL after 1 seconds.....

 ***ERROR Unsuccessful installation of unbound detected

Mar 15 15:59:21 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 15:59:56 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 16:00:55 RT-AC68U (unbound_manager.sh): 16260 Starting Script Execution (menu)
Mar 15 16:02:58 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 16:03:25 RT-AC68U (gen_adblock.sh): 17451 Warning unbound NOT running!
[1584288206] unbound[17723:0] notice: Start of unbound 1.9.6.
Mar 15 16:03:26 unbound[17723:0] debug: increased limit(open files) from 1024 to 1684
Mar 15 16:03:26 unbound[17723:0] debug: creating udp4 socket 127.0.0.1 53535
Mar 15 16:03:26 unbound[17723:0] debug: creating tcp4 socket 127.0.0.1 53535
Mar 15 16:03:26 unbound[17723:0] error: Setting TCP Fast Open as server failed: Protocol not available
Mar 15 16:03:26 unbound[17723:0] debug: creating tcp4 socket 127.0.0.1 953
Mar 15 16:03:26 unbound[17723:0] error: Setting TCP Fast Open as server failed: Protocol not available
Mar 15 16:03:26 unbound[17723:0] debug: setup SSL certificates
Mar 15 16:03:27 unbound[17723:0] debug: chdir to /opt/var/lib/unbound
Mar 15 16:03:27 unbound[17723:0] debug: chroot to /opt/var/lib/unbound
Mar 15 16:03:27 unbound[17723:0] debug: drop user privileges, run as nobody
Mar 15 16:03:27 unbound[17723:0] debug: switching log to /opt/var/lib/unbound/unbound.log

Anyone else experience similar?