Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

dave14305

Part of the Furniture
[email protected]:/tmp# cat /jffs/scripts/dnsmasq.postconf sh /jffs/addons/unbound/unbound.postconf "$1"
You need this as the first line of dnsmasq.postconf:
Code:
#!/bin/sh
Run these 3 commands:
Code:
sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq
 

krgck

Regular Contributor
You need this as the first line of dnsmasq.postconf:
Code:
#!/bin/sh
Run these 3 commands:
Code:
sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq
That did the trick. Thanks for saving my time and sleep :rolleyes:
 

tomsk

Very Senior Member
You need this as the first line of dnsmasq.postconf:
Code:
#!/bin/sh
Run these 3 commands:
Code:
sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq
Oh the shebang is there for me because the dnsmasq.postconf was added by diversion ... but this install was written by the manager script?
 

dave14305

Part of the Furniture
Oh the shebang is there for me because the dnsmasq.postconf was added by diversion ... but this unbound install was written by the manager script?
unbound.postconf is from Martineau. krgck must never have had a dnsmasq.postconf and unbound_manager doesn’t (yet) account for that scenario.
 

Twiglets

Senior Member
Yes fairly close depending on that else you might have bolted onto diversion
Code:
A:Option ==> ad

Analysed Diversion file: 'blockinglist'     Type=pixelserv, (Adblock Domains=55204) would add 620 entries
Analysed Diversion file: 'blacklist'     Type=pixelserv, (Adblock Domains=55204) would add 2 entries
Analysed Diversion file: 'whitelist'     Type=URL, (Adblock URLs=19) would add 22 entries

I guess the ultimate if you wanted identical lists would to be to import them from diversion... the code is in the script to do the dnsmasq to unbound type lists conversion already ( the steven black one downloads in dnsmasq form and needs conversion)
The script already has a 'merge' option that is not tied to a menu option.
A small 'edit' :rolleyes:;) can run the 'merge' code which does appear to work !!!???

Not sure why it has been left out ?
 

tomsk

Very Senior Member
The script already has a 'merge' option that is not tied to a menu option.
A small 'edit' :rolleyes:;) can run the 'merge' code which does appear to work !!!???

Not sure why it has been left out ?
Thats not just the icing... that's the cherry on top too.
 

joe scian

Very Senior Member
Hi Martineau

when I issue the command below I get
Code:
A:Option ==> ad


cp: write error: No space left on device
sed: write error

and now my syslog is filled up with. A router reboot fixes the syslog spam below but I wont issue that ad command again. !

Code:
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:14 RT-AC5300-0680 syslog: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device
 
Last edited:

Martineau

Part of the Furniture
I just had a look at my unbound.conf just now ... With dnsmasq enabled shouldn't the unbound interface on 127.0.0.1 for port 53 be commented out?
Code:
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
port: 53535                                 # v1.08 If 53, requires 'port=0' in '/etc/dnsmasq.conf' to 'disable' dnsmasq to answer queries direct from LAN clients
interface: [email protected]                  # v1.01 As per @dave14305 minimal config; Will be overwritten by $(nvram get lan_ipaddr_rt) if dnsmasq 'disabled'
interface: [email protected]                    # v1.10 Required by router if dnsmasq 'disabled'
#access-control: 0.0.0.0/0 allow            # v1.10 Will be overwritten by LAN subnet "${lan_ip_addr_rt}/24" if 'dnsmasq disabled' aka bypassed
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Whoops! :oops:

In trying to tidy up 'unbound.conf' v1.10, I 'helpfully' added a comment....see this commit

upload_2020-5-11_12-9-57.png


Clearly the original code
Code:
[ -n "$(grep "^interface: [email protected]$" /opt/var/lib/unbound/unbound.conf)" ] && sed   's/^interface: 127\.0\.0\[email protected]$/#interface: 127\.0\.0\[email protected]/' /opt/var/lib/unbound/unbound.conf
will now never match/find an uncommented line containing ONLY the text 'interface: [email protected]' :rolleyes:

I'll push a Hotfix later.
 
Last edited:

tomsk

Very Senior Member
I was having a look around for other organisations suppling RPZ data other than spamhaus . seems the other main player is SURBL .... the SURBL combined list is interesting in the way they use ip to show what list its from http://www.surbl.org/lists

multi.surbl.org - Combined SURBL list
All of the SURBL data sources are combined into a single, bitmasked list: multi.surbl.org.
Bitmasking means that there is only one entry per domain name or IP address, but that entry will resolve into an address (DNS A record) whose last octet indicates which lists it belongs to. The bit positions in that last octet for membership in the different lists are:
8 = listed on PH
16 = listed on MW
64 = listed on ABUSE
128 = listed on CR
If an entry belongs to just one list it will have an address where the last octet has that value. For example 127.0.0.8 means it's on the phishing list, while 127.0.0.64 means it's listed on the ABUSE list. An entry on multiple lists gets the sum of those list numbers as the last octet, so 127.0.0.80 means a record is on both MW and ABUSE (comes from: 16 + 64 = 80). In this way, membership in multiple lists is encoded into a single response. Octets other than the first and last one are reserved for future use and should be ignored.
/QUOTE]
 
Last edited:

Martineau

Part of the Furniture
Hi Martineau

when I issue the command below I get
Code:
A:Option ==> ad


cp: write error: No space left on device
sed: write error

and now my syslog is filled up with. A router reboot fixes the syslog spam below but I wont issue that ad command again. !

Code:
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:14 RT-AC5300-0680 syslog: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device
OK, what size blocking list do you use?
 

juched

Very Senior Member
@juched

FYI,,,
Code:
 _____   _ _   _         _ 
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 28116 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
should be
Code:
Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 2 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 3 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
'gen_adblock.sh' needs a minor tweak
Code:
    for url in $(echo $line); do
      [ "${url:0:1}" == "#" ] && continue # skip commented out lines - Thanks @Martineau
      echo "Attempting to Download $count of $(awk 'NF && !/^[:space:]*#/' $sites | wc -l) from $url."
      curl --progress-bar $url | grep -o '^[^#]*' | grep -v "::1" | grep -v "0.0.0.0 0.0.0.0" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $NF}' | grep -o '^[^\\]*' | grep -o '^[^\\$]*' | sort >> $list
      dos2unix $list
      count=$((count + 1)) 
    done


Thanks, fixed and pushed to master. I also pushed YT ad block script to master as well.
 

Martineau

Part of the Furniture
yes it should be perhaps an introduced "feature" since last hotfix.
What? :confused:
Also noted that
Code:
# log-local-actions - yes
remains commented out during adblock installation when going back and forth enabling Dnsmasq and subsequently choosing Dnsmasq disable and Adblock install. Had to manually uncomment this.
The option is only enabled if explicitly requested, and logging to 'syslog-ng/scribe' is also explicitly ENABLED.

The design/reasoning was that if unbound logging was ENABLED, then previously as there was no housekeeping performed on the size of the log, it could silently fill the disk a lot quicker.

e.g. explicitly ENABLE the setting
Code:
e  = Exit Script [?]

A:Option ==> adblock track

Option Auto Reply 'y'    Installing Ads and Tracker (Ad Block) Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                       
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 3200 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow


Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 3200 Number of adblocked hosts: 77143

Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 77143 zones
added 77143 zones
Removing temporary files...
Adblock update complete!

Logging Ad Block BLOCKED domains to scribe
I suppose now that logging housekeeping should no longer be an issue, as @juched's GUI stats can use the metrics, I suppose I could change the criteria to auto-enable if 'sgui' is ENABLED?
 
Last edited:

juched

Very Senior Member
What? :confused:

The option is only enabled if explicitly requested, and logging to 'syslog-ng/scribe' is also explicitly ENABLED.

The design/reasoning was that if unbound logging was ENABLED, then previously as there was no housekeeping performed on the size of the log, it could silently fill the disk a lot quicker.

e.g. explicitly ENABLE the setting
Code:
e  = Exit Script [?]

A:Option ==> adblock track

Option Auto Reply 'y'    Installing Ads and Tracker (Ad Block) Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                        
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 3200 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow


Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 3200 Number of adblocked hosts: 77143

Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 77143 zones
added 77143 zones
Removing temporary files...
Adblock update complete!

Logging Ad Block BLOCKED domains to scribe
I suppose now that logging housekeeping should no longer be an issue, as @juched's GUI stats can use the metrics, I suppose I could change the criteria to auto-enable if 'sgui' is enabled?

I think so, since every hour I flush those message as well, and support both scribe and non-scribe logging.

Would be good to have an "option" to use "sgui extended" to enable log-replies as well, so you can get extended stats showing all requested queries. Works even if dnsmasq disable is not used. No need to enable log-queries for extended stats, and my script will flush those lines from the log as well, so the disk should not fill up.

Thoughts?
 

Martineau

Part of the Furniture
I've uploaded v3.12

Version=3.12
Github md5=4a207524e455366859549c3cce137e95


use 'u' to update when prompted on screen

Use of the 'i = Update unbound Installation' ** not required **

Code:
FIX:    If '/jffs/scripts/dnsmasq.postconf doesn't exist then create it with '#!/bin/sh' (and execute attributes) before adding 'sh /jffs/scripts/addons/unbound/unbound.postconf' - @dave14305
FIX:    dnsmasq bypass, 'unbound.conf' v1.10 added a comment, so switching back to dnsmasq as Primary DNS, unbound now correctly releases '[email protected]' - @tomsk
FIX:    'ad' to use '/opt/tmp/' rather than '/tmp/' - @joe scian
CHANGE: If 'sgui' ENABLED and 'adblock' requested then ENABLE tracking of blocked domains 'log-local-actions: yes' - @joe scian
CHANGE: 'youtube' YT Video Ad blocking will now retrieve script from @juched's Github master branch rather than his dev branch.
CHANGE: Enhance 'debug' diagnostics
ADD:    'youtube [view | edit]' to allow access (for the curious) to the current YT blocked domains
CHANGE: '?' command, include clickable URL link to unbound v1.10.0 manual

    About unbound: https://nlnetlabs.nl/projects/unbound/about/ , Manual https://nlnetlabs.nl/documentation/unbound/unbound.conf/

Thanks again to the usual suspects for proof reading/bug reporting.
 
Last edited:

Martineau

Part of the Furniture
I think so, since every hour I flush those message as well, and support both scribe and non-scribe logging.

Would be good to have an "option" to use "sgui extended" to enable log-replies as well, so you can get extended stats showing all requested queries. Works even if dnsmasq disable is not used. No need to enable log-queries for extended stats, and my script will flush those lines from the log as well, so the disk should not fill up.

Thoughts?
'lo = Enable Logging' currently ENABLEs both queries and replies, and 'lx Disable Logging' DISABLEs them.

I have added 'sgui all' which will now enable tracking of the Ad Blocking ('log-local-actions: yes') but if logging is DISABLED then it is obviously irrelevant until logging is ENABLED.
 

juched

Very Senior Member
'lo = Enable Logging' currently ENABLEs both queries and replies, and 'lx Disable Logging' DISABLEs them.

I have added 'sgui all' which will now enable tracking of the Ad Blocking ('log-local-actions: yes') but if logging is DISABLED then it is obviously irrelevant until logging is ENABLED.

Log local actions should always work, even if log replies or log queries is disabled. So what do you mean that logging needs to be enabled? What does logging enabled or disabled mean to you? Verbosity?
 

Martineau

Part of the Furniture
Log local actions should always work, even if log replies or log queries is disabled. So what do you mean that logging needs to be enabled?

What does logging enabled or disabled mean to you? Verbosity?
Yes
 

Martineau

Part of the Furniture

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top