What's new

AB-Solution 1.07 is out!

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

@thelonelycoder, is possible when ab-solution do update once a week that i still have my router IP in adblock hosts? now when do update adblock hosts file goes to 0.0.0.0. and then pixelserv-tls not working corectly? where in your script can change 0.0.0.0 to my router IP?

ab_dnsmasq_postconf.sh
Code:
cru a UpdateHosts "00 02 * * 0 /tmp/mnt/sda1/adblocking/scripts/update-hosts.sh"
CONFIG=$1
source /usr/sbin/helper.sh
logger "AB-Solution added entries via ab_dnsmasq_postconf.sh"
pc_append "address=/0.0.0.0/0.0.0.0" $CONFIG
pc_append "ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0" $CONFIG
pc_append "addn-hosts=/tmp/mnt/sda1/adblocking/hosts-adblock" $CONFIG
pc_append "addn-hosts=/tmp/mnt/sda1/adblocking/blacklist.txt" $CONFIG

update-hosts.sh
Code:
#!/bin/sh
# generated by AB-Solution 1.07

# set directory
dir=/tmp/mnt/sda1/adblocking

# removing blank, empty and Windows EOL in white- and blacklist
    sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/whitelist.txt
    sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/blacklist.txt

# get hosts files and combine
wget -qO- \
"http://winhelp2002.mvps.org/hosts.txt" \
"http://someonewhocares.org/hosts/zero/hosts" \
"http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" \
"http://www.malwaredomainlist.com/hostslist/hosts.txt" \
"http://adaway.org/hosts.txt" \
"http://hosts-file.net/emd.txt" \
"http://hosts-file.net/ad_servers.txt" \
"http://hosts-file.net/grm.txt" \
"http://hosts-file.net/mmt.txt" \
> $dir/temp1

# amalgamate the hosts files, removing duplicates and sort in alphabetical order
cat $dir/temp1 | sed 's/127.0.0.1/0.0.0.0/g;s/\r$//' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u > $dir/temp2
You replace all occurrences of 0.0.0.0 in both files exactly as they are in. Then don't run a hosts file type change, toggle adblocking or logging. These actions replace the complete respective file.
 
You replace all occurrences of 0.0.0.0 in both files exactly as they are in. Then don't run a hosts file type change, toggle adblocking or logging. These actions replace the complete respective file.

i change now all 0.0.0.0 to 192.168.1.1.....is this ok or i got 0.0.0.0 again when will script run weekly update?

ab_dnsmasq_postconf.sh
Code:
cru a UpdateHosts "00 02 * * 0 /tmp/mnt/sda1/adblocking/scripts/update-hosts.sh"
CONFIG=$1
source /usr/sbin/helper.sh
logger "AB-Solution added entries via ab_dnsmasq_postconf.sh"
pc_append "address=/192.168.1.1/192.168.1.1" $CONFIG
pc_append "ptr-record=192.168.1.1.in-addr.arpa,192.168.1.1" $CONFIG
pc_append "addn-hosts=/tmp/mnt/sda1/adblocking/hosts-adblock" $CONFIG
pc_append "addn-hosts=/tmp/mnt/sda1/adblocking/blacklist.txt" $CONFIG

update-hosts.sh
Code:
#!/bin/sh
# generated by AB-Solution 1.07

# set directory
dir=/tmp/mnt/sda1/adblocking

# removing blank, empty and Windows EOL in white- and blacklist
    sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/whitelist.txt
    sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/blacklist.txt

# get hosts files and combine
wget -qO- \
"http://winhelp2002.mvps.org/hosts.txt" \
"http://someonewhocares.org/hosts/zero/hosts" \
"http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=192.168.1.1" \
"http://www.malwaredomainlist.com/hostslist/hosts.txt" \
"http://adaway.org/hosts.txt" \
"http://hosts-file.net/emd.txt" \
"http://hosts-file.net/ad_servers.txt" \
"http://hosts-file.net/grm.txt" \
"http://hosts-file.net/mmt.txt" \
> $dir/temp1

# amalgamate the hosts files, removing duplicates and sort in alphabetical order
cat $dir/temp1 | sed 's/127.0.0.1/192.168.1.1/g;s/\r$//' | grep -w ^192.168.1.1 | awk '{print $1 " " $2}' | sort -u > $dir/temp2
 
Round 2 of enhancements:

log-stats.sh gives you detailed statistics from your current log file:

Code:
#!/bin/sh
dir=/tmp/mnt/USB/adblocking/logs
L=5
LINE=--------------------------------
AL=1
printf "\nThe top $L requested domains were:\n$LINE\n"
awk '/query\[A]/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked ad domains were:\n$LINE\n"
awk '/is 0.0.0.0/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked custom domains were:\n$LINE\n"
awk '/is 10.0.0.1/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
rm -f temp*
printf "\nThe top $L noisiest name clients:\n$LINE\n"
awk '/from 192.168.1./ {print $8}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L > temp1
for i in $(awk '{print $2}' temp1); do
grep -e "$i" $dir/dnsmasq.log | awk '{print $6}'| awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n %s %s",a[o],o}' | sort -nr | head -1 >> temp2
CH="$(awk 'END{print $1}' temp2)"
TH="$(awk -v AL="$AL" 'FNR==AL{print $1}' temp1)"
let AL=AL+1
awk -v CH="$CH" -v TH="$TH" 'BEGIN{print ((CH * 100)/TH)"%"}' >> temp3
done
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp2 temp3 > temp4
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp1 temp4
rm -f temp*

Example output (domains replaced with snbforums.com):

admin@AC88U:/tmp/mnt/USB/adblocking/scripts# sh log-stats.sh

The top 5 requested domains were:
--------------------------------
3914 snbforums.com
1925 snbforums.com
1700 snbforums.com
1027 snbforums.com
909 snbforums.com

The top 5 blocked ad domains were:
--------------------------------
798 snbforums.com
321 snbforums.com
317 snbforums.com
196 snbforums.com
190 snbforums.com

The top 5 blocked custom domains were:
--------------------------------
3914 snbforums.com
1700 snbforums.com
858 snbforums.com
103 snbforums.com
2 snbforums.com

The top 5 noisiest name clients:
--------------------------------
23560 192.168.1.5 - 3914 snbforums.com - 16.6129%
5048 192.168.1.15 - 1027 snbforums.com - 20.3447%
2530 192.168.1.10 - 677 snbforums.com - 26.7589%
2361 192.168.1.13 - 258 snbforums.com - 10.9276%
1651 192.168.1.9 - 190 snbforums.com - 11.5082%

All fairly self explanatory except the last section which shows:
-the total requests for each noisiest client
-the most requested domain from that client
-the number of hits for that domain from that client
-a percentage value showing how much that domain constituted of that client's overall requests

You can also change the verbosity by changing the $L variable to 10 for example to show the top 10 instead of the default top 5 for each section.


update-hosts.sh has been updated to:
Code:
#!/bin/sh
# generated by AB-Solution 1.07

dir=/tmp/mnt/USB/adblocking
ldir=/mnt/USB/adblocking/logs
# removing blank, empty and Windows EOL in white- and blacklist
        sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/whitelist.txt
        sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/blacklist.txt

# backup existing stats
L=5
line=______________________________
aline=***************************************************************************
dline=---------------------------
echo -e "$aline\n\n\n$aline\n$dline\n$(date)\n$dline" >> $ldir/logstats
echo -e "\nThe top $L requested domains were:\n$line\n" >> $ldir/logstats
grep -e "query\[A]" $ldir/dnsmasq.log | awk '{print $6}' | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s %s ",a[o],o}' | sort -nr | head -$L >> $ldir/logstats
echo -e "\nThe top $L blocked ad domains were:\n$line\n" >> $ldir/logstats
grep -e "is 0.0.0.0" $ldir/dnsmasq.log | awk '{print $6}' | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s %s ",a[o],o}' | sort -nr | head -$L >> $ldir/logstats
echo -e "\nThe top $L blocked custom domains were:\n$line\n" >> $ldir/logstats
grep -e "is 10.0.0.1" $ldir/dnsmasq.log | awk '{print $6}' | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s %s ",a[o],o}' | sort -nr | head -$L >> $ldir/logstats
echo -e "\nThe top $L noisiest name clients:\n$line\n" >> $ldir/logstats
grep -e "192.168.1." $ldir/dnsmasq.log | awk '{print $8}' | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s %s ",a[o],o}' | sort -nr | head -$L >> $ldir/logstats

# get hosts files and combine

SQ=1
rm -f $dir/db/names
rm -f $dir/db/*.txt
for i in $(cat $dir/db/db); do
wget -qO $dir/db/$SQ.txt $i
if [ $? -eq 0 ]; then
echo $i >> $dir/db/names
printf "$SQ - $i downloaded\n"
cat $dir/db/$SQ.txt >> $dir/temp1
let SQ=SQ+1
else
printf "$i was NOT downloaded\n"
fi
done

# amalgamate the hosts files, removing duplicates and sort in alphabetical order
printf "Files are being processed. Please wait...\n"
cat $dir/temp1 | sed 's/127.0.0.1/0.0.0.0/g;s/\r$//' | grep -w ^0.0.0.0 | awk '{print $1 " " $2"\n:: " $2}' > $dir/temp2
printf "File processing completed.\nDuplicates are being removed. Please wait...\n"
sort -u $dir/temp2 > $dir/temp3
TOTALLINES=$(wc -l < $dir/temp2)
UNIQUELINES=$(wc -l < $dir/temp3)
printf "Duplicates removed.\n"

# remove localhost and whitelisted entries
cat $dir/temp3 | sed '/\b\(localhost\|local\)\b/d;/localhost.localdomain/d' | fgrep -vf $dir/whitelist.txt > $dir/hosts-adblock
printf "Localhost and whitelisted entries removed.\n"
NUMOFLINES=$(wc -l < "$dir/hosts-adblock")
printf "\nDuplicates removed: %s" "$(($TOTALLINES - $UNIQUELINES))"
printf "\nUnique ad domains blocked: %s" "$NUMOFLINES"
#remove temp files
rm $dir/temp*

#rotate the logs when logging is enabled
if [ -f $dir/logs/dnsmasq.log ];then
        mv $dir/logs/dnsmasq.log $dir/logs/dnsmasq.log.old
        logger "AB-Solution rotated dnsmasq log file"
fi

#restart dnsmasq to reload the files
service restart_dnsmasq
logger "AB-Solution updated hosts file and restarted dnsmasq"

Differences compared to the last one:
- IPv6 fix added thanks to charlie2alpha
- Friendly messages explaining what the script is currently doing
- Cut down version of log-stats.sh that appends the output to /logs/logstats


And finally, the IPv6 fix added to update-blacklist.sh in a transparent manner so you only have to deal with a single blacklist:

Code:
dir=/tmp/mnt/USB/adblocking
sed -i '/^[[:blank:]]*$/d;s/\r$//' $dir/blacklist.txt
NUMOFLINES=$(wc -l < "$dir/hosts-adblock")
NUMOFNAMESTEMP=$NUMOFLINES
printf "Unique ad domains blocked: %s\n" "$NUMOFLINES"
NUMOFLINES=$(cat $dir/blacklist.txt | sort -u | grep -c 10.0.0.1)
printf "Unique custom domains blocked: %s" "$NUMOFLINES"
NUMOFNAMES=$(($NUMOFLINES + $NUMOFNAMESTEMP))
mv $dir/blacklist.txt $dir/tmp
awk '{print $1 " " $2"\n:: " $2}' $dir/tmp > $dir/blacklist.txt
service restart_dnsmasq
mv $dir/tmp $dir/blacklist.txt
logger "EAZ update the ad-blocker blacklist - currently $NUMOFLINES custom and $NUMOFNAMES ad domains blocked"
 
Last edited:
Round 2 of enhancements:

log-stats.sh gives you detailed statistics from your current log file:

I wonder if people mine the big data in pixelserv's log, what kind of stats will you be able to produce..
 
i am using private proxy last two days and now i got all ads back...i put proxy in network settings on windows.

sent from Kodi 17 Krypton
 
adv-log-stats.sh
Code:
#!/bin/sh
L=10
AL=1
LINE=--------------------------------
dir=/tmp/mnt/USB/adblocking/logs
printf "\nThe top $L requested domains were:\n$LINE\n"
awk '/query\[A]/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked ad domains were:\n$LINE\n"
awk '/is 0.0.0.0/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked custom domains were:\n$LINE\n"
awk '/is 10.0.0.1/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
rm -f temp*
printf "\nThe top $L noisiest name clients:\n$LINE\n"
awk '/from 192.168.1./ {print $8}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L > temp1
for i in $(awk '{print $2}' temp1); do
grep -e "$i" $dir/dnsmasq.log | awk '{print $6}'| awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n %s %s",a[o],o}' | sort -nr | head -1 >> temp2
CH="$(awk 'END{print $1}' temp2)"
TH="$(awk -v AL="$AL" 'FNR==AL{print $1}' temp1)"
let AL=AL+1
awk -v CH="$CH" -v TH="$TH" 'BEGIN{print ((CH * 100)/TH)"%"}' >> temp3
done
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp2 temp3 > temp4
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp1 temp4
printf "\nTop $L domains for top $L clients:\n$LINE\n"
for i in $(awk '{print $2}' temp1); do
printf "$i:\n"
grep -e "$i" $dir/dnsmasq.log | awk '{print $6}'| awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n %s %s",a[o],o}' | sort -nr | head -$L
done
rm -f temp*

Gives you the same output as log-stats.sh but also includes the top 10 domains for the top 10 clients with the number of hits for each - change default of 10 by modifying L variable.
 
adv-log-stats.sh
Code:
#!/bin/sh
L=10
AL=1
LINE=--------------------------------
dir=/tmp/mnt/USB/adblocking/logs
printf "\nThe top $L requested domains were:\n$LINE\n"
awk '/query\[A]/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked ad domains were:\n$LINE\n"
awk '/is 0.0.0.0/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
printf "\nThe top $L blocked custom domains were:\n$LINE\n"
awk '/is 10.0.0.1/ {print $6}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L
rm -f temp*
printf "\nThe top $L noisiest name clients:\n$LINE\n"
awk '/from 192.168.1./ {print $8}' $dir/dnsmasq.log | awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n%s  %s ",a[o],o}' | sort -nr | head -$L > temp1
for i in $(awk '{print $2}' temp1); do
grep -e "$i" $dir/dnsmasq.log | awk '{print $6}'| awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n %s %s",a[o],o}' | sort -nr | head -1 >> temp2
CH="$(awk 'END{print $1}' temp2)"
TH="$(awk -v AL="$AL" 'FNR==AL{print $1}' temp1)"
let AL=AL+1
awk -v CH="$CH" -v TH="$TH" 'BEGIN{print ((CH * 100)/TH)"%"}' >> temp3
done
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp2 temp3 > temp4
awk 'NR==FNR{a[FNR]=$0 " -";next} {print a[FNR],$0}' temp1 temp4
printf "\nTop $L domains for top $L clients:\n$LINE\n"
for i in $(awk '{print $2}' temp1); do
printf "$i:\n"
grep -e "$i" $dir/dnsmasq.log | awk '{print $6}'| awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(o in a) printf "\n %s %s",a[o],o}' | sort -nr | head -$L
done
rm -f temp*

Gives you the same output as log-stats.sh but also includes the top 10 domains for the top 10 clients with the number of hits for each - change default of 10 by modifying L variable.
I need to get other things in my life out of the way before I have time to get your goodies into the next release. But it will be done. Thanks for contributing.
 
  • Like
Reactions: eaz
i am using private proxy last two days and now i got all ads back...i put proxy in network settings on windows.

sent from Kodi 17 Krypton
Since you mix my script with other scripts on this board you are largely on your own. AB-Solution can't handle that.
 
curius about the gui page that you want to create what are your plans on showing on it ?

what kinda stats in general etc :)
 
Since you mix my script with other scripts on this board you are largely on your own. AB-Solution can't handle that.
I will do fresh install your ab-solution and without others scripts and will see if it working with my proxy....

sent from Kodi 17 Krypton
 
curius about the gui page that you want to create what are your plans on showing on it ?

what kinda stats in general etc :)
This is for the script only 1.08 release ATM. Currently, my plan is on maintaining them in parallel as both are worthwhile to me.
The GUI version will be an entirely different beast!
 
@thelonelycoder thinking of doing a simple stats page as an optional feature since i want to keep my script minimal

is adding pages to asuswrt merlin documented somewhere ?
 
@thelonelycoder thinking of doing a simple stats page as an optional feature since i want to keep my script minimal

is adding pages to asuswrt merlin documented somewhere ?
The one you're likely thinking of is the http://<your-router-ip>/user/ directory. It's empty but can be used to add your own scripts. As long as they are supported by the internal httpd server they will work: .html and .asp work but with limitation.
And it's a volatile storage, so you will have to copy the content over from somewhere else at boot-up.
There is not much documentation for that I am aware of, just a few mentions by @RMerlin here and there in the forum. Be aware that the httpd is closed source and not much is known about it's capabilities. It's trial and error most of the time.

Edit: The user directory is physically located at /www/user/ on the router.
 
@thelonelycoder thinking of doing a simple stats page as an optional feature since i want to keep my script minimal

is adding pages to asuswrt merlin documented somewhere ?

@swetoast

Have you seen this?
http://www.snbforums.com/threads/ntp-daemon-for-asuswrt-merlin.28041/

Maybe consider option #3 here:
http://www.snbforums.com/threads/transmission-for-asuswrt-merlin.31278/#post-247945

A minimalist approach IMO. I'm sure you can find a way not to duplicate the whole /www as used in my NTP Daemon package...
 
@kvic tnx havent seen those gonna read thru em.

see that you have a git called goodies-asuswrt is that where the source is ?

edit: seems it was tnx @kvic
 
Last edited:
Be aware that the httpd is closed source and not much is known about it's capabilities. It's trial and error most of the time.

httpd is actually mostly open sourced. Very little of it is closed (what's closed mostly has to do with router model validation and password encoding for the authenticated SMTP support).

But there isn't any documentation on how to develop a webui, most of it is indeed done by looking at the existing pages.
 
Hey TLC,

Is there any way you could make an option in the menu to allow Google services (not sure what its called exactly).
With ad block set to medium, when I google a product the first few links are generally paid for ads that link you to a shop, these no longer work.
Also when using Google shopping it wont link to the shops. Its pretty annoying.

Is there any way you could give us/me an option to white-list these?
 
Hey TLC,

Is there any way you could make an option in the menu to allow Google services (not sure what its called exactly).
With ad block set to medium, when I google a product the first few links are generally paid for ads that link you to a shop, these no longer work.
Also when using Google shopping it wont link to the shops. Its pretty annoying.

Is there any way you could give us/me an option to white-list these?
Option "p" perhaps?
Add the domains to the <yourdevice>/adblocking/whitelist.txt and read it in with "p".
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top