AB-Solution - The Ad Blocking Solution

[Twiglets]

Your original proposal only has the blocked domains colored, that is still built in:
https://www.snbforums.com/threads/a...ing-solution-v3-9-2.37511/page-21#post-319223

But this looks much better!
I've tidied up the if clauses (you can use elif instead of else if) and I'm going to also use the theme colors since they appear best on the users terminal. But since the themes only have three text colors defined, I will have to add another one to it.
Also, your code does not work for IPv6 domains, use "$blockingFile\|blacklist.txt" instead of "is $custom_ipV4".

I'll include it in the next update.

Thanks for tidying up the 'bad' code
Not using IPv6 so missed that one.

 

[Xentrk]

TLC,
I am reposting the latest version of the changes I made to 'functions.add' to give a more colourful and useful (to me) listing for 'Follow the logfile' Option 1.
They seem to have been lost a few changes ago in 'AB-Solution'.
If you still feel the change is useful please update the 'functions.add' as per below.

Sample screencap attached.

Green = Original DNS Query.
Light Blue = Data from DNS Cache (via dnsmasq).
Yellow = Forwarded to DNS Server
White = DNS Reply data from Query.

[Please ignore the 'validation' lines in screencap, as I am using DNSSEC ..... badly !!! ]


Changes:
=======
In functions.add change lines 1098-1111 to:

# part of # [Ff] follow the logfile (advanced)
tail_logfile(){
trap : INT
if [ $1 == standard ];then
echo -e " --> following the logfile now (tail -F)"
echo -e " $ansi_red blocked domains are color marked$nc:\n"
tail -F $abSolutionPath/$logsDir/dnsmasq.log | \
while read line;do
if echo "$line" | grep -q "is $custom_ipV4";then
echo -e "$ansi_red$line$nc"
else
if echo "$line" | grep -q "127.0.0.1"; then
echo -e "$ansi_yellow$line$nc"
else
if echo "$line" | grep -q ": query"; then
echo -e "$ansi_green$line$nc"
else
if echo "$line" | grep -q ": cached "; then
echo -e "$ansi_lt_blue$line$nc"
else
echo "$line"
fi
fi
fi
fi
done

Add to top of file the following to define colours used:

# Set colours for use in 'Follow the Logfile' Option 1
ansi_red="\033[1;31m";
ansi_green="\033[1;32m";
ansi_yellow="\033[1;33m";
ansi_lt_blue="\033[1;36m";

That is all.

Sample Screencap:

View attachment 10886

Sorry seem to have lost indent in the listing.
Appears in the edit window but not in the posting. ???!!!

I like your updates.

This is off topic. Sorry! If you have time, can do me a favor and post the domain names that get generated while watching BBC iPlayer?

What I do, is do a tail -f on the dnsmasq.log file and output to another file. e.g. tail -f dnsmasq.log > bbc

I then run this utility to clean up the results. You have to change the IP address to match the device that generated the traffic.

#!/bin/sh
#set -xo
# This script will format the output of tail -f dnsmasq.log > logfile
# where logfile is the output of tail -f dnsmasq.log
#  1. extract records whose contents contain the word "query"
#  2. output only the domain name
#  3. sort file for unique contents to elimnate duplicates
#  4. save to $1_output
#
# Parameters Passed
# $1 = provide the name of the source file when running the script
#     e.g. ./getdomainnames.sh logfile
#
source_file=/tmp/mnt/absolution/adblocking/logs/$1
output_file=$source_file"_output"
cat $source_file | grep query | grep 192.168.22.152 | awk '{ print $6 }' | sort -u > $output_file
#cat $source_file | grep cached | awk '{ print $6 }' | sort -u > $output_file

 

[iManuB]

Hi!

I say I did not touch anything. Today I wanted to put a site in the Whitelist, but by accessing SSH and saying the classic "ab-solution", the terminal returns the following error:

ab-solution: Input / output error

What happened? I did not touch anything and the USB stick has always been in place.

Thanks so much!

sd 0:0:0:0: [sda] Device not ready
sd 0:0:0:0: [sda]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda]  Sense Key : Not Ready [current]
sd 0:0:0:0: [sda]  Add. Sense: Medium not present
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 00 2b ca 40 00 00 08 00
end_request: I/O error, dev sda, sector 2869824

EDIT: There were errors in the USB Stick. She is probably dying. I'm formatting the usb key and if it gets the same problem again, I'll buy another one!

 

[Twiglets]

Xentrk,
This is what I got when going to iPlayer [https://www.bbc.co.uk/iplayer] from Firefox.


bbc01.sitestat.com
cd-megavolt.90fe2324ce3eb149.xhst.bbci.co.uk
e3891.dscf.akamaiedge.net
edigitalsurvey.com
ichef.bbci.co.uk
iplayer-web.files.bbci.co.uk
mvt.api.bbc.com
mybbc.files.bbci.co.uk
nav.files.bbci.co.uk
polling.bbc.co.uk
sa.bbc.co.uk
sb.scorecardresearch.com
search.files.bbci.co.uk
ssl.bbc.co.uk
ssl.bbc.net.uk
static.bbc.co.uk
static.bbci.co.uk

Hope it is of use.

 

[thelonelycoder]

The addon file pixelserv-tls.add v3.9.2.2 is now available

Use cu to update.

This adds support for the -T switch for the upcoming pixelserv-tls version Kl

 

[skeal]

Can't wait for the upcoming pixelserv-tls version Kl. This script gets better and better! Its already great right now!!

 

[Xentrk]

Xentrk,
This is what I got when going to iPlayer [https://www.bbc.co.uk/iplayer] from Firefox.


bbc01.sitestat.com
cd-megavolt.90fe2324ce3eb149.xhst.bbci.co.uk
e3891.dscf.akamaiedge.net
edigitalsurvey.com
ichef.bbci.co.uk
iplayer-web.files.bbci.co.uk
mvt.api.bbc.com
mybbc.files.bbci.co.uk
nav.files.bbci.co.uk
polling.bbc.co.uk
sa.bbc.co.uk
sb.scorecardresearch.com
search.files.bbci.co.uk
ssl.bbc.co.uk
ssl.bbc.net.uk
static.bbc.co.uk
static.bbci.co.uk

Hope it is of use.

Thanks for the list! I've been helping another forum member with selective routing for iPlayer. I see three domains that I did not have in my list.

 

[thelonelycoder]

AB-Solution users:
Let's show some love to @kvic for his work on pixelserv-tls.
If we all do, we may get a speed boost and a smaller memory print for ad-blocking in AB-Solution.

The long version
The maintainer of pixelserv-tls (kvic) compiled an interesting comparison on the benefits of using pixelser-tls instead of the standard blocking IP 0.0.0.0, which AB-Solution uses if pixelserv-tls is not installed or not enabled.
Read it, it's enlightening!

I happened to notice one small but interesting detail about his use of Dnsmasq on his RT-AC56U: DNS adblocker with pixelserv-tls and cascading hash tables in Dnsmasq

Naturally, I inquired about these elusive hash tables, google giving me no coherent results.
kvic being kvic, replied with a Let's make it a bit more fun..

I replied with some download statistics by AB-Solution users. This alone might not suffice to convince kvic to do a write-up.
Other users in this tread replied with support here and here.

The short version
What I ask you to do:
Go here and Like this post, if more than 100 likes are registered, kvic will do a write-up of cascading hash tables in Dnsmasq which could potentially be of very good use in AB-Solution.
Thank you.

 

[heysoundude]

AB-Solution users:
Let's show some love to @kvic for his work on pixelserv-tls.
If we all do, we may get a speed boost and a smaller memory print for ad-blocking in AB-Solution.

The long version
The maintainer of pixelserv-tls (kvic) compiled an interesting comparison on the benefits of using pixelser-tls instead of the standard blocking IP 0.0.0.0, which AB-Solution uses if pixelserv-tls is not installed or not enabled.
Read it, it's enlightening!

I happened to notice one small but interesting detail about his use of Dnsmasq on his RT-AC56U: DNS adblocker with pixelserv-tls and cascading hash tables in Dnsmasq

Naturally, I inquired about these elusive hash tables, google giving me no coherent results.
kvic being kvic, replied with a Let's make it a bit more fun..

I replied with some download statistics by AB-Solution users. This alone might not suffice to convince kvic to do a write-up.
Other users in this tread replied with support here and here.

The short version
What I ask you to do:
Go here and Like this post, if more than 100 likes are registered, kvic will do a write-up of cascading hash tables in Dnsmasq which could potentially be of very good use in AB-Solution.
Thank you.

Liked, and liked that you've noticed and pointed this out to us.
Is there any way I can send both (each) of you some bitcoin in thanks?

 

[thelonelycoder]

Liked, and liked that you've noticed and pointed this out to us.
Is there any way I can send both (each) of you some bitcoin in thanks?

Last I heard, kvic does not accept donations. I have a PayPal donation link on my website.
I still have my reservations with cryptocurrencies. Being Swiss does not help in making a decision to use Bitcoin, Ethereum, Litecoin, Ripple or any others of those currencies.

 

[2992]

it'll be interesting to see whether bild.de will work... as now it doesn't.

 

[thelonelycoder]

it'll be interesting to see whether bild.de will work... as now it doesn't.

Is it because you see this helpful hint?

Mit aktiviertem Adblocker können Sie BILD.de nicht mehr besuchen

(Deactivate Adblocker to view content)
I'm using the AdsBeGone! blocking file and if I whitelist acdn.adnxs.com then access is graciously allowed.
Clear OS and Browser cache after adding (and processing) it.

 

[2992]

Is it because you see this helpful hint?

(Deactivate Adblocker to view content)
I'm using the AdsBeGone! blocking file and if I whitelist acdn.adnxs.com then access is graciously allowed.
Clear OS and Browser cache after adding (and processing) it.

Whitelisting a site is always a working solution (thanks for the hints of how to do it as I didn't knew what to whitelist so I have always used LTE when reading such sites instead of WiFi), the idea is to find a way to block (at least some of) the ads on similar sites like bild.de (site which sometimes I'm reading it)... and when I've seen the comments above and the charts with how fast pixelserv blocks versus 0.0.0.0, I was thinking (as I am not that technical) that the next pixelserv can (hopefully/eventually) do that.

 

[thelonelycoder]

Whitelisting a site is always a working solution (thanks for the hints of how to do it as I didn't knew what to whitelist so I have always used LTE when reading such sites instead of WiFi), the idea is to find a way to block (at least some of) the ads on similar sites like bild.de (site which sometimes I'm reading it)... and when I've seen the comments above and the charts with how fast pixelserv blocks versus 0.0.0.0, I was thinking (as I am not that technical) that the next pixelserv can (hopefully/eventually) do that.

Blocking with the 0.0.0.0 IP or sending it through pixelserv-tls is basically the same. Pixelserv-tls adds support for https websites which cannot be done with the null IP.
Pixelserv also fills some of the gaps that would glaringly stare at you if you use the null IP.

For you to understand: For pixelserv-tls to work, it needs a helper to send those host/domain requests to it. This is what the blocking and blacklist files do in AB-Solution. Pixelserv cannot work on it's own.
So, selecting a blocking file type and white and blacklisting is what you can and have to do. The rest is just magic and goodies from pixelserv.

 

[Protik]

Is it possible to restore an Ab-solution installation?

It seems I might have to format my USB. I don't want to loose the stats and go through the whole process again. If not, what's the best way to do this?

 

[thelonelycoder]

Is it possible to restore an Ab-solution installation?

It seems I might have to format my USB. I don't want to loose the stats and go through the whole process again. If not, what's the best way to do this?

Copy all files first to a save place. For AB it's the ab-solution.sh file and the complete /adblocking/ and /entware/ folder.
Then re-format device and put these files back on it.
Plug it into the router and start AB with the long start command:
sh /mnt/<your device>/ab-solution.sh
AB-Solution will take care of everything while starting up.

 

[Protik]

Copy all files first to a save place. For AB it's the ab-solution.sh file and the complete /adblocking/ and /entware/ folder.
Then re-format device and put these files back on it.
Plug it into the router and start AB with the long start command:
sh /mnt/<your device>/ab-solution.sh
AB-Solution will take care of everything while starting up.

That's slick!

Thanks!!

 

[heysoundude]

Last I heard, kvic does not accept donations. I have a PayPal donation link on my website.
I still have my reservations with cryptocurrencies. Being Swiss does not help in making a decision to use Bitcoin, Ethereum, Litecoin, Ripple or any others of those currencies.

Silly me - I was under the impression you could buy bitcoin at any rail kiosk, or use it to buy tickets on the Swiss Rail system...and that it might be more widely adopted.

 

[thelonelycoder]

Silly me - I was under the impression you could buy bitcoin at any rail kiosk, or use it to buy tickets on the Swiss Rail system...and that it might be more widely adopted.

Yes, sorry. My comment was unnecessary.
I just never had the need to look into cryptocurrencies. That's why.
No derogatory meaning, just the way I see it. I appreciate you wanting to contribute to AB-Solution in a way you feel comfortable with.
Let me sit down in a quiet moment and reevaluate the situation, I might be too conservative in this matter.

 

[thelonelycoder]

@.TT. did you test the router stats function on your RT86U?
If not, could you run a manual job and check if IP's are resolved to the hostname as they appear in the devices list?
I'm asking because you reported the dnsmasq.leases file error in the Skynet thread.
AB uses the same file.