AB-Solution - The Ad Blocking Solution

[thelonelycoder]

I thought about a feature yesterday. To purge our blacklist file from time to time it could be useful to show similar between the hosts file which are downloaded and our blacklist.txt

For instance if i ad some ads to my blacklist file as it's not present on stevenblack one and i add it on stevenblack after on github. What do you think about thelonelycoder ?

Sounds like an interesting feature and if built in, it would make sense to expand this to the whitelist as well.
But I have my doubts such a purge is useful and helpful for the longer term.

Domains are added to the list because some app or website is not working as it should.
Changing the blocking file type may then include or exclude the domain, but you still want that domain in the list in case it is added or removed from the hosts file .

Example: I add baddomain.com to the blacklist because of reasons and it is not included in the current blocking file I use.
One day, baddomain.com is added to one of the hosts files. Good, I can remove it from the blacklist --> done.
Scenario:
1. I change the blocking file type and baddomain.com is not included in it --> ugh, better re-add it to the blacklist.
2. The maintainer decides to remove baddomain.com from the hosts file --> ugh, better re-add it to the blacklist.

The same applies for the whitelist. You added it for a specific reason and always want that domain whitelisted, no matter what the hosts file maintainers do or what blocking file type you select.

Since both lists are domains you added for a specific reason I'd rather not have it purged by an automatism that does not take into consideration the reason why it was added in the first place.
Having baddomain.com in the blocking file and the blacklist is no problem at all. Dnsmasq will block it as soon as it finds it in one of the lists, omitting the second instance in the other.
Usually, the white/blacklist is a relatively small file. My blacklist has about 400 entries while the whitelist only has about 50.

And as a final note to this, the hosts files sizes change considerably over time.
With each update of the ab-solution.sh, I have to significantly increase the domain count and file size for the blocking file types.

So, in my conclusion, this change may not be welcome at all, but I'm always open for suggestions and improvements if they make sense. Or users can convince me with good reasons to add it.

 

[steelskinz]

You convince me it's not so useful option..

 

[thelonelycoder]

You convince me it's not so useful option..

I appreciate input in any way. Don't hold back on ideas.

AB-Solution is at a threshold at the moment.
Ad-blocking wise, there ia not much left to improve it. The routers limited recources are a hard stop for any ideas I have to make it more useful or granular in its capability.
V3.9x is as far as I can/will go with the code basis I built to install and manage the services AB-Solution offers.

I mentioned this on several occasions before: AB-Solution 4.0 will be a completely different app.
To give you a hint where this is going: I have a more or less working copy of Pi-Hole running on one of my Asus test routers.
This is to test the WebUI features I plan to implement and how much resources this will take.
My idea is to NOT fork Pi-Hole and port it to Asuswrt-Merlin but offer a similar experience in AB 4.x.
It is planned to have both the current terminalUI and a WebUI.

 

[Henrik!]

To give you a hint where this is going

Uhh.. Now I can't wait

 

[superjet]

I appreciate input in any way. Don't hold back on ideas.

AB-Solution is at a threshold at the moment.
Ad-blocking wise, there ia not much left to improve it. The routers limited recources are a hard stop for any ideas I have to make it more useful or granular in its capability.
V3.9x is as far as I can/will go with the code basis I built to install and manage the services AB-Solution offers.

I mentioned this on several occasions before: AB-Solution 4.0 will be a completely different app.
To give you a hint where this is going: I have a more or less working copy of Pi-Hole running on one of my Asus test routers.
This is to test the WebUI features I plan to implement and how much resources this will take.
My idea is to NOT fork Pi-Hole and port it to Asuswrt-Merlin but offer a similar experience in AB 4.x.
It is planned to have both the current terminalUI and a WebUI.

This is A++++ right here. I use both in different networks at the moment and would love to merge the two. That is bounty worthy.

 

[heysoundude]

Forge ahead, @thelonelycoder! I'm hoping a web UI will help me figure out why my aljazeera phone app has stopped loading. It's fine when I'm not on my wifi, it's fine on the wifi when I'm connected to a vpn, and this has only been since the last update...and ad-blocking doesn't happen when on the vpn.
which (now that I think about it) makes sense - I'm in a tunnel to a non-adblocked server. duh!

 

[steelskinz]

Sounds exciting !

One thing which is missing in Pi Hole is the option to use dnscrypt.

I found that list of parsed txt file for blocking. Huge work !

https://github.com/deathbybandaid/piholeparser/tree/master/parsed

Could it be used in ABS? I don't know why it won't but i'm not at home to test 1 file.

 

[thelonelycoder]

Forge ahead, @thelonelycoder! I'm hoping a web UI will help me figure out why my aljazeera phone app has stopped loading. It's fine when I'm not on my wifi, it's fine on the wifi when I'm connected to a vpn, and this has only been since the last update...and ad-blocking doesn't happen when on the vpn.
which (now that I think about it) makes sense - I'm in a tunnel to a non-adblocked server. duh!

The f option 1 or 2 is made to see what is blocked. Make a good guess what domain needs whitelisting, this is trial and error.

 

[thelonelycoder]

One thing which is missing in Pi Hole is the option to use dnscrypt.

Dnscrypt is an upstream service. Roughly it encrypts all requests between dnsmasq and an upstream server.
This is not part of ad-blocking. In your Asus router this is trivial to add using @bigeyes0x0 installer found here.
As for your Pi-Hole device, this is a matter of apt-get, yum or whatever package manager the distro uses.
In other words, I and the Pi-Hole coders leave that to the respective package or script maintainers as it is outside of ad-blocking.

 

[thelonelycoder]

I found that list of parsed txt file for blocking. Huge work !
https://github.com/deathbybandaid/piholeparser/tree/master/parsed
Could it be used in ABS? I don't know why it won't but i'm not at home to test 1 file.

These are just stripped down hosts files and a lot of them are empty. The format is not compatible with the current AB version as it only contains domains and no blocking IP.
I'm not sure this will be supported in AB4.

It would certainly help to not have to do the stripping on the router but I rather rely on the source file than entirely on a third party script to prepare the files for my adblocker. I do this already by using combined lists for some of the blocking file types (stephenblack).

What will return in the blacklist is the wildcard option for domains.

 

[thelonelycoder]

Uhh.. Now I can't wait

As with all good things: It takes time. A lot of it.
Come to think of it, I regret my username . But then, that's how I prefer to work. Independent and free to decide what to do.

 

[Henrik!]

As with all good things: It takes time. A lot of it.
Come to think of it, I regret my username . But then, that's how I prefer to work. Independent and free to decide what to do.

Maybe it can be a good Christmas present [emoji2]

 

[JemTheWire]

I appreciate input in any way. Don't hold back on ideas.

AB-Solution is at a threshold at the moment.
Ad-blocking wise, there ia not much left to improve it. The routers limited recources are a hard stop for any ideas I have to make it more useful or granular in its capability.
V3.9x is as far as I can/will go with the code basis I built to install and manage the services AB-Solution offers.

I mentioned this on several occasions before: AB-Solution 4.0 will be a completely different app.
To give you a hint where this is going: I have a more or less working copy of Pi-Hole running on one of my Asus test routers.
This is to test the WebUI features I plan to implement and how much resources this will take.
My idea is to NOT fork Pi-Hole and port it to Asuswrt-Merlin but offer a similar experience in AB 4.x.
It is planned to have both the current terminalUI and a WebUI.

You will have more chocolate than you know what to do with, lol.

 

[thelonelycoder]

You will have more chocolate than you know what to do with, lol.

Better start planning for it then and get my weight down.
I've gone through the +/- 25kg within a year several times. I basically have two wardrobes. Containing the + and the - outfits.

 

[thelonelycoder]

Maybe it can be a good Christmas present [emoji2]

I released AB 2.0.1 on September 13, 2016. On November 4 the same year AB 3 came out.
AB 3 added pixelserv and was a huge step forward.
AB 2.0.1 was a total of 84 KB, all server files counted.
AB 3: 170 KB
AB 3.9.1: 310 KB
Estimate for AB 4: beyond the 1.5 MB range.

 

[steelskinz]

What will return in the blacklist is the wildcard option for domains.

Sounds good !

 

[MarCoMLXXV]

Installed Ubuntu Gnome 17.04 yesterday (and this morning again) which has some serious DNS resolving issues, which I was aware of prior to installing. Managed to fix them all but somehow Ubuntu finds a way to resolve IP's, bypassing AB-Solution and pixelserv-TLS completely, even when I have manually entered 192.168.1.1 as DNS-server in network settings and edited several files to make sure network-manager can only use 192.168.1.1 to resolve. But still... argh... ads everywhere. Feeling kinda spoiled now when I need to surf without AB's functionality. The Interwebs have more ads apparently then content by now. Yuk. Haven't found a solution yet, read dozens of topics, but most are simply about getting FQDN's resolved again, not forcing Ubuntu to use ones router for resolution (and ad-blocking). Getting somewhat frustrated here.

 

[visortgw]

Installed Ubuntu Gnome 17.04 yesterday (and this morning again) which has some serious DNS resolving issues, which I was aware of prior to installing. Managed to fix them all but somehow Ubuntu finds a way to resolve IP's, bypassing AB-Solution and pixelserv-TLS completely, even when I have manually entered 192.168.1.1 as DNS-server in network settings and edited several files to make sure network-manager can only use 192.168.1.1 to resolve. But still... argh... ads everywhere. Feeling kinda spoiled now when I need to surf without AB's functionality. The Interwebs have more ads apparently then content by now. Yuk. Haven't found a solution yet, read dozens of topics, but most are simply about getting FQDN's resolved again, not forcing Ubuntu to use ones router for resolution (and ad-blocking). Getting somewhat frustrated here.

Is DNS caching locally on the Ubuntu box?

 

[MarCoMLXXV]

Is DNS caching locally on the Ubuntu box?

Yup, but thanks for asking. I flushed it, after making the necessary changes, to get it to resolve again and that worked. When I found out it was still bypassing local dns (local, as in router) I wiped the partition of the laptop and started fresh, disabled systemd-resolve (which is suspected to be the cause of it all) and installed an alternative dns resolver (unbound), before even opening a browser, but still the same problem. As I don't want go off-topic and it isn't caused by AB-Solution (my ancient, made-out-of-a-peace-of-Asus-rock netbook works like a charm and some searching on 'ubuntu 17.04 dns' speaks for itself), we'd probably better leave it at this, as I don't want to p*** off @thelonelycoder (been there, done that, don't feel like doing it again ). Nevertheless, thanks for thinking along with me.

 

[thelonelycoder]

I wiped the partition of the laptop and started fresh, disabled systemd-resolve (which is suspected to be the cause of it all) and installed an alternative dns resolver (unbound), before even opening a browser, but still the same problem.

I'd have to fire up an old laptop with Ubuntu on it but lack the time, but have you tried disabling dnsmasq?
http://mark.orbum.net/2012/05/14/disabling-dnsmasq-as-your-local-dns-server-in-ubuntu/

As I don't want go off-topic and it isn't caused by AB-Solution

You are more than welcome to bring it up here. This is very much related to this thread.

(...) as I don't want to p*** off @thelonelycoder (been there, done that, don't feel like doing it again ).

Since you bring this up from the last beta testing in a private thread, I feel comfortable to respond here in public as well.
I was pissed off at the time for the way it played out for all of us then. But that's over and in the past now.
I'm happy you found your way back to this thread a little while ago.