AdBlocking with combined hosts file

[tijaune]

adblocking from first post stop working when i turned on DNS filtering in web-interface

Hi thelonelycoder,

Maybe you can edit the first post to add this advise of turning off dns filtering. I had Norton Connect Safe dns filtering on and fumbled for hour making changes, revising, rebooting before find out. Now it is working great.

Thanks, it is truly useful

 

[thelonelycoder]

I was hoping there was a better way -- without duplicating every line -- but if not, this seems to work for now!

Out of future interest I double checked your findings. You are correct that both IPv4 and IPv6 entries need to be in the hosts file to block them. There is - at the moment - no way around it.

Also, I'll give you a hint of how far you can go with the file size of the hosts file on the Asus Router: beyond 5 MB! I have tested this on my RT-AC66U for several days by including a hosts file from hpHosts in my script, along with the other three already included. It did take some time to process and combine the four files on the router, but it managed to do it.
While I had it running I did notice a drop in responsivness while loading Websites though. This and the too restrictive blocking of certain websites made me end that experiment and run the scripts as posted in this thread.
Still, I am impressed of what my RT-AC66U can do!

 

[thelonelycoder]

Hi thelonelycoder,

Maybe you can edit the first post to add this advise of turning off dns filtering. I had Norton Connect Safe dns filtering on and fumbled for hour making changes, revising, rebooting before find out. Now it is working great.

Thanks, it is truly useful

Thanks for pointing this out, I have edited the post.

 

[ScottW]

Out of future interest I double checked your findings. You are correct that both IPv4 and IPv6 entries need to be in the hosts file to block them. There is - at the moment - no way around it.

Also, I'll give you a hint of how far you can go with the file size of the hosts file on the Asus Router: beyond 5 MB!

Good to know. It appears dnsmasq is very efficient at handling a large hosts file.

The three recommended files combined, then doubled to create an ipv6 entry for each, results in over 49,000 entries and a file size of about 1.2mb. When active, it decreases available memory on my N66u by about 4mb.

Some kind of script could probably be used to lookup each domain from the 0.0.0.0 list, and only create ipv6 entries for those which actually have an ipv6 DNS record rather than just duplicating every entry. But given how efficient dnsmasq is with the large file, I'm not sure this is worth the effort.

As a crude performance test, I did several runs of the "dns benchmark" utility from grc.com, with and without the 49,000+ hosts file active. I was unable to identify any difference in cached or uncached lookup performance. If there is a difference, it is imperceptible (at least with this test).

Many THANKS to thelonelycoder and all the others (Xerxist, ryzhov_al, sinshiva, others) for putting together the method and especially the simple instructions in this thread!

 

[enr00ted]

I think this should be on Merlin's Github Wiki

 

[davejsb]

Since I updated to the latest Merlin version, 378.51, the cron job which updates my hosts file at midnight every Friday has stopped working. I can manually run update-hosts.sh no problem.

I've checked the entries in /jffs/scripts/init-start which appear to be in order.

Can anyone help me resolve the issue please?

Thanks in advance.

 

[octopus]

Have you done chmod on file?

chmod a+rx /jffs/scripts/init-start

 

[thelonelycoder]

Since I updated to the latest Merlin version, 378.51, the cron job which updates my hosts file at midnight every Friday has stopped working. I can manually run update-hosts.sh no problem.

I've checked the entries in /jffs/scripts/init-start which appear to be in order.

Can anyone help me resolve the issue please?

Thanks in advance.

1. Try to add the cron jobs in /jffs/scripts/services-start instead of /jffs/scripts/init-start
2. Check formatting, there needs to be a space between 5 and / (cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh")
3. Setting Administration/System/"Enable JFFS custom scripts and configs" set to 'yes'?
4. Check if cron job is scheduled in Terminal. Enter

cru l

and check the output. It will list all the cronjobs added at boot to /var/spool/cron/crontabs/admin

 

[ColinTaylor]

That should be

cru l

Note that it's a lower case L not a number one.

 

[thelonelycoder]

Note that it's a lower case L not a number one.

Thanks, edit is done.

 

[davejsb]

Thanks for all of your help. I've added the cron job in /jffs/scripts/services-start instead of /jffs/scripts/init-start and it appears to be working. Thanks so much.

 

[thelonelycoder]

Thanks for all of your help. I've added the cron job in /jffs/scripts/services-start instead of /jffs/scripts/init-start and it appears to be working. Thanks so much.

I am sure others will have the same problem. I never noticed because I've had it in the /jffs/scripts/init-start for some time. Edit in Post 1: Done.

 

[waffles]

I have been using the privoxy ad blocking, but keep running into issues. Before that I had tried thelonelycoder's solution. While I did not experience these kind of issues, there were features that I prefer and therefore made me switch:
- it uses AdblockPlus EasyList filters. These lists have been working very well on my PCs. Much better actually than the ones used here.
- I can define a range of IP addresses, rather than mandating it for all the devices on the LAN

I was wondering if these 2 features could not be included in thelonelycoder's package. For me this would be the best of both 'adblock worlds'.

 

[thelonelycoder]

I have been using the privoxy ad blocking, but keep running into issues. Before that I had tried thelonelycoder's solution. While I did not experience these kind of issues, there were features that I prefer and therefore made me switch:
- it uses AdblockPlus EasyList filters. These lists have been working very well on my PCs. Much better actually than the ones used here.
- I can define a range of IP addresses, rather than mandating it for all the devices on the LAN

I was wondering if these 2 features could not be included in thelonelycoder's package. For me this would be the best of both 'adblock worlds'.

Short answer: No

Long answer: As I understand, EasyList on your router filters not (only) by domain, they actively block the ad parts of a website and fill the void with with a filler box. It preprocesses the the document before it is sent to the browser.
This is impossible with a hosts file.
And as for the IP range, the same applies. It is not possible to do it with the hosts file.
As stated in my (recently updated) first post: This is a simple but effective way to block ads on all your devices.

 

[bilboSNB]

Anyone noticed ads sneaking through on android you tube app? This is on a tablet with wi-fi only, no cellular.

 

[waffles]

Short answer: No

Long answer: As I understand, EasyList on your router filters not (only) by domain, they actively block the ad parts of a website and fill the void with with a filler box. It preprocesses the the document before it is sent to the browser.
This is impossible with a hosts file.
And as for the IP range, the same applies. It is not possible to do it with the hosts file.
As stated in my (recently updated) first post: This is a simple but effective way to block ads on all your devices.

Thanks for taking the time to reply and to explain.

PS: I did not expect that the current version would already be able to do this or that it would be a simple tweak to implement all this. I was thinking more of an advanced / next / iteration or even version.

 

[thelonelycoder]

Anyone noticed ads sneaking through on android you tube app? This is on a tablet with wi-fi only, no cellular.

I am going to look into that asap.

 

[thelonelycoder]

Thanks for taking the time to reply and to explain.

PS: I did not expect that the current version would already be able to do this or that it would be a simple tweak to implement all this. I was thinking more of an advanced / next / iteration or even version.

Don't tell anyone... but I do some experimenting in that direction. It'll take some time but it looks promising.

 

[waffles]

Don't tell anyone... but I do some experimenting in that direction. It'll take some time but it looks promising.

Excellent! ... and no, I will not tell anyone

Please let me know if you need anyone to test.

 

[bilboSNB]

amongst this somewhere:

r 27 19:55:22 dnsmasq[822]: query[A] ssl.google-analytics.com from 192.168.0.9
Mar 27 19:55:22 dnsmasq[822]: cached ssl.google-analytics.com is <CNAME>
Mar 27 19:55:22 dnsmasq[822]: cached ssl-google-analytics.l.google.com is 216.58.211.72
Mar 27 19:55:33 dnsmasq[822]: query[A] s.youtube.com from 192.168.0.9
Mar 27 19:55:33 dnsmasq[822]: /jffs/hosts.clean s.youtube.com is 0.0.0.0
Mar 27 19:55:35 dnsmasq[822]: query[A] r1---sn-aigllnes.googlevideo.com from 192.168.0.9
Mar 27 19:55:35 dnsmasq[822]: forwarded r1---sn-aigllnes.googlevideo.com to 208.67.222.222
Mar 27 19:55:35 dnsmasq[822]: forwarded r1---sn-aigllnes.googlevideo.com to 208.67.220.220
Mar 27 19:55:35 dnsmasq[822]: query[A] r1---sn-aigllnes.googlevideo.com from 192.168.0.9
Mar 27 19:55:35 dnsmasq[822]: forwarded r1---sn-aigllnes.googlevideo.com to 208.67.222.222
Mar 27 19:55:35 dnsmasq[822]: reply r1---sn-aigllnes.googlevideo.com is <CNAME>
Mar 27 19:55:35 dnsmasq[822]: reply r1.sn-aigllnes.googlevideo.com is 74.125.105.134
Mar 27 19:55:35 dnsmasq[822]: reply r1---sn-aigllnes.googlevideo.com is <CNAME>
Mar 27 19:55:35 dnsmasq[822]: reply r1.sn-aigllnes.googlevideo.com is 74.125.105.134
Mar 27 19:55:35 dnsmasq[822]: query[A] googleads.g.doubleclick.net from 192.168.0.9
Mar 27 19:55:35 dnsmasq[822]: /jffs/hosts.clean googleads.g.doubleclick.net is 0.0.0.0
Mar 27 19:55:36 dnsmasq[822]: query[A] csi.gstatic.com from 192.168.0.9
Mar 27 19:55:36 dnsmasq[822]: cached csi.gstatic.com is 74.125.198.120
Mar 27 19:55:36 dnsmasq[822]: cached csi.gstatic.com is 74.125.198.94
Mar 27 19:55:36 dnsmasq[822]: query[A] yt3.ggpht.com from 192.168.0.9
Mar 27 19:55:36 dnsmasq[822]: forwarded yt3.ggpht.com to 208.67.222.222
Mar 27 19:55:36 dnsmasq[822]: reply yt3.ggpht.com is <CNAME>
Mar 27 19:55:36 dnsmasq[822]: reply photos-ugc.l.googleusercontent.com is 173.194.40.139
Mar 27 19:55:36 dnsmasq[822]: reply photos-ugc.l.googleusercontent.com is 173.194.40.138
Mar 27 19:55:36 dnsmasq[822]: reply photos-ugc.l.googleusercontent.com is 173.194.40.140
Mar 27 19:55:38 dnsmasq[822]: query[A] pagead2.googlesyndication.com from 192.168.0.9
Mar 27 19:55:38 dnsmasq[822]: /jffs/hosts.clean pagead2.googlesyndication.com is 0.0.0.0