AdBlocking with combined hosts file

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

Smacs

New Around Here
First of all, thanks, this works really good. Nice way to get rid of ads especially on tablets or phones, lacking an adblock extension.

But to be somewhat paranoid - what are the chances that any of the three lists could be used for injecting a couple of malicious entries into our routers, redirecting some valid URL's (PayPal etc.) to a phishing server?
 

thelonelycoder

Part of the Furniture
First of all, thanks, this works really good. Nice way to get rid of ads especially on tablets or phones, lacking an adblock extension.

But to be somewhat paranoid - what are the chances that any of the three lists could be used for injecting a couple of malicious entries into our routers, redirecting some valid URL's (PayPal etc.) to a phishing server?
None. Look at the script where it says 'grep -w ^0.0.0.0'
It will only add lines starting with 0.0.0.0. to the hosts.clean file.
 

Weble

New Around Here
Hey guys I have added a little extra into my script to grab host files from other sites that use the 127.0.0.1 standard. It will grab the file, convert the local host ip to 0.0.0.0 and the rest is the same. Warning this makes your host file quite large at 24mb ish but its working fine on my n66u

Code:
#!/bin/sh

# set directory
dir=/mnt/sda1
# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' > $dir/temp
#get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
wget -qO- "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://support.it-mate.co.uk/downloads/hosts.txt" "http://hostsfile.mine.nu/Hosts" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/temp | sort -u | fgrep -vf $dir/whitelist.txt > $dir/hosts.clean
rm $dir/temp

#restart dnsmasq to apply changes
sleep 3
service restart_dnsmasq

I also made a script to update the whitelist without having to rerun the whole script.
Code:
#!/bin/sh

# set directory
dir=/tmp/mnt/sda1
# remove whitelisted entries in temp and write final file, remove temp file
mv $dir/hosts.clean $dir/temp
cat $dir/temp | fgrep -vf $dir/whitelist.txt > $dir/hosts.clean
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq
 
Last edited:

thelonelycoder

Part of the Furniture
Hey guys I have added a little extra into my script to grab host files from other sites that use the 127.0.0.1 standard. It will grab the file, convert the local host ip to 0.0.0.0 and the rest is the same. Warning this makes your host file quite large at 24mb ish but its working fine on my n66u

Thanks for the addition. I have done this before, and added my 2¢ to it in post 122.
I am going to run my router with this for a few days to see how it performs.
It uses quite a bit of RAM, some tests reveal this:

RAM readings on my RT-AC66U:
Fresh reboot, w/o scripts (standard merlin settings): 61MB
Fresh reboot, with standard hosts scripts as in post 1: 64MB
Fresh reboot, with standard hosts scripts, then ran updatehosts.sh with Weble's addition: 171MB
Fresh reboot, with Weble's addition: 136MB

Looking good so far, will report back after a few days use.

Edit: Just to clarify:
The 'http://support.it-mate.co.uk/downloads/hosts.txt' file is a mirror of hpHosts (http://hosts-file.net/download/hosts.txt)

Edit 2: To make Google Drive work again I had to add these to the whitelist:
#allow google drive
clients2.google.com
clients3.google.com
clients4.google.com
clients5.google.com
 
Last edited:

enr00ted

Regular Contributor
Any news with the last modifications/additions ? o_O
 

thelonelycoder

Part of the Furniture
Any news with the last modifications/additions ? o_O
Are you referring to the one in the post above yours?
A follow up post is planned for today.
 
Last edited:

thelonelycoder

Part of the Furniture
I have tested Weble's findings (in a post above) and have added an Advanced method in post #1 in this thread.
Since this thread is about adblocking I have only added pure Adblock lists and removed the outdated hosts files in his script.
The Advanced method supports a whitelist and a blacklist. Note that the original (Standard) and Advanced method are not compatible. You will have to delete the old scripts to run the Advanced method.
 

tijaune

Regular Contributor
Hi thelonelycoder,

In post 1, Advanced method of update-hosts.sh, there is a slight error missing .txt at the end, should be as follow...
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked.txt
... this will match with /jffs/config/dnsmasq.conf.add.
 
Last edited:

thelonelycoder

Part of the Furniture
Hi thelonelycoder,

In post 1, Advanced method of update-hosts.sh, there is a slight error missing .txt at the end, should be as follow...

... this will match with /jffs/config/dnsmasq.conf.add.
Thank you for reporting, it now matches the script, without the .txt extension as intended.
 

decedion

Occasional Visitor
I have done this modifications:

/jffs/scripts/post-mount
Code:
#!/bin/sh

if [ $1 = "/tmp/mnt/usb" ]
then
  ln -sf $1/entware /tmp/opt

  wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0" "http://hosts-file.net/ad_servers.asp" "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/download/hosts.txt" "http://hostsfile.mine.nu/Hosts"  | egrep -w "^(127.0.0.1)|^(0.0.0.0)" | sed '/0.0.0.0 0.0.0.0/d' | sed 's/\r$//' | sed 's/127.0.0.1/0.0.0.0/' | sed '/localhost/d' | LC_ALL=C sort -fiu -k 2 > $1/hosts.clean
  sleep 5
  service restart_dnsmasq
fi

NOTE: The file created is big (25.7M).

There is no need of sleep before creating the adblock file if we include that process in 'if [ $1 = "/tmp/mnt/sda1" ]...'. In addition thus we ensure that the unit is mounted and, in case of several connected units, the process only it executes one time.
I don't sure if it's required the 'sleep' after 'wget...'. :cool:

/jffs/scripts/services-start
Code:
#!/bin/sh....
cru a reboot "0 6 */5 * * reboot"
....

With this change every 5 days at 6 o'clock in the morning the router is rebooted and the adblock file is regenerated.
-------------------------------------
If you don't want reboot the router every n days but you want update the file every n days, you have to create a file with the next code:
/jffs/scripts/create-adblock-file
Code:
#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0" "http://hosts-file.net/ad_servers.asp" "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/download/hosts.txt" "http://hostsfile.mine.nu/Hosts"  | egrep -w "^(127.0.0.1)|^(0.0.0.0)" | sed '/0.0.0.0 0.0.0.0/d' | sed 's/\r$//' | sed 's/127.0.0.1/0.0.0.0/' | sed '/localhost/d' | LC_ALL=C sort -fiu -k 2 > $1/hosts.clean
sleep 5
service restart_dnsmasq

And you have to change
/jffs/scripts/post-mount
Code:
#!/bin/sh

if [ $1 = "/tmp/mnt/usb" ]
then
  ln -sf $1/entware /tmp/opt

  . /jffs/scripts/create-adblock-file
fi

And you have to change
/jffs/scripts/services-start
Code:
#!/bin/sh....
cru a update-adblock "0 6 */5 * * /jffs/scripts/create-adblock-file"
....

You can change the first 2 numbers by the minutes and hours respectively and the third number by number of days to update.
Regards.
 

thelonelycoder

Part of the Furniture
NOTE: The file created is big (25.7M).

There is no need of sleep before creating the adblock file if we include that process in 'if [ $1 = "/tmp/mnt/sda1" ]...'.
I don't sure if it's required the 'sleep' after 'wget...'. :cool:
Thanks for that decedion. I am no fan of that big of a hosts file. It introduces too many blockages for sites that I never even visit, even as a third party by cookies. I did do a test for about a week with the HP hosts file and some others (see earlier post in this thread). I had to constantly add hosts to the whitelist to get sites and services working in my LAN. I am now running the Advanced method as described in Post #1 and have had no issues so far.
The sleep times are there for another reason. I want to give some breathing time for the cpu while it is starting up. Downloading, sed, combining and sort are quite memory and cpu heavy operations. That is the case especially with the 25MB hosts file you have. Give it some slack so it can do it's other jobs as well.
 

decedion

Occasional Visitor
The sleep times are there for another reason. I want to give some breathing time for the cpu while it is starting up. Downloading, sed, combining and sort are quite memory and cpu heavy operations. That is the case especially with the 25MB hosts file you have. Give it some slack so it can do it's other jobs as well.
ok for the sleep after the process of sort, combine, etc. but ¿before?
 

thelonelycoder

Part of the Furniture
ok for the sleep after the process of sort, combine, etc. but ¿before?
I agree with you, the sleep before running the scripts is not really needed, I just have it there because I've had scripts not running at reboot without it and just left it in for the time being. I may remove it after tests with the next merlin release (.53).
 

tijaune

Regular Contributor
I just used the advanced method on rt-n16 running with the last supported merlin 376.49_5. Normal method has been running for some weeks already. Tools page still show 64 MB free memory. Will need feedback from my old man if he experience slowdown or not.
 

thelonelycoder

Part of the Furniture
Will need feedback from my old man if he experience slowdown or not.
If he does, let me know. I'll handle the complaints.
 

thelonelycoder

Part of the Furniture
rt-n16 running with the last supported merlin 376.49_5
How does the reboot with the new scripts work? I am asking because update-hosts.sh is started in the services-start file at boot.
 

tijaune

Regular Contributor
How does the reboot with the new scripts work? I am asking because update-hosts.sh is started in the services-start file at boot.

oh, I did not reboot his router because I was using TeamViewer to help him clean remotely. However I expected it to work as on mine (ac66u), I remember it works correctly.
 

GregS

Occasional Visitor
I'm on an Asus RT-AC68 and for some reason this line just produces a blank hosts.blocked file:
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

There's no error and the rest of the script works fine. If I go through the process line by line I can see the temp file does exists and has tons of normal looking entries. I do have a whitelist.txt as specified in post #1 with a few dummy domains.

Any ideas?
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top