Blocking ADs with MVPS Host File

[Woogi]

So I was asked to write up what I have done in the last day or two. If this has been covered. I apologize.

Purpose: To block ads from all computers on the network at the router using the MVPS Custom Host file.

Requirements:
- You should have the Merlin Firmware with /jffs/ mounted. (I also used some space on an attached flash drive, but with ~7mb of usable space on /jffs, this probably isn't needed.)

- Custom /jffs/scripts/init-start
- Custom /jffs/configs/hosts
- I will assume you know/understand basic linux commands
- I will assume you know/understand what a host file does


Summary: After I have setup this up, the following happens:
- Upon boot, the router creates a CRONjob
- While still booting, the Merlin Firmware copies entries from /jffs/configs/hosts to /etc/hosts
- DNSMasq will read /etc/hosts and block all known ad servers
- At the specified time the cronjob from above will poll mvps.org, and download the newest host file to /jffs/configs/hosts
- When the router reboots, it will pull latest /jffs/configs/hosts to /etc/hosts


Process:

A) Create cronjob at boot
1) In order to create the cronjob, I created /jffs/scripts/init-start, and I have added the following to the file:

[strike]
#!/bin/sh
cru a 1001 @midnight wget http://winhelp2002.mvps.org/hosts.txt -O /jffs/configs/hosts
[/strike]
PLEASE SEE POST 8 BELOW

This adds a cron job with the following formatting:

cru a <unique id> <"min hour day month week command">

I found it best to use @daily, or @midnight, or @weekly, rather than the stanard time format. I had issues scheduling it if i used the standard cronjob syntax.


After the router reboots, any cronjobs are lost. However since we are placing this 'script' in the init-start file, the Merlin firmware will recreate this cronjob at every reboot.



That is it, however for since the /jffs/configs/hosts file is currently empty, if you want to 'jump start' this process I would also do the following:

wget http://winhelp2002.mvps.org/hosts.txt -O /jffs/configs/hosts
killall dnsmasq
dnsmasq

This will populate the /jffs/configs/hosts file, and will restart the dnsmasq service, to reload the host file. I would also run the following on your local windows machine:

ipconfig /flushdns

This will cause your local machine to quary your DNS server (router) rather than using a local cache.




Please understand the effects of this, No not all ads will be blocked, Yes there may be some negitive effects, for example Hulu.com will offten run a 30-60sec black screen if it cant load an ad.

Please feel free to ask any questions, and I am sorry for any babbling above



---------------------------------------
Side note:

You could also add the following to your init-start file

#!/bin/sh
cru a 1001 @midnight wget http://winhelp2002.mvps.org/hosts.txt -O /jffs/configs/hosts

sleep 5
killall dnsmasq
dnsmasq

This will restart your DNSMasq service every night at midnight, thereby loading the new host file. However I my router gets rebooted once a month or more, so I am not that worried about it.

 

[uhlhazard]

Works like a champ!

 

[Zirescu]

What are the advantages/disadvantages of using this method over the Adblock Plus / privoxy method under Entware?

Thanks.

 

[jcy]

What are the advantages/disadvantages of using this method over the Adblock Plus / privoxy method under Entware?

Thanks.

your ipads and other mobile devices would also block the ads

 

[Zirescu]

Yeah, but that's already possible using privoxy as well. I was thinking more along the lines of performance and ad blocking capabilities.

 

[Woogi]

Yeah, but that's already possible using privoxy as well. I was thinking more along the lines of performance and ad blocking capabilities.

Well, while I have never used Privoxy, I have done some reading. Here are my thoughts.

1) Privoxy is a fully blown proxy server. So depending on what you are running Privoxy on, you are relaying on another 'box' to be the proxy setup.

2) Because Privoxy is a proxy server, there is client configuration that is required in enjoy the benefits. This is also true for adblock. With this method, any internet device (TV, roku, ipad, computer etc) plugged in behind the router will be able to block ads without the need for any configuration.

3) Does privoxy 'auto update' or is there any maintenance required? Using the method above, there is no work needed on your part to make sure you are running the latest host file.

4) Using absolutely no method of measurement, (and never using privoxy), I would think running a host file, be it locally or at the router, would be quicker for the following reasons:

i) No extra box to send or process the request. So even if the proxy was lighting fast, its still two less hops. (one going and one coming back)

ii) a proxy server will do more 'processing' of the entire page rather then a simple redirect (see below)

iii) (this is more of a simple explanation of what a host file does) When you request any website, say, MSN dot com, you are requesting an HTML page. This HTML page will load different parts of the page, including calling other web sites. For example, if you look at the source of msn dot com, you will see the following

<a href="http://investing.money.msn.com/investments/stock-price?Symbol=$INDU">

this line calls investing.money.msn.com to load a stock quote. So, its up to your local machine to go and poll that link for the required data. So your machine will try to lookup up the ip address of the link from a list of sources.

Local DNS Cache - Has it looked up this link recently? If so, it 'knows' the ip address

Local host file - is there in entry for this link in the local host file?

Local DNS Server - this is typically the router

Remote DNS Server - this is typically your ISP, and this request is handled by the router on behalf of the client, and where 90% of the DNS names are resolved to an IP address



Now, what the host file on the router is doing, is when the same site loads the following

http://ads1.msads.net/library/dapmsn.js

the same process goes into effect with a different outcome


Local DNS Cache - Has it looked up this link recently? If so, it 'knows' the ip address , which is why you should flush your dns cache when you first do this

Local host file - is there in entry for this link in the local host file? in our case, no because we load the host file on the router

Local DNS Server - this is typically the router the dnsmasq server on the router will see that there is the following entry

127.0.0.1 a.ads1.msads.net

So, your local brower will try and load:

http://127.0.0.1/library/dapmsn.js

And since its safe to say your local machine is not running a webserver, you will either see a blank white spot where the add should be, or a PAGE CAN NOT BE DISPLAYED error, the size of the add


Remote DNS Server - this is typically your ISP, and this request is handled by the router on behalf of the client, and where 90% of the DNS names are resolved to an IP address - This does not come into play, as the local machine was already able to resolve the ip


So that is the long was of saying, I think the host file, while limited in what ELSE it can do, is probably faster then privoxy on simply removing ads while browsing the web.

 

[TheTux]

Hi!

Working much faster than privoxy which sometimes leads in overflows for too much connections.

But one thing - the /jffs/configs/hosts only gets written to /etc/hosts on reboot of the server.
Just killing and reloading dnsmasq isn't doing the needed stuff.
Is there a way to force entware to do the replacement without rebooting?

Thanks
TheTux

 

[Woogi]

Hi!

Working much faster than privoxy which sometimes leads in overflows for too much connections.

But one thing - the /jffs/configs/hosts only gets written to /etc/hosts on reboot of the server.
Just killing and reloading dnsmasq isn't doing the needed stuff.
Is there a way to force entware to do the replacement without rebooting?

Thanks
TheTux

Tux, I'm glad this is working for you!

Ok, well after editing this post three times...and reading your post correctly, I have discovered the issue. (It early, and I am not fully awake yet).


Let me explain-

The reason we store the host entries in the /jffs/configs/hosts is because this is the only part of the router that will remain after a reboot. (Untested, as I am using a friends router to do all of my testing). So when the router reboots, its will load any entry in /jffs/configs/hosts into /etc/hosts, that will then be copied to /tmp/etc/hosts when dnsmasq is loaded.

However what you are seeing, is that we are only updating the /jffs/configs/hosts, which as you pointed out, only gets copied at reboot. So to resolve this, we need to do a few extra steps:

1) Create a script to restart dnsmasq

a) create a file /jffs/scripts/dnsrestart.sh
b) Copy the following into the file

#!/bin/sh
killall dnsmasq
dnsmasq

c) make the file executable

chmod 755 /jffs/scripts/dnsrestart.sh

This creates a script that will restart the dnsmasq service. Now we need to call the script after we update the host file. Make the following edits to the init-start file.

#!/bin/sh
cru a 1001 @midnight wget http://winhelp2002.mvps.org/hosts.txt -O /jffs/configs/hosts
cru a 1002 @midnight wget http://winhelp2002.mvps.org/hosts.txt -O /etc/hosts
cru a 1003 "2 0 * * * /jffs/scripts/dnsrestart.sh > /dev/null"

Let me walk you (and myself) through what is happening:

1) This will create a cronjob to update the /jffs/configs/hosts file every night at midnight. Updating this file will allow the latest host file to be loaded at the time of a reboot

2) This will create a cronjob to update the /etc/hosts file every night at midnight. This will allow the 'current' host file to be updated without the need for a restart of the router

3) This will create a cronjob to run the dnsrestart.sh script every night at 00:02 (12:02am, two minutes after the host file updates)


This should resolve the issue. I will update the first post shortly. Let me know if this works!

 

[RMerlin]

The following command will restart dnsmasq properly (including updating all the related config files):

service restart_dnsmasq

 

[TheTux]

Thank you Woogi and RMerlin!

 

[Woogi]

@RMerlin

Thanks for the command i tried a bunch of other syntax's and couldnt get it to work.

But you are suggesting that restarting dnsmasq correctly will re-read the file from the jffs mount?

 

[saabirsa]

I too am having problems with the number of connections when using Privoxy, and this seems like a plausible alternate solution. However, I currently have Privoxy set up to invoke with only a few IP addresses, and was wondering if there was a commend whereby this would be possible though this method?

 

[Woogi]

I too am having problems with the number of connections when using Privoxy, and this seems like a plausible alternate solution. However, I currently have Privoxy set up to invoke with only a few IP addresses, and was wondering if there was a commend whereby this would be possible though this method?

Can you elaborate as I am not too familer with privoxy?

Do you only want some clients in the network to block ads? Or do you only want some ip's to be blocked?

If you only want say your Apple TV to block the ads, this will not help you, but I you only want everyone to only block ads.google.com or Facebook.com then this can be modified to help.

 

[saabirsa]

Woogi - Apologies for the late response, I forgot to subscribe to the post and therefore was never informed of your reply.

As TheTux as mentioned earlier in the thread, I often get overflows leading to an error of too many connections. Therefore I am looking to move to the MPVS format as it works well when I have used it on HOSTS files directly.

With Privoxy, one of the commands to get it working on Merlin are as follows;

echo \#!/bin/sh > /jffs/scripts/firewall-start
echo iptables -t nat -A PREROUTING --source 192.168.0.101 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128 >> /jffs/scripts/firewall-start
chmod +x /jffs/scripts/firewall-start

This naturally means that Privoxy is only invoked for the IP address 192.168.0.101 as an example. I have extended this by copying the same command and adding the IP addresses that I specifically would like to have ad-blocked.

In essence, the response to your question is yes; I would like only some clients to block ads, depending on my selection. I do not particularly mind if this is via MAC address or IP address, whichever is easier.

However based on your response, I assume that this is not be possible through this method?

 

[halex4u]

Tux, I'm glad this is working for you!

Ok, well after editing this post three times...and reading your post correctly, I have discovered the issue. (It early, and I am not fully awake yet).


Let me explain-

The reason we store the host entries in the /jffs/configs/hosts is because this is the only part of the router that will remain after a reboot. (Untested, as I am using a friends router to do all of my testing). So when the router reboots, its will load any entry in /jffs/configs/hosts into /etc/hosts, that will then be copied to /tmp/etc/hosts when dnsmasq is loaded.

However what you are seeing, is that we are only updating the /jffs/configs/hosts, which as you pointed out, only gets copied at reboot. So to resolve this, we need to do a few extra steps:

1) Create a script to restart dnsmasq

a) create a file /jffs/scripts/dnsrestart.sh
b) Copy the following into the file



c) make the file executable



This creates a script that will restart the dnsmasq service. Now we need to call the script after we update the host file. Make the following edits to the init-start file.


Let me walk you (and myself) through what is happening:

1) This will create a cronjob to update the /jffs/configs/hosts file every night at midnight. Updating this file will allow the latest host file to be loaded at the time of a reboot

2) This will create a cronjob to update the /etc/hosts file every night at midnight. This will allow the 'current' host file to be updated without the need for a restart of the router

3) This will create a cronjob to run the dnsrestart.sh script every night at 00:02 (12:02am, two minutes after the host file updates)


This should resolve the issue. I will update the first post shortly. Let me know if this works!

Has anyone tried this script with the latest release 374.40_beta1?

I have tried, but it doesn't work with RT-N66U.

 

[fsatsil]

Would it be possible to get something like this to work on the n56u with Padavan's firmware?

 

[Mars]

Would it be possible to get something like this to work on the n56u with Padavan's firmware?

Yes, you can do roughly the same thing.

Using this current thread and this issue thread you should be able to figure it out if you're handy with the command line.

- DNSMasq reads from /etc/hosts
- You can keep the mvps.org hosts in /etc/storage/dnsmasq/hosts (This is where the webGUI hosts editor saves to - tricky part is that it can't save the size of the mvps file across reboot and truncates it, so you need to either deal with that, or I'm just lazy and set it to run wget on every boot of the router)
- Startup script to copy hosts and cycle DNSmasq goes in /etc/storage/started_script.sh (Might also work in some other place, scripts here go in some order, but started_script works for me.)

 

[assplats]

You can also get the file without writing to the JFFS partition (or USB disk), if you want to cut down on writes to flash memory.

I also pipe the file into egrep to only use lines that start with 0.0.0.0, just in case the site goes rogue and puts bogus entries in the DNS.

cru a AdBlock "00 00,12 * * * wget http://winhelp2002.mvps.org/hosts.txt -q -O- | egrep '^0.0.0.0' > /etc/hosts.dnsmasq && killall dnsmasq && dnsmasq"

Hope this helps.

 

[assplats]

Hey everyone,

I went a little further with the whole adblock process and added the ability to whitelist and blacklist certain hosts. It requires 3 scripts and a blacklist/whitelist files. Whitelist files are regex patterns, so you can whitelist and entire domain.

Script should be put on your usb or JFFS folder. Make sure you update the dnsmasq-update.sh adblockDirectory variable that is set at the top to the folder you're using.

Here are the script contents:

dnsmasq-update.sh

#!/bin/sh
#
# SCRIPT FOR DNSMASQ ADBLOCK UPDATES

# SET VARIABLES
adblockDirectory="/mnt/USBDRIVE/adblock"

# GET NEW hosts.dnsmasq FILE
/usr/bin/wget http://winhelp2002.mvps.org/hosts.txt -q -O- | tr -d '\r' | egrep '^0.0.0.0' > $adblockDirectory/hosts.dnsmasq.temp

# WHITELIST ANY REQUIRED DOMAINS FROM THE hosts.dnsmasq.whitelist FILE
$adblockDirectory/adblock-whitelist.sh hosts.dnsmasq.whitelist hosts.dnsmasq.temp

# BLACKLIST ANY REQUIRED DOMAINS FROM THE hosts.dnsmasq.blacklist FILE
$adblockDirectory/adblock-blacklist.sh hosts.dnsmasq.blacklist hosts.dnsmasq.temp

# START DIFF
# COMPARE NEW TO OLD AND ONLY RUN IF DIFFERENCE
DIFF=$(diff -b -i /etc/hosts.dnsmasq $adblockDirectory/hosts.dnsmasq.temp)
if [ "$DIFF" != "" ]; then
# THE DNSMASQ FILE WAS MODIFIED

# PREP LOG FILE
echo "There has been a change to the Dnsmasq files." > $adblockDirectory/hosts.dnsmasq.delta

# DIFF THE NEW FILE AGAINST THE LAST FILE APPEND TO LOG FILE
diff -b -i /etc/hosts.dnsmasq $adblockDirectory/hosts.dnsmasq.temp >> $adblockDirectory/hosts.dnsmasq.delta

# MOVE NEW hosts.dnsmasq FILE IN FOR USE BY DNS
mv $adblockDirectory/hosts.dnsmasq.temp /etc/hosts.dnsmasq

# RESTART DNS
killall dnsmasq && dnsmasq

# PUT MESSAGE IN SYSLOG
logger -p user.info -t AdBlock-Diff "There has been a change to the Dnsmasq files."
else
echo "There has been no change to the Dnsmasq files." > $adblockDirectory/hosts.dnsmasq.delta

# REMOVE hosts.dnsmasq TEMP FILE
rm $adblockDirectory/hosts.dnsmasq.temp

# PUT MESSAGE IN SYSLOG
logger -p user.info -t AdBlock-Diff "There has been no change to the Dnsmasq files."
fi

 

[assplats]

Hey everyone,

I went a little further with the whole adblock process and added the ability to whitelist and blacklist certain hosts. It requires 3 scripts and a blacklist/whitelist files. Whitelist files are regex patterns, so you can whitelist and entire domain.

Script should be put on your usb or JFFS folder. Make sure you pass in the location to your adblock scripts as a parameter.

Looks like the site is stripping off my indentations.

Here are the script contents:

dnsmasq-update.sh

# SET VARIABLES
adblockDirectory=$1
adblockTempDirectory=~/adblock
now=$(date +"%Y%m%d_%H%M")
daysToKeepDeltas=30

# CREATE TEMP FOLDER IF IT DOESN'T EXIST
if [ ! -d "$adblockTempDirectory" ]; then
mkdir $adblockTempDirectory
fi

if [ ! -f /etc/hosts.dnsmasq ]; then
# CREATE /etc/hosts.dnsmasq FOR THE FIRST TIME IF IT DOESN'T EXIST
/usr/bin/wget http://winhelp2002.mvps.org/hosts.txt -q -O- | tr -d '\r' | egrep '^0.0.0.0' | tee /etc/hosts.dnsmasq >> $adblockTempDirectory/hosts.dnsmasq.temp
else
# GET NEW hosts.dnsmasq FILE IF WE DIDN'T ALREADY ABOVE
/usr/bin/wget http://winhelp2002.mvps.org/hosts.txt -q -O- | tr -d '\r' | egrep '^0.0.0.0' > $adblockTempDirectory/hosts.dnsmasq.temp
fi

# WHITELIST ANY REQUIRED DOMAINS FROM THE hosts.dnsmasq.whitelist FILE
$adblockDirectory/adblock-whitelist.sh $adblockDirectory/hosts.dnsmasq.whitelist $adblockTempDirectory/hosts.dnsmasq.temp

# BLACKLIST ANY REQUIRED DOMAINS FROM THE hosts.dnsmasq.blacklist FILE
$adblockDirectory/adblock-blacklist.sh $adblockDirectory/hosts.dnsmasq.blacklist $adblockTempDirectory/hosts.dnsmasq.temp

if [ ! -z "$(diff -q /etc/hosts.dnsmasq $adblockTempDirectory/hosts.dnsmasq.temp)" ]; then
# THE DNSMASQ FILE WAS MODIFIED

# PREP LOG FILE
echo "There has been a change to the Dnsmasq files." > $adblockTempDirectory/hosts.dnsmasq.delta

# DIFF THE NEW FILE AGAINST THE LAST FILE APPEND TO LOG FILE
diff -b -i /etc/hosts.dnsmasq $adblockTempDirectory/hosts.dnsmasq.temp >> $adblockTempDirectory/hosts.dnsmasq.delta

# MOVE NEW hosts.dnsmasq FILE IN FOR USE BY DNS
mv $adblockTempDirectory/hosts.dnsmasq.temp /etc/hosts.dnsmasq

# RESTART DNS
killall dnsmasq && dnsmasq

# PUT MESSAGE IN SYSLOG
logger -p user.info -t AdBlock-Diff "There has been a change to the Dnsmasq files."
else
echo "There has been no change to the Dnsmasq files." > $adblockTempDirectory/hosts.dnsmasq.delta

# REMOVE hosts.dnsmasq TEMP FILE
rm $adblockTempDirectory/hosts.dnsmasq.temp

# PUT MESSAGE IN SYSLOG
logger -p user.info -t AdBlock-Diff "There has been no change to the Dnsmasq files."
fi

# TAG DELTA FILE WITH DATE
mv $adblockTempDirectory/hosts.dnsmasq.delta $adblockTempDirectory/hosts.dnsmasq.delta.$now

# CLEAN UP DELTA FILE OLDER THAN $daysToKeepDeltas
find $adblockTempDirectory/hosts.dnsmasq.delta* -mtime +$daysToKeepDeltas -exec rm {} \;

----------------------------------
adblock-blacklist.sh:

#!/bin/sh

#Insert header

sed -i -e "1i\\
#BEGIN BLACKLIST\\
\\" $2

#Insert each blacklist entry using the linecount variable to increment the line number to insert on
linecount=3
for entry in $(cat $1 | egrep -v "^#"); do
sed -i -e "$((linecount))i\\
0.0.0.0 $entry" $2
linecount=$((linecount + 1))
done

#Insert footer

sed -i -e "$((linecount))i\\
\\
#END BLACKLIST\\
\\
#BEGIN DOWNLOADED HOSTS" $2

---------------------

That's it for the script. Set them as executable. You'll need to create the hosts.dnsmasq.whitelist and .blacklist files in the same folder as the scripts.

An example of the inside of the hosts.dnsmasq.whitelist file:
www\.ojrq\.net
.*googleadservices\.com

Those are regex patterns. The \ escapes the . and makes it literal. .* means anything before whatever is after it.

An example of the inside of the hosts.dnsmasq.blacklist file:
www.ojrq.net
googleadservices.com

Those are not regex patterns. They are simple strings.

Keep in mind that the blacklist wins if there is a conflict between your blacklist and whitelist. Also, you can use the # symbol to commit out lines in the blacklist and whitelist. The scripts that read those files filter out any lines beginning with #

Enjoy.