AdBlocking with combined hosts file

[Deleted member 42539]

With my N66U I cannot get this to work.
The scripts run, and the hosts.blocked is generated each time I restart my router, so that part I'm pretty sure works as intended.

But nothing is blocked even though the files are generated. Neither the ones from the blacklist or the ones from the generated file.
I have tripple-checked the paths, and they are correct.

What can I do to troubleshoot this?

Merlin-Firmware: 378.55

 

[ColinTaylor]

Try running an nslookup on one of the blocked domains from your PC. You should see something like this:

C:\>nslookup doubleclick.net
Server:  router.asus.com
Address:  192.168.1.1

Name:    doubleclick.net
Address:  0.0.0.0

PS Do you use the routers DNS Filter feature?

 

[Deleted member 42539]

Try running an nslookup on one of the blocked domains from your PC. You should see something like this:

C:\>nslookup doubleclick.net
Server:  router.asus.com
Address:  192.168.1.1

Name:    doubleclick.net
Address:  0.0.0.0

PS Do you use the routers DNS Filter feature?

No, I don't use the filtering found under Parental Controls

I get the following from the lookup:

C:\Users\Squazz>nslookup doubleclick.net
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
Name:    doubleclick.net
Address:  70.32.146.212

I use Googles DNS, can this be the problem?

 

[john9527]

I use Googles DNS, can this be the problem?

You have to use the router (dnsmasq) as your DNS server, so it's can process the blocks. The DNS server address in the nslookup should be your router address.

 

[ColinTaylor]

You have to use the router (dnsmasq) as your DNS server, so it's can process the blocks. The DNS server address in the nslookup should be your router address.

Further to what John said, if you have set values in LAN > DHCP Server > DNS Server 1 & 2 you should remove them. That way your DHCP clients will get the routers IP as their DNS server.

 

[Deleted member 42539]

You have to use the router (dnsmasq) as your DNS server, so it's can process the blocks. The DNS server address in the nslookup should be your router address.

Further to what John said, if you have set values in LAN > DHCP Server > DNS Server 1 & 2 you should remove them. That way your DHCP clients will get the routers IP as their DNS server.

You two just saved my day Now it works
Really hadn't considered that my forcing of the DNS could be the problem

How about the WAN DNS, can I set this one without problems?

 

[ColinTaylor]

How about the WAN DNS, can I set this one without problems?

Yes, that's not a problem. That defines the upstream DNS server used after any modifications by dnsmasq.

 

[Deleted member 42539]

Yes, that's not a problem. That defines the upstream DNS server after any modifications by dnsmasq.

The reason why I asked is, that when I do set the WAN DNS to Googles, the adds reappear

 

[ColinTaylor]

The reason why I asked is, that when I do set the WAN DNS to Googles, the adds reappear

You said it was working. What was it set to before?

 

[Deleted member 42539]

You said it was working. What was it set to before?

Before it was set to 212.10.10.4 (my ISPs DNS).

 

[ColinTaylor]

Before it was set to 212.10.10.4 (my ISPs DNS).

Strange. It shouldn't make any difference which WAN DNS server you use. What do you get from the nslookup command now. Is it using 192.168.1.1?

Your clients might be caching the results. You might have to reboot them.

 

[Deleted member 42539]

Strange. It shouldn't make any difference which WAN DNS server you use. What do you get from the nslookup command now. Is it using 192.168.1.1?

Your clients might be caching the results. You might have to reboot them.

It seems like a reboot did the trick, I don't have the issue anymore
Thanks everyone for the help. I'm now also running the "entware-free" version, no probs

 

[Eet_46]

I'm trying to get this working, but i'm having PATH problems.
I have edited entries in update-hosts.sh and dnsmasq.conf.add from /tmp/mnt/sda1/hosts to /tmp/mnt/Data/hosts to apply to my system.
Got an external HD mounted, which i would use for the external files for the script. The paths seem fine, the directory DOES exist, but I somehow get errors when trying to run update-hosts.sh manually:

So what am i doing wrong here?

 

[thelonelycoder]

@Eet_46 Try copy/paste this here into update-hosts.sh:

#!/bin/sh

# set directory
dir=/tmp/mnt/Data/hosts

# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > $dir/temp

#get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
wget -qO- "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/ad_servers.txt" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

#remove temp file
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

 

[waffles]

I was wondering what the best way would be to verify that the adblocking as described in the 1. post has been successfully installed and configured (I understand that there may always be some that may slip through, unless you add them to the black list).

I use the "2. Advanced Method".

Reason I ask is that even though the hosts.blocked file is generated, I am still getting a ton of ads in my browser. It seems in particularly bad with German websites, e.g. Spiegel.de.
(If I activate the adblockplus plugin in FF, they are all gone.)
I have flushed the DNS Cache in between as suggested.

Thanks!

 

[thelonelycoder]

I was wondering what the best way would be to verify that the adblocking as described in the 1. post has been successfully installed and configured (I understand that there may always be some that may slip through, unless you add them to the black list).

I use the "2. Advanced Method".

Reason I ask is that even though the hosts.blocked file is generated, I am still getting a ton of ads in my browser. It seems in particularly bad with German websites, e.g. Spiegel.de.
(If I activate the adblockplus plugin in FF, they are all gone.)
I have flushed the DNS Cache in between as suggested.

Thanks!

Add this to dnsmasq.conf.add (adjust path to your liking):

# Logging
log-facility=/tmp/mnt/sda1/logs/dnsmasq.log
log-queries

Then open a terminal and log into your router.
Enter

service restart_dnsmasq

And then

tail -f /tmp/mnt/sda1/logs/dnsmasq.log

Pick some of the entries in the hosts.blocked file and paste it into the address field of your brwoser and open the page.

For blocked hosts you should see something like this:

query[A] settings-win.data.microsoft.com from 192.168.2.200
/tmp/mnt/sda1/hosts/blacklist.txt settings-win.data.microsoft.com is 0.0.0.0
or:
query[A] 7457.accessaw.blueseek.com from 192.168.2.200
/tmp/mnt/sda1/hosts/hosts.blocked.txt 7457.accessaw.blueseek.com is 0.0.0.0

Edit: The log file will grow indefinitely and fill up your USB stick. To stop the router from doing that rotate the log file as per post #321

 

[Eet_46]

@Eet_46 Try copy/paste this here into update-hosts.sh:

#!/bin/sh

# set directory
dir=/tmp/mnt/Data/hosts

# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > $dir/temp

#get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
wget -qO- "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/ad_servers.txt" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

#remove temp file
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

Got it working. Forgot to convert the file to UNIX instead of Windows format.. Doh..

 

[Eet_46]

Okay, next problem I'm trying to whitelist "nyhederne.tv2.dk", but it seems i still can't watch the sites videos. (I can read the articles, but not watch the videos) Is there a way to pinpoint what is blocked on that particular site, so i can start excluding elements one by one, to get the videos working? (Not sure if the above example with logging added to dnsmasq.conf can be used somehow?)

 

[thelonelycoder]

@Eet_46 Activate dnsmasq logging and see what is blocking your content. How to is in an earlier post in this thread.
Add the sites you think block your video content to the whitelist file and rerun update-hosts.sh with:

sh /jffs/scripts/update-hosts.sh

If you only change the blacklist file, a restart of dnsmasq will do:

service restart_dnsmasq

 

[zekesdad]

probably a really dumb question, but what fs does the usb stick need to be formatted in?