What's new

AdBlocking with combined hosts file

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

The empty hosts.blocked file suggests the whitelist.txt file has an empty line. It's matching and removing every blocked site.

Wow it may do, it was created with vi, and it looks empty, but how can I make sure there are no lines?
I did 'dd' and deleted any lines if there were (I couldn't see any).

I did change the wget and got this:
Code:
services-start   update-hosts.sh
admin@RT-AC88U-89C0:/jffs/scripts# ./update-hosts.sh
--2016-01-15 23:21:41--  http://winhelp2002.mvps.org/hosts.txt
Resolving winhelp2002.mvps.org... 216.155.126.40
Connecting to winhelp2002.mvps.org|216.155.126.40|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 508717 (497K) [text/plain]
Saving to: 'STDOUT'

-                  100%[==================>] 496.79K   445KB/s   in 1.1s

2016-01-15 23:21:42 (445 KB/s) - written to stdout [508717/508717]

--2016-01-15 23:21:42--  http://someonewhocares.org/hosts/zero/hosts
Resolving someonewhocares.org... 209.97.222.140
Connecting to someonewhocares.org|209.97.222.140|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: 'STDOUT'

-                      [    <=>            ] 340.71K   398KB/s   in 0.9s

2016-01-15 23:21:43 (398 KB/s) - written to stdout [348890]

--2016-01-15 23:21:43--  http://pgl.yoyo.org/adservers/serverlist.php?hostfo                                                                                                                                                                 rmat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mi                                                                                                                                                                 metype=plaintext&useip=0.0.0.0
Resolving pgl.yoyo.org... 213.230.210.230
Connecting to pgl.yoyo.org|213.230.210.230|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: 'STDOUT'

-                      [ <=>               ]  58.73K  --.-KB/s   in 0.03s

2016-01-15 23:21:43 (2.29 MB/s) - written to stdout [60135]

FINISHED --2016-01-15 23:21:43--
Total wall clock time: 2.8s
Downloaded: 3 files, 896K in 2.0s (448 KB/s)
--2016-01-15 23:21:44--  http://www.malwaredomainlist.com/hostslist/hosts.tx                                                                                                                                                                 t
Resolving www.malwaredomainlist.com... 143.215.130.61
Connecting to www.malwaredomainlist.com|143.215.130.61|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48878 (48K) [text/plain]
Saving to: 'STDOUT'

-                  100%[==================>]  47.73K   167KB/s   in 0.3s

2016-01-15 23:21:44 (167 KB/s) - written to stdout [48878/48878]

--2016-01-15 23:21:44--  http://hosts-file.net/ad_servers.txt
Resolving hosts-file.net... 107.22.171.143
Connecting to hosts-file.net|107.22.171.143|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1795072 (1.7M) [text/plain]
Saving to: 'STDOUT'

-                                                           100%[==========================================================================================================================================>]   1.71M  39.0KB/s   in 35s

2016-01-15 23:22:20 (49.7 KB/s) - written to stdout [1795072/1795072]

FINISHED --2016-01-15 23:22:20--
Total wall clock time: 36s
Downloaded: 2 files, 1.8M in 36s (50.7 KB/s)

Done.
admin@RT-AC88U-89C0:/jffs/scripts#

But after I look in the host.blocked file with vi and its still empty.

How can I create an empty whitelist/blacklist if that is the issue?

EDIT:
Ha you were 100% correct about the blank line.
I added in www.google.com to the top line re-ran the .sh scrip and boom host.blocked if full...
 
Last edited:
Ok so now even tho my host.blocked file is full I am still seeing ads on all my devices.
I have tried on my laptop ipconfig /flushdns
Restarted laptop and router.
Restarted tablets and phones.

All are still seeing ads, any ideas?
 
Ok so now even tho my host.blocked file is full I am still seeing ads on all my devices.
I have tried on my laptop ipconfig /flushdns
Restarted laptop and router.
Restarted tablets and phones.

All are still seeing ads, any ideas?
You could add logging for dnsmasq and see what's going on.
Add this to the file dnsmasq.conf.add, changing path to logfile as needed:
Code:
log-facility=/tmp/mnt/sda1/logs/dnsmasq.log
log-async
log-queries
Then restart dnsmasq in a terminal with
Code:
service restart_dnsmasq
And then look at the logfile output with this, again changing the path as needed:
Code:
tail -f /tmp/mnt/sda1/logs/dnsmasq.log
You should see something like this when dnsmasq is restarted:
Code:
Jan 16 12:16:56 dnsmasq[10175]: read /tmp/mnt/sda1/adblock-hosts/adblock.hosts - 356162 addresses
In my case it successfully read a bunch of entries in the adblock.hosts file.
Then while still having the log window still open, open one of the blocked sites in your browser. Dnsmasq will log it with an entry such as this:
Code:
Jan 16 12:19:32 dnsmasq[10175]: 7 192.168.2.161/59955 /tmp/mnt/sda1/adblock-hosts/adblock.hosts www.whateverdomain.com is 0.0.0.0
Make sure you remove the logging entries (and then restart dnsmasq again) when your adblocking works sucessfully as it will gradually fill up your usb-device with the growing logfile.

Logging will be part of my automated script but with a logfile rotation built in.
 
Umm so I had to restart my router as I put in a line between:
Code:
# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt

log-facility=/tmp/mnt/sda1/logs/dnsmasq.log
log-async
log-queries

And then internet all died.
Removed the space between the old script text and this and its working again.

I noticed after the restart that host.blocked was empty again.
I ran ./update-hosts.sh and it fulled the host.blocked again.

Shouldn't this be filled automatically when the router starts?

Now that I ran the update-hosts.sh script my log now shows:
Jan 16 12:45:43 dnsmasq[6865]: read /tmp/mnt/sda1/hosts/hosts.blocked - 73776 addresses

I can see this in the log too:
Jan 16 12:47:25 dnsmasq[6865]: query[A] cm.g.doubleclick.net from 192.168.1.162
Jan 16 12:47:25 dnsmasq[6865]: /tmp/mnt/sda1/hosts/hosts.blocked cm.g.doubleclick.net is 0.0.0.0

So it does seem like its working?
But I am see lots of adds still...
 
You will still see some ads--some sites pull them from their own domain rather than pulling them in from elsewhere. But you should see notable space.

Still, you said the clients are using the router for DNS. Is the router using the ISP for DNS? Specifically, you haven't filled in the DNS fields under LAN/DHCP Server, you have "advertise router's IP" set to yes, and under WAN you have "connect to DNS server" set to yes?
 
You will still see some ads--some sites pull them from their own domain rather than pulling them in from elsewhere. But you should see notable space.

Still, you said the clients are using the router for DNS. Is the router using the ISP for DNS? Specifically, you haven't filled in the DNS fields under LAN/DHCP Server, you have "advertise router's IP" set to yes, and under WAN you have "connect to DNS server" set to yes?

No I am using OpenDNS for my DNS servers.
 
Ok I'm all working :)

The ads that are left behind are being served by the domain themselves so not much that can be done :(
Thanks to all for the help, really appreciated.
 
You will still see some ads--some sites pull them from their own domain rather than pulling them in from elsewhere. But you should see notable space.

Still, you said the clients are using the router for DNS. Is the router using the ISP for DNS? Specifically, you haven't filled in the DNS fields under LAN/DHCP Server, you have "advertise router's IP" set to yes, and under WAN you have "connect to DNS server" set to yes?

I'm glad you mentioned this: "Advertise router's IP in addition to user-specified DNS", mine is set to "yes" (must be the default setting). Would I be correct in thinking that, when selected to "yes", if the user-specified DNS IP addresses fail to connect/resolve, then DNS resolution would be attempted from the router's own DNS resources as a last resort?
 
Ok I'm all working :)

The ads that are left behind are being served by the domain themselves so not much that can be done :(
Thanks to all for the help, really appreciated.
Don't forget to disable logging. Or you could add this at the end to update-hosts.sh, it rotates the files every Friday:
Code:
# set directory
logrotate=/tmp/mnt/sda1/logs
#rotate the logs
if [ -f $logrotate/dnsmasq.log ]
then
    mv $logrotate/dnsmasq-1.log $logrotate/dnsmasq-2.log
    mv $logrotate/dnsmasq.log $logrotate/dnsmasq-1.log
fi
Make sure the following lines come after this snippet, it creates a new logfile. It's fine to move the existing ones down:
Code:
#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq
 
Could some one help me to set this up?
Im trying too get this working but its failing hard on my side.
I inputted a USB stick in the router but it doesnt see it as sda1 so i changed all lines to make the host file on /jffs/script.
When i reboot the device there is no hostfile being made, when i manually execute the post-mount file it ends with a "done" message and i can see the host file there.
Even though the host file is then there i still have adds in all my apps on my android phone wich is connected to the WiFi network of the AC68U.

Code:
Jan 16 19:11:56 rc_service: service 947:notify_rc restart_dnsmasq
Jan 16 19:11:56 dnsmasq[902]: exiting on receipt of SIGTERM
Jan 16 19:11:56 custom config: Appending content of /jffs/configs/dnsmasq.conf.add.
Jan 16 19:11:56 dnsmasq[950]: started, version 2.75 cachesize 1500
Jan 16 19:11:56 dnsmasq[950]: asynchronous logging enabled, queue limit is 5 messages
Jan 16 19:11:56 dnsmasq-dhcp[950]: DHCP, IP range 192.168.1.2 -- 192.168.1.254, lease time 1d
Jan 16 19:11:56 dnsmasq-dhcp[950]: DHCP, sockets bound exclusively to interface br0
Jan 16 19:11:56 dnsmasq[950]: read /etc/hosts - 6 addresses
Jan 16 19:11:56 dnsmasq[950]: read /jffs/scripts/hosts.clean - 25435 addresses
Jan 16 19:11:56 dnsmasq[950]: using nameserver 84.116.46.23#53
Jan 16 19:11:56 dnsmasq[950]: using nameserver 84.116.46.22#53

The code above is from the log, it shows it has adresses but i dont get what the nameserver part is.
I manually set DNS to Google 8.8.8.8 and 8.8.4.4.
Could that be my problem?
 
Hmm...

Still trying to figure out if my AC3200 firewall works, so I did iptables -L, and a lot came out of course (se below). What I am most interesting in is of course the input chain. It seems to say:
ACCEPT all -- anywhere anywhere state NEW
ACCEPT all -- anywhere anywhere state NEW

Is that correct, I get the feeling everything can pass...

Not to mention FORWARD and then what is "FUPNP" - BamseRIP is a WHS2011 server... Any chance anyone could explain, first and most INPUT but I love som help on the rest :)

Code:
admin@RT-AC3200-2B50:/# iptables -L
Chain INPUT (policy ACCEPT)
target  prot opt source  destination
DROP  all  --  anywhere  anywhere  match-set BlockedCountries src
DROP  all  --  anywhere  anywhere  match-set TorNodes src
DROP  icmp --  anywhere  anywhere  icmp echo-request
DROP  all  --  anywhere  anywhere  state INVALID
ACCEPT  all  --  anywhere  anywhere  state RELATED,ESTABLISHED
ACCEPT  all  --  anywhere  anywhere  state NEW
ACCEPT  all  --  anywhere  anywhere  state NEW
ACCEPT  udp  --  anywhere  anywhere  udp spt:bootps dpt:bootpc
ACCEPT  icmp --  anywhere  anywhere  icmp !echo-request
DROP  all  --  anywhere  anywhere

Chain FORWARD (policy DROP)
target  prot opt source  destination
DROP  ipv6-auth--  anywhere  anywhere
DROP  ipv6-crypt--  anywhere  anywhere
DROP  udp  --  anywhere  anywhere  udp dpt:4500
DROP  udp  --  anywhere  anywhere  udp dpt:500
DROP  udp  --  anywhere  anywhere  udp dpt:1701
DROP  gre  --  anywhere  anywhere
DROP  tcp  --  anywhere  anywhere  tcp dpt:1723
ACCEPT  all  --  anywhere  anywhere  state RELATED,ESTABLISHED
DROP  all  --  anywhere  anywhere
DROP  all  --  anywhere  anywhere  state INVALID
ACCEPT  all  --  anywhere  anywhere
ACCEPT  tcp  --  anywhere  anywhere  tcpflags: FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5
ACCEPT  tcp  --  anywhere  anywhere  tcpflags: FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
ACCEPT  icmp --  anywhere  anywhere  icmp echo-request limit: avg 1/sec burst 5
ACCEPT  tcp  --  anywhere  anywhere  tcp dpt:www
ACCEPT  tcp  --  anywhere  anywhere  tcp dpt:domain
ACCEPT  udp  --  anywhere  anywhere  udp dpt:domain
ACCEPT  tcp  --  anywhere  anywhere  tcp dpt:https
ACCEPT  udp  --  anywhere  anywhere  udp dpt:ntp
ACCEPT  tcp  --  anywhere  anywhere  tcp dpt:943
ACCEPT  udp  --  anywhere  anywhere  udp dpt:1194
DROP  all  --  anywhere  anywhere
ACCEPT  all  --  anywhere  anywhere  ctstate DNAT
ACCEPT  all  --  anywhere  anywhere

Chain OUTPUT (policy ACCEPT)
target  prot opt source  destination

Chain FUPNP (0 references)
target  prot opt source  destination
ACCEPT  udp  --  anywhere  BamseRIP  udp dpt:49882
ACCEPT  tcp  --  anywhere  BamseRIP  tcp dpt:49882
ACCEPT  udp  --  anywhere  BamseRIP  udp dpt:63580
ACCEPT  tcp  --  anywhere  BamseRIP  tcp dpt:63580

Chain PControls (0 references)
target  prot opt source  destination
ACCEPT  all  --  anywhere  anywhere

Chain logaccept (0 references)
target  prot opt source  destination
LOG  all  --  anywhere  anywhere  state NEW LOG level warning tcp-sequence tcp-options ip-options prefix "ACCEPT "
ACCEPT  all  --  anywhere  anywhere

Chain logdrop (0 references)
target  prot opt source  destination
LOG  all  --  anywhere  anywhere  state NEW LOG level warning tcp-sequence tcp-options ip-options prefix "DROP "
DROP  all  --  anywhere  anywhere
 
Hmm...

Still trying to figure out if my AC3200 firewall works, so I did iptables -L, and a lot came out of course (se below). What I am most interesting in is of course the input chain. It seems to say:
ACCEPT all -- anywhere anywhere state NEW
ACCEPT all -- anywhere anywhere state NEW

Is that correct, I get the feeling everything can pass...

Not to mention FORWARD and then what is "FUPNP" - BamseRIP is a WHS2011 server... Any chance anyone could explain, first and most INPUT but I love som help on the rest :)
Would you mind opening your own thread for this?
It's not related to my scripts and has no relevance here.
Thanks
 
Could some one help me to set this up?
Im trying too get this working but its failing hard on my side.
I inputted a USB stick in the router but it doesnt see it as sda1 so i changed all lines to make the host file on /jffs/script.
Bad idea to use jffs for this. Reboot you router and look into the System Log to find the name for your USB Stick.
The line where it says so starts with "hotplug"
Mine looks like this:
hotplug[674]: USB ext3 fs at /dev/sda1 mounted on /tmp/mnt/sda1
 
Bad idea to use jffs for this. Reboot you router and look into the System Log to find the name for your USB Stick.
The line where it says so starts with "hotplug"
Mine looks like this:
hotplug[674]: USB ext3 fs at /dev/sda1 mounted on /tmp/mnt/sda1

after my post i looked at the log to find errors after a reboot.
i found the name of the usb stick and changed all files with the correct names.
i see the file being made on the usb stick yet i still have adds everywhere.
big amazon one for example at the top of this page.
i have the idea i shouldnt see those of things are working correctly.
any ideas whats going wrong on my side?
 
after my post i looked at the log to find errors after a reboot.
i found the name of the usb stick and changed all files with the correct names.
i see the file being made on the usb stick yet i still have adds everywhere.
big amazon one for example at the top of this page.
i have the idea i shouldnt see those of things are working correctly.
any ideas whats going wrong on my side?
Did your read the 3. Notes part in the first post?
Also clear your browser cache after this.
 
I have now verified my scripts, and they are 100% identical to YOUR scripts - this is relevant.
I still don't see why your iptables question is related to my scripts. Please keep the discussion in your own thread.
It is not relevant to other users of my scripts.
Thanks
 
Did your read the 3. Notes part in the first post?
Also clear your browser cache after this.
yes i have read them.
you mention windows computers.
I am testing on my android smartphone wich i have rebooted after each attempt.
So i hope cache aint a problem.
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top