AdBlocking with combined hosts file

[thelonelycoder]

Prefer to keep the router simple.

One way or the other, you'd probably still need a USB Stick in one of the ports to have the temp storage of the adblock files.
Don't get too excited, I'll have to first get familiar with these things before I give it a go. And time is a precious thing at the moment.

 

[Ars92]

One way or the other, you'd probably still need a USB Stick in one of the ports to have the temp storage of the adblock files.
Don't get too excited, I'll have to first get familiar with these things before I give it a go. And time is a precious thing at the moment.

Damn...hopefully there's some way to keep it in the flash or something. Take all the time you need, im sure you will be able to figure out something. Will be a great learning experience for you too!!

 

[BreakingDad]

What Software tools would one need to do that? I _may_ be interested to venture in this area.
I know RMerlin will not do it, for obvious reasons.

I'd be happy to test it on my rt-n66u.

Best of luck to you.

 

[Phk]

Damn...hopefully there's some way to keep it in the flash or something. Take all the time you need, im sure you will be able to figure out something. Will be a great learning experience for you too!!

Why can't you use something like /jffs/adblock instead of /mnt/sda1?

Whats the issue of using JFFS?

Thanks

 

[Phk]

I've managed to put a simple version working without external storage. Didn't use the post-mount script, but instead the init-start.

1. mkdir /jffs/adblock/

2. vi /jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/jffs/adblock/hosts.clean

3. vi /jffs/scripts/update-hosts.sh

#!/bin/sh

wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /jffs/adblock/hosts.clean
sleep 5
service restart_dnsmasq

4. chmod a+rx /jffs/scripts/update-hosts.sh

5. vi /jffs/scripts/init-start

#!/bin/sh

cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"

6. chmod a+rx /jffs/scripts/init-start

7. Run 1st time manualy:

/jffs/scripts/update-hosts.sh

And... So far its working:

C:\Users\Somo Nozes>nslookup
Default Server: resolver2.opendns.com
Address: 208.67.220.220
> server 192.168.1.254
> ads.pof.com
Server: [192.168.1.254]
Address: 192.168.1.254
Name: ads.pof.com
Address: 0.0.0.0
> server 208.67.220.220
Default Server: resolver2.opendns.com
Address: 208.67.220.220
> ads.pof.com
Server: resolver2.opendns.com
Address: 208.67.220.220
Non-authoritative answer:
Name: ads.pof.com
Address: 199.182.216.202

Its the first time i'm modding my ASUS so actually I'm not sure if I'm missing something here! Please tell me what you think.

Cheers
Phk

 

[sleepless007]

Not working for me

Hey guys, so i'm trying to set up adblocking on my N66U but it's not working for me i followed the first post and for me everything seems to be setup in the right way. maybe someone can help me. i tested if it's working by just calling some of the urls in the hosts.clean file in my browser, and all of them were working so something is going on - also thepiratebay is plasterd with ads...

Here are my scripts:

/jffs/scripts/post-mount:

#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ] 
then
  # Entware setup
  ln -sf $1/entware /tmp/opt

  # Entware services. Do NOT use services-start for this bec usb may not be ready
  if [  -x /opt/etc/init.d/rc.unslung ]
  then
    /usr/bin/logger -t $0  "Found rc.unslung; starting Entware services"
    /opt/etc/init.d/rc.unslung start
    /usr/bin/logger -t $0  "Entware services started"
  else
    /usr/bin/logger -t $0  "ERROR: - rc.unslung not found - Entware services NOT STARTED"
  fi  
fi

sleep 12
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

/jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean

/jffs/scripts/update-hosts.sh

#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

I also created the init-start, but i have a question here, since this is just a cronjob, can i also add it to services-start instead of init-start, since i have some other working cronjobs there? I would think so, but tell me if i'm wrong on this!

All the files are executeable, when i manually start ./update-hosts.sh it's working, also in the syslog i can see that dnsmasq is reading the hosts-clean

dnsmasq[372]: read /tmp/mnt/sda1/hosts.clean - 23822 addresses
Jan  1 01:00:19 dnsmasq[372]: read /etc/hosts.dnsmasq - 6 addresses
Jan  1 01:00:19 dnsmasq-dhcp[372]: read /etc/ethers - 6 addresses
Jan  1 01:00:20 dnsmasq[372]: using local addresses only for domain blablabla......

The file permissions are also properly set...

-rwxrwxrwx    1 admin    root           196 Sep 11 08:06 SpdnsUpdate.sh
-rwxrwxrwx    1 admin    root            83 Sep 11 09:55 init-start
-rwxrwxrwx    1 admin    root            55 Sep 10 22:14 ledsoff.sh
-rwxrwxrwx    1 admin    root            55 Sep 10 22:14 ledson.sh
-rwxrwxrwx    1 admin    root           866 Sep 11 09:50 post-mount
-rwxrwxrwx    1 admin    root           232 Sep 11 09:54 services-start
-rwxrwxrwx    1 admin    root            43 Sep 11 00:33 services-stop
-rwxrwxrwx    1 admin    root           372 Sep 11 08:40 update-hosts.sh

I also did several reboots and also double and triple checked the code in the scripts (maybe i'm a bit blind from coding the last two days ). I hope somone can help me out

kind regards

 

[Phk]

Hi! Have you tried without the mount and the SDA as a stated in the last post?
It's really much simpler and its working for me.

Cheers

 

[sleepless007]

Hi! Have you tried without the mount and the SDA as a stated in the last post?
It's really much simpler and its working for me.

Cheers

Thx for the reply,

Haven't tried this solution, but i don't want to write to /jffs too often.
I don't know if it's a problem with my usb mounting on /sda1, but i don't think so, because the hosts.clean is found by dnsmasq and it also reads the entries.

The wget is called after the usb has been mounted, because i'm also looking if entware is started

 

[Phk]

Thx for the reply,

Haven't tried this solution, but i don't want to write to /jffs too often.
I don't know if it's a problem with my usb mounting on /sda1, but i don't think so, because the hosts.clean is found by dnsmasq and it also reads the entries.

The wget is called after the usb has been mounted, because i'm also looking if entware is started

Hey. Yes, you'd rather write to external storage; JFFS won't live long
However what I was telling you was to start with a simpler solution and then add the "mount" condition if the rest is working.

Still about the JFSS MTBF, does anyone really has a metric that shows this degradation? I'd like to avoid that the router needs the external storage to boot OK, so I'm thinking that the 600k JFFS file is read once on boot (then it's paged into memory) and written once on Fridays (which is my cron). So I don't see actually a JFFS abuse here..

Cheers

 

[sleepless007]

Still about the JFSS MTBF, does anyone really has a metric that shows this degradation? I'd like to avoid that the router needs the external storage to boot OK, so I'm thinking that the 600k JFFS file is read once on boot (then it's paged into memory) and written once on Fridays (which is my cron). So I don't see actually a JFFS abuse here..

Cheers

No idea from my side But in general i don't like the wget command in jffs i played around with a dyndns update script and wget always writes some s%$t anywhere. But this was just a feeling i don't really know.

How i see it my problem lies somwhere in the forwarding to 0.0.0.0 part, because how i understand the system log

dnsmasq[372]: read /tmp/mnt/sda1/hosts.clean - 23822 addresses

dnsmasq reads the file fine.

I do get one or two errors directly after the kernel is up

Jan  1 01:00:09 kernel: br0: port 1(vlan1) entering forwarding state
Jan  1 01:00:09 dnsmasq[355]: started, version 2.72test3 cachesize 1500
Jan  1 01:00:09 dnsmasq[355]: warning: interface ppp1* does not currently exist
Jan  1 01:00:09 dnsmasq[355]: asynchronous logging enabled, queue limit is 5 messages
Jan  1 01:00:09 dnsmasq-dhcp[355]: DHCP, IP range 192.168.1.100 -- 192.168.1.254, lease time 1d
Jan  1 01:00:09 dnsmasq[355]: using local addresses only for domain blabla
Jan  1 01:00:09 dnsmasq[355]: read /etc/hosts - 5 addresses
Jan  1 01:00:09 dnsmasq[355]: failed to load names from /tmp/mnt/sda1/hosts.clean: No such file or directory
Jan  1 01:00:09 dnsmasq[355]: read /etc/hosts.dnsmasq - 6 addresses
Jan  1 01:00:09 dnsmasq-dhcp[355]: read /etc/ethers - 6 addresses
Jan  1 01:00:09 dnsmasq[355]: using local addresses only for domain blabla

but later on my mounts are loading

Jan  1 01:00:10 kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
Jan  1 01:00:10 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Jan  1 01:00:10 kernel: scsi 1:0:0:0: Direct-Access     Multi    Flash Reader     1.00 PQ: 0 ANSI: 0
Jan  1 01:00:10 kernel: sd 1:0:0:0: [sdb] Attached SCSI removable disk
Jan  1 01:00:10 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0

and then the post-mount script gets executed

Jan  1 01:00:13 /jffs/scripts/post-mount: Found rc.unslung; starting Entware services
Jan  1 01:00:13 /jffs/scripts/post-mount: Entware services started

and after that dnsmasq reads the hosts.clean file

an  1 01:00:17 dnsmasq[372]: read /etc/hosts - 5 addresses
Jan  1 01:00:19 kernel: Ebtables v2.0 registered
Jan  1 01:00:19 rc_service: udhcpc 511:notify_rc start_firewall
Jan  1 01:00:19 rc_service: udhcpc 511:notify_rc stop_upnp
Jan  1 01:00:19 rc_service: waitting "start_firewall" via udhcpc ...
Jan  1 01:00:19 start_nat_rules: apply the nat_rules(/tmp/nat_rules_eth0_eth0)!
Jan  1 01:00:19 WAN Connection: WAN was restored.
Jan  1 01:00:19 dnsmasq[372]: read /tmp/mnt/sda1/hosts.clean - 23822 addresses
Jan  1 01:00:19 dnsmasq[372]: read /etc/hosts.dnsmasq - 6 addresses
Jan  1 01:00:19 dnsmasq-dhcp[372]: read /etc/ethers - 6 addresses
Jan  1 01:00:20 dnsmasq[372]: using local addresses only for domain blabla

There are also some other errors and warnings i don't really understand, especially the first one, maybe it has something to do with this

Jan  1 01:00:09 dnsmasq[372]: warning: interface ppp1* does not currently exist
Jan  1 01:00:09 dnsmasq[372]: asynchronous logging enabled, queue limit is 5 messages
Jan  1 01:00:09 WAN Connection: ISP's DHCP did not function properly.

 

[sleepless007]

Working now

Stupid me

I found my error, i had set a custom DNS on my Client, now i set it to my router and everything is working...

 

[thelonelycoder]

Stupid me

I found my error, i had set a custom DNS on my Client, now i set it to my router and everything is working...

Glad you found the error and hopfully it works as it should now!

 

[zy_zhao]

does this work with ac68u ??

 

[sardelkin]

adblocking from first post stop working when i turned on DNS filtering in web-interface

 

[ColinTaylor]

adblocking from first post stop working when i turned on DNS filtering in web-interface

That is correct.

 

[sammyano]

Same here - so does the DNS Filter restrict the Adblock?

 

[ColinTaylor]

Same here - so does the DNS Filter restrict the Adblock?

Yes. DNS Filter uses iptables to intercept all DNS requests from the specified client and redirects them to the DNS service you have chosen. Therefore it bypasses the routers DNS server (where the ad blocking happens) completely.

 

[sammyano]

Yes. DNS Filter uses iptables to intercept all DNS requests from the specified client and redirects them to the DNS service you have chosen. Therefore it bypasses the routers DNS server (where the ad blocking happens) completely.

oh - so its either Adblock or DNS Filter

 

[Phk]

I've managed to put a simple version working without external storage. Didn't use the post-mount script, but instead the init-start.

1. mkdir /jffs/adblock/

2. vi /jffs/configs/dnsmasq.conf.add


3. vi /jffs/scripts/update-hosts.sh


4. chmod a+rx /jffs/scripts/update-hosts.sh

5. vi /jffs/scripts/init-start


6. chmod a+rx /jffs/scripts/init-start

7. Run 1st time manualy:


And... So far its working:


Its the first time i'm modding my ASUS so actually I'm not sure if I'm missing something here! Please tell me what you think.

Cheers
Phk

Guys, just managed to get this working with DNSCRYPT

Now i'm happy with this solution.

AD filtering, and DNS requests encrypted from ISP perspective.

The only difference is:

vi /jffs/configs/dnsmasq.conf.add

no-resolv
server=127.0.0.1#65053
addn-hosts=/jffs/adblock/hosts.clean

and the rest is from tutorial: https://github.com/RMerl/asuswrt-merlin/wiki/Secure-DNS-queries-using-DNSCrypt

 

[bilboSNB]

I use this method but I use a static file created the first time implemented and it works great in the most part.

However my other half likes going on this site:

http://www.digitalspy.co.uk/

which wont work at all as it is an ad fest, I have removed all of the digital spy references but none of the links work on the site unless ad blocking is disabled.

Any suggestions on how to get it to work?

Also, is there a 'cut down' version of the domain file anywhere which isn't as comprehensive and maybbe only covers core ad domains?