AdBlocking with combined hosts file

[Chrysalis]

adding an alias ip is trivial, but here is a problem.

tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN

the system gui does listen on 0.0.0.0 meaning it listens on all ips. Until that can be changed adding new ip's is useless for this.

looking at configs I can find and nvram settings I cannot find anything which can adjust the listen ip, it might be hardcoded.

An alternative idea is to.

1 - add 2nd lan ip
2 - setup lighttpd to listen on an alternative port eg. port 81
3 - using iptables add a forward rule so port 80 on the 2nd lan ip is rerouted to port 81.

Are people going to be bothered to do this tho? as the idea would be just to keep the fast gui whilst using lighttpd.

 

[bilboSNB]

Can we add a ping condition to this, my wan seems to take a little while to connect and therefore I get an empty hosts file.

  GNU nano 2.2.6              File: /jffs/scripts/post-mount

#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -sf $1/entware /tmp/opt
fi

[B]if ping 8.8.8.8
then[/B]
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/host$
sleep 10
service restart_dnsmasq

I am not sure how to do this properly though, do you?

 

[bilboSNB]

This *seems* to work

#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -sf $1/entware /tmp/opt
fi

until ping -c 1 8.8.8.8
do
        sleep 10;
done
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/host$
sleep 10
service restart_dnsmasq

My wan seems to take a longer to come up now since iplementing dnscrypt and ipset for blocking tor and countries.

 

[Chrysalis]

guys I suggest removing http://someonewhocares.org/hosts/zero/hosts from the script, it has questionable listing including bet365 a major uk betting site not a ads/spam/malware site.

also I whitelisted the following by adding this to end of update-hosts.sh to allow skysports videos to work (without ads).

sed -i 's/192.168.1.253 secure-uk.imrworldwide.com/#192.168.1.253 secure-uk.imrworldwide.com/g' /tmp/mnt/OPTWARE/hosts.clean
sed -i 's/192.168.1.253 servedby.flashtalking.com/#192.168.1.253 servedby.flashtalking.com/g' /tmp/mnt/OPTWARE/hosts.clean

 

[thelonelycoder]

also I whitelisted the following by adding this to end of update-hosts.sh to allow skysports videos to work (without ads).
[/code]

I suggest to keep all three hosts files as all of them combined have it's merits.

Instead, you could create a whitelist.txt file of the domains you want to allow.
With a few changes to the files /jffs/scripts/post-mount and /jffs/scripts/update-hosts.sh you can do it. The way I do it is with the example below.
I have added the variable dir to make things simpler. It creates a temp file that is deleted after the script finishes. Also added comments.

First create /tmp/mnt/sda1/whitelist.txt and add the domains you want removed as they appear in the hosts.clean file.
Make sure line endings are UNIX (LF) and only one exeption per line!
Sample content:

secure-uk.imrworldwide.com
servedby.flashtalking.com

Then modify/replace post-mount and update-hosts.sh files as follows:

#!/bin/sh

# set directory
dir=/tmp/mnt/sda1/hosts
# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > $dir/temp
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/temp | fgrep -vf $dir/whitelist.txt > $dir/hosts.clean
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

You can update the whitelist with new entries whenever you want. Just re-run update-hosts.sh manually.

 

[test011]

I was wondering why does the script have 'sort' in it.
Sorting it has any merit?

Another thing, would it be better to download hosts files to /tmp first and whitelist it to tmp/mnt/sda1/hosts/ ?

 

[thelonelycoder]

I was wondering why does the script have 'sort' in it.
Sorting it has any merit?

Another thing, would it be better to download hosts files to /tmp first and whitelist it to tmp/mnt/sda1/hosts/ ?

You may leave the sort pipe out if you want to. It helps me to check for similar entries and keeps the file - well - sorted.

I am not sure what you mean with the tmp thing.
Since this started without the whitelist file I wanted to keep the some seblance of "add in" functionality instead of completely rewriting the original.
However, you are welcome to modify to your needs.

The whitelisting along with combining and ereasing duplicates as implemented in my previous post is for a short while taxing for the processor. It will take about 20' to fully process it when ececuted manually. There may be a better way to do it. Please post if you do.

 

[ColinTaylor]

Tell dnsmasq to add the new hosts file and where to send the unwanted requests to (sending them to 0.0.0.0)
Create or add to /jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean

Hi thelonelycoder, thanks for the scripts.

Can I ask what is the purpose of "address=/0.0.0.0/0.0.0.0"?

It seems to mean: Resolve all hosts in the domain called "0.0.0.0" to IP address 0.0.0.0. All the hosts in the hosts.clean file resolve to IP address 0.0.0.0 so this statement seems to do nothing useful. I've tried removing it and haven't noticed any difference in the ad blocking.

 

[test011]

I thought /tmp is a ramdrive, not sure now though.
Putting a temporary processing file on ramdrive is a good practice.

 

[thelonelycoder]

I thought /tmp is a ramdrive, not sure now though.
Putting a temporary processing file on ramdrive is a good practice.

/tmp/mnt/sda1/ is not a temporary file system. It is a mounted drive in the /tmp/ tree.

 

[bilboSNB]

Hi thelonelycoder, thanks for the scripts.

Can I ask what is the purpose of "address=/0.0.0.0/0.0.0.0"?

It seems to mean: Resolve all hosts in the domain called "0.0.0.0" to IP address 0.0.0.0. All the hosts in the hosts.clean file resolve to IP address 0.0.0.0 so this statement seems to do nothing useful. I've tried removing it and haven't noticed any difference in the ad blocking.

I'd been wondering this as well.

Separately, there was an issue with my sky set top box with this blocking method in that it would not turn on if the router was on because it was trying to access www.google-analytics.com on 0.0.0.0. I am running the update script every now and then manually, so I changed the entry for this to 127.0.0.1 and all seems okay. Does 0.0.0.0 have any advantage over 127.0.0.1 for other devices in the household?

 

[thelonelycoder]

Hi thelonelycoder, thanks for the scripts.
Can I ask what is the purpose of "address=/0.0.0.0/0.0.0.0"?

I'll answer two questions here as it needs to be adressed, there is a lot of confusion about them.
- 127.0.0.1 vs 0.0.0.0
- what does "address=/0.0.0.0/0.0.0.0" do

First of all, the question of hosts: 127.0.0.1 vs 0.0.0.0 is discussed at length elsewhere (google it) and the use of 0.0.0.0 has the upper hand.

127.0.0.1 is your comuter's or devices internal address (localhost) and it will try to answer any request made to it. If it cannot answer, it will timeout.

0.0.0.0 is a null address. It will not answer and therefore timeout immediately. In addition to that a device will not try again to send requests to 0.0.0.0 as it is a dead end.

Going further on the 0.0.0.0 path, telling your router that 0.0.0.0 is effectively a dead end will lower timeout even more (my own tests). Also the dnsmasq entry "address=/0.0.0.0/0.0.0.0" fixes netstat an traceroute problems for some.

Some consistent timeout results I measured for blocked hosts in Firefox, various blocked hosts on page:
With 127.0.0.1 --> 250 to 500 ms
With 0.0.0.0 without address=/0.0.0.0/0.0.0.0 entry --> 10 to 35 ms
With 0.0.0.0 with address=/0.0.0.0/0.0.0.0 entry --> 1 to 2 ms (!)

However, someonewhocares.org states on the 127.0.0.1 vs 0.0.0.0 debate:
On some machines this may run minutely faster, however the zero version may not be compatible with all systems.

I am using Windows, Linux, Android, Mac and iOS devices on my network. None of them have issues with this setup.
In a previous post I have explained the use of a whitelist file. This may help for the special cases of google-analytics.com

 

[bilboSNB]

Thank you very much indeed for the explanation.

 

[test011]

/tmp/mnt/sda1/ is not a temporary file system. It is a mounted drive in the /tmp/ tree.

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 25024 25024 0 100% /
tmpfs 119780 2624 117156 2% /tmp
devfs 119780 0 119780 0% /dev
/dev/mtdblock5 32768 1260 31508 4% /jffs
/dev/sda1 1888958 595863 1195559 33% /tmp/mnt/sda1

/tmp as tmpfs. not /dev/sda1

so, wget initial file to > /tmp/temp.host
cat /tmp/temp.host | fgrep -vf /tmp/mnt/sda1/whitelist.txt > /tmp/mnt/sda1/hosts.clean
wouldn't be better?

 

[Mivis]

Hi, I´ve been to follow every step to get the adblocking on my ac68u with no success. Could someone make a guide? I would really apreciate it!

Thank you very much!

 

[thelonelycoder]

Hi, I´ve been to follow every step to get the adblocking on my ac68u with no success. Could someone make a guide? I would really apreciate it!

Thank you very much!

The How-to is in my first post in this thread.
If you need more assistance ask specific questions.

 

[BreakingDad]

What would be really cool would be if someone took the latest Merlin Firmware, Added Adblock, Then Added a simple GUI to it and reuploaded it as Merlin+Adblock.

 

[thelonelycoder]

What would be really cool would be if someone took the latest Merlin Firmware, Added Adblock, Then Added a simple GUI to it and reuploaded it as Merlin+Adblock.

What Software tools would one need to do that? I _may_ be interested to venture in this area.
I know RMerlin will not do it, for obvious reasons.

 

[thelonelycoder]

OK found it, it's all described in the Wiki!
Now, all I need is some time to learn and understand.

 

[Ars92]

Yeah I dint have a USB stick in my router and am not planning to install stuffs on it, have my NAS for those. Prefer to keep the router simple. It would be crazy fantastic to have this integrated with merlin! I think if you get in contact with merlin, he might be fine with it, as long as you dint expect him to develop it lol. Great stuff, man. With this ill be able to H gave no ads on my phone without installing any adblock.

Sent from my SM-N9005 using Tapatalk