ad blocking on the router

[ryzhov_al]

Having ipset support would be ideal in terms of performance for those kind of blacklists. Unfortunately my last attempt at porting ipset to our kernel wasn't entirely successful - I had issues with the client itself. Maybe someday I'll give it another try.

ipset 4.5 is ready.

opkg install ipset4

If kernel modules are ready, lets test them.

 

[RMerlin]

ipset 4.5 is ready.

opkg install ipset4

If kernel modules are ready, lets test them.

Since ipset requires the command to be usable, it wouldn't make much sense to have the kernel modules but not the command itself in the FW.

I just lack the time to take another look at it these days.

 

[ryzhov_al]

I just lack the time to take another look at it these days.

I've got utility on hands, you've got a kernel modules.

The ipset utility is not so kernel depended, so you may try to use it.

 

[alk630]

I've got utility on hands, you've got a kernel modules.

The ipset utility is not so kernel depended, so you may try to use it.

I think this really would make a great addition for the less technical people here, myself included. Personally, I've never seen any firmware with working ad-blocking so one that did would be a real bonus

 

[RMerlin]

I think this really would make a great addition for the less technical people here, myself included. Personally, I've never seen any firmware with working ad-blocking so one that did would be a real bonus

ipset IS targeted at technical people. I have no intention of integrating any actual ad blocker in the firmware, for many reasons. ipset is just an iptable extensions that makes it more efficient to block a whole range of IPs, and can be used for many other reasons beyond just ad blocking.

 

[felix]

Privoxy

I have been running privoxy on my PCs/Macs for some years, and it is easy to use, even the default action files work fine in my opinion. I just installed the Merlin firmware and activated privoxy via entware today, so I cannot give any sufficient experience report regarding its use on the router itself, except for that the setup was straightforward and the only skill required was understanding the privoxy configuration manual.

 

[ryzhov_al]

I've got utility on hands, you've got a kernel modules.

The ipset utility is not so kernel depended, so you may try to use it.

Now I get it^)
I've recompiled kernel to get ipt_{set,SET} modules with appropriate patch from wl500g project and I've got both modules and ipset command.

Now I can define ipset rules but… I can't apply it, because iptables need to be patched too. Will deal with it later.

 

[RMerlin]

Now I get it^)
I've recompiled kernel to get ipt_{set,SET} modules with appropriate patch from wl500g project and I've got both modules and ipset command.

Now I can define ipset rules but… I can't apply it, because iptables need to be patched too. Will deal with it later.

Ah, that might explain it. When the ipset command failed to apply any new rules, I thought at the time it was because I was missing a few shared libraries into my rootfs (I noticed I had built a dynamically linked version of ipset). At that point I was already quite busy with the 3TB + Busybox stuff (which was more important), so I simply dropped the project for now, hoping to get back to it at a later time.

I used the 630-netfilter_ipset patch from WL500G to get kernel support - probably the same thing you also used.

I'll create an ipset branch on Github, and reapply my work-in-progress to it.

 

[RMerlin]

ipset branch pushed to Github. I actually finalized the Makefile so it will also install all the ipset shared libraries as well. Creating new sets worked, I haven't had the time yet to test if creating iptable rules that refer to ipsets is working.

If you give it a shot, don't forget to also load the required kernel modules.

I'll spend some more time with it this week (for now it's way past bedtime for me).

 

[AnthonyArmato]

The ad blocking method from tutorial is based on hosts blacklist. I'll describe how to do it, but IMHO, it's a simplest and worst one. Later I tell you why.

You'll need to create two files. A first one is /jffs/scripts/wan-start:

#!/bin/sh
wget -O - http://www.mvps.org/winhelp2002/hosts.txt | grep 127.0.0.1 | sed '2,$s/127.0.0.1/0.0.0.0/g; s/[[:space:]]*#.*$//g;' | grep -v localhost | tr ' ' '\t' |tr -s '\t' | tr -d '\015' | sort -u > /tmp/hosts0
killall dnsmasq && dnsmasq --log-async

The second one is /jffs/configs/dnsmasq.conf.add with this content:

addn-hosts=/tmp/hosts0

Make first one executable:

chmod +x /jffs/scripts/wan-start

and reboot router.

Was this the final way of doing this ?

 

[bazzly]

I think this was the only version of this type, and last I tried it does not work.

I tried to set up a different way, but need help with the script, and havent had any bites yet.

 

[RMerlin]

Now I get it^)
I've recompiled kernel to get ipt_{set,SET} modules with appropriate patch from wl500g project and I've got both modules and ipset command.

Now I can define ipset rules but… I can't apply it, because iptables need to be patched too. Will deal with it later.

I got ipset working I created an ipset, and inserted a rule in my INPUT table. I was able to block SSH access from one of my machines that way.

The syntax seems a bit different from the example I found on the web however (the correct argument was --set, not --set-match).

I'll push the working code in a few mins so you can start tinkering with it.

 

[ryzhov_al]

Great! It works!

Here is an example to block outgoing connections to Google DNS servers:

# ping 8.8.4.4

PING 8.8.4.4 (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: seq=0 ttl=50 time=56.394 ms
64 bytes from 8.8.4.4: seq=1 ttl=50 time=56.358 ms
64 bytes from 8.8.4.4: seq=2 ttl=50 time=56.463 ms

# ipset -N NoMoreGoogleDNS iphash
# ipset -A NoMoreGoogleDNS 8.8.8.8
# ipset -A NoMoreGoogleDNS 8.8.4.4
# iptables -A OUTPUT -m set --set NoMoreGoogleDNS dst -j DROP

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Operation not permitted

And here — to block incoming connections from whole countries:

ipset -N BlockedCountries nethash
# Block connections from Pakistan, Peru and Brunei
for country in pk pe bn
do
    for IP in $(wget -q -O - http://www.ipdeny.com/ipblocks/data/countries/$country.zone)
    do
        ipset -A BlockedCountries $IP
    done
done
iptables -A INPUT -m set --set BlockedCountries src -j DROP

This is an asuswrt-merlin, ipset branch. Both ipset from firmware and Entware is working, will keep mine in repo for other firmwares.

 

[RMerlin]

ipset code was merged into master, and will be part of the next release.

We'll need a tutorial or two on how to use it for efficient blocking of blacklisted IPs

 

[ryzhov_al]

We'll need a tutorial or two on how to use it for efficient blocking of blacklisted IPs

And we need a volunteer who will test its perfomance too

wget -q -O - http://www.ipdeny.com/ipblocks/data/countries/cn.zone | wc -l
2950

Not sure I'll be able to catch a perfomance degradation on my 30Mbps connection.

 

[ryzhov_al]

[HOW-TO] Using ipset

As I suspected, no performance degradation revealed while using over a 6000 strings ipset filter on my 30Mbps internet connection. This is a first approach to new HOW-TO.

Ipset feature will be a part of next Merlin's release, if you wish to try it right now, please use this build for RT-N66U: RT-N66U_3.0.0.4_270.26_ipset.trx and please note:

…ipset IS targeted at technical people. …ipset is just an iptable extensions that makes it more efficient to block a whole range of IPs, and can be used for many other reasons beyond just ad blocking.

There will be no any WEB UI ipset page or something, it's a command line tool for geeks


It's my working script, fast and dirty. This is NOT for ad-blocking, but for preventing hacker's/scanner's attacks by blocking incoming connections from certain IP addresses. Will add some fool-proof later. Script divided to three parts, started with (#) signs:

• loading ipset modules,
• defining rules to block traffic from Tor nodes,
• defining rules to block traffic from specific countries,

Please, enable JFFS form WEB UI, put this content to /jffs/scripts/firewall-start file:

#!/bin/sh
# load ipset modules
IPSET_PATH=/lib/modules/2.6.22.19/kernel/net/ipv4/netfilter
insmod $IPSET_PATH/ip_set.ko
insmod $IPSET_PATH/ip_set_nethash.ko
insmod $IPSET_PATH/ip_set_iphash.ko
insmod $IPSET_PATH/ipt_set.ko

# block traffic from Tor nodes
iptables -D INPUT -m set --set TorNodes src -j DROP
ipset --destroy TorNodes
ipset -N TorNodes iphash
for IP in $(wget -q -O - http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv)
do
    ipset -A TorNodes $IP
done
iptables -A INPUT -m set --set TorNodes src -j DROP

# block incoming traffic from some countries. 
iptables -D INPUT -m set --set BlockedCountries src -j DROP
ipset --destroy BlockedCountries
ipset -N BlockedCountries nethash
for country in pk cn
do
    for IP in $(wget -q -O - http://www.ipdeny.com/ipblocks/data/countries/$country.zone)
    do
        ipset -A BlockedCountries $IP
    done
done
iptables -A INPUT -m set --set BlockedCountries src -j DROP

and make it executable:

chmod +x /jffs/scripts/firewall-start

You may put (un)desired countries codes to this string

for country in pk cn

where pk is for Pakistan and cn is for Chine, just for example. Please refer to ipdeny.com/ipblocks/ for full countries list.

 

[Barry]

I followed all the steps and everything worked, but when I reboot all the files in the /jffs/configs and /jffs/scripts are gone and adsuck too. Entware doesn't seem to stay either.

Edit: Might be that formatting bug. I'll report back later.

 

[RMerlin]

I followed all the steps and everything worked, but when I reboot all the files in the /jffs/configs and /jffs/scripts are gone and adsuck too. Entware doesn't seem to stay either.

Edit: Might be that formatting bug. I'll report back later.

Yeah, just disable the formatting option manually after the first reboot:

nvram set jffs2_format=0
nvram commit

 

[Barry]

Did the downgrade/upgrade and it worked. Running those commands would have saved me a lot of time.

Adsuck is working! Only thing is it is not collapsing blocked elements on my phone. If I want to try something else, how do I go about removing adsuck? Also, how do I manually format the jffs partition?

Edit: Used the MVPS list in adsuck's Hosts.prc and all is good. Their list hadn't been updated in over a year, so now I get the best of both worlds.

 

[kfmfe04]

ip lists to feed ipset

Thanks for incorporating ipset into the FW!

Any suggestions for where I can get lists of useful IPs to feed into ipset?


It'd be great if there were lists categorized by type-of-vice - I was thinking of just using dig or nslookup on that giant list of URLs provided by URLBlacklist to generate an equivalent ip, but I suspect there must be a better way...

UPDATE
Looks like there may be some good candidates here:

http://www.selectrealsecurity.com/public-block-lists