What's new

Using ipset revisited

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Neurophile

Occasional Visitor
As part of a round of beefing up the security of my network I decided to set up ipset/iptables to filter traffic from the darkest corners of the interwebs. I started with the Malware Filter script from the wiki here: https://github.com/RMerl/asuswrt-merlin/wiki/Using-ipset
Unfortunately, using 380.58 on an N66U this script is broken...so I fixed it and tweaked it to my needs. Here is the result.

First create the scripts (full code can be found on my github: https://github.com/Neurophile/ASUSWRT-scripts )
1) /jffs/scripts/firewall-start
2) /jffs/scripts/firewall-update
3) /jffs/scripts/services-start

set each one executable with, e.g.: chmod +x /jffs/scripts/firewall-start

Next create the list of blacklists that you want to use:
/jffs/filters/malware-filter.list

Mine is a very minimal set for now, more are listed in my master-blacklist-list or use your own favorites. Manually run the firewall-update script once to create and populate the IP blocklist cache file /jffs/filters/malware-filter.txt

You can now run firewall-start or reboot the router and your new blocklist will be blocking! The blacklist updates every night at 3 am via the cron job setup in the services-start script.

Enjoy!
 
Next create the list of blacklists that you want to use:
/jffs/filters/malware-filter.list

Mine is a very minimal set for now, more are listed in my master-blacklist-list or use your own favorites. Manually run the firewall-update script once to create and populate the IP blocklist cache file /jffs/filters/malware-filter.txt

You can now run firewall-start or reboot the router and your new blocklist will be blocking! The blacklist updates every night at 3 am via the cron job setup in the services-start script.

Enjoy!


Haha nice of ya to use my script tnx :D hope that it works out for ya and tnx for the revision
feel free to update the wiki both me and rmerlin would be thankful :)
 
Haha nice of ya to use my script tnx :D hope that it works out for ya and tnx for the revision
feel free to update the wiki both me and rmerlin would be thankful :)

Thanks for writing the original! The main failure is that wget doesn't recognize the ssl certificates. I hacked around this by adding --no-check-certificate but that is not the safest fix. I am 99.9% sure that spamhaus has a valid ssl, so it has to be something with either the wget config or with ssl certs as a whole in the router. If I can figure that out I will fix up the wiki.
 
Thanks for writing the original! The main failure is that wget doesn't recognize the ssl certificates. I hacked around this by adding --no-check-certificate but that is not the safest fix. I am 99.9% sure that spamhaus has a valid ssl, so it has to be something with either the wget config or with ssl certs as a whole in the router. If I can figure that out I will fix up the wiki.

Wget does not let you specify the location of the CA bundle at compile time, it's hardcoded to the default location used by OpenSSL. Unfortunately, Asus decided to use that default location to store the httpd's SSL certificate, making it hard to include a default bundle without having to shuffle a lot of things around.

As a workaround, you can use curl - curl is compiled to point at the CA bundle stored in /rom/ .
 
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.
 
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.

I'll take a look to see if it could easily be done in a global wgetrc file.
 
This setup works with my RT-AC87U. Using both malware-filter and windows 10 filter. Crazy number of ips being blocked.
 
FYI if you want to see how much (if any) traffic is being blocked by the list do:
iptables -L -v | less

up near the top of the first page you should see something like this
Code:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source        destination         
 4    336  DROP       all  --  any    any     anywhere      anywhere       set malware-filter dst
This is what it looked like after I sent 4 ping packets to an ip on the list to make sure it is working.
 
wget will look for certs in the directory pointed at by the SSL_CERT_DIR env variable. I use the little snippet below to make wget work with ssl in my scripts

Code:
# Define path to certs so HTTPS works in wget
# Certificates are installed using 'opkg install ca-certificates'
export SSL_CERT_DIR=/opt/etc/ssl/certs
 
Nice too see my little script benefit so many :)

yepp and i can agree to how many ips are being banned but the net is a shirtty place with spammer and malware spreaders so why not try and make the most of it and ban em :)
 
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.

Fixed.
 
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.

One is for the testing of which version of ipset is running for the loading of the modules that @john9527 has pointed out in his thread http://www.snbforums.com/threads/fo...ilable-v18e3-v19b4.18914/page-209#post-278253

The other is where the script tests for an "Uknown set"

Code:
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep 'Unknown set')" != "" ]; then

We can change it to test for both version 4 and version 6 with this small change.

Code:
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then

Cheers.
 
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.

One is for the testing of which version of ipset is running for the loading of the modules that @john9527 has pointed out in his thread http://www.snbforums.com/threads/fo...ilable-v18e3-v19b4.18914/page-209#post-278253

The other is where the script tests for an "Uknown set"

Code:
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep 'Unknown set')" != "" ]; then

We can change it to test for both version 4 and version 6 with this small change.

Code:
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then

Cheers.

Thanks very much. This info was very helpful.

Huge thanks and shutout to @HardCat

All working now as expected, I need to read the change log more often, I guess I get complacent at times, which I can only blame myself :rolleyes:
 
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.
Regarding the new ipset 6.x version, I'm still figuring ipset out, and I'm using the sample scripts on the RMerlin wiki here, but I'm not sure yet what I really need to all change to get it working. Could someone point me in the correct direction, or update the wiki?

At the moment, this is all i get from the first script sample:
Code:
./firewall-start
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: 'ip_set_nethash.ko': module not found
insmod: 'ip_set_iphash.ko': module not found
insmod: 'ipt_set.ko': module not found
Bad argument `TorNodes'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `BlockedCountries'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `MicrosoftSpyServers'
Try `iptables -h' or 'iptables --help' for more information.

Current script sample:
Code:
#!/bin/sh

# Loading ipset modules
lsmod | grep "ipt_set" > /dev/null 2>&1 || \
for module in ip_set ip_set_nethash ip_set_iphash ipt_set
do
   insmod $module
done

# Preparing folder to cache downloaded files
IPSET_LISTS_DIR=/jffs/ipset_lists
[ -d "$IPSET_LISTS_DIR" ] || mkdir -p $IPSET_LISTS_DIR

# Different routers got different iptables syntax
case $(uname -m) in
  armv7l)
   MATCH_SET='--match-set'
   ;;
  mips)
   MATCH_SET='--set'
   ;;
esac

# Block traffic from Tor nodes
if [ "$(ipset --swap TorNodes TorNodes 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N TorNodes iphash
   [ -e $IPSET_LISTS_DIR/tor.lst ] || wget -q -O $IPSET_LISTS_DIR/tor.lst http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
   for IP in $(cat $IPSET_LISTS_DIR/tor.lst)
   do
       ipset -A TorNodes $IP
   done
fi
[ -z "$(iptables-save | grep TorNodes)" ] && iptables -I INPUT -m set $MATCH_SET TorNodes src -j DROP

# Block incoming traffic from some countries. cn and pk is for China and Pakistan. See other countries code at http://www.ipdeny.com/ipblocks/
if [ "$(ipset --swap BlockedCountries BlockedCountries 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N BlockedCountries nethash
   for country in pk cn
   do
       [ -e $IPSET_LISTS_DIR/$country.lst ] || wget -q -O $IPSET_LISTS_DIR/$country.lst http://www.ipdeny.com/ipblocks/data/countries/$country.zone
       for IP in $(cat $IPSET_LISTS_DIR/$country.lst)
       do
           ipset -A BlockedCountries $IP
       done
   done
fi
[ -z "$(iptables-save | grep BlockedCountries)" ] && iptables -I INPUT -m set $MATCH_SET BlockedCountries src -j DROP

# Block Microsoft telemetry spying servers
if [ "$(ipset --swap MicrosoftSpyServers MicrosoftSpyServers 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N MicrosoftSpyServers iphash
   for IP in 23.99.10.11 63.85.36.35 63.85.36.50 64.4.6.100 64.4.54.22 64.4.54.32 64.4.54.254 \
             65.52.100.7 65.52.100.9 65.52.100.11 65.52.100.91 65.52.100.92 65.52.100.93 65.52.100.94 \
             65.55.29.238 65.55.39.10 65.55.44.108 65.55.163.222 65.55.252.43 65.55.252.63 65.55.252.71 \
             65.55.252.92 65.55.252.93 66.119.144.157 93.184.215.200 104.76.146.123 111.221.29.177 \
             131.107.113.238 131.253.40.37 134.170.52.151 134.170.58.190 134.170.115.60 134.170.115.62 \
             134.170.188.248 157.55.129.21 157.55.133.204 157.56.91.77 168.62.187.13 191.234.72.183 \
             191.234.72.186 191.234.72.188 191.234.72.190 204.79.197.200 207.46.223.94 207.68.166.254
   do
       ipset -A MicrosoftSpyServers $IP
   done
fi
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
Last edited:
Ah, thanks! The second link was the one I couldn't find! :)

Still having a problem though...
Code:
Bad argument `TorNodes'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `BlockedCountries'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `MicrosoftSpyServers'
Try `iptables -h' or 'iptables --help' for more information.
But if I run the commands manually, they all seem to work. Anyone see anything wrong?

Code:
#!/bin/sh

# Loading ipset modules
ipset -v | grep -i "v4" > /dev/null 2>&1                                      
if [ $? -eq 0 ];                                                              
then                                                                    
   # old ipset                                                          
   ipsetv=4                                                              
   lsmod | grep "ipt_set" > /dev/null 2>&1 || \                          
   for module in ip_set ip_set_nethash ip_set_iphash ipt_set            
   do                                                                    
       insmod $module                                                
   done                                                                  
else                                                                          
   # new ipset                                                  
   ipsetv=6                                                    
   lsmod | grep "xt_set" > /dev/null 2>&1 || \                  
   for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set  
   do                                                          
       insmod $module                                      
   done                                                        
fi        

# Preparing folder to cache downloaded files
IPSET_LISTS_DIR=/jffs/ipset_lists
[ -d "$IPSET_LISTS_DIR" ] || mkdir -p $IPSET_LISTS_DIR

# Different routers got different iptables syntax
case $(uname -m) in
  armv7l)
    MATCH_SET='--match-set'
    ;;
  mips)
    MATCH_SET='--set'
    ;;
esac

# Block traffic from Tor nodes
if [ "$(ipset --swap TorNodes TorNodes 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create TorNodes hash:ip
    [ -e $IPSET_LISTS_DIR/tor.lst ] || wget -q -O $IPSET_LISTS_DIR/tor.lst http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
    for IP in $(cat $IPSET_LISTS_DIR/tor.lst)
    do
        ipset add TorNodes $IP
    done
fi
[ -z "$(iptables-save | grep TorNodes)" ] && iptables -I INPUT -m set $MATCH_SET TorNodes src -j DROP

# Block incoming traffic from some countries. cn and pk is for China and Pakistan. See other countries code at http://www.ipdeny.com/ipblocks/
if [ "$(ipset --swap BlockedCountries BlockedCountries 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create BlockedCountries hash:net
    for country in pk cn
    do
        [ -e $IPSET_LISTS_DIR/$country.lst ] || wget -q -O $IPSET_LISTS_DIR/$country.lst http://www.ipdeny.com/ipblocks/data/countries/$country.zone
        for IP in $(cat $IPSET_LISTS_DIR/$country.lst)
        do
            ipset add BlockedCountries $IP
        done
    done
fi
[ -z "$(iptables-save | grep BlockedCountries)" ] && iptables -I INPUT -m set $MATCH_SET BlockedCountries src -j DROP

# Block Microsoft telemetry spying servers
if [ "$(ipset --swap MicrosoftSpyServers MicrosoftSpyServers 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create MicrosoftSpyServers hash:ip
    for IP in 23.99.10.11 63.85.36.35 63.85.36.50 64.4.6.100 64.4.54.22 64.4.54.32 64.4.54.254 \
              65.52.100.7 65.52.100.9 65.52.100.11 65.52.100.91 65.52.100.92 65.52.100.93 65.52.100.94 \
              65.55.29.238 65.55.39.10 65.55.44.108 65.55.163.222 65.55.252.43 65.55.252.63 65.55.252.71 \
              65.55.252.92 65.55.252.93 66.119.144.157 93.184.215.200 104.76.146.123 111.221.29.177 \
              131.107.113.238 131.253.40.37 134.170.52.151 134.170.58.190 134.170.115.60 134.170.115.62 \
              134.170.188.248 157.55.129.21 157.55.133.204 157.56.91.77 168.62.187.13 191.234.72.183 \
              191.234.72.186 191.234.72.188 191.234.72.190 204.79.197.200 207.46.223.94 207.68.166.254
    do
        ipset add MicrosoftSpyServers $IP
    done
fi
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
I see you have already modified script to match ipset 6.29.
Try to remove "--" before swap...........

Can you see your rule in INPUT, FORWARD chain?
Code:
iptables -vnL INPUT
iptables -vnL FORWARD

Remove possible "CR"
Code:
dos2unix /jffs/scripts/firewall-start
 
I see you have already modified script to match ipset 6.29.
Try to remove "--" before swap...........

Can you see your rule in INPUT, FORWARD chain?
Code:
iptables -vnL INPUT
iptables -vnL FORWARD

Remove possible "CR"
Code:
dos2unix /jffs/scripts/firewall-start

Nope same problem, and no the rules don't show up. However, after I've manually run the below once for each of the three, the script works fine.
Code:
MATCH_SET='--match-set'
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
Last edited:

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top