Malware Filter / bad host IPSET

[icornish]

Hi,

I have implemented this script, and seems to work OK. Is it possible to include the Country & TorNode blocking that is covered in the first script on the wiki page by adding the appropriate URL's to the list file?

Scratch this... I can see the problem - this script is intended to block connections from LAN through to known malware sources... the other one needs to prevent them from coming into the LAN in the first place (FORWARD vs INPUT)

 

[swetoast]

no idea havent look at those script

 

[Xentrk]

Thanks for pointing me in the right direction. It appears that the following hosts on the list are unavailable at the moment based on ping:

http://cinsscore.com/list/ci-badguys.txt
http://sanyalnet-cloud-vps.freeddns.org/mirai-ips.txt

I commented those entries out and the script worked.

I could access the sites via web browser.

 

[swetoast]

then use em nothing wrong with em just that they are inconsistant for some users (including me)

 

[HardCat]

@swetoast - installed as per wiki. Initially I had a problem of only getting around 1158 ip's in the ipset list after the script ran. Turns out, I had a version of wget installed from entware that was not playing nice. So, I removed it with "opkg remove wget" and now the script is working great 33389 ip's in the ipset list since the version /usr/sbin/wget is used instead. Great Work!

 

[swetoast]

if you have entware wget you need ca-certificates also but using firmware wget is perfectly fine also

 

[spalife]

@swetoast

I noticed that about 30-50% of time is reduced by speeding up / enabling parallel processing of additions to ipset

by changing the below line in Malware Filter script

for i in `cat $path/malware-filter.txt`; do nice ipset $SYNTAX malware-update $i ; done

TO

xargs -P10 -I “PARAM” -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-filter PARAM > /dev/null 2>&1

xargs has the capability to "parallel" process whenever possible in the above case about 10 max parallel procs

 

[spalife]

xargs -P10 -I “PARAM” -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-filter PARAM > /dev/null 2>&1

should read

xargs -P10 -I “PARAM” -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-update PARAM > /dev/null 2>&1

 

[swetoast]

Hey @spalife will try it out and if it all works then ill merge it

 xargs -P10 -I  ^`^|PARAM ^`^} -n1

when i paste your script when i correct it to "" it still doesnt work.

please use the code tag when posting

 

[swetoast]

also remember that the xargs busybox version is not like the standard version

BusyBox v1.25.1 (2017-01-31 13:38:47 EST) multi-call binary.

Usage: xargs [OPTIONS] [PROG ARGS]

Run PROG on every item given by stdin

       -p      Ask user whether to run each command
       -r      Don't run command if input is empty
       -0      Input is separated by NUL characters
       -t      Print the command on stderr before execution
       -e[STR] STR stops input processing
       -n N    Pass no more than N args to PROG
       -s N    Pass command line of no more than N bytes
       -I STR  Replace STR within PROG ARGS with input line
       -x      Exit if size is exceeded

the only way to get it the way you want it is to use the entware package "findutils" we could do a check if /opt/bin/xargs exists and then do it the way you want it if not then fallback on my way of doing it.

but that requires the line to be fixed

xargs -P10 -I 'PARAM' -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-filter PARAM > /dev/null 2>&1

 

[spalife]

It is double-quotes i.e., (")around PARAM.

You are right I have been using the xargs from the entware.
Perhaps it was installed as a dependency of one of the packages either dnscrypt or privoxy.

xargs --version
xargs (GNU findutils) 4.6.0


xargs -P10 -I “PARAM” -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-update PARAM > /dev/null 2>&1

 

[swetoast]

use code tag when posting lines.

 

[swetoast]

i didnt use codetag and got it running used ' instead

12862 xxxx     4216 S    /opt/bin/xargs -P10 -I PARAM -n1 -a /opt/var/cache/malware-filter/malware-filter.txt ipset add malware-filter PARAM
28835 xxxx     4400 S    grep ipset
28846 xxxx      868 R    ipset add malware-filter 212.227.105.182
28849 xxxx      852 R    ipset add malware-filter 212.227.29.131
28850 xxxx      348 R    ipset add malware-filter 212.227.31.159
28852 xxxx      300 R    ipset add malware-filter 212.227.91.80

 

[swetoast]

could do something like this

        if ! /opt/bin/xargs; then
        for i in `cat $path/malware-filter.txt`; do ipset $SYNTAX malware-update $i ; done; else
        /opt/bin/xargs -P10 -I 'PARAM' -n1 -a $path/malware-filter.txt ipset $SYNTAX malware-update PARAM; fi

 

[spalife]

nice
I was using the BB Code Editor option while posting,
looks like it does not work out so well

 

[swetoast]

your idea is accepted just tweaking it to my style gonna add you as a contributor and post a revision as soon as its tested.

 

[spalife]

Nice of you.
I see that in your attempt it spawned 4 procs

 

[swetoast]

tested it with watch command running every second and it can spawn up to 10

 

[swetoast]

Revision 10, feel free to try it out and see if it all works as intended. So with Entwares findutils package installed the script takes about 5 min to complete usually takes about 10 min when running without.

Changelog:

• Checks if the package "findutils" xargs is available from entware and if it is speeds up the process of adding the set to the firewall (tnx spalife)

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 10

path=/opt/var/cache/malware-filter                      # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;

*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
        mkdir -p $path
        wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.pre
        cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-filter.txt
 }

run_ipset () {

get_list

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-filter PARAM
    else for i in `cat $path/malware-filter.txt`; do nice -n 2 ipset $SYNTAX malware-filter $i ; done; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-update PARAM
    else for i in `cat $path/malware-filter.txt`; do nice -n 2 ipset $SYNTAX malware-update $i ; done; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi
}

run_ipset

logger -s -t system "Malware Filter loaded $(cat $path/malware-filter.txt | wc -l) unique ip addresses."
exit $?

 

[swetoast]

So ive been running test since the new revision and 30000+ takes around 5~ min on a RT-AC56U kinda curious to hear results from others. I am interested in both routers with and without entware installed.

just do this command

time malware-block

and when the script ends it prints out 3 stats i used the "real" as measurement for my tests so please post avg time and router model and if entware was present.