Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Malware Filter / bad host IPSET

Discussion in 'Asuswrt-Merlin' started by kobejo34, Oct 27, 2016.

  1. wallyg8r

    wallyg8r Occasional Visitor

    Joined:
    Feb 10, 2017
    Messages:
    19
    Mine is 34295. Thanks for the script.
     
    swetoast likes this.
  2. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    recently broke a new record

    Mar 2 12:07:14 system: Malware Filter loaded 52282 unique ip addresses.

    freaking insane numbers of infected or infecting clients outthere
     
    visortgw and popxunga like this.
  3. Cedarhillguy

    Cedarhillguy New Around Here

    Joined:
    Jan 15, 2017
    Messages:
    7
    I tried upgrading to version 17, but doesn't work as-is for a 68u, version 380.65 without entware. It reports the find command, used near the end for cleanup, doesn't support -type and -delete arguments. I was able to get the find command to work using the below code.
    Code:
    find /tmp -name 'malware-filter-*.part' -exec rm {} +
    
    I would also suggest for consideration changing the logger to show the actual number of addresses in IPSET. Current statement assumes everything in the file is loaded, which may not always be the case. The below, I believe would show what is actually in the IPSET tables.
    Code:
    logger -s -t system "Malware Filter loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."
    
     
  4. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    kewl, ill do some testing and if it work for me aswell then ill merge it :)
     
  5. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Boom revision bump due to incompability without entware, tnx Cedarhillguy
    it also it states the correct number for ips blocked in syslog.

    Code:
    #!/bin/sh
    # Author: Toast
    # Contributers: Octopus, Tomsk, Neurophile, jimf, spalife, visortgw, Cedarhillguy
    # Testers: shooter40sw
    # Revision 18
    blocklist=/jffs/malware-filter.list                     # Set your path here
    failover=eth0                                           # Change only if WAN interface is not detected.
    retries=3                                               # Set number of tries here
    regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) # Value for ARM Routers
        MATCH_SET='--match-set'
        HASH='hash:ip'
        SYNTAX='add'
        SWAPPED='swap'
        DESTROYED='destroy'
        OPTIONAL='family inet hashsize 2048 maxelem 65536'
         ipsetv=6
         lsmod | grep "xt_set" > /dev/null 2>&1 || \
         for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
         do
              insmod $module
         done
    ;;
    *v4) # Value for Mips Routers
        MATCH_SET='--set'
        HASH='iphash'
        SYNTAX='-q -A'
        SWAPPED='-W'
        DESTROYED='--destroy'
        OPTIONAL=''
        ipsetv=4
         lsmod | grep "ipt_set" > /dev/null 2>&1 || \
         for module in ip_set ip_set_nethash ip_set_iphash ipt_set
         do
              insmod $module
         done
    ;;
    esac
    check_online () {
    if [ -z "$(which nvram)" ]; then
    iface=`grep "$failover" /proc/net/dev`
    if   [ -n "$iface" ]; then
         if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
         then get_list; fi
         else exit 1; fi
    else iface=`nvram get wan0_ifname`
    if   [ -n "$iface" ]; then
         if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
         then get_list; fi
         else exit 1; fi
    fi }
    get_list () {
    url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
    if [ ! -f $blocklist ]
    then wget $url -O $blocklist; get_source; else get_source; fi
    }
    get_source () {
        wget -q --tries=$retries --show-progress -i $blocklist -O /tmp/malware-filter-raw.part
            awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' /tmp/malware-filter-raw.part > /tmp/malware-filter-presort.part
        cat /tmp/malware-filter-presort.part | grep -oE "$regexp" | sort -u > /tmp/malware-filter-sorted.part
    }
    run_ipset () {
    echo "adding ipset rule to firewall this will take time."
    ipset -L malware-filter >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
        nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
        if [ -f /opt/bin/xargs ]; then
        /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-filter PARAM
        else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-filter {}; fi
    fi
    else
        nice -n 2 ipset -N malware-update $HASH $OPTIONAL
        if [ -f /opt/bin/xargs ]; then
        /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-update PARAM
        else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-update {}; fi
        nice -n 2 ipset $SWAPPED malware-update malware-filter
        nice -n 2 ipset $DESTROYED malware-update
    fi
    iptables -L | grep malware-filter > /dev/null 2>&1
    if [ $? -ne 0 ]; then
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    else
        nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    fi }
    cleanup () {
    logger -s -t system "Malware Filter loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."
    find /tmp -name 'malware-filter-*.part' -exec rm {} +
    }
    check_online
    run_ipset
    cleanup
    exit $?
     
  6. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    125
    I have been experiencing inconsistent results with the check_online routine, where it does not always report the IP address to confirm online status. I have changed to the following (Thank you, Google!):

    if [ "$(ip addr show eth0 | grep inet | awk '{print $2}' | sed 's#/.*##')" \
    != "" ] then get_list;
    else exit 1; fi​
     
  7. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    447
    I've used [ $(nvram get wan0_state_t) -eq 2 ] to check if router is online in the past. Not sure if that is correct...
     
    visortgw likes this.
  8. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    125
    That appears to be what has been used in rstats source. It also seems to be quickest method to verify WAN connection status.
     
  9. Xentrk

    Xentrk Very Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    706
    Location:
    Chiang Mai, Thailand
    ipset on DD-WRT

    Hi @swetoast,

    I know you are trying to make the malware-filter and privacy-filer generic. So I wanted to see if I could get them to work on DD-WRT.

    I got ipset to work on DD-WRT build Firmware: DD-WRT v3.0-r29837 std (06/06/16). ipset is not included in dd-wrt builds and the iptables version is old. To get it working, I followed the instructions here http://www.dd-wrt.com/phpBB2/viewtopic.php?p=1035801 with the caveat of compiling the included xp_set.ko file.

    Code:
    mkdir /jffs/usr
    cd /jffs/usr
    wget -O ipset_ipt_libmnl.K3.Arm.tar http://www.dd-wrt.com/phpBB2/download.php?id=31626
    wget -O dnsmasq_ipset.tar http://www.dd-wrt.com/phpBB2/download.php?id=35637
    tar xf ipset_ipt_libmnl.K3.Arm.tar
    tar xf dnsmasq_ipset.tar
    rm ipset_ipt_libmnl.K3.Arm.tar
    rm dnsmasq_ipset.tar
    Then, get the xp_set.zip file from here:
    https://www.dd-wrt.com/phpBB2/viewtopic.php?t=306131

    Download to PC and unzip. Copy xt_set.ko to /jffs/usr/lib/modules/

    Code:
    insmod /jffs/usr/lib/modules/xt_set.ko
    Code:
    # ipset -v
    ipset v6.21.1, protocol version: 6
    Code:
    #lsmod | grep xt_set
    xt_set                  6435  0 [permanent] 
    I tried to use the version if ipset on entware. But it is very old and did not work. So I removed the package. Following is the version information. Entware could use an update on ipset!

    Code:
    ipset v4.5, protocol version 4.
    ipset v4.5: Kernel ip_set module is of protocol version 6.I'm of protocol version 4.
    Please upgrade your kernel and/or ipset(8) utillity. 
    The iptables that came with the above tar files and installed in jffs/usr/sbin is dated as well

    Code:
    # iptables -v
    iptables v1.3.7: no command specified
    So, I installed iptables from entware.

    Code:
    #opkg install iptables
    [CODE]# /opt/sbin/iptables -v
    iptables v1.4.21: no command specified
    But I had to change my PATH to make sure the script used the newer version:

    Code:
    PATH=/opt/sbin:/opt/bin:/bin:/bin:/usr/bin:/sbin:/usr/sbin:/jffs/usr/sbin:/jffs/usr/mmc/sbin:/mmc/bin:/mmc/usr/sbin:/mmc/usr/bin
    This allowed me to run the Tor and Countries Block script. I then added the following to the Administration - Commands – Startup so it would start the script at boot up:


    Code:
    insmod /jffs/usr/lib/modules/xt_set.ko
    PATH=/opt/sbin:/opt/bin:/bin:/bin:/usr/bin:/sbin:/usr/sbin:/jffs/usr/sbin:/jffs/usr/mmc/sbin:/mmc/bin:/mmc/usr/sbin:/mmc/usr/bin
    sh /jffs/scripts/create-ipset-lists.sh

    Code:
    ~# iptables -L | grep set
    DROP       all  --  anywhere             anywhere             match-set BlockedCountries src
    DROP       all  --  anywhere             anywhere             match-set TorNodes src
    DROP       all  --  anywhere             anywhere             match-set MicrosoftSpyServers dst
    REJECT     tcp  --  anywhere             anywhere             reject-with tcp-reset
    I then tried to run malware-block and privacy-filter. But they both exit on the check_online function. I wonder if you can help me figure out why. Is it because my NVRAM is limited? I have 64KB total of NVRAM and 46KB is being used. I don’t have a /usr/sbin/nvram folder so 0 length gets returned on that line.

    Code:
    check_online
    + check_online
    + which nvram
    + [ -z /usr/sbin/nvram ]
    + nvram get wan0_ifname
    + iface=
    + [ -n  ]
    + exit 1
    
    /jffs/scripts # grep "eth0" /proc/net/dev
      eth0: 1299364610 1292030    0  48    0     0          0         0 1221297441 1254742    0    0    0     0       0          0[\CODE]
    
    /jffs/scripts # nvram show | grep wan0_ifname
    size: 46387 bytes (19149 left)
     
    Last edited: Mar 8, 2017
  10. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    447
    Maybe you can try swapping the check_online function with:
    Code:
    check_online () {
      [ $(nvram get wan0_state_t) -eq 2 ] && get_list || exit 1
    }
     
  11. Xentrk

    Xentrk Very Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    706
    Location:
    Chiang Mai, Thailand
    Thanks for the suggestion. I get an unkown operand error with the revised function. I will continue to analyze..
    Code:
    check_online
    + check_online
    + nvram get wan0_state_t
    + [ -eq 2 ]
    sh: 2: unknown operand
    + exit 1
    At the suggestion of https://www.shellcheck.net/#, I quoted the nvram:
    Code:
    check_online () {
      [ "$(nvram get wan0_state_t)" -eq 2 ] && get_list || exit 1
    }
    I now get
    Code:
    + check_online
    + nvram get wan0_state_t
    + [  -eq 2 ]
    sh: out of range
    + exit 1
    Getting closer, on AC88U, I get two lines returned:
    Code:
    nvram show | grep wan0_state_t
    wan0_state_t=2
    size: 60870 bytes (70202 left)
    On dd-wrt, only the size information is returned. I need to find the equivalent nvram parameter in dd-wrt. Getting closer..

    I found the equivalent nvram parameter in dd-wrt. It is "wanup". It was set to "1" so that is what I used in the check:
    Code:
    }
    check_online () {
      [ "$(nvram get wanup)" -eq 1 ] && get_list || exit 1
    }
    I made the same change to privacy-filter and that worked too. So this is very kewl!!! I have been working on this for 5 days off and on and it is good to see it working.
    Code:
    # iptables -L | grep match
    DROP       all  --  anywhere             anywhere             match-set BlockedCountries src
    DROP       all  --  anywhere             anywhere             match-set TorNodes src
    REJECT     all  --  anywhere             anywhere             match-set privacy-filter_ipv4 src,dst reject-with icmp-port-unreachable
    REJECT     all  --  anywhere             anywhere             match-set malware-filter src,dst reject-with icmp-port-unreachable
    DROP       all  --  anywhere             anywhere             match-set MicrosoftSpyServers dst
    DD-WRT does not make this simple. Not including ipset and having old iptables version in the kernel does not help. Flashing any newer firmware past June 2016 bricks the D-Link 880L. Many guys complaining about this in the dd-wrt Broadcom Forum, D-Link 880L thread. I had to use my USB to TTL cable more than once to unbrick when trying newer dd-wrt versions. Entware came through on providing updated iptables. But the ipset version on entware could use an update.

    I may post my findings on the dd-wrt.com forum pending feedback from @swetoast. This was done using privacy-filter version 12 and version 18 on malware-filter.

    The children's home where this router is installed is going to let me replace this router with another. It will definitely be an ASUS. I will then turn the D-Link into an AP to give improve wifi range at the home. I just could not resist the challenge to make these scripts work on another platform.
     
    Last edited: Mar 9, 2017
  12. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    447
    Yes, sorry. It appears your dd-wrt router does not have that variable defined in nvram, here is an alternate way:
    Code:
    check_online () {
      ping -q -c 1 google.com >/dev/null 2>&1 && get_list || exit 1
    }
     
  13. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    447
    @swetoast Do you know why the checks for nvram in path or checking if router is online are even necessary? Shouldn't all asuswrt routers have that and be expected to always be online? I mean if the router is not online, that's a major issue.
     
  14. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    447
    Have you tested this? Quick test would be to disconnect the wan ethernet and checking if that flips.
     
  15. Xentrk

    Xentrk Very Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    706
    Location:
    Chiang Mai, Thailand
    Excellent point @redhat27! I will test it tomorrow and let you know.
     
  16. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Hi guys, been at work and i see you got some suggestions and im sure those are excellent so im gonna make the changes when i got some time over based on your suggestions cause i want to work with *wrt and x86 out of the box :)
     
  17. Xentrk

    Xentrk Very Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    706
    Location:
    Chiang Mai, Thailand
    I can help test on wrt platform for you. From what I read on the forums, ipset is not included in the dd-wrt builds to save space. And, the iptables verison on dd-wrt is also old. Having ipset version updated on entware will allow someone to easily get around the issues I had. I wonder if @ryzhovau can update ipset4 entware package to more recent version? And maybe iptables as well?
     
    skeal likes this.
  18. skeal

    skeal Senior Member

    Joined:
    Apr 30, 2016
    Messages:
    225
    Location:
    Moose Jaw Saskatchewan Canada
    This is great it installed on my AC68U real well. Great script!
     
    swetoast likes this.
  19. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    tnx @skeal, feel free to check out my other scripts :)
     
  20. Xentrk

    Xentrk Very Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    706
    Location:
    Chiang Mai, Thailand
    I confirmed the DD-WRT nvram parameter wanup=1 when the WAN is UP and wanup=0 when the WAN is DOWN.
     
    Last edited: Mar 9, 2017
    redhat27 likes this.

Share This Page