Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Malware Filter / bad host IPSET

Discussion in 'Asuswrt-Merlin' started by kobejo34, Oct 27, 2016.

  1. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
  2. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    282
    I found on the web this example for logging:

    Code:
    iptables -N LOGGING
    iptables -A INPUT -j LOGGING
    iptables -A OUTPUT -j LOGGING
    iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
    iptables -A LOGGING -j DROP
    and on the iptables -L I can see there is already a chain called logdrop, with the code above can I edit to reject ?
    thanks
     
  3. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    dunno never bothered with it but its worth trying :) if anyone else got experience please chime in :D in anycase its easy to reset if it doesn't work.
     
  4. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Revision Bump to 20

    Code:
    #!/bin/sh
    # Author: Toast
    # Contributers: Octopus, Tomsk, Neurophile, jimf, spalife, visortgw, Cedarhillguy, redhat27
    # Testers: shooter40sw
    # Supporters: lesandie
    # Revision 20
    
    blocklist=/jffs/malware-filter.list                     # Set your path here
    fwoption=REJECT                                         # DROP/REJECT    (Default Value: REJECT)
    retries=3                                               # Set number of tries here
    regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value
    
    case $(ipset -v | grep -o "v[4,6]") in
      v6)   MATCH_SET='--match-set'; CREATE='create'; ADD='add'; SWAP='swap'; IPHASH='hash:ip'; DESTROY='destroy';
            lsmod | grep -q "xt_set" || \
            for module in ip_set ip_set_nethash ip_set_iphash xt_set; do
                insmod $module
            done ;;
      v4)   MATCH_SET='--set'; CREATE='--create'; ADD='--add'; SWAP='--swap'; IPHASH='iphash'; DESTROY='--destroy';
            lsmod | grep -q "ipt_set" || \
            for module in ip_set ip_set_nethash ip_set_iphash ipt_set; do
                insmod $module
            done ;;
      *)    logger -t system "$0 unsupported ipset version"; exit 1 ;;
    esac
    
    check_online () {
    while ! ping -q -c 1 google.com >/dev/null 2>&1; do
        sleep 1
        WaitSeconds=$((WaitSeconds+1))
        [ $WaitSeconds -gt 300 ] && logger -t system "$0 Router not online! Aborting after a wait of 5 minutes..." && exit 1
    done
    }
    
    get_list () {
    url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
    if [ ! -f $blocklist ]
    then wget $url -O $blocklist; get_source; else get_source; fi
    }
    
    get_source () {
    wget -q --tries=$retries --show-progress -i $blocklist -O /tmp/malware-filter-raw.part
        awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' /tmp/malware-filter-raw.part > /tmp/malware-filter-presort.part
        cat /tmp/malware-filter-presort.part | grep -oE "$regexp" | sort -u > /tmp/malware-filter-sorted.part
    }
    
    run_ipset () {
    echo "adding ipset rule to firewall this will take time."
    ipset -L malware-filter >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
        nice -n 2 ipset $CREATE malware-filter $IPHASH
        if [ -f /opt/bin/xargs ]; then
        /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset  $ADD malware-filter PARAM
        else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $ADD malware-filter {}; fi
    fi
    else
        nice -n 2 ipset $CREATE malware-update $IPHASH
        if [ -f /opt/bin/xargs ]; then
        /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset  $ADD malware-update PARAM
        else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $ADD malware-update {}; fi
        nice -n 2 ipset $SWAP malware-update malware-filter
        nice -n 2 ipset $DESTROY malware-update
    fi
    iptables -L | grep malware-filter > /dev/null 2>&1
    if [ $? -ne 0 ]; then
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j $fwoption
    else
        nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j $fwoption
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j $fwoption
    fi }
    
    cleanup () {
    logger -t system "$0 loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."
    find /tmp -name 'malware-filter-*.part' -exec rm {} +
    }
    
    check_online
    get_list
    run_ipset
    cleanup
    
    exit $?
     
    sasax and wallyg8r like this.
  5. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    501
    FYI, I had done exactly that: Put the choice to the user, both in the iblocklist-loader.sh and on the tor/country/etc. block
     
  6. florid

    florid Occasional Visitor

    Joined:
    Jan 15, 2017
    Messages:
    34
    wow, your list integrated with everything.
    Again I am saying DROP is more common in enterprise firewalls.
    As for home router users, if the block is to filter the outbound traffic, like privacy filter, AD block, please use REJECT (I personally like to use DNSMASQ to filter these URL based list, not iptables).
    If the block is inbound, please consider to use DROP instead of REJECT.
     
  7. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    please discuss redhat27 script at his threads, keep it about my scripts less crosstalk its kinda counter productive
     
  8. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    501
    swetoast likes this.
  9. elorimer

    elorimer Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    322
    You might mention in the wiki to make malware-block executable.

    Also, running .20 manually, I got an error I think in the get_source() module:

    Code:
    awk: /tmp/malware-filter-raw.part: No such file or directory
    I ran it a second time and it worked okay.
     
    Last edited: Mar 25, 2017
  10. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Revision 21
    Changelog:
    • Persistent firewall rules
    • Ipset rewrite
    • Minor fixes and tweaks
    Code:
    #!/bin/sh
    # Author: Toast
    # Contributers: Octopus, Tomsk, Neurophile, jimf, spalife, visortgw, Cedarhillguy, redhat27
    # Testers: shooter40sw
    # Supporters: lesandie
    # Revision 21
    
    blocklist=/jffs/malware-filter.list                     # Set your path here
    fwoption=REJECT                                         # DROP/REJECT    (Default Value: REJECT)
    retries=3                                               # Set number of tries here
    regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value
    
    case $(ipset -v | grep -o "v[4,6]") in
      v6)   MATCH_SET='--match-set'; CREATE='create'; ADD='add'; SWAP='swap'; IPHASH='hash:ip'; DESTROY='destroy';LIST='list'; 
            lsmod | grep -q "xt_set" || \
            for module in ip_set ip_set_nethash ip_set_iphash xt_set; do
                insmod $module
            done ;;
      v4)   MATCH_SET='--set'; CREATE='--create'; ADD='--add'; SWAP='--swap'; IPHASH='iphash'; DESTROY='--destroy';LIST='--list'; 
            lsmod | grep -q "ipt_set" || \
            for module in ip_set ip_set_nethash ip_set_iphash ipt_set; do
                insmod $module
            done ;;
      *)    logger -t system "$0 unsupported ipset version"; exit 1 ;;
    esac
    
    check_online () {
    while ! ping -q -c 1 google.com >/dev/null 2>&1; do
        sleep 1
        WaitSeconds=$((WaitSeconds+1))
        [ $WaitSeconds -gt 300 ] && logger -t system "$0 Router not online! Aborting after a wait of 5 minutes..." && exit 1
    done
    }
    
    get_list () {
    url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
    if [ ! -f $blocklist ]
    then wget $url -O $blocklist; get_source; else get_source; fi
    }
    
    get_source () {
    wget -q --tries=$retries --show-progress -i $blocklist -O /tmp/malware-filter-raw.part
        awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' /tmp/malware-filter-raw.part > /tmp/malware-filter-presort.part
        cat /tmp/malware-filter-presort.part | grep -oE "$regexp" | sort -u > /tmp/malware-filter-sorted.part
    }
    
    run_ipset () {
    echo "adding malware-filter rules to firewall this will take time."
    ! ipset $LIST malware-filter &>/dev/null
    if [ $? -ne 0 ]
    then    nice -n 15 ipset $CREATE malware-update $IPHASH
            if [ -f /opt/bin/xargs ]; then
            /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 15 ipset $ADD malware-update PARAM
            else cat /tmp/malware-filter-sorted.part | xargs -I {} nice -n 15 ipset $ADD malware-update {}; fi
            nice -n 15 ipset $SWAP malware-update malware-filter
            nice -n 15 ipset $DESTROY malware-update
    else    nice -n 15 ipset $CREATE malware-filter $IPHASH
            if [ -f /opt/bin/xargs ]; then
            /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 15 ipset $ADD malware-filter PARAM
            else cat /tmp/malware-filter-sorted.part | xargs -I {} nice -n 15 ipset $ADD malware-filter {}; fi
    fi }
    
    set_firewall () {
    for ipSet in $(ipset -L | sed -n '/^Name:/s/^.* //p'); do
        case $ipSet in
            malware-filter) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j $fwoption ;;
        esac
    done
    }
    
    cleanup () {
    logger -t system "$0 loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."
    find /tmp -name 'malware-filter-*.part' -exec rm {} +
    }
    
    check_online
    get_list
    run_ipset
    set_firewall
    cleanup
    
    exit $?
     
    Last edited: Mar 27, 2017
    shooter40sw likes this.
  11. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    135
    The line "! ipset list malware-filter &>/dev/null" does not work for mips routers (e.g., RT-N66R) with the older version of ipset. It appears that "! ipset --list malware-filter &>/dev/null" will work for both ipset v4 and ipset v6.
     
    Last edited: Mar 27, 2017
    swetoast likes this.
  12. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    kewl ill adjust the revision in the thread no point in uping a new revision for an easy fix, in anycase the above sample should work now
     
  13. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    282
    HI, I have been searching the net but have not found the proper list to use with this script, to be able to block tor exit nodes? I asume that the list on the other script is not compatible with this one, I dont want to be making scripts for everything so does anybody know a list for tor exits IP that I can use with this script?
    thanks
     
  14. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    consult the wiki, this is only for blocking malware filters.. think redhat27 supports tor blocking scripts not me
     
  15. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    282
    Hi @swetoast what is the difference between rev 21 vs 22?
     
  16. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    ohh forgot i upped it

    changelog:
    • syntax cleanup
    • minor tweaks and fixes
     
  17. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Hi everyone

    been working on the wiki while i was sick (didnt have anything better to do) so there is some changes coming up new versions of my script will be posted on my github or if i want testing done ill post here in the thread this is so the wiki will become easier to grasp so there is less confusion over ipset versions and how to deploy the scripts on your home router.

    https://www.snbforums.com/threads/cleaning-up-the-wiki.38338/
    https://github.com/RMerl/asuswrt-merlin/wiki/Using-ipset
    https://github.com/RMerl/asuswrt-merlin/wiki/Ipset-script-installation-instructions
     
    HeMaN likes this.
  18. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    i know i get questions from time to time on how to see if this filter is working or not and i finally took the time to make a simple line that shows it easily for the user.

    Code:
    iptables -L -v | grep "malware-filter" | awk '{print "Malware Filter Blocked: " $1 " packets", $2 " is the size of the transmission"}'
    it will print something like this

    Code:
    Malware Filter Blocked: 0 packets 0 is the size of the transmission
    one could make an alias on this and have it as a easy command
     
  19. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Will link to gitlab from now on

    Revision 23 is up

    Link
    • Bugfix (noticed that sometimes after sorting the script gets confused and looses a file so i added a failover if the file with the ip are missing that its downloaded again.)
     
  20. skeal

    skeal Senior Member

    Joined:
    Apr 30, 2016
    Messages:
    381
    Location:
    Moose Jaw Saskatchewan Canada

    The above command is taken by terminal but doesn't show any results as described.

    Whoops rebooted tried again and it works.
     
    Last edited: Apr 2, 2017

Share This Page