Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Yet another malware block script using ipset (v4 and v6)

Discussion in 'Asuswrt-Merlin' started by redhat27, May 4, 2017.

  1. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    As if we don't have enough already :p

    Here is yet another malware blocking script that uses ipset (v4 and v6). So why put up another one?
    • Over 650 Million unique IPs blocked
    • Simpler (minimalist script with only 25 lines (which includes a couple of blank lines and comment lines)
    • Quicker (about 20 secs typical run time with default lists depending on CPU load, processing power and bandwidth)
    • Less CPU cycles for your router to load the lists (kinda saying the same thing as above)
    • Entware not required (Can run without extended packages or external storage)
    • Only one single preprocess file created in /tmp (that is removed right afterwards)
    • Also supports both ipset v4 and ipset v6
    • Overcomes ipset-v4 limitation to handle more than 64k elements by splitting into multiple sets.
    • Can also be run in cru
    • You can manually whitelist specific IPs
    • Starting with version 2.3, you can manually blacklist discrete IPs and/or CIDR ranges in addition to the lists defined in the urls file.
    • Blocks on the PREROUTING chain of the iptables raw table (affects both traffic initiated by malware sources and also traffic initiated inside your LAN trying to connect to malware destinations)
    • Works on other firmwares (Tomato by Shibby, for example). Script here.
    I am not asking anyone who is using an existing script that they are happy with to switch. I'd hate to get into an argument as to which script is better and why. But in case any one is willing to try this, that is the purpose of this thread.

    The sources lists are mainly from the Cybercrime IP Feeds by FireHOL site:
    As per the site, there are 4 blocking tiers:

    Level1 (around 652 Million unique IPs): A firewall blacklist composed from IP lists, providing maximum protection with minimum false positives. Suitable for basic protection on all internet facing servers, routers and firewalls. (includes: bambenek_c2 dshield feodo fullbogons spamhaus_drop spamhaus_edrop sslbl zeus_badips ransomware_rw)

    Level2 (around 47 Thousand unique IPs): An ipset made from blocklists that track attacks, during about the last 48 hours. (includes: blocklist_de dshield_1d greensnow)

    Level3 (around 189 Thousand unique IPs): An ipset made from blocklists that track attacks, spyware, viruses. It includes IPs than have been reported or detected in the last 30 days. (includes: bruteforceblocker ciarmy dragon_http dragon_sshpauth dragon_vncprobe dshield_30d dshield_top_1000 malc0de maxmind_proxy_fraud myip shunlist snort_ipfilter sslbl_aggressive talosintel_ipfilter zeus vxvault)

    Level4 (around 9.6 Million unique IPs): An ipset made from blocklists that track attacks, but may include a large number of false positives. (includes: cleanmx_viruses blocklist_net_ua botscout_30d cruzit_web_attacks cybercrime haley_ssh iblocklist_hijacked iblocklist_spyware iblocklist_webexploit ipblacklistcloud_top iw_wormlist malwaredomainlist)

    Level1 through Level3 are enabled on this script by default.

    Note on Level4: Because of a large number of false positives in the Level4 ipset, I've disabled it in the sources list. After you install the script, you can, (if you so want) edit the sources list /jffs/ipset_lists/ya-malware-block.urls (default location) to remove the comment '#' on the level4 source. But you may need to whitelist, if you discover some of your desired domains are blocked.


    Install/Reinstall:
    Code:
    wget --no-check-certificate -O /jffs/scripts/ya-malware-block.sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
    chmod +x /jffs/scripts/ya-malware-block.sh
    

    Running:
    Run via cru or on the command line as many times as you want:
    cru (example run every 6 hours):
    Code:
    cru a UpdateYAMalwareBlock "0 */6 * * * /jffs/scripts/ya-malware-block.sh"
    Put the above line in your existing /jffs/scripts/init-start to create a cron schedule at boot time.
    To schedule the job right away (without waiting for the next reboot of your router), just run the above command at the shell prompt. But please note that if you run the cru command from the command line, the schedule will only remain in effect till the next router boot unless also put in init-start

    In addition, you can also run the script once at router boot:
    append the following 2 lines at end of your existing /jffs/scripts/services-start:
    Code:
    sleep 600
    sh /jffs/scripts/ya-malware-block.sh
    Doing so will ensure that the script is run also at boot time, so that you are covered in the interval your router has just rebooted but your scheduled run via cron has not occurred yet.

    You can also just run it at anytime from your router prompt:
    Code:
    /jffs/scripts/ya-malware-block.sh
    This is a sample run on my (slow) router from the terminal with the default blocking (Level1 through Level3)
    Code:
    [email protected]:/tmp/home/root# ya-malware-block.sh
    /jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
    >>> Downloading and aggregating malware sources (also processing whitelists)...[131428/122042/9386] ~11s
    >>> Adding data and processing rule for YAMalwareBlock1IP... ~4s
    >>> Adding data and processing rule for YAMalwareBlock2IP... ~4s
    >>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
    >>> Cleaning up... ~0s
    /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507) and YAMalwareBlockCIDR (9386) in 20 seconds
    
    And this is with all 4 Levels enabled:
    Code:
    [email protected]:/tmp/home/root# ya-malware-block.sh
    /jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
    >>> Downloading and aggregating malware sources (also processing whitelists)...[250472/238198/12274] ~23s
    >>> Adding data and processing rule for YAMalwareBlock1IP... ~5s
    >>> Adding data and processing rule for YAMalwareBlock2IP... ~5s
    >>> Adding data and processing rule for YAMalwareBlock3IP... ~5s
    >>> Adding data and processing rule for YAMalwareBlock4IP... ~3s
    >>> Adding data and processing rule for YAMalwareBlockCIDR... ~2s
    >>> Cleaning up... ~0s
    /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (65535) YAMalwareBlock3IP (65535) YAMalwareBlock4IP (41593) and YAMalwareBlockCIDR (12274) in 43 seconds
    

    Whitelisting [manual]:
    This script does have provision for whitelisting discrete IPs that have been blocked on the YAMalwareBlock?IP sets. Although you'd typically not want to whitelist identified malware, sometimes you may find that you need to, especially if you enable FireHOL Level4.
    You can add any IP you would want to whitelist to the file: /jffs/ipset_lists/ya-malware-block.whites (default location)

    You should only append to this file, and only add IPs (no other characters or comments).

    Notes on whitelisting:
    You should keep the regex entries on the top (private/unroutable over internet) and then append regular IPs down after that.

    The regex entries ensure that your LAN IPs are never blocked. You do not need to whitelist your LAN IPs.

    I've added 3 additional myself (as an example: You may remove these if you want, but be aware of the below, see why I added those):
    • A regex ^216\.239\.3[2468]\.21 for Nintendo users, as per @VZ3 's post
    • AB solution uses a hosts list provider from pgl.yoyo.org. That was being blocked on the default Level1 through Level3 blocking of FireHOL tiers. That 213.230.210.230 IP is for pgl.yoyo.org
    • I use all 4 Levels and noticed that androidfilehost.com (needed by most files hosted in xda-developers.com) was blocked on Level4. That 192.124.249.10 IP is for androidfilehost.com

    The reasoning here of adding IPs without any regex is that the dots need not be escaped (though you could if you wanted to) as dots will replace dots in the regex.


    Blacklisting [manual|optional]
    :
    This script also has provision for blacklisting discrete IPs and/or CIDR ranges in addition to the blocking provided by the lists defined by the urls file.

    You can add any IP/CIDR entries you would want to blacklist to the file: /jffs/ipset_lists/ya-malware-block.blacks (default location)

    Unlike the whitelists file, this file is totally optional, and the script will run fine without it.
    The custom blacklist supports line comments, but not in-line comments. Sample here. Data in that sample file are just test data demonstrating usage/syntax. Please don't use that file as is!


    Uninstall :(:
    Code:
    rm -f /jffs/ipset_lists/ya-malware-block.*
    rm -f /jffs/scripts/ya-malware-block.sh
    
    In addition,
    Undo your changes to /jffs/scripts/init-start and /jffs/scripts/services-start
    Reboot (optional)
    
    Enjoy!
     
    Last edited: Aug 14, 2017
    hunne, eclp, cybrnook and 5 others like this.
  2. steelskinz

    steelskinz Regular Contributor

    Joined:
    Mar 9, 2015
    Messages:
    114
    Location:
    France
    "/jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
    /jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (38675) and YAMalwareBlockCIDR (865) sets."
    Great ! :) Thanks !
     
    redhat27 likes this.
  3. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    Thank you for trying it out. I am curious... How long did it take for the script to run?
    You can time the run with:
    Code:
    time /jffs/scripts/ya-malware-block.sh
    The real value is the total time taken.
     
  4. steelskinz

    steelskinz Regular Contributor

    Joined:
    Mar 9, 2015
    Messages:
    114
    Location:
    France
    R7000. I only got 3Mbps. Don't know if it slow the "real" as it has to dl the lists i guess.
     
  5. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    I would say it is a good run for 3mbps bandwidth. I get around 13 seconds on mine (rt-ac66r), but with higher bandwidth.
     
  6. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    266
    Hi, great script, took less than a minute in my router, aprox 45 to 50 seconds
     
    redhat27 likes this.
  7. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    It should typically be quicker. what is your cpu load just before you run the script? Your cpu load can be found with the uptime command. If your current cpu load (the first of three numbers) is less than 0.5, I would expect the script to finish quicker
     
  8. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    266
    This is an IP from the list, I cant ping from the router, but I can from my PC, I know that when its on the forward chain it blocks from the PC but the router does let the packets pass, is this the correct behavior?

    Pinging 190.214.66.206 with 32 bytes of data:
    Reply from 190.214.66.206: bytes=32 time=232ms TTL=48
    Reply from 190.214.66.206: bytes=32 time=230ms TTL=48
    Reply from 190.214.66.206: bytes=32 time=239ms TTL=48

    uptime after reboot, run the script and after the execution of the script. I have 3mbs bandwith, took 40 seconds
    Code:
     uptime
     16:30:07 up 6 min,  load average: 0.09, 0.48, 0.27
    
    ./ya-malware-block.sh
    ./ya-malware-block.sh: Adding malware-block rules to firewall...
    ./ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (38618) and YAMalwareBlockCIDR (864) sets.
    
    uptime
     16:31:59 up 8 min,  load average: 0.72, 0.55, 0.31
     
  9. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    Very good catch, thank you! :)
    I have moved the block on the FORWARD chain. We should be more focussed on protecting the LAN from malware (router is *nix OS, and less prone to malware) Although we can have the block on both, but that is a judgement call

    If you are re-downloading the script, either delete the existing rules with
    Code:
    iptables -D INPUT -m set --set YAMalwareBlockCIDR src -j DROP
    iptables -D INPUT -m set --set YAMalwareBlockIP src -j DROP
    
    (note if you have ipset-v6 replace the --set with --match-set)
    
    OR
    simply reboot your router

    before running it again
     
    shooter40sw likes this.
  10. shooter40sw

    shooter40sw Senior Member

    Joined:
    Mar 3, 2013
    Messages:
    266
    Perfect I run the commands again and its on the forward chain and blocking on the LAN side! Thanks,
    Im already using the TOR and country block + this one
     
    redhat27 likes this.
  11. Adamm

    Adamm Very Senior Member

    Joined:
    Mar 26, 2013
    Messages:
    690
    Good implementation in efficiently adding a large amount of rules, much faster than other methods.
     
    redhat27 likes this.
  12. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    2,754
    Location:
    In the heart of Switzerland
    I had to manually create folder and file and populate /jffs/ipset_lists/ya-malware-block.url_list.
    Is that intended?
    Might be confusing for newbies as you do not include this step in the intro post.
    Without the file, Syslog happily reports the grand total of 1 address added:
    Code:
    May  5 07:40:49 Firewall: /jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
    May  5 07:40:50 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (1) and YAMalwareBlockCIDR (1) sets.
    After adding the file I get the expected YAMalwareBlockIP (37949) and YAMalwareBlockCIDR (865).

    And look how fast it loads these rules, amazing!

    Please don't change the ya-malware-block.url_list file name. Else, let me know as soon as you do.
    I will add it to the auto-whitelisting in AB with the next update to make sure these domains are not blocked, as I do with other scripts posted here..
     
    redhat27 likes this.
  13. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    2,754
    Location:
    In the heart of Switzerland
    Just tested with another router. If you create the folder /jffs/ipset_lists manually first, it will download the ya-malware-block.url_list.
    Might want to create the folder in your script or change location to /jffs/ where it can be written directly.
     
  14. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    You are correct, my mistake. I had assumed that folder will exist, as it is also used by some other scripts... I put the fix in
     
  15. steelskinz

    steelskinz Regular Contributor

    Joined:
    Mar 9, 2015
    Messages:
    114
    Location:
    France
    I didn't say it but i had to do the same.

    Can you put revision on file so we know if we are late.. :) Thanks
     
  16. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    2,754
    Location:
    In the heart of Switzerland
    I agree, version number in script helps keeping track of what version is installed @redhat27
    I also put the release date in every script in AB:
    Code:
    appVersion=3.8
    devVersion=1
    releaseDate=201704xx
    Date being YYYYMMDD, referenced in AB UI.
     
  17. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
    Added a version on the top.
     
    thelonelycoder likes this.
  18. skeal

    skeal Senior Member

    Joined:
    Apr 30, 2016
    Messages:
    263
    Location:
    Moose Jaw Saskatchewan Canada
    Hey I'm not finding the saved info on my usb drive from this script it gets lost in a reboot. Did I screw something up here? EXT4 is my usb drive root.
    Code:
    ################################################Customise for local use #############################################
    if [ -d  "/tmp/mnt/"EXT4 ];then
        DIR="/tmp/mnt/"EXT4                # <== USB Location of IPSET save/restore configuration
    else
        DIR="/tmp"                                #         NOTE: /TMP isn't permanent! ;-) but allows testing of save/restore
    fi
    
    HHMMSS="168:00:00"                            # <== Specify retention period to keep Blacklist entries or passed via 'init reset' hh:mm:ss' invocation
                                                #            e.g. 168 hrs = 7 days
    #####################################################################################################################
     
  19. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    469
  20. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    2,754
    Location:
    In the heart of Switzerland
    Make that
    "/tmp/mnt/EXT4"
    for both instances.
     

Share This Page