Yet another malware block script using ipset (v4 and v6)

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

redhat27

Very Senior Member
As if we don't have enough already :p

Here is yet another malware blocking script that uses ipset (v4 and v6). So why put up another one?
  • Over 650 Million unique IPs blocked
  • Simpler (minimalist script with only 26 lines (which includes a couple of blank lines and comment lines)
  • Quicker (about 20 secs typical runtime with default lists depending on CPU load, processing power and bandwidth)
  • Less CPU cycles for your router to load the lists (kinda saying the same thing as above)
  • Entware not required (Can run without extended packages or external storage)
  • Only one single preprocess file created in /tmp (that is removed right afterward)
  • Also supports both ipset v4 and ipset v6
  • Overcomes ipset-v4 limitation to handle more than 64k elements by splitting into multiple sets.
  • Can also be run in cru
  • You can manually whitelist specific IPs
  • Starting with version 2.3, you can manually blacklist discrete IPs and/or CIDR ranges in addition to the lists defined in the urls file.
  • Blocks on the PREROUTING chain of the iptables raw table (affects both traffic initiated by malware sources and also traffic initiated inside your LAN trying to connect to malware destinations)
  • Works on other firmwares (Tomato by Shibby, for example). Script here.
I am not asking anyone who is using an existing script that they are happy with to switch. I'd hate to get into an argument as to which script is better and why. But in case anyone is willing to try this, that is the purpose of this thread.

The sources lists are mainly from the Cybercrime IP Feeds by FireHOL site:
As per the site, there are 4 blocking tiers:

Level1 (around 652 Million unique IPs): A firewall blacklist composed from IP lists, providing maximum protection with minimum false positives. Suitable for basic protection on all internet facing servers, routers, and firewalls. (includes: bambenek_c2 dshield feodo fullbogons spamhaus_drop spamhaus_edrop sslbl zeus_badips ransomware_rw)

Level2 (around 47 Thousand unique IPs): An ipset made from blocklists that track attacks, during about the last 48 hours. (includes: blocklist_de dshield_1d greensnow)

Level3 (around 119 Thousand unique IPs): An ipset made from blocklists that track attacks, spyware, viruses. It includes IPs that have been reported or detected in the last 30 days. (includes: bruteforceblocker ciarmy dragon_http dragon_sshpauth dragon_vncprobe dshield_30d dshield_top_1000 malc0de maxmind_proxy_fraud myip shunlist snort_ipfilter sslbl_aggressive talosintel_ipfilter zeus vxvault)

Level4 (around 9.4 Million unique IPs): An ipset made from blocklists that track attacks, but may include a large number of false positives. (includes: cleanmx_viruses blocklist_net_ua botscout_30d cruzit_web_attacks cybercrime haley_ssh iblocklist_hijacked iblocklist_spyware iblocklist_webexploit ipblacklistcloud_top iw_wormlist malwaredomainlist)

Level1 through Level3 is enabled on this script by default.

Note on Level4: Because of a large number of false positives in the Level4 ipset, I've disabled it in the sources list. After you install the script, you can, (if you so want) edit the sources list /jffs/ipset_lists/ya-malware-block.urls (default location) to remove the comment '#' on the level4 source. But you may need to whitelist if you discover some of your desired domains are blocked.


Install/Reinstall:
Code:
wget --no-check-certificate -O /jffs/scripts/ya-malware-block.sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
chmod +x /jffs/scripts/ya-malware-block.sh

Running:
Run via cru or on the command line as many times as you want:
cru (example run every 6 hours):
Code:
cru a UpdateYAMalwareBlock "0 */6 * * * /jffs/scripts/ya-malware-block.sh"
Put the above line in your existing /jffs/scripts/init-start to create a cron schedule at boot time.
To schedule the job right away (without waiting for the next reboot of your router), just run the above command at the shell prompt. But please note that if you run the cru command from the command line, the schedule will only remain in effect till the next router boot unless also put in init-start

In addition, you can also run the script once at router boot:
append the following 2 lines at end of your existing /jffs/scripts/services-start:
Code:
sleep 600
sh /jffs/scripts/ya-malware-block.sh
Doing so will ensure that the script is run also at boot time so that you are covered in the interval your router has just rebooted but your scheduled run via cron has not occurred yet.

You can also just run it at any time from your router prompt:
Code:
/jffs/scripts/ya-malware-block.sh
This is a sample run on my (slow) router from the terminal with the default blocking (Level1 through Level3)
Code:
[email protected]:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[131428/122042/9386] ~11s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~4s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~4s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507) and YAMalwareBlockCIDR (9386) in 20 seconds
And this is with all 4 Levels enabled:
Code:
[email protected]:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[250472/238198/12274] ~23s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock3IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock4IP... ~3s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~2s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (65535) YAMalwareBlock3IP (65535) YAMalwareBlock4IP (41593) and YAMalwareBlockCIDR (12274) in 43 seconds

Whitelisting [manual]:
This script does have provision for whitelisting discrete IPs that have been blocked on the YAMalwareBlock?IP sets. Although you'd typically not want to whitelist identified malware, sometimes you may find that you need to, especially if you enable FireHOL Level4.
You can add any IP you would want to whitelist to the file: /jffs/ipset_lists/ya-malware-block.whites (default location)

You should only append to this file, and only add IPs (no other characters or comments).

Notes on whitelisting:
You should keep the regex entries on the top (private/unroutable over the internet) and then append regular IPs down after that.

The regex entries ensure that your LAN IPs are never blocked. You do not need to whitelist your LAN IPs.

I've added 3 additional myself (as an example: You may remove these if you want, but be aware of the below, see why I added those):
  • A regex ^216\.239\.3[2468]\.21 for Nintendo users, as per @VZ3 's post
  • AB solution uses a hosts list provider from pgl.yoyo.org. That was being blocked on the default Level1 through Level3 blocking of FireHOL tiers. That 213.230.210.230 IP is for pgl.yoyo.org
  • I use all 4 Levels and noticed that androidfilehost.com (needed by most files hosted in xda-developers.com) was blocked on Level4. That 192.124.249.10 IP is for androidfilehost.com

The reasoning here of adding IPs without any regex is that the dots need not be escaped (though you could if you wanted to) as dots will replace dots in the regex.


Blacklisting [manual|optional]
:
This script also has provision for blacklisting discrete IPs and/or CIDR ranges in addition to the blocking provided by the lists defined by the urls file.

You can add any IP/CIDR entries you would want to blacklist to the file: /jffs/ipset_lists/ya-malware-block.blacks (default location)

Unlike the whitelists file, this file is totally optional, and the script will run fine without it.
The custom blacklist supports line comments, but not in-line comments. Sample here. Data in that sample file are just test data demonstrating usage/syntax. Please don't use that file as is!


Uninstall :(:
Code:
rm -f /jffs/ipset_lists/ya-malware-block.*
rm -f /jffs/scripts/ya-malware-block.sh

In addition,
Undo your changes to /jffs/scripts/init-start and /jffs/scripts/services-start
Reboot (optional)
Enjoy!
 
Last edited:

steelskinz

Regular Contributor
"/jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
/jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (38675) and YAMalwareBlockCIDR (865) sets."
Great ! :) Thanks !
 

redhat27

Very Senior Member
Thank you for trying it out. I am curious... How long did it take for the script to run?
You can time the run with:
Code:
time /jffs/scripts/ya-malware-block.sh
The real value is the total time taken.
 

steelskinz

Regular Contributor
/jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
/jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (3856 5) and YAMalwareBlockCIDR (865) sets.
real 0m 17.41s
user 0m 4.63s
sys 0m 1.54s
R7000. I only got 3Mbps. Don't know if it slow the "real" as it has to dl the lists i guess.
 

redhat27

Very Senior Member
I would say it is a good run for 3mbps bandwidth. I get around 13 seconds on mine (rt-ac66r), but with higher bandwidth.
 

redhat27

Very Senior Member
It should typically be quicker. what is your cpu load just before you run the script? Your cpu load can be found with the uptime command. If your current cpu load (the first of three numbers) is less than 0.5, I would expect the script to finish quicker
 

shooter40sw

Senior Member
It should typically be quicker. what is your cpu load just before you run the script? Your cpu load can be found with the uptime command. If your current cpu load (the first of three numbers) is less than 0.5, I would expect the script to finish quicker
This is an IP from the list, I cant ping from the router, but I can from my PC, I know that when its on the forward chain it blocks from the PC but the router does let the packets pass, is this the correct behavior?

Pinging 190.214.66.206 with 32 bytes of data:
Reply from 190.214.66.206: bytes=32 time=232ms TTL=48
Reply from 190.214.66.206: bytes=32 time=230ms TTL=48
Reply from 190.214.66.206: bytes=32 time=239ms TTL=48

uptime after reboot, run the script and after the execution of the script. I have 3mbs bandwith, took 40 seconds
Code:
 uptime
 16:30:07 up 6 min,  load average: 0.09, 0.48, 0.27

./ya-malware-block.sh
./ya-malware-block.sh: Adding malware-block rules to firewall...
./ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (38618) and YAMalwareBlockCIDR (864) sets.

uptime
 16:31:59 up 8 min,  load average: 0.72, 0.55, 0.31
 

redhat27

Very Senior Member
Very good catch, thank you! :)
I have moved the block on the FORWARD chain. We should be more focussed on protecting the LAN from malware (router is *nix OS, and less prone to malware) Although we can have the block on both, but that is a judgement call

If you are re-downloading the script, either delete the existing rules with
Code:
iptables -D INPUT -m set --set YAMalwareBlockCIDR src -j DROP
iptables -D INPUT -m set --set YAMalwareBlockIP src -j DROP

(note if you have ipset-v6 replace the --set with --match-set)
OR
simply reboot your router

before running it again
 

shooter40sw

Senior Member
Very good catch, thank you! :)
I have moved the block on the FORWARD chain. We should be more focussed on protecting the LAN from malware (router is *nix OS, and less prone to malware) Although we can have the block on both, but that is a judgement call

If you are re-downloading the script, either delete the existing rules with
Code:
iptables -D INPUT -m set --set YAMalwareBlockCIDR src -j DROP
iptables -D INPUT -m set --set YAMalwareBlockIP src -j DROP

(note if you have ipset-v6 replace the --set with --match-set)
OR
simply reboot your router

before running it again
Perfect I run the commands again and its on the forward chain and blocking on the LAN side! Thanks,
Im already using the TOR and country block + this one
 

Adamm

Part of the Furniture
Good implementation in efficiently adding a large amount of rules, much faster than other methods.
 

thelonelycoder

Part of the Furniture
I had to manually create folder and file and populate /jffs/ipset_lists/ya-malware-block.url_list.
Is that intended?
Might be confusing for newbies as you do not include this step in the intro post.
Without the file, Syslog happily reports the grand total of 1 address added:
Code:
May  5 07:40:49 Firewall: /jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
May  5 07:40:50 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (1) and YAMalwareBlockCIDR (1) sets.
After adding the file I get the expected YAMalwareBlockIP (37949) and YAMalwareBlockCIDR (865).

And look how fast it loads these rules, amazing!

Please don't change the ya-malware-block.url_list file name. Else, let me know as soon as you do.
I will add it to the auto-whitelisting in AB with the next update to make sure these domains are not blocked, as I do with other scripts posted here..
 

thelonelycoder

Part of the Furniture
I had to manually create folder and file and populate /jffs/ipset_lists/ya-malware-block.url_list.
Is that intended?
Might be confusing for newbies as you do not include this step in the intro post.
Without the file, Syslog happily reports the grand total of 1 address added:
Code:
May  5 07:40:49 Firewall: /jffs/scripts/ya-malware-block.sh: Adding malware-block rules to firewall...
May  5 07:40:50 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded ip addresses to YAMalwareBlockIP (1) and YAMalwareBlockCIDR (1) sets.
After adding the file I get the expected YAMalwareBlockIP (37949) and YAMalwareBlockCIDR (865).

And look how fast it loads these rules, amazing!

Please don't change the ya-malware-block.url_list file name. Else, let me know as soon as you do.
I will add it to the auto-whitelisting in AB with the next update to make sure these domains are not blocked, as I do with other scripts posted here..
Just tested with another router. If you create the folder /jffs/ipset_lists manually first, it will download the ya-malware-block.url_list.
Might want to create the folder in your script or change location to /jffs/ where it can be written directly.
 

redhat27

Very Senior Member
You are correct, my mistake. I had assumed that folder will exist, as it is also used by some other scripts... I put the fix in
 

thelonelycoder

Part of the Furniture
Can you put revision on file so we know if we are late.. :) Thanks
I agree, version number in script helps keeping track of what version is installed @redhat27
I also put the release date in every script in AB:
Code:
appVersion=3.8
devVersion=1
releaseDate=201704xx
Date being YYYYMMDD, referenced in AB UI.
 

skeal

Part of the Furniture
Hey I'm not finding the saved info on my usb drive from this script it gets lost in a reboot. Did I screw something up here? EXT4 is my usb drive root.
Code:
################################################Customise for local use #############################################
if [ -d  "/tmp/mnt/"EXT4 ];then
    DIR="/tmp/mnt/"EXT4                # <== USB Location of IPSET save/restore configuration
else
    DIR="/tmp"                                #         NOTE: /TMP isn't permanent! ;-) but allows testing of save/restore
fi

HHMMSS="168:00:00"                            # <== Specify retention period to keep Blacklist entries or passed via 'init reset' hh:mm:ss' invocation
                                            #            e.g. 168 hrs = 7 days
#####################################################################################################################
 

thelonelycoder

Part of the Furniture
Hey I'm not finding the saved info on my usb drive from this script it gets lost in a reboot. Did I screw something up here? EXT4 is my usb drive root.
Code:
################################################Customise for local use #############################################
if [ -d  "/tmp/mnt/"EXT4 ];then
    DIR="/tmp/mnt/"EXT4                # <== USB Location of IPSET save/restore configuration
else
    DIR="/tmp"                                #         NOTE: /TMP isn't permanent! ;-) but allows testing of save/restore
fi

HHMMSS="168:00:00"                            # <== Specify retention period to keep Blacklist entries or passed via 'init reset' hh:mm:ss' invocation
                                            #            e.g. 168 hrs = 7 days
#####################################################################################################################
Make that
"/tmp/mnt/EXT4"
for both instances.
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top