Yet another malware block script using ipset (v4 and v6)

[redhat27]

I already had wget installed through entware-ng as one of the pre-requisites for running an NTP-daemon on Asuswrt-Merlin, so the output of which wget is, as expected

That makes it simple. Your entware-ng wget is not setup or working properly. You've three options to get it to work. You can try either one.

• Replace the 3 wgets in this script on lines 18, 19, 20 from just wget to /usr/sbin/wget (the easiest solution, but you'd need to keep doing this on every update of the script, if you choose to keep using this)
• Remove entware wget with opkg remove wget and see if NTP-deaemon still works. I believe it should (but I'm not sure), as your existing entware wget has issues.
• Voice your wget issue on the NTP-deaemon support thread or entware-ng thread and/or see if any solution is posted for your wget error.

 

[MarCoMLXXV]

As my memory is slightly affected after thee burnouts, I went for option number two.

• Remove entware wget with opkg remove wget and see if NTP-deaemon still works. I believe it should (but I'm not sure), as your existing entware wget has issues.

Works like a charm and NTPd seems unnaffected, as it synced correctly after rebooting. Thanks for the assistance!

marco@RT-AC68U:/tmp/home/root# which wget
/usr/sbin/wget
marco@RT-AC68U:/tmp/home/root# wget --no-check-certificate -O /jffs/scripts/ya-malware-block.
sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
--2017-05-25 19:03:18--  https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
Resolving raw.githubusercontent.com... 151.101.128.133, 151.101.0.133, 151.101.192.133, ...
Connecting to raw.githubusercontent.com|151.101.128.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3101 (3.0K) [text/plain]
Saving to: '/jffs/scripts/ya-malware-block.sh'

/jffs/scripts/ya-malwar 100%[==============================>]   3.03K  --.-KB/s   in 0.002s

2017-05-25 19:03:19 (1.67 MB/s) - '/jffs/scripts/ya-malware-block.sh' saved [3101/3101]

marco@RT-AC68U:/tmp/home/root# /jffs/scripts/ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (12425) and YAMalwareBlockCIDR (8923) in 10 seconds

Me happy

And then realized that I thought the entware-ng version of wget might be better, so I could keep it up to date...

 

[chibby]

I just updated to 380.66_4 on my AC68-U and the script didn't load any rules on startup:

/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (0) and YAMalwareBlockCIDR (0) in 5 seconds

I tried to call the script manually again - same output.
Everything worked before the update.

Then i read about custom wget versions - i only have the /usr/sbin/wget version installed.
And the /jffs/ipset_lists/ folder is already populated with ya-malware-block.* files.

When i manually deleted these and re-ran the script everything works fine.

/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (9707) and YAMalwareBlockCIDR (8849) in 14 seconds

What could have gone wrong here? (script version 2.1)
Thank you for your great work!!

EDIT:
files before deletion:

-rw-rw-rw-    1 admin    root           445 May 20 11:03 ya-malware-block.url_list
-rw-rw-rw-    1 admin    root           445 May 25 16:35 ya-malware-block.urls
-rw-rw-rw-    1 admin    root           395 May 25 17:50 ya-malware-block.whites

files after deletion and re-execution:

-rw-rw-rw-    1 admin    root           445 May 27 10:40 ya-malware-block.urls
-rw-rw-rw-    1 admin    root           118 May 27 10:40 ya-malware-block.whites

 

[eclp]

After update to 380.66_4 on my RT-AC87U the script works fine.

May 27 11:50:19 Firewall: /jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
May 27 11:50:30 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (9612) and YAMalwareBlockCIDR (8840) in 11 seconds

 

[redhat27]

the script didn't load any rules on startup

I suspect there was a momentary loss of wan connectivity and wget timed out at 5 seconds.

When i manually deleted these and re-ran the script everything works fine.

That is a bit puzzling. Might be coincidental to connectivity. Let me know if that happens again.

 

[chibby]

I suspect there was a momentary loss of wan connectivity and wget timed out at 5 seconds.

seems unlikely - there is a "sleep 600" in front of the call, like suggested by you (which rules out any WAN problems connected to the booting of the router. My internet connection is very stable and fast - i can't imagine it being the problem.

I'll watch it and come back to you. Thank you anyway!

 

[redhat27]

I will put some more activity and timing logging in the next version (when run from the terminal). That may help diagnose the issue where its failing (when it does).

EDIT: Just noticed:

-rw-rw-rw-    1 admin    root           395 May 25 17:50 ya-malware-block.whites

It should be 118 bytes by default, unless you added some entries manually. If that file is not in the proper format, it may cause problems. That may have been the issue.

 

[chibby]

That may have been the issue.

Huh - i completely overlooked that too.. I thought i deleted my whitelist experiments before. That must have been it!

Btw: what is the proper format for whitelist-entries (what is possible here)?

 

[redhat27]

what is the proper format for whitelist-entries (what is possible here)?

Its in post #1.

You should keep the regex entries on the top (private/unroutable over internet) and then append regular IPs down after that. I've added 2 myself, see the last comments on post #217. The reasoning here is that the dots need not be escaped (though you could if you wanted to) as dots will replace dots in the regex.

 

[Sebastien Bougie]

Hello,

I most be missing a little thing. When I'm trying to run

jffs/scripts/ya-malware-block.sh

I'm getting the following error:

wget: invalid option -- 'i'
BusyBox v1.25.1 () multi-call binary.

Usage: wget [-c|--continue] [-s|--spider] [-q|--quiet] [-O|--output-document FILE]
[--header 'header: value'] [-Y|--proxy on/off] [-P DIR]
[-U|--user-agent AGENT] URL...

Retrieve files via HTTP or FTP

-s Spider mode - only check file existence
-c Continue retrieval of aborted transfer
-q Quiet
-P DIR Save to DIR (default .)
-O FILE Save to FILE ('-' for stdout)
-U STR Use STR for User-Agent header
-Y Use proxy ('on' or 'off')

Can someone help

thanks

 

[octopus]

Try: /jffs/scripts/ya-malware-block.sh
Missing "/"

 

[redhat27]

@Sebastien Bougie and @HRearden Please download the latest version of the script. I've put a version 2.2 of the script that has an alternate way of handling -i option in wget.

 

[redhat27]

All, I've uploaded a new version of the script (2.2) a few minutes ago.

Changelog for version 2.2:

• Should now be able to run in an environment where wget is limited from BusyBox, like Tomato by Shibby (Post #204). [Edit: I've switched to using curl now]
• The script will now display what it is doing and how long that took when run in the terminal (not from cron)

This is a sample run on my (slow) router from the terminal with the default blocking (Level1 through Level3)

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)... ~13s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~3s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~2s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (9737) and YAMalwareBlockCIDR (8842) in 19 seconds

And this is with all 4 Levels enabled:

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)... ~24s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~4s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~4s
>>> Adding data and processing rule for YAMalwareBlock3IP... ~2s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~2s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (65535) YAMalwareBlock3IP (30362) and YAMalwareBlockCIDR (11654) in 36 seconds

 

[Sebastien Bougie]

@Sebastien Bougie and @HRearden Please download the latest version of the script. I've put a version 2.2 of the script that has an alternate way of handling -i option in wget.

this is what i'm getting with the new script

/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...w get: not an http or ftp url: https://raw.githubusercontent.com/shounak-de/misc-s cripts/master/telemetry_and_scanners.txt
wget: not an http or ftp url: https://raw.githubusercontent.com/firehol/blocklis t-ipsets/master/firehol_level1.netset
wget: not an http or ftp url: https://raw.githubusercontent.com/firehol/blocklis t-ipsets/master/firehol_level2.netset
wget: not an http or ftp url: https://raw.githubusercontent.com/firehol/blocklis t-ipsets/master/firehol_level3.netset
wget: not an http or ftp url: #https://raw.githubusercontent.com/firehol/blockli st-ipsets/master/firehol_level4.netset
~0s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~1s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~0s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (0) and YAMalwa reBlockCIDR (0) in 1 seconds

 

[redhat27]

@Sebastien Bougie can you check if the --no-check-certificate flag works with your wget? How did you download the script from github? Copy/paste or wget?

 

[Sebastien Bougie]

@Sebastien Bougie can you check if the --no-check-certificate flag works with your wget? How did you download the script from github? Copy/paste or wget?

Download with WGET

the no --no-check-certificate work no problem

 

[redhat27]

I made a small change. Can you re-download the script and let me know if that worked?

 

[Sebastien Bougie]

its worst i'm getting alot of error

No I get doesnt recognize the no certificat

 

[redhat27]

Does the --no-check-certificate flag work or not? I did not understand.

Can you paste this exact command and let me know the output?

wget --no-check-certificate -O /tmp/ya-malware-block.sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh

 

[Sebastien Bougie]

Does the --no-check-certificate flag work or not? I did not understand.

Can you paste this exact command and let me know the output?

wget --no-check-certificate -O /tmp/ya-malware-block.sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh

this what i get

/jffs/scripts$ wget --no-check-certificate -O /tmp/ya-malware-block.sh https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/root/.wget-hsts'. HSTS will be disabled.
--2017-05-29 14:52:40-- https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.sh
Resolving raw.githubusercontent.com... 151.101.64.133, 151.101.128.133, 151.101.192.133, ...
Connecting to raw.githubusercontent.com|151.101.64.133|:443... connected.
WARNING: cannot verify raw.githubusercontent.com's certificate, issued by 'CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US':
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 3762 (3.7K) [text/plain]
Saving to: '/tmp/ya-malware-block.sh'

0K ... 100% 7.26M=0s

2017-05-29 14:52:40 (7.26 MB/s) - '/tmp/ya-malware-block.sh' saved [3762/3762]