How to Dynamically Ban Malicious IP's using IPSet (Martineau version)

[Csection]

I suspect the fact that you have to 'try and fix the code' due to the download being 'corrupt' is the root cause of your current issues.

I have just created a test6.sh from the script hosted here

https://pastebin.com/zQ3KEe8P

and it runs fine

./test6.sh init nolog

(test6.sh): 21788 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
(test6.sh): 21788 IPSET restore from '/tmp/mnt/RT-AC68U/IPSET_Block.config' starting.....
(test6.sh): 21788 Dynamic IPSET Blacklist banning enabled.
(test6.sh): 21788 Hacker Port Activity report scheduled every 06:05 daily

 Summary Blacklist: 0 Successful blocks! ( 3427 IPs currently banned - 0 added since: Apr 25 19:19 ), Entries auto-expire after 168:00:00 hrs

 ./test6.sh

(test6.sh): 21957 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....

 Summary Blacklist: 0 Successful blocks! ( 3427 IPs currently banned - 0 added since: Apr 25 19:22 ), Entries auto-expire after 168:00:00 hrs

Please try creating the script from pastebin rather than the GitHub version.

EDIT: Just created a test0.sh script from the GitHub version and that too runs correctly.

The problem I have with "Paste-bin" is that when I highlight it in Win 10 and copy it to a file in /jffs/scripts, it copies the line numbers and code.
I used the version that "SweeToast" put in the wiki. That does not copy the line numbers.
By the way! I put it into firewall-start and services-start as he suggests and it seems to run ok. The last time it ran at 5pm. It listed a lot of "Banned ip's" in my syslog and the current ban said, "Blocked 210" new ip's.
So!
Maybe it's working ok now!
How can I get rid of those "Line-Numbers" from "Paste-Bin"???

 

[Martineau]

The problem I have with "Paste-bin" is that when I highlight it in Win 10 and copy it to a file in /jffs/scripts, it copies the line numbers and code.

How can I get rid of those "Line-Numbers" from "Paste-Bin"???

On the pastebin page , scroll down to the bottom of the page and you will see the separate scrollable code box marked:

RAW Paste Data

Copy'n'paste the code from this box rather than the nicely coloured/annotated code!

You may run the script at the command prompt without args (see the help) at any time to get a real-time status or even schedule a cru/cron job to have a status report every hour.
NOTE: If you have a USB flash drive attached, it is advised that you carefully edit one line in the script to set the $DIR variable from '/tmp' to a persistent storage/drive location

################################################Customise for local use #############################################
if [ -d  "/tmp/mnt/"$MYROUTER ];then
 DIR="/tmp/mnt/"$MYROUTER    # <== USB Location of IPSET save/restore configuration
else
 DIR="/tmp"        #         NOTE: /TMP isn't permanent! ;-) but allows testing of save/restore
fi

This allows the script to retrieve a saved copy of the banned IPs when the router reboots, otherwise the script will start with an empty IPSET.

 

[octopus]

Why don't you use "download" button on Pastebin? Then you get a .sh file

 

[Xentrk]

...
I do have a problem that when I copy the script from git, it does not copy correctly(Character overrun, lines too long.). I have to go through and try and fix the code.
I'll try the code you suggest and see if it makes a difference.

Ouch!?

Try this handy script to download the script to your /jffs/scripts directory in the file IPSET_Block.sh. Call it whatever you want (e.g. get-ipset-block-script). Then chmod 755 on the file before running:

#!/bin/sh
#set -xv
#location and name of file that will get updated.
IPSET_Block_script=/jffs/scripts/IPSET_Block.sh
#source code on git-hub
site=https://raw.githubusercontent.com/MartineauUK/IPSET_Block/master/IPSET_Block.sh

get_source () {
wget $site -O $IPSET_Block_script
}

Or, run wget from the command line as follows:

wget https://raw.githubusercontent.com/MartineauUK/IPSET_Block/master/IPSET_Block.sh -O /jffs/scripts/IPSET_Block.sh

Or, click on the Raw button at the top before copy/paste to the file to avoid getting special characters. See below

 

[Xentrk]

 

[Csection]

On the pastebin page , scroll down to the bottom of the page and you will see the separate scrollable code box marked:

RAW Paste Data

Copy'n'paste the code from this box rather than the nicely coloured/annotated code!

You may run the script at the command prompt without args (see the help) at any time to get a real-time status or even schedule a cru/cron job to have a status report every hour.
NOTE: If you have a USB flash drive attached, it is advised that you carefully edit one line in the script to set the $DIR variable from '/tmp' to a persistent storage/drive location

################################################Customise for local use #############################################
if [ -d  "/tmp/mnt/"$MYROUTER ];then
 DIR="/tmp/mnt/"$MYROUTER    # <== USB Location of IPSET save/restore configuration
else
 DIR="/tmp"        #         NOTE: /TMP isn't permanent! ;-) but allows testing of save/restore
fi

This allows the script to retrieve a saved copy of the banned IPs when the router reboots, otherwise the script will start with an empty IPSET.

Thank you for this!
It helps a lot for future use!
I don't exactly understand the $dir reference. Do I set it to a USB drive to save it?
I only have one USB drive connected to my router and that has Joe's "Nvram-Save" on it. I hate to use that, but if it helps, I guess it would be ok!

 

[Csection]

View attachment 9117

This helps a lot! Thanks!
I didn't see that on the site cause I use larger text setting(Bad eyesight!).

 

[Xentrk]

This helps a lot! Thanks!
I didn't see that on the site cause I use larger text setting(Bad eyesight!).

You're welcome. Using the curl command is another option. Navigate to /jffs/scripts, then type:

curl -O https://raw.githubusercontent.com/MartineauUK/IPSET_Block/master/IPSET_Block.sh

You should see the file in the /jffs/scripts folder.

There is another way but these methods will make life easier for you. No more copy/paste. I'm writing scripts to detect changes for scripts I use the most on github and if there are updates, download it for me.

 

[Csection]

You're welcome. Using the curl command is another option. Navigate to /jffs/scripts, then type:

curl -O https://raw.githubusercontent.com/MartineauUK/IPSET_Block/master/IPSET_Block.sh

You should see the file in the /jffs/scripts folder.

There is another way but these methods will make life easier for you. No more copy/paste. I'm writing scripts to detect changes for scripts I use the most on github and if there are updates, download it for me.

I haven't used those commands in a long time! I am refreshing myself on them from time to time.
Thanks, again!

 

[Martineau]

I don't exactly understand the $dir reference. Do I set it to a USB drive to save it?
I only have one USB drive connected to my router and that has Joe's "Nvram-Save" on it. I hate to use that, but if it helps, I guess it would be ok!

If you mean @john9527's nvram-save utility, then either just set the variable to the same USB drive

e.g. if the USB drive is labelled as 'xxxxx' (or mounted as 'sd??') then simply edit the line

DIR='/mnt/xxxxx'
or
DIR='/mnt/sd??'

or create a new directory on the USB say IPSET and set the DIR variable accordingly.

mkdir /mnt/xxxxx/IPSET

 

[Csection]

If you mean @john9527's nvram-save utility, then either just set the variable to the same USB drive

e.g. if the USB drive is labelled as 'xxxxx', then simply edit the line

DIR='/mnt/xxxxx'

or create a new directory on the USB say IPSET and set the DIR variable accordingly.

mkdir /mnt/xxxxx/IPSET

Thanks so much!
I thought that was what you meant, but wanted to be sure. I don't want anything to happen to the "John9527" USB drive.
Sorry for the mis-statement on the name! I hope John don't get pissed off and ban me from his firmware(John! I've got a nasty headache today. Sorry!).

 

[Xentrk]

Also, I don't remove the DROP messages from Syslog, because I have a supplementary script that reports on the Blacklist entries:

./HackerPorts.sh

(HackerPorts.sh): 8647 Syslog Hacker report starting.....
(HackerPorts.sh): 8647 Hacker report created '/tmp/mnt/RT-AC68U/HackerReport.txt' (Total Ports attacked: 218)

Spoiler: Blacklist Hacker Report

Top 10 Ports attacked:
2560 http://www.speedguide.net/port.php?port=23
460 http://www.speedguide.net/port.php?port=22
427 http://www.speedguide.net/port.php?port=5358
279 http://www.speedguide.net/port.php?port=1433
179 http://www.speedguide.net/port.php?port=2323
62 http://www.speedguide.net/port.php?port=3389
39 http://www.speedguide.net/port.php?port=2222
34 http://www.speedguide.net/port.php?port=80
33 http://www.speedguide.net/port.php?port=445
17 http://www.speedguide.net/port.php?port=3306

Last 10 attackers:
1 95.9.87.17 1433
1 94.233.198.64 23
1 94.102.49.190 27017
1 93.80.238.198 23
1 91.236.144.4 1433
1 91.197.235.10 3389
1 91.197.234.3 23500
1 91.187.14.11 23
1 91.147.23.18 23
1 90.178.194.34 23

@Martineau, Can you share the report you wrote with us? Much appreciated.

 

[swetoast]

can someone that has a github and is using this script update some info on the wiki for this script

https://github.com/RMerl/asuswrt-me...on-instructions#dynamically-ban-malicious-ips

 

[Martineau]

@Martineau, Can you share the report you wrote with us? Much appreciated.

It will have to wait until I am back and have access to the source.

P.S. It's only a bit of 'grep/sed/awk/sort' - I'm sure you could code the script.

 

[bayern1975]

i think this is not working as well? why is ther zero blocked?

Apr 30 14:00:00 crond[519]: USER admin pid 6733 cmd /jffs/scripts/IPSET_Block.sh save
Apr 30 14:00:00 (IPSET_Block.sh): 6734 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
Apr 30 14:00:00 (IPSET_Block.sh): 6734 Saving IPSET Block rules to /tmp/IPSET_Block.config.....
Apr 30 14:00:01 (IPSET_Block.sh): 6734 Summary Blacklist: 57 0 Successful blocks! ( 192 IPs currently banned - 7 added since: Apr 30 13:50 )
Apr 30 15:00:00 crond[519]: USER admin pid 7558 cmd /jffs/scripts/IPSET_Block.sh save
Apr 30 15:00:00 (IPSET_Block.sh): 7559 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
Apr 30 15:00:00 (IPSET_Block.sh): 7559 Saving IPSET Block rules to /tmp/IPSET_Block.config.....
Apr 30 15:00:01 (IPSET_Block.sh): 7559 Summary Blacklist: 175 0 Successful blocks! ( 199 IPs currently banned - 61 added since: Apr 30 14:00 )

 

[Martineau]

i think this is not working as well? why is ther zero blocked?

Apr 30 14:00:00 crond[519]: USER admin pid 6733 cmd /jffs/scripts/IPSET_Block.sh save
Apr 30 14:00:00 (IPSET_Block.sh): 6734 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
Apr 30 14:00:00 (IPSET_Block.sh): 6734 Saving IPSET Block rules to /tmp/IPSET_Block.config.....
Apr 30 14:00:01 (IPSET_Block.sh): 6734 Summary Blacklist: 57 0 Successful blocks! ( 192 IPs currently banned - 7 added since: Apr 30 13:50 )
Apr 30 15:00:00 crond[519]: USER admin pid 7558 cmd /jffs/scripts/IPSET_Block.sh save
Apr 30 15:00:00 (IPSET_Block.sh): 7559 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
Apr 30 15:00:00 (IPSET_Block.sh): 7559 Saving IPSET Block rules to /tmp/IPSET_Block.config.....
Apr 30 15:00:01 (IPSET_Block.sh): 7559 Summary Blacklist: 175 0 Successful blocks! ( 199 IPs currently banned - 61 added since: Apr 30 14:00 )

Only v4.01 (un-released) explicitly uses 2 figures (separated by the '+' character) for the $HITS variable reporting.

You have already posted that the firewall scripts are firing twice, as is normal since firewall-start and nat-start can and will fire multiple times even for environments where Dual-Wan and VPN Client connections are NOT configured.

So unfortunately you have multiple IPSET_Block firewall rules

i.e. If you issue the following commands you will see that two values are returned for the $HITS variable.

iptables --line -nvL INPUT | grep -E "set.*Blacklist"
iptables --line -nvL INPUT | grep -E "set.*Blacklist" | awk '{print $2}'

So since the first rule DROPs the Hacker Port attempts '175 Sucessful Hits!', the second rule never fires - hence the additional '0' being spuriously reported.

 

[Martineau]

It will have to wait until I am back and have access to the source.

IPSET_Block.sh Hacker report script:

/jffs/scripts/HackerPorts.sh

EDIT: 07/05/2017 The (too large to post in-line) current version (v2.03) is hosted here:

https://pastebin.com/LRHJih8Z

Spoiler: v1.01 deprecated

#!/bin/sh
VER="v1.01"
#======================================================================================================= © 2016-2017 Martineau, v1.01
#
# Scan Syslog and report on Hacker attempts to attack local LAN ports. (Usually scheduled 05:00 daily by '/jffs/scripts/IPSET_Block.sh')
#      (If 'nolog' arg is used for v3.0x '/jffs/scripts/IPSET_Block.sh' then this report will NOT be able to report on the attack instance messages!)
#
#      The console display report is also created to disk and allows double-clicking on the URL to help identify the port being attacked and its attacker.
#            (v4.0x of '/jffs/scripts/IPSET_Block.sh' uses an IPSET as a crude database for the attacks so this script is no longer used.)
#
#     HackerPorts   [help | -h] | [file_name_for report] [verbose]
#
#     HackerPorts
#                   Will produce a summary display of the top 10 in three categories:
# 
#                   e.g. Thu Mar 9 13:31:11 DST 2017 (Ports attacked Total=324)
#
#                              Top 10 Ports attacked:
#                          4227 http://www.speedguide.net/port.php?port=23    from  https://dnsquery.org/ipwhois/1.10.130.6
#                              <...>
#                              Top 10 attackers:
#                            3 https://dnsquery.org/ipwhois/52.174.156.242
#                              <...>
#                              Last 10 most recent attackers:
#                              https://dnsquery.org/ipwhois/146.185.239.117
#                              <...>
#
#     HackerPorts   verbose
#                   Will produce a summary display as above, but will also list ALL of the attacked ports and by whom! i.e. 324 in this example!!
#
# https://dnsquery.org/ipwhois/ is FREE but not 100% accurate?, whereas https://www.whoisxmlapi.com/ is accurate but ONLY 20 FREE IP lookups per Guest

# Print between line beginning with '#==' to first blank line inclusive
ShowHelp() {
 awk '/^#==/{f=1} f{print; if (!NF) exit}' $0
}
# Function Parse(String delimiter(s) variable_names)
Parse() {
 #
 #  Parse  "Word1,Word2|Word3" ",|" VAR1 VAR2 REST
 #    (Effectivley executes VAR1="Word1";VAR2="Word2";REST="Word3")
 
 local string IFS
 
 TEXT="$1"
 IFS="$2"
 shift 2
 read -r -- "$@" <<EOF
$TEXT
EOF
}

#==============================================Main=============================================

# Need assistance!???
if [ "$1" == "help" ] || [ "$1" == "-h" ]; then
 ShowHelp
 exit 0
fi

MYROUTER=$(nvram get computer_name)

if [ -d "/tmp/mnt/"$MYROUTER ];then
   MOUNT="/tmp/mnt/"$MYROUTER
else
   MOUNT="/tmp"
fi

if [ ! -z $1 ] && [ "$1" != "verbose" ];then
   LOGFILE=$1
else
   LOGFILE=${MOUNT}/HackerReport.txt
fi

rm $LOGFILE".tmp" 2> /dev/null
rm $LOGFILE".new" 2> /dev/null

logger -st "($(basename $0))" $$ $VER "Hacker Port attacks Syslog Report starting....."

VERBOSE=0
if [ "$( echo $@ | grep -o "verbose" | wc -w )" -eq 1 ];then
   VERBOSE=1         # List ALL report lines rather than just the summary
fi

# Extract the relevant Hacker attempt msgs (created by /jffs/scripts/IPSET_Block.sh etc.) from Syslog
grep -E "DROP IN=|Block IN=$(nvram get wan0_ifname)" /tmp/syslog.log | grep -oE "SRC.*DPT=.*\SEQ" \
  | awk '{ print $1" " $(NF-1)}' | sort -t " " -nk 2.5n | uniq -f 1 -c \
  | sed -e 's/^[ \t]*//' | while read ITEM
 do
    Parse "$ITEM" " =" PORTCNT v2 SRC v4 PORT
    echo -e "$(printf "%5d %-45s from %s" $PORTCNT "http://www.speedguide.net/port.php?port="$PORT )" "https://dnsquery.org/ipwhois/"$SRC >> $LOGFILE.tmp
 done

TOTAL=$(wc -l ${LOGFILE}.tmp | cut -d" " -f1)

TIMESTAMP=$(date)

echo -e "\n\n"$TIMESTAMP "(Ports attacked Total="$TOTAL")" >> $LOGFILE.new # New period's report Header
logger -t "($(basename $0))" $$ "Hacker report created '"$LOGFILE"' (Total Ports attacked:" $TOTAL")" $TIMESTAMP

echo -e "\n\tTop 10 Ports attacked:" >> $LOGFILE.new
head $LOGFILE".tmp" | sort -nr       >> $LOGFILE.new

echo -e "\n\tTop 10 attackers:"      >> $LOGFILE.new
cat $LOGFILE".tmp" | uniq -f 3 -c | sort -nr | head | awk '{printf "%5d %s\n", $1, $5}' >> $LOGFILE.new

echo -e "\n\tLast 10 most recent attackers:" >> $LOGFILE.new
tail -n 10 $LOGFILE".tmp" | awk '{print "      "$4}'  >> $LOGFILE.new  # ...in chronological order last is 'most recent'

cat $LOGFILE".new" >> $LOGFILE       # Update master report with this period's attack summary

echo -e "\n\tPorts attacked:" >> $LOGFILE
cat $LOGFILE".tmp" >> $LOGFILE       # Update master report with this period's attack details

# Show report just created ...either in FULL or just the summary.
if [ "$VERBOSE" == "1" ];then        # Just the summary report or ALL report details?
 awk -v pattern="${TIMESTAMP}" ' $0 ~ pattern { matched = 1 }; matched { print }' "$LOGFILE"  # Display ALL report lines
else
 cat $LOGFILE.new          # Only display the new period's summary report
fi

echo -e

rm $LOGFILE".new" 2> /dev/null
rm $LOGFILE".tmp" 2> /dev/null

exit 0

Spoiler: Example v2.03 Hacker Report.txt created from Syslog messages rather than from BlacklistTRK IPSET

 ./HackerPorts.sh

(HackerPorts.sh): 10260 v2.03 Hacker Port attacks Report.....

Scanning /tmp/syslog.log for 'eth0' violations, please wait.....

4726 records scanned from Syslog ('/tmp/syslog.log')

07 May 15:37:05: # Unique Ports attacked via 'eth0': 175 (out of 2188 attempts) tracked via SYSLOG, May 5 20:39:18 - May 7 15:37:02
 ***ATTENTION 5 GRE (Generic Route Encapsulation) aka Virtual Private Network (VPN) attacks.

 Top 10 Ports attacked:
 1136 http://www.speedguide.net/port.php?port=23    e.g.  https://dnsquery.org/ipwhois/1.164.18.251
  233 http://www.speedguide.net/port.php?port=1433  e.g.  https://dnsquery.org/ipwhois/1.30.218.43
  230 http://www.speedguide.net/port.php?port=22    e.g.  https://dnsquery.org/ipwhois/1.52.153.173
   62 http://www.speedguide.net/port.php?port=2323  e.g.  https://dnsquery.org/ipwhois/103.207.45.166
   39 http://www.speedguide.net/port.php?port=1900  e.g.  https://dnsquery.org/ipwhois/1.223.243.124
   35 http://www.speedguide.net/port.php?port=5060  e.g.  https://dnsquery.org/ipwhois/104.243.40.162
   34 http://www.speedguide.net/port.php?port=35092 e.g.  https://dnsquery.org/ipwhois/107.158.43.150
   27 http://www.speedguide.net/port.php?port=5358  e.g.  https://dnsquery.org/ipwhois/114.43.81.103
   24 http://www.speedguide.net/port.php?port=3389  e.g.  https://dnsquery.org/ipwhois/101.167.216.61
   22 http://www.speedguide.net/port.php?port=2222  e.g.  https://dnsquery.org/ipwhois/111.200.228.50

 Top 10 attackers:
    1 https://dnsquery.org/ipwhois/1.164.18.251
    1 https://dnsquery.org/ipwhois/1.30.218.43
    1 https://dnsquery.org/ipwhois/1.52.153.173
    1 https://dnsquery.org/ipwhois/103.207.45.166
    1 https://dnsquery.org/ipwhois/1.223.243.124
    1 https://dnsquery.org/ipwhois/104.243.40.162
    1 https://dnsquery.org/ipwhois/107.158.43.150
    1 https://dnsquery.org/ipwhois/114.43.81.103
    1 https://dnsquery.org/ipwhois/101.167.216.61
    1 https://dnsquery.org/ipwhois/111.200.228.50

 Last 10 most recent attackers:
      https://dnsquery.org/ipwhois/104.82.68.238
      https://dnsquery.org/ipwhois/107.170.37.212
      https://dnsquery.org/ipwhois/62.24.131.142
      https://dnsquery.org/ipwhois/49.159.28.198
      https://dnsquery.org/ipwhois/192.0.73.2
      https://dnsquery.org/ipwhois/31.214.227.139
      https://dnsquery.org/ipwhois/106.75.15.57
      https://dnsquery.org/ipwhois/161.69.165.23
      https://dnsquery.org/ipwhois/172.217.6.227
      https://dnsquery.org/ipwhois/65.55.158.119
[/SPOILER]

 

[bayern1975]

here are my outputs....i would like to solve my problem but i do not know where is problem?

ASUSWRT-Merlin RT-AC3200 380.66-beta2-g7682479 Fri Apr 28 02:30:18 UTC 2017
admin@RT-AC3200-0000:/tmp/home/root# iptables --line -nvL INPUT | grep -E "set.*
Blacklist"
11      24  3152 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set Blacklist src
admin@RT-AC3200-0000:/tmp/home/root# iptables --line -nvL INPUT | grep -E "set.*
Blacklist" | awk '{print $2}'
24
admin@RT-AC3200-0000:/tmp/home/root#

 

[Martineau]

here are my outputs....i would like to solve my problem but i do not know where is problem?

ASUSWRT-Merlin RT-AC3200 380.66-beta2-g7682479 Fri Apr 28 02:30:18 UTC 2017
admin@RT-AC3200-0000:/tmp/home/root# iptables --line -nvL INPUT | grep -E "set.*
Blacklist"
11      24  3152 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set Blacklist src
admin@RT-AC3200-0000:/tmp/home/root# iptables --line -nvL INPUT | grep -E "set.*
Blacklist" | awk '{print $2}'
24
admin@RT-AC3200-0000:/tmp/home/root#

It would appear that you have rebooted?, so the duplicate rules no longer exist.

If you issue

cd /jffs/scripts
./IPSET_Block.sh

hopefully the report now shows the correct number of successful hits, and the extra '0' is no longer shown?

 

[bayern1975]

It would appear that you have rebooted?, so the duplicate rules no longer exist.

If you issue

cd /jffs/scripts
./IPSET_Block.sh

hopefully the report now shows the correct number of successful hits, and the extra '0' is no longer shown?

yes, i rebooted more times.....another output....

admin@RT-AC3200-0000:/jffs/scripts# ./IPSET_Block.sh
(IPSET_Block.sh): 4241 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....

        Summary Blacklist: 24 Successful blocks! ( 46 IPs currently banned - 38 added since: Apr 30 19:00 ), Entries auto-expire after 168:00:00 hrs