1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Yet another malware block script using ipset (v4 and v6)

Discussion in 'Asuswrt-Merlin' started by redhat27, May 4, 2017.

  1. Przem

    Przem New Around Here

    Joined:
    Jul 29, 2017
    Messages:
    9
    @redhat27:
    Thank you VERY MUCH! The script does not show any errors in Tomato logs neither when run via SSH.
    Is there other simple way to check if its working?

    P.
     
    redhat27 likes this.
  2. VZ3

    VZ3 Regular Contributor

    Joined:
    Nov 4, 2016
    Messages:
    52
    You can use next command to monitor malware packets been dropped

    iptables -vL -t raw

    If you see chain pkts counter goes up that means your firewall is doing it's thing.
     
  3. Przem

    Przem New Around Here

    Joined:
    Jul 29, 2017
    Messages:
    9
    @VZ3:
    Thank you, I have run command you send, and it is working. :)


    Wysłane z iPad za pomocą Tapatalk Pro
     
  4. moooooooo

    moooooooo Occasional Visitor

    Joined:
    Jan 11, 2017
    Messages:
    11
    thanks for this as i've been wondering if it was possible to do it!
    /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (53445) and YAMalwareBlockCIDR (5811) in 11 seconds

    RT-AC68U here on a gigabit fibre connection. I'll see if any of my regular sites break and then play around with the whitelist mentioned on page 1!
    cheers
    peter
     
    Last edited: Oct 18, 2017
  5. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    492
    Location:
    The Netherlands
    Have a look at post #420.
    You can make the script even better by implementing what's mentioned there.
    (Put the code in between the "esac" and "startTS" command to the ya-malware-block.sh script.)

    I hope @redhat27 will add this, but he seems to be offline quite a lot.
     
    Last edited: Oct 18, 2017
    moooooooo and hervon like this.
  6. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    Version 2.5 (just uploaded in github) will take care of the issue of removing older ipsets and rules if they are no longer needed.
     
    Builder71 likes this.
  7. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    492
    Location:
    The Netherlands
    Thx!

    Also read your Git comment.
    To make sure I understand it correct.

    I don't change my ya-malware-block.urls file at all.
    It's just the active urls in it make I often go up and down around 65k.
    Simply because the content changes a bit grabbed from the urls.

    Does your change account for that?
     
  8. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    Apologies, I had misunderstood. Regardless, this version should take care of it. Thanks for pushing me to do it. BTW: I should post a similar fix to your other gihub issue soon (create-ipset-lists.sh)

    Yes, and just to test it, you can edit the /jffs/ipset_lists/ya-malware-block.urls file and uncomment the level4 url, run it (it creates few more ipsets) check the iptable rules and ipsets, and then comment it back and run it again. You should see the older ipsets and iptables rules removed.
     
    Last edited: Oct 20, 2017
  9. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    492
    Location:
    The Netherlands
    Tested as you suggested and works like a charm. :D

    Just want to make sure the script isn't looking at the ya-malware-block.urls file being edited or something like that.
    Because, in my case, that file is always the same and not the issue here.

    As a script noob I can't understand how you fixed it because the script is just too complicated for me. :oops:
    Hence my question.
     
  10. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    No, it's not looking at .urls being edited :)

    I've added some new lists to the .urls file in github. These are not included in FireHOL levels 1 through 4:

    Counts are as of the time of writing this post and will vary over time:
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/alienvault_reputation.ipset (68255 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bbcan177_ms1.netset (2565 subnets, 5268567 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bbcan177_ms3.netset (1146 subnets, 30151694 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bds_atif.ipset (5022 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/blocklist_de_bots.ipset (143 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/blocklist_de_ssh.ipset (11261 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/blocklist_de_strongips.ipset (104 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/dyndns_ponmocup.ipset (163 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/et_block.netset (1980 subnets, 24411811 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/et_botcc.ipset (728 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/et_compromised.ipset (1801 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/hphosts_exp.ipset (314 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/hphosts_hjk.ipset (57 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/hphosts_mmt.ipset (1136 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ransomware_feed.ipset (5216 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ransomware_locky_ps.ipset (3 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/taichung.ipset (10694 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/urandomusto_ssh.ipset (410 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/urandomusto_telnet.ipset (445 unique IPs)
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/uscert_hidden_cobra.ipset (627 unique IPs)
    Users of this script can update their ya-malware-block.urls file from the GitHub version if they choose to include these additional lists
     
    Builder71 likes this.
  11. Csection

    Csection Senior Member

    Joined:
    Oct 20, 2016
    Messages:
    315
    Should these new lists be just appended to the bottom of the .url file.
    I have already updated to 2.5.
     
  12. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    You can simply replace your .urls file. Let the script redownload it on the next run:
    Code:
    rm /jffs/ipset_lists/ya-malware-block.urls
    or download it yourself:
    Code:
    wget --no-check-certificate -O /jffs/ipset_lists/ya-malware-block.urls https://raw.githubusercontent.com/shounak-de/misc-scripts/master/ya-malware-block.urls
    
    Uncomment Level4 if you want
     
  13. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    492
    Location:
    The Netherlands
    Version 2.5 is working great! :)

    Also deleted ya-malware-block.urls to get the new one.
    A lot of new urls!
    I was still using the below urls because of the recent GitHub drama. ;)

    Is it OK if we use Git now?

    In the new ya-malware-block.urls file a lot is active. (No # sign in front.)
    What about false positives?
    Run into that before and then the family goes :mad:. :D
     
  14. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    I would tend to be an optimist and think that GitHub incident to be a one-off. Let's see if that recurs.
    Regarding false positives, I think if a site gets blocked, or something stops working, it should be fairly easy to whitelist: Just ping the domain to verify its blocked, and if it is, then just add that IP to the .whites file and rerun the script. A family typically has a handful of favourite sites (at least mine does) and I've whitelisted what I've seen blocked (I have Level4 active)
     
    Builder71 likes this.
  15. Mircica

    Mircica New Around Here

    Joined:
    Nov 3, 2017
    Messages:
    5
    Need some help

    I am having this

    /jffs/scripts/ya-malware-block.sh
    /jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
    >>> Downloading and aggregating malware sources (also processing whitelists)...[87619/80687/6932] ~18s
    >>> Adding data and processing rule for YAMalwareBlock1IP...Can't find library for match `webstr'
    ~4s
    >>> Adding data and processing rule for YAMalwareBlock2IP...Can't find library for match `webstr'
    ~1s
    >>> Adding data and processing rule for YAMalwareBlockCIDR...Can't find library for match `webstr'
    ~1s
    >>> Cleaning up... ~0s
    /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (15152) and YAMalwareBlockCIDR (6932) in 24 seconds


    but this shows right

    iptables -vL -t raw
    Chain PREROUTING (policy ACCEPT 223K packets, 185M bytes)
    pkts bytes target prot opt in out source destination
    63 3507 DROP all -- any any anywhere anywhere match-set YAMalwareBlockCIDR src
    126 5967 DROP all -- any any anywhere anywhere match-set YAMalwareBlock2IP src
    194 11880 DROP all -- any any anywhere anywhere match-set YAMalwareBlock1IP src

    Chain OUTPUT (policy ACCEPT 16415 packets, 4212K bytes)
    pkts bytes target prot opt in out source destination

    Any idea?
     
  16. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    492
    Location:
    The Netherlands
    We have to guess which router you use?
     
  17. Mircica

    Mircica New Around Here

    Joined:
    Nov 3, 2017
    Messages:
    5
    ASUS RT-AC66U, ASUSWRT Merlin 380 68 4
     
  18. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    Thank you. Can you post the output of these?
    Code:
    ipset --version
    iptables --version
    
    Also, do you get the 'webstr' library error on each run of the script, or just the first time?
     
  19. Mircica

    Mircica New Around Here

    Joined:
    Nov 3, 2017
    Messages:
    5
    ipset --version
    ipset v4.5, protocol version 4.
    Kernel module protocol version 4.

    iptables --version
    iptables v1.4.21

    each time
     
  20. redhat27

    redhat27 Very Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    508
    Sorry for the late reply. I'm assuming that you are running the script unmodified. Let me know if that is not the case.

    Do you get any output when you issue these commands:
    Code:
    iptables-save | grep -q YAMalwareBlockCIDR && echo "found"
    
    iptables -t raw -I PREROUTING -m set --set YAMalwareBlockCIDR src -j DROP