Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Yet another malware block script using ipset (v4 and v6)

Discussion in 'Asuswrt-Merlin' started by redhat27, May 4, 2017.

  1. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    478
    FYI to all who is using this script. I've noticed that sometimes the google resolver 8.8.8.8 appears in one of the blocklists. That affects the default resolver configured for the hostip utility.

    Please include that in the /jffs/ipset_lists/ya-malware-block.whites file (default location) I've also included that in the default whitelist
     
    Supernova and Builder71 like this.
  2. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    I'm using the default ya-malware-block.urls
    So this uses firehol_level1, 2, 3 and telemetry_and_scanners.txt

    When I try,
    Code:
    [email protected]:/jffs# MatchIP 8.8.8.8
    8.8.8.8 not found in YAMalwareBlockCIDR
    8.8.8.8 not found in YAMalwareBlock1IP
    
    the output gives me the impression not all lists from the default ya-malware-block.urls are used.
    (Only two output lines.)

    Is this as expected?
     
  3. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Looking at it, it seems like "ya-malware-block.urls" is not used.
    It always goes for YAMalwareBlock1IP and YAMalwareBlockCIDR, no matter what.
    What am I missing?

    pastebin
     
  4. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    478
    Do not comment out the sources in the .urls file. You may leave level4 commented, but I use all of the lists (none of the lists are commented). Sometimes having level4 enabled may have some false positives. (Like 8.8.8.8 may sometimes be mistakenly included). The sources in these lists are very dynamic in nature and frequently change with time. That is why I'd suggested running it every 6 hours

    I would recommend you read post #1 of this thread and let me know if anything is missing or confusing there.
     
  5. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Yes, I know I should not comment out the sources .urls file.
    This was only my example to show the file seems to be ignored.
    Whatever I do with it, only YAMalwareBlockCIDR and YAMalwareBlock1IP are active.

    With a default .urls file (only level4 commented out), this is what the logging shows:

    Code:
    Aug 27 06:00:16 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (55323) and YAMalwareBlockCIDR (6649) in 16 seconds
    Code:
    [email protected]:/tmp/home/root# iptables -L -t raw
    Chain PREROUTING (policy ACCEPT)
    target     prot opt source               destination
    DROP       all  --  anywhere             anywhere            set YAMalwareBlockCIDR src
    DROP       all  --  anywhere             anywhere            set YAMalwareBlock1IP src
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    [email protected]:/tmp/home/root#
    
    I expect also to see something from firehol_level2, 3 and telemetry_and_scanners.txt in the logging.
    Or is above OK? :confused:
     
    Last edited: Aug 27, 2017
  6. redhat27

    redhat27 Senior Member

    Joined:
    Jul 29, 2016
    Messages:
    478
    There should be more YAMalwareBlock?IP ipsets, generally. The pastebin output you gave had this:
    Code:
    [email protected]:/jffs/ipset_lists# cat ya-malware-block.urls
    #https://raw.githubusercontent.com/shounak-de/misc-scripts/master/telemetry_and_scanners.txt
    #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
    #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
    #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
    
    And that is why I said that. You can try to un-comment all the sources (even level4) and post your output here. The first line from the terminal run gives summary counts.
    See post #336
     
  7. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    No problem, I start all over.
    First run is with all urls active. (No # sign before any line.)
    This seems to work better only no log if telemetry_and_scanners.txt became active?

    After that I only put a # sign before the last url.
    This is the default situation with level4 not active.

    Now you see only two urls become active.
    There is something going wrong here.

    pastebin
     
  8. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Fixed it!

    If I add a space after the # sign in the ya-malware-block.urls file it works. :D

    So it should look like this:

    Code:
    [email protected]:/jffs/ipset_lists# cat ya-malware-block.urls
    https://raw.githubusercontent.com/shounak-de/misc-scripts/master/telemetry_and_scanners.txt
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
    https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
    # https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
    
    If it's #https:// (without space) it doesn't work correctly on my RT-N66U.
     
  9. VZ3

    VZ3 Occasional Visitor

    Joined:
    Nov 4, 2016
    Messages:
    49
    That's perfectly normal... This output shows that out of default .urls file with level4 commented out you ONLY have 55323 IP addresses to block this particular time. So, 55323 < 65535 - this way you will have ONLY one YAMalwareBlock1IP active...
    Some days, when sources IP count goes more then 65k you will see YAMalwareBlock2IP etc.

    PS: you last post with "space in between # and https" does not make sense. Show me output of the next command:

    curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
     
    Builder71 likes this.
  10. VZ3

    VZ3 Occasional Visitor

    Joined:
    Nov 4, 2016
    Messages:
    49
    @redhat27,

    btw, there is small bug in your script. When block IP count goes above 65k you are creating YAMalwareBlock2IP etc. BUT, when other days IP count stays below 65k those YAMalwareBlock2IP, ...3IP etc. sets are still active in drop chain and did not get deleted.

    I guess this is no biggy.
     
  11. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Code:
    [email protected]:/jffs/scripts# curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
    curl: no URL specified!
    curl: try 'curl --help' for more information
    [email protected]:/jffs/scripts#
    
    Ahh, thanks for explaining, I didn't know that.
    When I add the extra space, the ya-malware script always shows level1,2,3 and CIDR are loaded.
    Because of this output I assumed this is what it should do.

    Without the space I currently see level1,2 and CIDR are loaded.
    But from what I understand from your comment is that level3 is also loaded but is not shown in script output.
    Correct?

    Edit: tested with the MatchIP script and level3 IP's are indeed shown. (As level2 but who cares.)
    :)
     
    Last edited: Aug 31, 2017
  12. VZ3

    VZ3 Occasional Visitor

    Joined:
    Nov 4, 2016
    Messages:
    49
    Your curl output command with commented out URL as expected - meaning no IP list are downloaded from that URL.

    Those level1,2,3 and 4 URLs are sources for YAMB block list which later aggregated into YAMalwareBlock 1, 2, 3 etc. lists. YAMB single set cannot hold more then 65k addresses, so it will "overfill" into next set and so on...

    Due to the bug in YAM script, if you run it without router restart, ALL your old/previous "overfill" YAMB sets with grater number WILL stay (will not be cleared up) if on next run you will have less IP addresses in source list. So, that can explain your experience when you see YAMB 1, 2 and 3 sets even if you commented out the source URLs.

    And again, source URLs level 1, 2, 3 and 4 has no direct relations (one to one) to YAMB1, 2, 3 etc.
     
    Builder71 likes this.
  13. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    That indeed was my mistake.

    Your comment in post #410, is this only "cosmetic" or could it mean we have a possiblity to get false positives?
     
  14. VZ3

    VZ3 Occasional Visitor

    Joined:
    Nov 4, 2016
    Messages:
    49
    When previously some "BAD" IP will become "GOOD" one, maybe due to time or other criteria, your router will still treat it as "BAD" until next "overfill" or router reboot.

    As for me, it's like good reputation - one have to treasure it from the very beginning :).
     
  15. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Sure thing. :D

    I think redhat27 will fix it when he feels like it. :)
     
  16. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    I added below code block in between "esac" and "startTS" command to the ya-malware-block.sh script.
    It seems to work well, but I'm horrible at scripting.
    So please shoot at it to make it better. :D

    Code:
    # Delete existing YAMalwareBlock3IP rule.
    number=`iptables -t raw -L --line-numbers | grep YAMalwareBlock3IP | cut -d' ' -f1`
    logger -t Firewall "Deleting YAMalwareBlock3IP rule if exist."
    echo Deleting YAMalwareBlock3IP rule if exist.
    iptables -t raw -D PREROUTING $number > /dev/null 2>&1
    # Delete existing YAMalwareBlock2IP rule.
    number=`iptables -t raw -L --line-numbers | grep YAMalwareBlock2IP | cut -d' ' -f1`
    logger -t Firewall "Deleting YAMalwareBlock2IP rule if exist."
    echo Deleting YAMalwareBlock2IP rule if exist.
    iptables -t raw -D PREROUTING $number > /dev/null 2>&1
    
     
    Last edited: Sep 5, 2017
  17. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    @redhat27 @VZ3
    A co-worker helped me making it a one liner. :)
    Works well on my RT-N66U. (ipset v4)

    Maybe the only "downside" of deleting the old block rules is that the blockstats counter also resets to zero.
    (With 'blockstats' command you can see how well your blocklists are doing.)

    Code:
    # Delete old YAMalwareBlock rules.
    logger -t Firewall "Delete old YAMalwareBlock rules."
    echo Delete old YAMalwareBlock rules.
    iptables -t raw -L --line-numbers | grep YAMalwareBlock | cut -d' ' -f1 | sed '1!G;h;$!d' | while read number; do iptables -t raw -D PREROUTING $number; done
    
    Put above code in between the "esac" and "startTS" command to the ya-malware-block.sh script.
     
  18. VZ3

    VZ3 Occasional Visitor

    Joined:
    Nov 4, 2016
    Messages:
    49
    Nice job, you are deleting block set from iptables but you forgeting about actual ipset destroying its unused named set.
     
  19. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    Thx.

    Correct, but removing the iptables rules are enough preventing "false positives".
    The ipset named set is rewritten anyway when used again.
    But true, it's cleaner to remove that from memory as well. (Although it is never used without iptables rule.)
     
  20. Builder71

    Builder71 Senior Member

    Joined:
    Oct 14, 2012
    Messages:
    464
    Location:
    The Netherlands
    @VZ3 OK, I could not resist to fix that. :)

    Code:
    # Delete old YAMalwareBlock rules.
    logger -t Firewall "Delete old YAMalwareBlock rules."
    echo Delete old YAMalwareBlock rules.
    iptables -t raw -L --line-numbers | grep YAMalwareBlock | cut -d' ' -f1 | sed '1!G;h;$!d' | while read number; do iptables -t raw -D PREROUTING $number; done
    ipset -L | grep YAMalwareBlock | cut -d' ' -f2 | while read setname; do ipset --destroy $setname; done
    
     

Share This Page