What's new

Skynet Filter Validator v0.7 - Skynet Firewall Filter List IPv4 Integrity Validator

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Viktor Jaep

Part of the Furniture
Filter Validator v0.7
A collaborative project between @Viktor Jaep and @SomeWhereOverTheRainBow!


Filter Validator tests the IPv4 addresses (and IPv6 if present) on a given filter list that are to be used with the Skynet Firewall running on Asus-Merlin Firmware in order to block incoming/outgoing IPs. This script arose out of the need to determine exactly which blacklist URL contained an invalid IP that was causing our Skynet firewalls to fail importing the correct IP sets due to an invalid IP somewhere on these lists.

**EDIT** -- as of March 10, 2023 -- Skynet v7.3.6 now has the necessary regex fixes included to filter out invalid addresses that were breaking the script, wholly inspired by Filter Validator, with many thanks to @SomeWhereOverTheRainBow for his excellent contributions making both scripts even better! Filter Validator may still play a role to determine if blacklist operators are continuing to maintain IP address data integrity, but also giving you a sense on how many entries are in your filter lists. Skynet has a hard limit of 500,000 entries at this point, and Filter Validator may tell you if you're getting too close. Last, other blacklists may stop being supported, or even disappear -- and while Skynet wouldn't give you any indication of this happening, Filter Validator will.

1678533684338.png


If there's an issue with one of the lists itself, or an IP on any of these lists, you'll see something like this:

1678536033496.png


After it's completed validating each entry, it will give you a summary at the end:

1678536120354.png



Usage Guide​

Execute the script as such: sh /jffs/scripts/filtervalidator.sh

Upon execution, it will ask for a valid URL to the specified filter list to be tested. For example, here is are a few valid filter list URLs that will be used if you press enter:

https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

NOTE: Should any list come back with any invalid IP entries (marked in Red), it would be advisable to remove or #COMMENT out the offending entry in your filter list in order to get Skynet back in working condition, or get in touch with the entity that takes care of the list in order to correct their mistake.


Download​

Code:
curl --retry 3 "https://raw.githubusercontent.com/ViktorJp/FilterValidator/master/filtervalidator.sh" -o "/jffs/scripts/filtervalidator.sh" && chmod 755 "/jffs/scripts/filtervalidator.sh"
 
Last edited:
<RESERVED>
 
Decided to just create it's own thread instead of keeping it buried under another thread. ;) I'm sure we'll be seeing some worthy additions to this soon! Thanks for your collaborative partnership on this, @SomeWhereOverTheRainBow!
 
Last edited:
Last edited:
Who is submitting a pull request to @Adamm to catch these malformed IPs during banmalware? He won’t bite (I think).

This is all news to me. Is the regex broken? o_O
 
This is all news to me. Is the regex broken? o_O
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
 
Just saying... @SomeWhereOverTheRainBow is a master at optimizing regex... just published v0.3 that runs the more optimized code combining the validation of both IPv4 and IPv6 addresses. :)
 
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
Don't forget about octal formats like 01.01.01.01 if those are present in the list the regex skynet uses will pull those in as well.
 
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
And some lists mix IPv6 addressing into their IPv4 lists as well... fun stuff. ;)
 
grep: warning: stray \ before #
 
Sorry about the abrupt message — I was in a hurry. This error is displayed when running v0.3 for every file using your filter list even though every file reports valid.
 
Sorry about the abrupt message — I was in a hurry. This error is displayed when running v0.3 for every file using your filter list even though every file reports valid.
Could you please elaborate on the error? Also, could you please provide me the URL for the list you're trying to validate?
 
Could you please elaborate on the error? Also, could you please provide me the URL for the list you're trying to validate?
Code:
   _____ ____            _   __     ___    __     __         
  / __(_) / /____ ____  | | / /__ _/ (_)__/ /__ _/ /____  ____
 / _// / / __/ -_) __/  | |/ / _ '/ / / _  / _ '/ __/ _ \/ __/
/_/ /_/_/\__/\__/_/     |___/\_,_/_/_/\_,_/\_,_/\__/\___/_/   v0.3
        By @Viktor Jaep and @SomewhereOverTheRainbow

Filter Validator was designed to run through your Skynet filter lists to determine
if all IP addresses fall within their normal ranges. Should any entries not follow
standard IP rules, they will be identified below. NOTE: Having invalid IPs within
these filter sets will cause the Skynet firewall to malfunction due to regex issues
that are not filtering out bad IPs, causing a loss of blocked IPs and ranges to occur.

Please enter a valid filter list URL, or hit <ENTER> to use example below:
Example 1: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
Example 2: https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

URL: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list

Testing against: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list

[Downloading Filter List]...OK
[Checking Filter List Contents]...OK

grep: warning: stray \ before #
Checking https://feodotracker.abuse.ch/downloads/ipblocklist.txt
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/alienvault_reputation.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/bds_atif.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/bi_any_2_30d.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/blocklist_net_ua.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/cybercrime.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dm_tor.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dshield.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dyndns_ponmocup.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_block.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_compromised.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_spamhaus.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_tor.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level1.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level2.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level3.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/greensnow.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/ciarmy.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/iblocklist_pedophiles.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/malc0de.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/maxmind_proxy_fraud.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/normshield_high_attack.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/spamhaus_drop.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/spamhaus_edrop.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/urlvir.ipset
[Valid]

grep: warning: stray \ before #
Checking https://sigs.interserver.net/iprbl.txt
[Valid]

grep: warning: stray \ before #
Checking https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
[Valid]

grep: warning: stray \ before #
Checking https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt
[Valid]

grep: warning: stray \ before #
Checking https://www.talosintelligence.com/documents/ip-blacklist
[Valid]

grep: warning: stray \ before #
Checking https://voipbl.org/update
[Valid]
 
It’s from /opt/bin/grep versus /bin/grep.
Maybe I'm missing something... but I'm not seeing any screenshot or other info to come to that conclusion... but I bet you're right. ;)
 
Maybe I'm missing something... but I'm not seeing any screenshot or other info to come to that conclusion... but I bet you're right. ;)
It is caused from this line of code

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -v '^\s*$\|^\s*\#' | sed -n $listcount'p') 2>&1

try

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -vE '^[[:space:]]*#' | sed -n $listcount'p') 2>&1
 
Last edited:
It is caused from this line of code

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -v '^\s*$\|^\s*\#' | sed -n $listcount'p') 2>&1

try

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -vE '^[[:space:]]*#' | sed -n $listcount'p') 2>&1
@SomeWhereOverTheRainBow That fixes it! Thanks! (It appears to run much more quickly, too!)
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top