Skynet Filter Validator v0.7 - Skynet Firewall Filter List IPv4 Integrity Validator

[SomeWhereOverTheRainBow]

@visortgw

The second test has even better results.


and third test.

This definitely shows the download aspect is the biggest factor.

 

[Viktor Jaep]

Who's ready for v0.6 with some awesome performance stats... as @SomeWhereOverTheRainBow and I found out, the biggest factor that affects processing time is the amount of time it takes to download the list from the server... probably due to internet weather, or current server load. Enjoy!

Some sweet sweet stats courtesy of @SomeWhereOverTheRainBow!

 

[Kingp1n]

Hello @Viktor Jaep ....I appreciate the script and I've yet to try it! I did have a question...I only use the default list (settings) within Diversion (lite) and Skynet. Just curious if I run this script and notice an invalid IP...what would be the steps to correct this within Skynet and/or Diversion? Or does the only work with custom made list?

 

[Viktor Jaep]

Hello @Viktor Jaep ....I appreciate the script and I've yet to try it! I did have a question...I only use the default list (settings) within Diversion (lite) and Skynet. Just curious if I run this script and notice an invalid IP...what would be the steps to correct this within Skynet and/or Diversion? Or does the only work with custom made list?

It would be highly unlikely that his list would break, but if it did - you're right - you would be dead in the water until he fixed the list... or, unless you switched over to another list. These are the sources he uses in his list:

https://iplists.firehol.org/files/alienvault_reputation.ipset
https://iplists.firehol.org/files/bds_atif.ipset
https://iplists.firehol.org/files/cybercrime.ipset
https://iplists.firehol.org/files/dyndns_ponmocup.ipset
https://iplists.firehol.org/files/et_block.netset
https://iplists.firehol.org/files/et_compromised.ipset
https://iplists.firehol.org/files/firehol_level2.netset
https://iplists.firehol.org/files/firehol_level3.netset
https://iplists.firehol.org/files/spamhaus_edrop.netset
https://iplists.firehol.org/files/urlvir.ipset

Oh, and not quite sure about Diversion... I've not dived into figuring out where those lists live.

 

[visortgw]

Who's ready for v0.6 with some awesome performance stats... as @SomeWhereOverTheRainBow and I found out, the biggest factor that affects processing time is the amount of time it takes to download the list from the server... probably due to internet weather, or current server load. Enjoy!

Some sweet sweet stats courtesy of @SomeWhereOverTheRainBow!
View attachment 48377

And mine:

[Valid List Entries]: 33
[Invalid List Entries]: 0
[Total Items Checked]: 329019
[Total Processing Runtime]: 22130 ms or 22.130 sec

 

[SomeWhereOverTheRainBow]

And mine:

[Valid List Entries]: 33
[Invalid List Entries]: 0
[Total Items Checked]: 329019
[Total Processing Runtime]: 22130 ms or 22.130 sec

Not bad.

 

[SomeWhereOverTheRainBow]

And mine:

[Valid List Entries]: 33
[Invalid List Entries]: 0
[Total Items Checked]: 329019
[Total Processing Runtime]: 22130 ms or 22.130 sec

Here is my best run:

---------------------------------------------
[Valid List Entries]: 40
[Invalid List Entries]: 0
[Total Items Checked]: 544451
[Total Processing Runtime]: 17100 ms or 17.100 sec

@Viktor Jaep

Clocked me.

 

[dave14305]

Why download the lists again when they should already be sitting there in /tmp/mnt/<label>/skynet/lists/?

 

[Viktor Jaep]

Here is my best run:

---------------------------------------------
[Valid List Entries]: 40
[Invalid List Entries]: 0
[Total Items Checked]: 544451
[Total Processing Runtime]: 17100 ms or 17.100 sec

@Viktor Jaep

Clocked me.

This was my worst sofar...

 

[SomeWhereOverTheRainBow]

Why download the lists again when they should already be sitting there in /tmp/mnt/<label>/skynet/lists/?

That is assuming they haven't become borked in between load failures. I am sure it is doable if the lists are intact. I imagine we could add an option to check the lists if they are intact. I think the original idea of the script was to check a filter list before attempting to load it. Basically a tool users can use to see if the custom filter list will bork skynet loading process. We wouldn't want to load a filter.list if it is problematic.

 

[Adamm]

*looks at pull requests*

While I can appreciate the effort here, is the better solution to not just improve the built-in regex to ignore these edge cases?

 

[SomeWhereOverTheRainBow]

*looks at pull requests*

While I can appreciate the effort here, is the better solution to not just improve the built-in regex to ignore these edge cases?

Pull request submited:

Fix asn link, Add Real IP address and ranges regex. by jumpsmm7 · Pull Request #114 · Adamm00/IPSet_ASUS

Resolve string parsing errors with bad ASN links. Add Real IP Address and Range detection logic. Capture any remaining missed Ranges [30-32] in regex detection.

github.com

I think I added in all possible places to consider.

 

[Kingp1n]

Just saw a minor update to Skynet...might be a fix maybe?

 

[dave14305]

Just saw a minor update to Skynet...might be a fix maybe?

Not yet. Adamm merged an earlier pull request from me for an unrelated issue.

 

[SomeWhereOverTheRainBow]

Not yet. Adamm merged an earlier pull request from me for an unrelated issue.

@dave14305 thank you for still being around the community, you have a natural ability to help this community. And you do a wonderful job at it.

 

[dave14305]

@dave14305 thank you for still being around the community, you have a natural ability to help this community. And you do a wonderful job at it.

And sometimes I’m a schmuck, so it balances out…

 

[SomeWhereOverTheRainBow]

And sometimes I’m a schmuck, so it balances out…

I think we all are at times, but sometimes that is how we learn to be better instead of just good.

 

[Mutzli]

Looks like Adamm implemented the regex fix here with an updated version to 7.3.6.

 

[Mogsy]

A result for Is default firewall good enough? discussion?
Will give it a go

 

[Viktor Jaep]

Looks like Adamm implemented the regex fix here with an updated version to 7.3.6.

Actually, @SomeWhereOverTheRainBow is the one who submitted all the regex changes to be included in this latest Skynet update! Huge kudos and thanks to him!