Skynet Issue with skynet stats -Skynet Stats not properly displaying Ban Reason

[SomeWhereOverTheRainBow]

So, I was playing with skynet stats today because I thought something was odd about the stats.

Top 10 Blocks From (Outbound);


--------   | --------------       | --------------                                          | --------------                                | ----------------------
| Hits |   | | IP Address |       | | AlienVault |                                          | | Ban Reason |                                | | Associated Domains |
--------   | --------------       | --------------                                          | --------------                                | ----------------------

747x       | 192.28.144.124  (AD) | https://otx.alienvault.com/indicator/ip/192.28.144.124  | BanMalware: ipv4.feed*                        |
552x       | 192.28.147.68   ()   | https://otx.alienvault.com/indicator/ip/192.28.147.68   | *                                             |
440x       | 199.15.214.243  (US) | https://otx.alienvault.com/indicator/ip/199.15.214.243  | *                                             |
422x       | 203.191.149.239 (CN) | https://otx.alienvault.com/indicator/ip/203.191.149.239 | Imported: Apples                             |
385x       | 206.81.13.146   (US) | https://otx.alienvault.com/indicator/ip/206.81.13.146   | Imported: Apples                             |
369x       | 103.143.248.129 (AU) | https://otx.alienvault.com/indicator/ip/103.143.248.129 | BanMalware: ipv4.feed*                        |
349x       | 66.115.184.26   (US) | https://otx.alienvault.com/indicator/ip/66.115.184.26   | Imported: Apples                             |
317x       | 66.115.184.27   (US) | https://otx.alienvault.com/indicator/ip/66.115.184.27   | Imported: Apples                             |
272x       | 116.196.106.34  (CN) | https://otx.alienvault.com/indicator/ip/116.196.106.34  | Imported: Apples                             |
262x       | 198.27.88.77    (US) | https://otx.alienvault.com/indicator/ip/198.27.88.77    | Imported: Apples                             |

I found it odd that in the Top 10 Blocks From (Outbound) it would list the reason "Imported: Apples" as the ban reason when the only lists I have imported using the comment "Apples" was whitelisted items.

firewall import whitelist file.txt "Apples"

I am just using the comment "Apples" as it fits the example. The issue is, I do not have any manually imported blacklists that use the comment "Apples", so how could the ban reason be "Imported: Apples" when I am only using that comment for whitelist import lists?

Anybody wanting to see if they can "reproduce" this issue, I posted more information about it here.

Skynet Stats not properly displaying Ban Reason. · Issue #129 · Adamm00/IPSet_ASUS

Brief Description Of Issue So, I was playing with skynet stats today because I thought something was odd about the stats. I found it odd that in the Top 10 Blocks From (Outbound) it would list the ...

github.com

@Adamm

 

[dave14305]

If it can’t find a full IP match in the banned IPs, it tries again with the first three octets. But it doesn’t seem to ensure it’s only the first 3 octets and not the last 3 octets on this second try. Try to escape the first dot where there are 2 in a row. It seems to be testing for a CIDR range.

https://github.com/Adamm00/IPSet_ASUS/blob/7f5cab97ea297df87a983168ae901069bfed1f96/firewall.sh#L670

Test case:

root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)..*/" test.txt
10.11.12.0/24
10.11.123.0/16
30.10.11.120/31
root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)\..*/" test.txt
10.11.12.0/24

 

[Viktor Jaep]

I am just using the comment "Apples" as it fits the example. The issue is, I do not have any manually imported blacklists that use the comment "Apples", so how could the ban reason be "Imported: Apples" when I am only using that comment for whitelist import lists?

Why do you have such a thing against "Apples"!! LOL. Hope you resolve this mystery!

 

[SomeWhereOverTheRainBow]

Why do you have such a thing against "Apples"!! LOL. Hope you resolve this mystery!

Lol that's the point, I only use the comment "Apples" for my white listed entries yet some how it is being misinterpreted as the ban reason from skynet stat parsing code.

 

[SomeWhereOverTheRainBow]

If it can’t find a full IP match in the banned IPs, it tries again with the first three octets. But it doesn’t seem to ensure it’s only the first 3 octets and not the last 3 octets on this second try. Try to escape the first dot where there are 2 in a row. It seems to be testing for a CIDR range.

https://github.com/Adamm00/IPSet_ASUS/blob/7f5cab97ea297df87a983168ae901069bfed1f96/firewall.sh#L670

Test case:

root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)..*/" test.txt
10.11.12.0/24
10.11.123.0/16
30.10.11.120/31
root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)\..*/" test.txt
10.11.12.0/24

I will take a look when I am off work tonight.

 

[SomeWhereOverTheRainBow]

If it can’t find a full IP match in the banned IPs, it tries again with the first three octets. But it doesn’t seem to ensure it’s only the first 3 octets and not the last 3 octets on this second try. Try to escape the first dot where there are 2 in a row. It seems to be testing for a CIDR range.

https://github.com/Adamm00/IPSet_ASUS/blob/7f5cab97ea297df87a983168ae901069bfed1f96/firewall.sh#L670

Test case:

root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)..*/" test.txt
10.11.12.0/24
10.11.123.0/16
30.10.11.120/31
root@router:~# grep -E "$(echo 10.11.12.13 | cut -d '.' -f1-3)\..*/" test.txt
10.11.12.0/24

Unless I am completely missing something, I believe I figured it out.

https://github.com/Adamm00/IPSet_ASUS/blob/7f5cab97ea297df87a983168ae901069bfed1f96/firewall.sh#L668-671

needs an adjustment.

            banreason="$(grep -F " ${ipaddr} " "$skynetipset" | grep -vE "(Skynet-Whitelist){1}(Domains){0,1}" | awk -F '"' '{print $2}')"
            if [ -z "$banreason" ]; then
                banreason="$(grep -E "$(echo "$ipaddr" | cut -d '.' -f1-3)\..*/" "$skynetipset" | grep -vE "(Skynet-Whitelist){1}(Domains){0,1}" | awk -F '"' '{print $2}')*"
            fi

From what I can tell | grep -m1 -vF "Skynet-MasterWL" at one time may have served an actual purpose when the whitelist was only Skynet-MasterWL, but it appears that all the additions has split the whitelists namings.

ipset list | grep "Skynet"
Name: Skynet-Whitelist
Name: Skynet-WhitelistDomains
Name: Skynet-Blacklist
Name: Skynet-BlockedRanges
Name: Skynet-IOT
Name: Skynet-Master
Skynet-Blacklist
Skynet-BlockedRanges
Name: Skynet-MasterWL
Skynet-Whitelist
Skynet-WhitelistDomains

The $skynetipset location lists the ipset add entries with Skynet-Whitelist and Skynet-WhitelistDomains which are entries we do not wish to match. Skynet-MasterWL is no longer the naming used in this ipset file which renders | grep -m1 -vF "Skynet-MasterWL" invalid.

 

[dave14305]

which renders | grep -m1 -vF "Skynet-MasterWL" invalid.

So changing it to filter out Skynet-Whitelist would be enough to catch either IPs or Domains.

Using your original report in the OP as a test case, how was 203.191.149.239 getting through? Was it a partial IP match from a blocked range? Is there something in your whitelist with 203.191.149?

 

[SomeWhereOverTheRainBow]

So changing it to filter out Skynet-Whitelist would be enough to catch either IPs or Domains.

Using your original report in the OP as a test case, how was 203.191.149.239 getting through? Was it a partial IP match from a blocked range? Is there something in your whitelist with 203.191.149?

The IP was in the whitelist, but the Skynet-MasterWL did not apply to properly filtering out comments of whitelisted entries. Otherwise it would have known that the Ban reason was a blocked range.

 

[dave14305]

The IP was in the whitelist, but the Skynet-MasterWL did not apply to properly filtering out comments of whitelisted entries. Otherwise it would have known that the entry was a Range.

Ok, but if it was in the whitelist, why was it being blocked in the first place? Or was it only recently whitelisted?

 

[SomeWhereOverTheRainBow]

Ok, but if it was in the whitelist, why was it being blocked in the first place? Or was it only recently whitelisted?

Recently whitelisted.

 

[SomeWhereOverTheRainBow]

@dave14305 thanks for your help on this. I would not have had the patience to figure this out if you had not pointed me in the right place to investigate.

 

[JTnola]

God hears your prayers.
Or reads your forum posts.

Skynet v7.4.6

GitHub - Adamm00/IPSet_ASUS: Skynet - Advanced IP Blocking For ASUS Routers Using IPSet.

Skynet - Advanced IP Blocking For ASUS Routers Using IPSet. - Adamm00/IPSet_ASUS

github.com

 

[SomeWhereOverTheRainBow]

View attachment 53140

God hears your prayers.
Or reads your forum posts.

Skynet v7.4.6

GitHub - Adamm00/IPSet_ASUS: Skynet - Advanced IP Blocking For ASUS Routers Using IPSet.

Skynet - Advanced IP Blocking For ASUS Routers Using IPSet. - Adamm00/IPSet_ASUS

github.com

Whelp I had to do one more small patch with the '-m1'. I think @Adamm happen to be dropping by. I hope adamm is doing well.


@Adamm just pushed a final patch, and the issue is closed.

 

[SomeWhereOverTheRainBow]

Looks like @dave14305 got tired of all my putzing arounding. He has located more places in the script to fix. He added in some changes for consistency as well. Overall, great work!

Escape other CIDR dots and make grep consistent by dave14305 · Pull Request #132 · Adamm00/IPSet_ASUS

Fix remaining banreason commands, and restore consistency in the grep expression for all banreason commands. Fixes: fab7656

github.com

So everyone be ready for another incoming update once @Adamm approves the request.

 

[Tomo]

minor update online.