Sorry I missed this. I appreciate the investigative work. Quick question, does your outbound blocklist only block malware, or are there tracking/advertising IPs in there as well? I only ask because I see it's blocking a Netfilx-owned IP - 45.57.40.1. This is on my wife's iPhone, so I'm not sure if she was having any issues watching something, or if this IP is only tracking related and the streaming service doesn't rely on it. We watch netflix all the time without issue, so I assume blocking outbound connections to this IP isn't breaking the service.@jorgsmash
Here is my investigative work....
If your CDN whitelisting had properly loaded in from skynet, you would not have had to whitelist the teams IP address.
Code:{ printf "AS714\nAS12222\nAS16625\nAS33438\nAS20446\nAS54113\nAS36459" | xargs -I {} sh -c 'curl -fsL --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors https://asn.ipinfo.app/api/text/list/{} | awk -v asn={} '\''/^(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: %s\"\n", $1, asn }'\''' curl -fsL --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors https://www.cloudflare.com/ips-v4 | awk '/^(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: CloudFlare\"\n", $1 }' curl -fsL --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors https://ip-ranges.amazonaws.com/ip-ranges.json | awk 'BEGIN{RS="(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT)printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Amazon\"\n", RT }' curl -fsL --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors https://api.github.com/meta | awk 'BEGIN{RS="(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT)printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Github\"\n", RT }' curl -fsL --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors https://endpoints.office.com/endpoints/worldwide?clientrequestid="$(awk '{printf "%s", $1}' /proc/sys/kernel/random/uuid)" | awk 'BEGIN{RS="(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT)printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Microsoft365\"\n", RT }'; wait; } 2>/dev/null | awk '!x[$0]++' | grep -E '.*[[:space:]]52.*CDN-Whitelist:.*Microsoft365.*' add Skynet-Whitelist 52.96.0.0/14 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.100.0.0/14 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.238.78.88/32 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.112.0.0/14 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.122.0.0/15 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.238.119.141/32 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.244.160.207/32 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.104.0.0/14 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.108.0.0/14 comment "CDN-Whitelist: Microsoft365" add Skynet-Whitelist 52.244.37.168/32 comment "CDN-Whitelist: Microsoft365"
add Skynet-Whitelist 52.112.0.0/14 comment "CDN-Whitelist: Microsoft365"
would have covered the 52.113.194.132 IP.
View attachment 50198
View attachment 50199
I am using the version of skynet I have submitted a pull request for the CDN whitelisting issue.
Here is my ping test from a client on my network.
Code:ping 52.113.194.132 PING 52.113.194.132 (52.113.194.132): 56 data bytes 64 bytes from 52.113.194.132: seq=0 ttl=118 time=22.885 ms 64 bytes from 52.113.194.132: seq=1 ttl=118 time=22.747 ms 64 bytes from 52.113.194.132: seq=2 ttl=118 time=23.672 ms 64 bytes from 52.113.194.132: seq=3 ttl=118 time=22.091 ms 64 bytes from 52.113.194.132: seq=4 ttl=118 time=19.494 ms 64 bytes from 52.113.194.132: seq=5 ttl=118 time=22.420 ms 64 bytes from 52.113.194.132: seq=6 ttl=118 time=23.725 ms 64 bytes from 52.113.194.132: seq=7 ttl=118 time=19.866 ms 64 bytes from 52.113.194.132: seq=8 ttl=118 time=22.166 ms ^C --- 52.113.194.132 ping statistics --- 9 packets transmitted, 9 packets received, 0% packet loss round-trip min/avg/max = 19.494/22.118/23.725 ms
And here are the entries present from my patched skynet.
Code:RT-AX88U_Pro-29B8:/tmp/home/root# ipset list | grep -E '^52.*CDN-Whitelist:.*Microsoft365.*' 52.244.160.207 comment "CDN-Whitelist: Microsoft365" 52.100.0.0/14 comment "CDN-Whitelist: Microsoft365" 52.108.0.0/14 comment "CDN-Whitelist: Microsoft365" 52.238.119.141 comment "CDN-Whitelist: Microsoft365" 52.122.0.0/15 comment "CDN-Whitelist: Microsoft365" 52.96.0.0/14 comment "CDN-Whitelist: Microsoft365" 52.244.37.168 comment "CDN-Whitelist: Microsoft365" 52.104.0.0/14 comment "CDN-Whitelist: Microsoft365" 52.238.78.88 comment "CDN-Whitelist: Microsoft365" 52.112.0.0/14 comment "CDN-Whitelist: Microsoft365"
This is the pull request I have open
V7.4.1 by jumpsmm7 · Pull Request #124 · Adamm00/IPSet_ASUS
Improve curl hangs by defining the maximum amount of time allowed per try. This change came about because curl to some servers during whitelist processing caused an indefinite hang due a connection...github.com
As you can tell, I discovered this issue three days ago by coincidence. I noticed the amount of CDN-Whitelisting entries would greatly vary in number between list processing done by skynet. It took a little investigative work, but it turns out that a couple of the curl commands would hang indefinitely producing no output, until the connection closed resulting in numerous missing CDN whitelist entries (the Microsoft365 list was the main culprit). This is when I made the connection to whitelist processing hangs reported to @thelonelycoder during diversion/skynet shared lists processing.
Code:
Jun 9 13:17:29 kernel: [BLOCKED - OUTBOUND] IN=br0 OUT= MAC= SRC=192.168.50.38 DST=45.57.40.1 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=62229 DPT=443 SEQ=1539659743 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT (020405B4010303060101080A0E6F38630000000004020000)
