Skynet How do I use a VPN blocklist as a VPN whitelist instead?

Spud

Occasional Visitor
I came across a NordVPN blocklist that’s updated daily; so, as someone who uses NordVPN and experiences firewall blocks in Skynet, I wondered how I could use this list to whitelist NordVPN servers instead of blocking them:


The trouble I’m having is that Skynet refuses to import the entries from the URL directly, or from the txt file downloaded via a browser (Option 5). Is there something about the blocklist format that doesn’t work with whitelisting?

In addition, I originally aimed to download the txt file with wget or curl to update the whitelist as part of a script, but those methods throw up 404 or 418 errors.

I’m pretty new to Linux and I expect I’m missing something. Any pointers would be much appreciated.
 

Viktor Jaep

Very Senior Member
I’m pretty new to Linux and I expect I’m missing something. Any pointers would be much appreciated.

As long as the list is just a single column of IP4 addresses, it should import fine? I'm having trouble reaching that domain and downloading this list to see for some reason.

This is the method I'm using to import all NordVPN servers in order to whitelist them in Skynet with my VPNMON-R2 tool:

Code:
firewall import whitelist /jffs/scripts/NordVPN.txt

EDIT: Are you trying to whitelist all possible NordVPN servers in your particular country? If so, let me know... I have some code for you to try.
 

Spud

Occasional Visitor
As long as the list is just a single column of IP4 addresses, it should import fine? I'm having trouble reaching that domain and downloading this list to see for some reason.

This is the method I'm using to import all NordVPN servers in order to whitelist them in Skynet with my VPNMON-R2 tool:

Code:
firewall import whitelist /jffs/scripts/NordVPN.txt

EDIT: Are you trying to whitelist all possible NordVPN servers in your particular country? If so, let me know... I have some code for you to try.

Thanks Viktor! Yes, downloading and importing that list has me stumped - I can't see why it shouldn't work.

In fact, the background to this post very much involves your script, but indirectly. I'll explain...

When you first released your two scripts, I was running UK and DE connections via two client instances in VPN Manager. I reused your code (thanks for this) to pull all the servers into Skynet's whitelist with a cron job.

Code:
                curl --silent "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[]                           >
                firewall import whitelist /jffs/scripts/NordVPN-UK.txt "UK NordVPN"
                        sleep 10
                curl --silent "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[]                           >
                firewall import whitelist /jffs/scripts/NordVPN-DE.txt "DE NordVPN"
                        sleep 10
                rm /jffs/scripts/NordVPN-UK.txt  #Cleanup UK
                rm /jffs/scripts/NordVPN-DE.txt  #Cleanup DE

exit 0

This has worked well on the whole, but every so often I run into a bunch of blocks, which got me thinking.

A few days ago this happened, and I reran the script and noticed that the servers pulled into the txt files included the Public IP, but the "connected" IP in yellow was missing (example, not the actual IP). The yellow IP was also missing from the aggregated whitelist in Skynet, pulled over a couple of months.

Screenshot 2022-04-25 at 09.28.57.png



A quick search in Google with the IP brought me to the NordVPN blocklist at the top of the post, which included the missing server (that's why I wanted to try it out).

So, coming round in a big circle, I'm wondering if the problem could be solved differently.

I guess that VPN Manager auto-downloads a config file to repopulate a client slot, but the server IP in the config file doesn't get whitelisted. Sometimes this IP differs from the Public IPs, which your script appears to pull and whitelist.

In parallel with your script, do you think there's a way to grab and whitelist the IP from the VPN config file as VPN Manager updates? I'm thinking this might pretty much solve my issues.
 

Viktor Jaep

Very Senior Member
When you first released your two scripts, I was running UK and DE connections via two client instances in VPN Manager. I reused your code (thanks for this) to pull all the servers into Skynet's whitelist with a cron job.
Nice work, @Spud ... but take a look at your curl statement. It seems it's whitelisting the entire world. ;) Not sure if that's what you intended, but you probably want to limit it just to the UK and DE, right? Here are some corrections:

Code:
curl --silent --retry 3 "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[] | select(.locations[].country.name == "United Kingdom") | .station' > /jffs/scripts/NordVPN-UK.txt

curl --silent --retry 3 "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[] | select(.locations[].country.name == "Germany") | .station' > /jffs/scripts/NordVPN-DE.txt

Please know that these are just the public IPs of the NordVPN endpoints that you're whitelisting. When you make an actual connection to the VPN server, you will be assigned a different Public IP in order for you to get out to the internet. None of these IPs or ranges are known. Like, if 50 people all connect to the same VPN endpoint, what addresses will they all get?

I added this function simply because Skynet would at times block the IP for a known good NordVPN server that I wasn't able to then connect to, and it would just retry endlessly, leaving my VPN connection down. Since doing this, I have not had a single issue. When you say you're running into a bunch of blocks, what do you mean? And how are you determining this? If you can give me a little more explanation here, perhaps I can help you with a workaround? :)
 
Last edited:

Viktor Jaep

Very Senior Member
I guess that VPN Manager auto-downloads a config file to repopulate a client slot, but the server IP in the config file doesn't get whitelisted. Sometimes this IP differs from the Public IPs, which your script appears to pull and whitelist.

VPNMGR basically does an API call based on the location you have configured for your slot, and based on your distance, latency, etc., picks the most appropriate server that closest/fastest for you. The script you're running above should be able to capture all available NordVPN servers for your region, and should include even the ones that VPNMGR recommends for you.
 

Spud

Occasional Visitor
Nice work, @Spud ... but take a look at your curl statement. It seems it's whitelisting the entire world. ;) Not sure if that's what you intended, but you probably want to limit it just to the UK and DE, right? Here are some corrections:

Code:
curl --silent --retry 3 "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[] | select(.locations[].country.name == "United Kingdom") | .station' > /jffs/scripts/NordVPN-UK.txt

curl --silent --retry 3 "https://api.nordvpn.com/v1/servers?limit=16384" | jq --raw-output '.[] | select(.locations[].country.name == "Germany") | .station' > /jffs/scripts/NordVPN-DE.txt

Please know that these are just the public IPs of the NordVPN endpoints that you're whitelisting. When you make an actual connection to the VPN server, you will be assigned a different Public IP in order for you to get out to the internet. None of these IPs or ranges are known. Like, if 50 people all connect to the same VPN endpoint, what addresses will they all get?

I added this function simply because Skynet would at times block the IP for a known good NordVPN server that I wasn't able to then connect to, and it would just retry endlessly, leaving my VPN connection down. Since doing this, I have not had a single issue. When you say you're running into a bunch of blocks, what do you mean? And how are you determining this? If you can give me a little more explanation here, perhaps I can help you with a workaround? :)


Thanks for coming back to me.

The code line breaks came out badly - seems some stuff is missing - but I can confirm I'm only whitelisting the UK and DE :)

The recent blocks have tapered off and, unfortunately, they've been refreshed out of my Skynet stats. All were IPs belonging to providers NordVPN uses in the UK: Clouvider, Hydra Communications and UK Dedicated Servers. (At the moment, I'm using two UK servers vs one UK and one DE.)

Your explanation about VPN endpoints and the way they're selected pretty much aligns with my thoughts, so I think that's OK.

The problem I'm having seem to be with the VPN client's connection to NordVPN, i.e. the IP in yellow. I've come across instances where this IP hasn't found its way onto the whitelist, despite the script whitelisting many IPs for weeks. Sometimes this correlates to the connection dying off, or an authentication error when trying to connect or reconnect.

(Much more often, the yellow IP is already on the whitelist and the connection holds just fine.)

The challenge I see is how to ensure the IP in yellow always gets whitelisted too. I can only assume it's pulled from the VPN config file downloaded by VPN Manager at each refresh. The IP would then need to be whitelisted just before VPN Manager sets up the new connection.

I hope I've explained it well enough. It seems you've been fortunate in the US and haven't encountered the issue.
 

Viktor Jaep

Very Senior Member
The code line breaks came out badly - seems some stuff is missing - but I can confirm I'm only whitelisting the UK and DE :)
OK whew! I just noticed the "more" > marks to the right of the code window. lol
The problem I'm having seem to be with the VPN client's connection to NordVPN, i.e. the IP in yellow. I've come across instances where this IP hasn't found its way onto the whitelist, despite the script whitelisting many IPs for weeks. Sometimes this correlates to the connection dying off, or an authentication error when trying to connect or reconnect.

(Much more often, the yellow IP is already on the whitelist and the connection holds just fine.)

The challenge I see is how to ensure the IP in yellow always gets whitelisted too. I can only assume it's pulled from the VPN config file downloaded by VPN Manager at each refresh. The IP would then need to be whitelisted just before VPN Manager sets up the new connection.

I hope I've explained it well enough. It seems you've been fortunate in the US and haven't encountered the issue.

I just ran a test, downloaded a list of the UK servers, and was able to match the IP being in there that you've got highlighted in yellow:

Screenshot 2022-04-25 16.21.58.png


So did you export your list out of Skynet to validate that this server wasn't listed in Skynet, but it was in your NordVPN-UK.txt file?

I know NordVPN servers are sometimes volatile, as in, they come and go, because the total number of servers in the US fluctuates by a couple on a daily basis... and perhaps that's why you just happened to hit that one server at the right time and the right place, as it was still in a transition. Or perhaps servers are being shuffled between countries in Europe more liberally so that it's no longer listed on the UK list? I guess it would be interesting to do like a reverse lookup of sorts with the API and see which country a particular NordVPN IP is being assigned to? You could use this:

Code:
curl --silent https://api.nordvpn.com/server | jq '.[] | select(.ip_address == "81.92.203.111")'

But from what I can tell, as long as you're downloading the latest list on a regular basis (perhaps increase it to 2x or 4x/day) for import into Skynet, you shouldn't have any issue with your yellow-highlighted server not being listed?

Definitely a good mystery! :)
 

Spud

Occasional Visitor
OK whew! I just noticed the "more" > marks to the right of the code window. lol


I just ran a test, downloaded a list of the UK servers, and was able to match the IP being in there that you've got highlighted in yellow:

View attachment 40981

So did you export your list out of Skynet to validate that this server wasn't listed in Skynet, but it was in your NordVPN-UK.txt file?

I know NordVPN servers are sometimes volatile, as in, they come and go, because the total number of servers in the US fluctuates by a couple on a daily basis... and perhaps that's why you just happened to hit that one server at the right time and the right place, as it was still in a transition. Or perhaps servers are being shuffled between countries in Europe more liberally so that it's no longer listed on the UK list? I guess it would be interesting to do like a reverse lookup of sorts with the API and see which country a particular NordVPN IP is being assigned to? You could use this:

Code:
curl --silent https://api.nordvpn.com/server | jq '.[] | select(.ip_address == "81.92.203.111")'

But from what I can tell, as long as you're downloading the latest list on a regular basis (perhaps increase it to 2x or 4x/day) for import into Skynet, you shouldn't have any issue with your yellow-highlighted server not being listed?

Definitely a good mystery! :)

Yes, a good mystery!

So, the yellow IP I gave earlier was just an example for the screenshot (VPN client working normally at the time) and like the vast majority, it appears in the Skynet whitelist via the txt file created with your script.

The other day, however, I recall the connection stalled. I went to check the whitelist for the yellow IP, but it wasn't there. I then ran the script, but it still didn't show up in the txt file either.

But I did find it in the global blocklist I shared initially, run by a guy from NL:


I think you could spot on about servers being shared across on Europe.

I'll have to try out a much broader whitelist, hopefully tomorrow. I guess you must have a comparable number in the US as all Europe combined, while I've only tapped two countries.

BTW, do you know if it's possible to pull "Europe" as a whole out of the API?

Re servers changing quickly, I've had the script running every three hours, so pretty often, but you never know.

Thanks for the reverse API tip. It's quite late now, so I'll try it out tomorrow.
 

Viktor Jaep

Very Senior Member
I think you could spot on about servers being shared across on Europe.

I'll have to try out a much broader whitelist, hopefully tomorrow. I guess you must have a comparable number in the US as all Europe combined, while I've only tapped two countries.

BTW, do you know if it's possible to pull "Europe" as a whole out of the API?

Re servers changing quickly, I've had the script running every three hours, so pretty often, but you never know.

Thanks for the reverse API tip. It's quite late now, so I'll try it out tomorrow.

Hi @Spud ... I'm not seeing any way to extract all of Europe... it looks to me like you would need to call each country individually. But perhaps just broaden out your circle, including NL, BE, perhaps FR? Good thing you have cron doing all the heavy lifting for you. Would love to hear if this helps solve your issue! :)

As an alternative, and also as a shameless plug for VPNMON-R2... :p but after thinking about your situation, this potentially could help solve your issue as well? If you designate the "United Kingdom" as your NordVPNSuperRandom country, there would be a total of 608 servers you could possibly connect to if for some reason something went wrong with your connection, or the load exceeded a certain amount? If VPNMON-R2 couldn't make a connection due to it not being on a whitelist, it would just try again until it finds a server that works.
 

Spud

Occasional Visitor
Hi @Spud ... I'm not seeing any way to extract all of Europe... it looks to me like you would need to call each country individually. But perhaps just broaden out your circle, including NL, BE, perhaps FR? Good thing you have cron doing all the heavy lifting for you. Would love to hear if this helps solve your issue! :)

As an alternative, and also as a shameless plug for VPNMON-R2... :p but after thinking about your situation, this potentially could help solve your issue as well? If you designate the "United Kingdom" as your NordVPNSuperRandom country, there would be a total of 608 servers you could possibly connect to if for some reason something went wrong with your connection, or the load exceeded a certain amount? If VPNMON-R2 couldn't make a connection due to it not being on a whitelist, it would just try again until it finds a server that works.

Many thanks for checking that, and your help these last couple of days.

Indeed, I spent a while trying to pull the Europe servers in one go through the API with no luck. It would have been overkill. As you say, makes sense to whitelist nearby countries only. So, I've now expanded the original script, and I'll see how it goes.

VPNMON-R2 has come a long way, it's very good work. Would it accommodate the following setup across countries? Normally, I'd use slots VPN 2 and 5 (DE and UK, NordVPN), the slot 5 with an optional kill switch, and both controlled by VPN Manager.
 

Viktor Jaep

Very Senior Member
Many thanks for checking that, and your help these last couple of days.
Happy to assist! :)

Indeed, I spent a while trying to pull the Europe servers in one go through the API with no luck. It would have been overkill. As you say, makes sense to whitelist nearby countries only. So, I've now expanded the original script, and I'll see how it goes.
I'd definitely be interested to find out if you come across this situation again, and determining where the missing VPN server is located!

VPNMON-R2 has come a long way, it's very good work. Would it accommodate the following setup across countries? Normally, I'd use slots VPN 2 and 5 (DE and UK, NordVPN), the slot 5 with an optional kill switch, and both controlled by VPN Manager
Right now it's only able to handle 1 country at a time... and it would need to be using VPN slots 1 and 2 in your case. VPNMON-R2 doesn't really care how you have your VPN setup, so if you want to use VPNMGR/killswitches, that's your call! But that's a great idea, @Spud! Wow... Let me think through how I could make this happen. Perhaps assigning a different country to each 1-5 VPN slot, and let you specify that in the setup. I'll be devoting serious thought to this... Thanks for the suggestion!
 

Spud

Occasional Visitor
Happy to assist! :)


I'd definitely be interested to find out if you come across this situation again, and determining where the missing VPN server is located!


Right now it's only able to handle 1 country at a time... and it would need to be using VPN slots 1 and 2 in your case. VPNMON-R2 doesn't really care how you have your VPN setup, so if you want to use VPNMGR/killswitches, that's your call! But that's a great idea, @Spud! Wow... Let me think through how I could make this happen. Perhaps assigning a different country to each 1-5 VPN slot, and let you specify that in the setup. I'll be devoting serious thought to this... Thanks for the suggestion!
All the best with your next iteration, sounds like it could be very useful.

If I hit more issues I’ll report back, hopefully with some logs next time.

Cheers!
 
Last edited:

Viktor Jaep

Very Senior Member
All the best with your next iteration, sounds like it could be very useful.

If I hit more issues I’ll report back, hopefully with some logs next time.

Cheers!
Hi @Spud!

I wanted to let you know that I have tried to tackle your issue/request, and hoping that this might be a good enough compromise. So first the good news.

1.) I have added the capability of using up to 3 different countries in total which get randomly selected during a reset sequence.
2.) When a random country is selected, it will import the entire range of NordVPN servers for that country, and whitelist them in Skynet.

Now for the bad news...

1.) It really goes against the spirit of VPNMON-R2 to statically use selected VPN slots... like your #2 and #5 that you currently have in use, with one using a killswitch. In a more usable scenario, you would really need to move these to slots #1 and #2, and they would need to be configured identically. So if one is configured with a killswitch, they both would need to be configured in that same manner. The reason is, when you randomize, and when you use loops in code, it really helps when things are in sequential order... Slots that are statically assigned just don't seem to work well in this manner. I'll keep thinking about how to get around this issue, but for now, really, the easiest thing is to just use the VPN slots in their intended sequence... 1... 2... in use with 3, 4, 5 blank/disabled.

But the bad news really isn't all that bad...

In actuality, it would be beneficial for you to configure all 5 VPN slots identically... as this increases your random pool and ability to connect to a random city. If it picks 5 different cities across the UK vs. 2 cities... you just have a much bigger pool to pick from. And each VPN slot is just configured identically in this case. That's really how I have mine setup as well. In your case, those 5 slots would get prefilled with random UK cities... or random GE cities that it would pick 1 from to connect to.

The other thing to know is that the moment VPNMON-R2 determines that your link is down with that killswitch enabled, it will kill that VPN, and immediately try to reconnect again, limiting any traffic exposure to an absolute minimum.

I have uploaded a beta version out there for you to try if you want to give this a go? If you have good luck with it, I'll publish it in a few days. I have also added the capability for it to interface with YazFi, and automatically updates the guest networks with the current VPN slot, if you use this tool as well. Here's some more info/screenshots about that: https://www.snbforums.com/threads/r...alth-of-your-vpn-connection.76748/post-760236

Let me know if I can be of any assistance to help you through this, OK? Once you copy it over, make sure you rename it back to "vpnmon-r2.sh", run the script with the "-config" flag. During the setup process it will ask if you want to use multiple countries, and lets you specify which ones.

Here's the script: https://raw.githubusercontent.com/ViktorJp/VPNMON-R2/main/vpnmon-r2-1.4beta.sh
 

Spud

Occasional Visitor
Hi @Spud!

I wanted to let you know that I have tried to tackle your issue/request, and hoping that this might be a good enough compromise. So first the good news.

1.) I have added the capability of using up to 3 different countries in total which get randomly selected during a reset sequence.
2.) When a random country is selected, it will import the entire range of NordVPN servers for that country, and whitelist them in Skynet.

Now for the bad news...

1.) It really goes against the spirit of VPNMON-R2 to statically use selected VPN slots... like your #2 and #5 that you currently have in use, with one using a killswitch. In a more usable scenario, you would really need to move these to slots #1 and #2, and they would need to be configured identically. So if one is configured with a killswitch, they both would need to be configured in that same manner. The reason is, when you randomize, and when you use loops in code, it really helps when things are in sequential order... Slots that are statically assigned just don't seem to work well in this manner. I'll keep thinking about how to get around this issue, but for now, really, the easiest thing is to just use the VPN slots in their intended sequence... 1... 2... in use with 3, 4, 5 blank/disabled.

But the bad news really isn't all that bad...

In actuality, it would be beneficial for you to configure all 5 VPN slots identically... as this increases your random pool and ability to connect to a random city. If it picks 5 different cities across the UK vs. 2 cities... you just have a much bigger pool to pick from. And each VPN slot is just configured identically in this case. That's really how I have mine setup as well. In your case, those 5 slots would get prefilled with random UK cities... or random GE cities that it would pick 1 from to connect to.

The other thing to know is that the moment VPNMON-R2 determines that your link is down with that killswitch enabled, it will kill that VPN, and immediately try to reconnect again, limiting any traffic exposure to an absolute minimum.

I have uploaded a beta version out there for you to try if you want to give this a go? If you have good luck with it, I'll publish it in a few days. I have also added the capability for it to interface with YazFi, and automatically updates the guest networks with the current VPN slot, if you use this tool as well. Here's some more info/screenshots about that: https://www.snbforums.com/threads/r...alth-of-your-vpn-connection.76748/post-760236

Let me know if I can be of any assistance to help you through this, OK? Once you copy it over, make sure you rename it back to "vpnmon-r2.sh", run the script with the "-config" flag. During the setup process it will ask if you want to use multiple countries, and lets you specify which ones.

Here's the script: https://raw.githubusercontent.com/ViktorJp/VPNMON-R2/main/vpnmon-r2-1.4beta.sh

Hi Viktor,

I really appreciate your effort, but I’m going to be a pooper - sorry! Since whitelisting more countries with your earlier script, there hasn’t been a NordVPN block in Skynet, no dying connections, and no reconnect errors. Page load times are also snappier.

I’m doing nothing. Before, I’d be restarting clients up to three or four times a day. Often I’d have to reboot.

It’s basically fantastic, so I’m going to leave it well alone and enjoy :)
 
Last edited:

Viktor Jaep

Very Senior Member
Hi Viktor,

I really appreciate your effort, but I’m going to be a pooper - sorry! Since whitelisting more countries with your earlier script, there hasn’t been a NordVPN block in Skynet, no dying connections, and no reconnect errors. Page load times are also snappier.

I’m doing nothing. Before, I’d be restarting clients up to three or four times a day. Often I’d have to reboot.

It’s basically fantastic, so I’m going to leave it well alone and enjoy :)

Lol, it's not a problem. It will be there for when you might need it in the future. ;) Thanks for making the suggestion, and contributing to the VPNMON-R2 project to make it even better! I'm really glad to hear everything is working smoothly on your end!!
 

Spud

Occasional Visitor
Lol, it's not a problem. It will be there for when you might need it in the future. ;) Thanks for making the suggestion, and contributing to the VPNMON-R2 project to make it even better! I'm really glad to hear everything is working smoothly on your end!!
Well, I just had my first NordVPN block ;). A little investigation revealed some IP trickery.

The server was 188.241.156.217, which shows up as UK, run by Hydra Communications. Oddly, Skynet blocked it because it's in my banned country ranges, but the UK isn't banned...

I pulled the global NordVPN server list (5.5K entries), but it doesn't appear.

The privacy API feature on this website shows it's a relay to an address in India (click "read more").

I'd banned a range of countries reported as top sources of attacks, which included India. Now I know, and it's off the ban list.

Have a great weekend :cool:
 

Viktor Jaep

Very Senior Member
Well, I just had my first NordVPN block ;). A little investigation revealed some IP trickery.

The server was 188.241.156.217, which shows up as UK, run by Hydra Communications. Oddly, Skynet blocked it because it's in my banned country ranges, but the UK isn't banned...

I pulled the global NordVPN server list (5.5K entries), but it doesn't appear.

The privacy API feature on this website shows it's a relay to an address in India (click "read more").

I'd banned a range of countries reported as top sources of attacks, which included India. Now I know, and it's off the ban list.

Have a great weekend :cool:
Some good sleuthing there, @Spud! Yeah, I just tried that "reverse nordvpn lookup" api script and it comes up blank. Looks like Hydra could have recently acquired a new netblock which was assigned to India, and may take some time before everything reflects the "UK". Still, doesn't make your life any easier, right? :)
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top