VPNMON VPNMON-R3 Custom OVPN Server List Generation Tutorials and Examples

[Viktor Jaep]

I guess this was the case indeed. I read all the way down that issue page looking for answers, as I was really concerned about v1 suddenly getting removed as it's already considered deprecated. Well, there you go, now!

LOL, yeah, to my surprise.

Thank you for your time in advance. You might already know from my other contributions, I'm always ready whenever you need testing and/or troubleshooting with changes or updates, I'll do whatever I have to do. But in this case particularly, before installing the script

Well, if it wasn't for Claude, I would have still been stuck on the first statement. Please go ahead and take a look. I've reformulated each statement for the v2 API. Please note, these are only for UDP OVPN. I've also fixed a bunch of the Wireguard ones, and will post that on the other thread for @iTyPsIDg to take a look at.

Feel free to try VPNMON-R3 with these v2 statements. Let me know if I can be of any help!

 

[Tarek Yag]

Feel free to try VPNMON-R3 with these v2 statements. Let me know if I can be of any help!

That is totally great, good job! I just quickly reviewed your changes in the first post.
Now I can and will try VPNMON-R3 at my earliest convenience within the next 48 hours.
I have a great idea for this thread that I'll provide after I finish setting up my VPNMON instance. Stay tuned!

 

[Tarek Yag]

I just started learning and trying to generate my own server list before installing VPNMON-R3. So, this is the first time I deal with the generation commands for both v1 and v2.
I noticed a few things, including an issue, and a huge difference in performance when using v1 versus v2, in terms of input/output size, and processing time.
I already figured out the cause of the issue I experienced. I'll do some research and get back here with results and recommendations as soon as possible, within a few hours.

 

[Viktor Jaep]

I noticed a few things, including an issue, and a huge difference in performance when using v1 versus v2, in terms of input/output size, and processing time.

The idea is to only have this run 1x a day, typically in the middle of the night so you wouldn't have to deal with any issues in processing time. For me, running it against NordVPN and returning 2909 possible servers runs in a matter of a few seconds, so I'm not terribly disappointed.

I already figured out the cause of the issue I experienced. I'll do some research and get back here with results and recommendations as soon as possible, within a few hours.

Would love to know what you encountered. Thanks for contributing!

 

[Tarek Yag]

Sorry for my late feedback, but I got carried away with work and a lot of errands last week. And since I wanted to give an accurate and well written feedback, I decided to do it later. So here it is...

I'll start with when I first tried the commands of the V2 API. The first command took very long until I got any process feedback (as it was a silent cURL call), until more than a minute, I got a jQuery error stating that the syntax is invalid (referring to its input value originally retrieved from web). I tried it again and it worked the second time but it took almost the same amount of time to show the output.

So, I had to try the V1 API command and it worked much quicker, unlike the V2 API command. I completed my experiment to evaluate the average success rate, and it was almost a 50/50 success rate with the V2 API command. Given this, I wanted to verify what was the bottleneck in my test, and to my surprise it was both my connection and the jQuery process. I ruled this out by downloading the NordVPN API output to a local file for both versions.

I was shocked as soon as I discovered the data size difference between the output of each API version. For your reference, I just downloaded the output of the commands in your first post (and more) to self-indicating file names.

-rw-rw-rw-    1 atr      root        337202 Nov 28 17:53 v1.servers
-rw-rw-rw-    1 atr      root         30223 Nov 28 17:53 v1.servers.countries
-rw-rw-rw-    1 atr      root         43565 Nov 28 17:53 v1.servers.recommendations
-rw-rw-rw-    1 atr      root       8887860 Nov 28 17:56 v2.servers.16354.limit
-rw-rw-rw-    1 atr      root        109819 Nov 28 17:53 v2.servers.default.limit

These files downloaded now with a max single connection speed of 50 Kbps (unexplained speed limit), which made it take forever to download the 8.47MB v2.servers.16354.limit file, and it failed once! Which totally explains the very long time it took to execute the V2 API commands in the first post of this thread.
So, in my case I must use V1 API calls specifically, as V2 API output is too large for my connection and would fail execution very often.

I'll do some research and get back here with results and recommendations as soon as possible, within a few hours.

I tried to figure out a way to get the URL's filters GET parameter to work with V2 API, but I failed to get it to work even though I came up with some ideas. Have you made any trials within this approach?

The idea is to only have this run 1x a day, typically in the middle of the night so you wouldn't have to deal with any issues in processing time.

Just a quick question; is it possible to manually specify the time of the API call from within the script? I'm asking because I usually have a high data usage until late night.

I have a great idea for this thread that I'll provide after I finish setting up my VPNMON instance. Stay tuned!

I wanted to suggest that you start a new thread specific for NordVPN V2 API calls, or moving them to another post within this thread, in order to split the instructions of each API version. It got me super confused reading through the first post, when I first started experimenting with the commands. Additionally, the first post has an improper text formatting for Dark Mode view of this forums (See the attachments)

 

[Viktor Jaep]

I'll start with when I first tried the commands of the V2 API. The first command took very long until I got any process feedback (as it was a silent cURL call), until more than a minute, I got a jQuery error stating that the syntax is invalid (referring to its input value originally retrieved from web). I tried it again and it worked the second time but it took almost the same amount of time to show the output.

So, I had to try the V1 API command and it worked much quicker, unlike the V2 API command. I completed my experiment to evaluate the average success rate, and it was almost a 50/50 success rate with the V2 API command. Given this, I wanted to verify what was the bottleneck in my test, and to my surprise it was both my connection and the jQuery process. I ruled this out by downloading the NordVPN API output to a local file for both versions.

These files downloaded now with a max single connection speed of 50 Kbps (unexplained speed limit), which made it take forever to download the 8.47MB v2.servers.16354.limit file, and it failed once! Which totally explains the very long time it took to execute the V2 API commands in the first post of this thread.
So, in my case I must use V1 API calls specifically, as V2 API output is too large for my connection and would fail execution very often.

I tried to figure out a way to get the URL's filters GET parameter to work with V2 API, but I failed to get it to work even though I came up with some ideas. Have you made any trials within this approach?

Just a quick question; is it possible to manually specify the time of the API call from within the script? I'm asking because I usually have a high data usage until late night.

You might benefit from tweaking the curl parameters a bit more seeing that you might be coming from a low bandwidth connection. By default, I'm using:

curl --silent --retry 3 --connect-timeout 3 --max-time 30 --retry-delay 1 --retry-all-errors "https://api.nordvpn.com/v2/...

Perhaps you should try:

curl --silent --retry 3 --connect-timeout 3 --max-time 120 --retry-delay 1 --retry-all-errors "https://api.nordvpn.com/v2/...

But obviously tweak it to your particular needs.

I wanted to suggest that you start a new thread specific for NordVPN V2 API calls, or moving them to another post within this thread, in order to split the instructions of each API version. It got me super confused reading through the first post, when I first started experimenting with the commands. Additionally, the first post has an improper text formatting for Dark Mode view of this forums (See the attachments)

Sorry about the text formatting. I don't use dark mode, but I fixed this up for you... guess I was specifying a black color instead of default: none. I haven't decided to remove the V1 instructions just yet until NordVPN releases some news when they will be deprecated. I'll work on clearing up the language a bit more to help reduce confusion. Thanks for letting me know!

 

[Tarek Yag]

Oops!
I just found out that other V1 API calls in the first post -which I didn't test earlier- uses a limit parameter to get the full list of servers, and I was horribly shocked again that it's very much larger than V2 API output. Outputting 28.19 MB for V1 API compared with 8.47 MB for V2 API), which expands the issue further more! OMG!

I'm already thinking of creating a web-based solution hosted on my own website which hosts and updates the only data we need to query for use in VPNMON-R3.
I don't know if you're interested in a long-term project. But, I can offer the free creation and/or permanent hosting of a PHP script with a special meaningful URL (using merlin subdomain name and vpnmon directory name) on my own IT Consulting Services website, with permanent special FTP access for you. A PHP script that handles NordVPN server list conditional queries, which users can customize to their choice through URL GET parameters, that gives a ready-to-use list of servers for your VPNMON-R3.

Free for all, accepting donations, and can always be expanded to other scripts and developers within Merlin Addons community, and I win partially promoting my website in the whole process. I think it should be a win-win deal, let me know what you think of this in a DM, and we can discuss this further more in full details.

 

[Tarek Yag]

Sorry about the text formatting. I don't use dark mode, but I fixed this up for you

No worries, I already guessed that you don't use dark mode. It got fixed now, thank you!

I'll work on clearing up the language a bit more to help reduce confusion. Thanks for letting me know!

I really appreciate it in advance!

 

[Viktor Jaep]

Oops!
I just found out that other V1 API calls in the first post -which I didn't test earlier- uses a limit parameter to get the full list of servers, and I was horribly shocked that it's very much larger than V2 API output (28.19 MB), which expands the issue further more! OMG!

I'm already thinking of creating a web-based solution hosted on my own website which hosts and updates the only data we need to query for use in VPNMON-R3.
I don't know if you're interested in a long-term project. But, I can offer the free creation and/or permanent hosting of a PHP script with a special meaningful URL (using merlin subdomain name and vpnmon directory name) on my own IT Consulting Services website, with permanent special FTP access for you. A PHP script that handles NordVPN server list conditional queries, which users can customize to their choice through URL GET parameters, that gives a ready-to-use list of servers for your VPNMON-R3.

Free for all, accepting donations, and can always be expanded to other scripts and developers within Merlin Addons community, and I win partially promoting my website in the whole process. I think it should be a win-win deal, let me know what you think of this in a DM, and we can discuss this further more in full details.

I obviously can't stop you from doing what you want to do, but I would advise you to approach this from a pure hobbyists angle, providing this as a PoC free service in the interest of helping others with some particular need. It would be a win-win for you as this solution would give you the needed fix instead of having to query a 28MB file, while providing some additional functionality for users while getting some more PHP experience in building something like this.

I personally have never run into any issues downloading the full list 1x a day in order to refresh my Nord server lists... nor have I ever heard of others bringing this up as a concern, but perhaps others with extremely low bandwidths would benefit from this? Perhaps you should create a new thread with a poll to determine interest?

I'm also not terribly interested in having to customize or modify VPNMON-R3 to work in conjunction with this PHP script... but if there's a way for people to configure a list for download in the required format that can be CURL'd from the VPNMON-R3 interface, then that might work. Either that, or building a local VPNMON-R3 Server List Generator helper script that interfaces with your PHP solution, that can be called with cron... but again, you would really need to determine the demand and need of others out there before sinking a lot of time and effort into this?

 

[Tarek Yag]

but perhaps others with extremely low bandwidths would benefit from this? Perhaps you should create a new thread with a poll to determine interest?

Good idea!

I'm also not terribly interested in having to customize or modify VPNMON-R3 to work in conjunction with this PHP script

I guess you didn't get my idea properly. No changes in VPNMON-R3 needed whatsoever.

but if there's a way for people to configure a list for download in the required format that can be CURL'd from the VPNMON-R3 interface, then that might work.

This exactly what I was talking about. A single cURL command (replacing the current cURL + jq commands) to a single URL (like merlin.domain.com/vpnmon/servers?country=United+StatesS&city=New+York) which retrieves a servers list customized by every user that requires no further processing at all. Now that I just clarified the idea, rethink my offer and let me know, please.

Still there are two questions you might have missed answering...

I tried to figure out a way to get the URL's filters GET parameter to work with V2 API, but I failed to get it to work even though I came up with some ideas. Have you made any trials within this approach?

Just a quick question; is it possible to manually specify the time of the API call from within the script? I'm asking because I usually have a high data usage until late night.

Thanks for your time.

 

[Viktor Jaep]

This exactly what I was talking about. A single cURL command (replacing the current cURL + jq commands) to a single URL (like merlin.domain.com/vpnmon/servers?country=United+StatesS&city=New+York) which retrieves a servers list customized by every user that requires no further processing at all. Now that I just clarified the idea, rethink my offer and let me know, please.

It might just work... but again, I'd use a poll to check for demand before going down this route.

Still there are two questions you might have missed answering...

Tarek Yag said:
I tried to figure out a way to get the URL's filters GET parameter to work with V2 API, but I failed to get it to work even though I came up with some ideas. Have you made any trials within this approach?

Tarek Yag said:
Just a quick question; is it possible to manually specify the time of the API call from within the script? I'm asking because I usually have a high data usage until late night.

I posted a few curl statements earlier that might help in your situation. I'm not using any GET statements. But with curl, you have some different parameters to work with to help extend the amount of time it takes before it times out, etc.

Hope that helps!

 

[Tarek Yag]

I'd use a poll to check for demand before going down this route.

Okay. Feel free to contact me in the future whenever you decide to give it a shot.

I'm not using any GET statements.

Yep, I noticed that you only used them with a single URL from the V1 API:

curl --silent --retry 3 --connect-timeout 3 --max-time 6 --retry-delay 1 --retry-all-errors "https://api.nordvpn.com/v1/servers/recommendations?filters\[country_id\]=228&limit=5" | jq --raw-output '.[].station'

That's why I wanted to ask whether you ever tried to use them with the V2 API, as using GET URL Parameters would have provided new possibilites for filtering the raw output, hence reducing the size dramatically.

But for my last question, I meant to ask whether there is an option in VPNMON-R3 to change the time of generating the server list automatically. Anyway, I'll figure it out as soon as I install the script as soon as possible.

 

[Viktor Jaep]

Okay. Feel free to contact me in the future whenever you decide to give it a shot.

Sorry... This was directed at you... for you to use a poll question, if you want to determine if there would be interest for the solution you're proposing. This is functionality that's built-into the SNB forum.

Yep, I noticed that you only used them with a single URL from the V1 API:

That's why I wanted to ask whether you ever tried to use them with the V2 API, as using GET URL Parameters would have provided new possibilites for filtering the raw output, hence reducing the size dramatically.

I don't believe I've ever used GET before in V1 or V2, but if you have any suggestions on how using it might provide better performance, I'm all ears.

But for my last question, I meant to ask whether there is an option in VPNMON-R3 to change the time of generating the server list automatically. Anyway, I'll figure it out as soon as I install the script as soon as possible.

Yes, you have this option... #7 under the configuration menu give you the option of refreshing custom server lists when using the vpnmon-r3 -reset command.

Using the "R" key on the operations menu allows you to set a cron job to run a -reset at the time of your choice.

 

[Tarek Yag]

Sorry... This was directed at you... for you to use a poll question

Yep, I already got it. I meant that I won't be doing a poll any soon, but whenever you get interested in proceeding with the offer, I can start soon enough without a poll.

I don't believe I've ever used GET before in V1 or V2, but if you have any suggestions on how using it might provide better performance, I'm all ears.

Ooops, I guess you already forgot about it. Please see the attached screenshot taken from your first post in this thread to find what I'm talking about. I guess now this is why you didn't fully understand me earlier.

Using the "R" key on the operations menu allows you to set a cron job to run a -reset at the time of your choice.

Great, this is what I was asking about. Thank you!

 

[Viktor Jaep]

Yep, I already got it. I meant that I won't be doing a poll any soon, but whenever you get interested in proceeding with the offer, I can start soon enough without a poll.

Thanks @Tarek Yag ... I'm not sure if we're dealing with a language barrier, or if I'm just not making myself clear, which is totally possible:

1. I can't stop you from doing whatever you want to do. You could start this project at any time... or wait... truly whatever you please, but it's completely up to you. I'm not going to tell you what to do.
2. I would be happy to give input or advice along the way if advice is needed, but not really interested in partnering or accepting any offers with you (or your company).
3. I was simply suggesting using a poll to ask the community if what you are proposing would be worth the time and effort to build. This would be a good first step before committing time and resources to doing whatever it is you want to accomplish.
4. Like I said before, I have not had any performance issues myself, and have never heard a single complaint from the community in all these years.
5. I gave you some previous insight on how to possibly treat this project of yours moving forward... making sure it's open to the community to help benefit others, but in the end this is something that you would want to commit time and energy to.

Ooops, I guess you already forgot about it. Please see the attached screenshot taken from your first post in this thread to find what I'm talking about. I guess now this is why you didn't fully understand me earlier.

I'm not sure what what you mean... this is the URL that we're pointing the CURL statement to. Unless you meant to say that it's missing a GET statement in there as well? Technically, there is no performance advantage to adding an explicit GET statement (using -X GET or --request GET) because CURL defaults to GET automatically.

 

[Tarek Yag]

I'm not sure what what you mean... this is the URL that we're pointing the CURL statement to.

Ah! I never thought that you didn't get the right terminology I'm using...

GET Parameters: Definition - Seobility Wiki

GET parameters are used when a client, such as a browser, requests a particular resource from a web server using the HTTP protocol.

www.seobility.net

So, "filters" within the URL string is the URL GET Parameter I'm pointing at since the very beginning. In that single URL from V1 API I told you about, you have "country_id" with the "filters" GET Parameter.
For more clarification terminology-wise on my side, when you execute a cURL command while using GET request method, it will simply append the supplied parameters (using --data) of the cURL command to the requested URL, then retrieve the output.
Please let me know if I can clarify any more. So, repeating the main old question again, did you try experimenting with the GET parameters of the URLs of the V2 API? In less professional words, did you play with the URL string or parameters?
I did briefly experiment with them, but never was able to filter the raw output.

but not really interested in partnering or accepting any offers with you (or your company).

Not a language barrier at all, and I already understood all the points you just mentioned again; maybe we both didn't clarify enough. The quote above is your answer that I already understood implicitly, so I told you -in different words- if you ever would like to partner with me for this project, I can proceed without doing any polls on my side. For the meantime, I won't start the project on my own. I hope and think it's all clear now for both of us.

 

[Tarek Yag]

For more clarification terminology-wise on my side, when you execute a cURL command while using GET request method, it will simply append the supplied parameters (using --data) of the cURL command to the requested URL, then retrieve the output.

curl - How To Use

curl.se