spdMerlin spdMerlin v4.4.14 [2025-Jul-12] - Internet Speedtest and Monitoring Tool for AsusWRT Merlin

[Martinski]

I totally agree. It's good to hear other opinions


I apologize for any inconvenience!

No worries, and there's no need to apologize. We simply have a different perspective; no harm was done, and you have nothing to apologize for.

 

[Martinski]

I just split my scroll screenshot into 3 proper screenshots in the original post. If you or any other user would ever need it.

Yes, those new screenshots are absolutely much better. Now, I assume that all 3 were taken *after* the problem had already occurred. Perhaps you can provide the output of the shell script now, while the problem still exists, and before you try to recreate it. There might still be some clues there.

 

[Tarek Yag]

Now, I assume that all 3 were taken *after* the problem had already occurred.

Exactly!

There might still be some clues there.

Unfortunately, I reconfigured the script an hour ago. I'll do the job at my earliest free time.

 

[Tarek Yag]

while the problem still exists, and before you try to recreate it. There might still be some clues there.

I tried to reproduce the problem but didn't succeed after a few tries. I'll try to figure it out as soon as possible.

Anyhow, I've got hundreds of SQLite error lines during one of my trials. I'll send you a DM with the error log from spdMerlin and the two debug outputs from the shell script you provided. I believe the logged errors are very related to the loss of configuration and data, and maybe I'll lose them in a couple of hours of automatic speed tests, because of all the errors that have happened.

 

[Martinski]

I tried to reproduce the problem but didn't succeed after a few tries. I'll try to figure it out as soon as possible.

Anyhow, I've got hundreds of SQLite error lines during one of my trials. I'll send you a DM with the error log from spdMerlin and the two debug outputs from the shell script you provided. I believe the logged errors are very related to the loss of configuration and data, and maybe I'll lose them in a couple of hours of automatic speed tests, because of all the errors that have happened.

Whenever you get SQLite3 errors, please provide the separate SQLite log file along with the syslog messages that you've already provided. This separate debug log file includes more details.

Debug logfile default location: /opt/share/tmp/

 

[Tarek Yag]

Hello again, and excuse me for the very long delay in this issue update.
Back in the time when I first reported the complete data loss issue, and before having enough time to do more troubleshooting and pin-point the bug by reproducing the problem, real-life drama kicked-in and took my concentration far away, well, at least for a while only.

Now back to the issue, I had a few spdMerlin's complete data loss incidents and was able to unintentionally reproduce the reported issue a few times, but without taking any notes of the exact steps that led to the exact same issue. Moreover, one of the quick troubleshooting times I spent back then, led to spdMerlin producing an endless number of SQLite errors after making a manual speed test.

At the very end, after a few quick troubleshooting tries, utilizing installing and uninstalling the script a few times, I ended up being unable to even reinstall spdMerlin where I got endless SQLite errors within the install process, noting that I tried to install it from within AMTM, but this is absolutely unrelated to the issue.

Currently, I have a super unstable internet connection throughout the day (due to a country-wide network congestion), though it gets better at night. So, for best results, I manually deleted all residue script files, both from JFFS and USB, before making a clean install two days ago.

Now, I'm inspecting the following issues that all seem (at least to me) to be related to the same bug:

1- Automatic speed test for interface VPNC1 (the only one I use) gets re-enabled every time I disable it in the script's config, most probably with every WAN disconnection, hence probably with every OpenVPN event. I'm working on figuring it out...

2- For the mean time, I'm avoiding changing any script configuration from the Web GUI, because since I first reported the issue, every time I reproduced the issue was when I changed config from the Web GUI. It might not be related at all, but I'm trying to rule out more scenarios. I'll work again on reproducing the complete data loss issue right after figuring out what exactly is repeatedly re-enabling VPNC1 automatic speed test.

3- Endless SQLite errors back then seemed to appear after I started having complete data losses, so I'll begin troubleshooting this after getting done with the previous two points of inspection.


I'll post any updates here as soon as I have exact details and issue reproduction steps!

 

[Tarek Yag]

After many hours of troubleshooting, code understanding, fixing, testing, and then doing my very first code commit in GitHub.
Finally, I submitted my very first pull request to the develop branch of spdMerlin script to address the bug of losing the "excluded" mark on interfaces when their status switch between up and down, with a minor improvement to relevant code.

Next, I'll start troubleshooting the data loss issue.. after a good break!

 

[ExtremeFiretop]

After many hours of troubleshooting, code understanding, fixing, testing, and then doing my very first code commit in GitHub.
Finally, I submitted my very first pull request to the develop branch of spdMerlin script to address the bug of losing the "excluded" mark on interfaces when their status switch between up and down, with a minor improvement to relevant code.

Next, I'll start troubleshooting the data loss issue.. after a good break!

Hi @Tarek Yag

I saw this message and went to check OSR to start a quick code review and noticed no pull requests opened (or closed) against OSR.
Just an FYI in case your waiting on something, we don't see any pull request on our end.

Thanks!

 

[Tarek Yag]

@ExtremeFiretop Thank you for informing me about this.
As this is my very first time making a contribution in GitHub, would you please just tell me where should I start editing the file and in which repo exactly to submit the pull request correctly?
The change right now is in a fork I had to create within my account, as I got told that I have no write access to the repository, so I'm a bit lost, a mini good hint would get me right on track.

You’re making changes in a project you don’t have write access to. Submitting a change will write it to a new branch in your fork tarek-yag/spdMerlin, so you can send a pull request.

 

[ExtremeFiretop]

@ExtremeFiretop Thank you for informing me about this.
As this is my very first time making a contribution in GitHub, would you please just tell me where should I start editing the file and in which repo exactly to submit the pull request correctly?
The change right now is in a fork I had to create within my account, as I got told that I have no write access to the repository, so I'm a bit lost, a mini good hint would get me right on track.

Sure; it can be a bit intimating when you get started.
Let me literally do a "explain it like I'm 5" version and bare with me if I explain something you already knew or have done.

1. Start by going to the desired repo and clicking "Fork" at the top of the page as seen below:

2. You'll notice this is creating a duplicate of the code in your own account.
- Make sure to UN-Select the "copy the master branch only" as you want both develop and master to play with.

3. Once it's done forking you'll find on your account the repo/project along with all the branches of the code.
- Make sure to select "Develop" to make your changes.

4. Once you've made your desired changes in the develop branch, click on "Compare & Pull Request" if the button is missing, you can also click on the "x commit ahead" as identified below.
- In my example below, I only edited the readme to add the word "TEST" and nothing else.

5. After clicking the "Compare & pull request" button or the "x commit ahead" you'll find this comparison screen below outlining all the changes and where to submit them.

6. Make sure before you click "Create Pull Request" that you include a title, a description of the change, and that it's going from your personal develop branch, to the upstream develop branch.
- This allows you to essentially "hand off" or "pass on" the develop changes upstream to the main repo, without it going directly into production.

Once we accept the pull request, the changes will be also found in our develop branch and will be "In Sync" with yours.
This allows us to review the code, test or modify as needed before pushing to production.

If you make your changes in your master branch and try to open a pull request against our master branch, it will automatically be denied by our workflow as we don't allow changes from outsiders to go straight to production.
Hope that clarifies!

 

[Tarek Yag]

Sure; it can be a bit intimating when you get started.
Let me literally do a "explain it like I'm 5" version and bare with me if I explain something you already knew or have done.

Thanks a lot for the big help! I just submitted it successfully!
I figured out that I was missing the very last step only

User Interfaces get misleading sometimes when you're not very familiar with the environment, I had everything ready earlier even the subject and description of the pull request, I just missed the exact page where to submit the pull request after I left it the first time.
There are too many ways to go through!

If you make your changes in your master branch and try to open a pull request against our master branch, it will automatically be denied by our workflow as we don't allow changes from outsiders to go straight to production.

Noted. I expected this by myself when I re-prepared my pull request to be my official first!
Well, this is a nice software developing methodology. Good luck everyone on board!

 

[ExtremeFiretop]

Thanks a lot for the big help! I just submitted it successfully!
I figured out that I was missing the very last step only

User Interfaces get misleading sometimes when you're not very familiar with the environment, I had everything ready earlier even the subject and description of the pull request, I just missed the exact page where to submit the pull request after I left it the first time.
There are too many ways to go through!

I fully understand and no worries. I can confirm I immediately got email notified as soon as you opened the PR against OSR. So seems you got it right this time

I myself almost exclusively used the GitHub website and it's interface for MerlinAU up until I started helping with OSR and realized it was a bit of a mess to handle multiple projects.

I now mostly use the GitHub CLI or the GitHub Desktop app. Both of which can do any of the steps identified above, and more.

Sometimes there's some tasks that simply cannot be done through either the GitHub Desktop app or the GitHub webpages. In those instances, you need to use the GitHub CLI commands which are the most powerful to get the desired result. However as with anything, more power can mean more potential mistakes.

Noted. I expected this by myself when I re-prepared my pull request to be my official first!
Well, this is a nice software developing methodology. Good luck everyone on board!

Happy you understand. We made the call early on to allow people to contribute freely to the develop branches, but you need to be apart of OSR to do anything with the master branches, it's just safer that way for everyone involved.

 

[Tarek Yag]

Now it's time to go back to spdMerlin issues!

After many hours of continuous troubleshooting, which I lost count already, I figured out the root cause of all consequent issues caused by the complete data loss bug! As I had already expected based on experience, and even speculated since the very beginning; At the end of the troubleshooting process, it all made sense to me, finally!

I'll jump straight to conclusions as it would take so long and too much effort to explain all the experiments, processes, and results.

2- For the mean time, I'm avoiding changing any script configuration from the Web GUI, because since I first reported the issue, every time I reproduced the issue was when I changed config from the Web GUI. It might not be related at all, but I'm trying to rule out more scenarios. I'll work again on reproducing the complete data loss issue right after figuring out what exactly is repeatedly re-enabling VPNC1 automatic speed test.

Actually, avoiding changing script config from the Web GUI at the time of inspecting the first issue (excluded interfaces issue), allowed me to get very oriented in troubleshooting the root cause of both bugs. Now that I believe I would never be able to reproduce the data loss bug from within the CLI, I saved myself a lot of time already!!

Straight to the root cause of all consequent issues, many duplicate files are left at the original storage location (JFFS) when the user moves the script storage location from JFFS to USB using the CLI (didn't try from the GUI) caused -and not limited to- the following issues:

1- The script randomly switches back from USB to JFFS storage when updating settings using Web GUI, driven by so many factors (which I pin-pointed some in my own setup and was able to reproduce the complete data loss issue so many times).
Factors include suspended and zombie processes, environment variables, code library differences, actual runtime processes, CPU workload, other system scripts, spdMerlin file attributes, and spdMerlin/SQLite file and memory locks, etc.

2- Different residue files in the JFFS storage folder lead to different ways and results of data loss when the script randomly switches back from USB storage to JFFS storage. So, the more residue files, the more possibilities of data loss you get.

3- One root cause (which made me the "luckiest" person to face this bug nightmare), a very unstable internet connection causes and accumulates issues and errors (in memory and file system) while doing more and more speed tests, all these make the perfect combination in the running router environment to lead to the script switching back to JFFS storage at the earliest config change from Web GUI. So, the better the internet connection the less possibility to encounter the bug.

[EDIT 1: Added]
4- Very unstable internet connections can also cause many SQLite errors even before the first faulty switch of storage location, and it even provides another perfect combination of factors to lead to the same bug. That's when I faced the bug twice or thrice with normal and usual script use, even before I first reported the bug about 2 months ago.

3- Endless SQLite errors back then seemed to appear after I started having complete data losses, so I'll begin troubleshooting this after getting done with the previous two points of inspection.

5- After the script switches itself back to JFFS storage, you'll start getting into real trouble through accumulating other errors and issues and that's when SQLite errors kick-in very seriously!! Even if you keep experimenting and testing, you'll end up with a total mess that might prevent you from even re-installing the script without going through a manual cleaning for all residue files both in JFFS and USB script storage locations.

I tried very very hard to sum up so so many information and experiments I did one by one to rule out causes and confirm others, and did my best by writing this update post over 3 typing sessions. So, please forgive me for any missing details that I'm open to provide whenever asked about, [EDIT 1: Added] as it seemed so overwhelming to me to summarize an endless number of possibilities and an uncountable number of tests and experiments I already did.

In conclusion, the ULTIMATE solution for this nightmare for me and for all other AMTM scripts' users regarding the data loss issue in spdMerlin, is that AMTM-OSR Team (@thelonelycoder, @ExtremeFiretop, @Martinski, @SomeWhereOverTheRainBow, and others) need to review and make strategic modifications to do perfect file clean-ups when moving storage location inside their scripts.

To be continued...

 

[Tarek Yag]

...
A unified method or algorithm for all AMTM-OSR scripts would be a great and perfect solution for everyone, since the following came to my attention too within the long troubleshooting process:

1- dn-vnStat script actually doesn't keep or maintain ANY duplicate file names between its JFFS and USB storage locations

2- The prompts to keep config and data files when uninstalling dn-vnstat and spdMerlin scripts, keeps the door wide open for more residue and duplicate file issues to users doing re-installs anytime later, especially that these scripts don't move kept files to a special leftover (.old/.keep/.backup) folder, isn't it a decent suggestion though?

3- Again, within dn-vnStat and spdMerlin scripts' uninstall processes, the prompt question itself in one script is the exact opposite in the other, one asks "Do you want to keep these files?" and the other asks "Do you want to delete these files too?", so, same answers opposite results!!
Hold on! I actually fell a few times into this linguistic trap for real when I was experimenting with both scripts in the past 3 days, as we all get used to answering these questions everywhere in a glimpse of an eye, so much better to get it unified within the same team and sibling products.

Finally, excuse me for this art attack as it all came-in at once!!
I appreciate y'all and I do really love this community and all the great products it makes.

Yours Sincerely,

 

[Ripshod]

It's strange. I've never had any issues with the current (and several previous) version of spdMerlin, not even when changing settings in the GUI.
Is there any specific setting and value I can try so I could duplicate what you're experiencing?

 

[Tarek Yag]

It's strange.

I know and I totally understand. Please keep in mind the huge technology advancement you have in your hands (RT-BE88U) compared to my ancient RT-AC68U router, which is barely keeping-up and struggling still. Can you just imagine that the process of "Retrieving data for WebUI charts..." step takes a bit less than a minute each time I get a successful speed test? See?
Accumulating this with all other factors, I start having frozen/zombie/suspended processes, until they pile up and cause bigger issues, as I explained earlier. So, optimizing the script and its stored files should solve all issues.

I've never had any issues with the current (and several previous) version of spdMerlin, not even when changing settings in the GUI.

I totally agree with you, even I myself didn't have any issues before I lost the last hope of getting a better internet connection in the near future, and then everything went to the worse and is still going to the worst!!
So, for a higher end router and a better connection I doubt this could be a problem, but since it occurs on slower devices and connections (slower execution), then software should be optimized to prevent any issues for everyone.

Is there any specific setting and value I can try so I could duplicate what you're experiencing?

Unfortunately, it's not a specific setting or value at all. I was able to make the script crash and lose data by simply submitting the same settings from the Web GUI while having duplicate files in both JFFS and USB storage locations.

However, diving into more details for your curiosity, I do have two compressed files each one containing the spdmerlin.d folder from /jffs/addons/ and /opt/share/ folders, respectivly. In the final stages of my testing and experimenting, each time, I clean up everything and kill all related processes then decompress both files each to its original place and test another crash scenario. That's exactly how I was able to conclude item #2 in my detailed post, and was even able to avoid the faulty storage location switch:

2- Different residue files in the JFFS storage folder lead to different ways and results of data loss when the script randomly switches back from USB storage to JFFS storage. So, the more residue files, the more possibilities of data loss you get.

As these files contain my own internet speed test results over a period of time, I'm open to providing these files and the code I used for clean-up between tests, and even more specifics to the AMTM-OSR Team and/or forum admins.

Thanks for your contribution!

 

[Ripshod]

That's a shame. Wouldn't hurt to have other eyes on this issue

*edit* I too have remnants of spdMerlin in the /jffs/addons folder, mainly just htm files for building the results graphs. All csv results are stored on the USB.

 

[Tarek Yag]

Wouldn't hurt to have other eyes on this issue

I totally agree! I'm still awaiting OSR team replies anyways.

And to be honest, I'm eager to do a few more quick tests, and to try a specific process I have in mind to trigger the bug in any normal setup, but you know, time is a cost and I've already spent a lot of time and effort into this and I'm already tired now.

I'll see if I can spend a little more time on this project in the following 48 hours. However, even if I become able to trigger the whole issue with a single step, hence a single spot-on fix for this bug, all other code fixes and improvements are still critical as I already said and explained.

 

[Tarek Yag]

*edit* I too have remnants of spdMerlin in the /jffs/addons folder, mainly just htm files for building the results graphs. All csv results are stored on the USB.

Thanks A LOT for bringing this up for me! While we were rapidly busy replying one after another, I missed your post edit!! This now should make a HUGE difference to you and the others, as it will surely show every single issue I had and every effort I spent, on a whole new level of clarity for everyone!!

Since what you mentioned are the only files you have in your JFFS location for spdMerlin, then just take a look, or maybe a sneak peek, at what I (((normally))) have in my own setup for both connmon (using USB storage) and spdMerlin (before and while troubleshooting)...

Repeating again, both scripts have their storage location set to USB and are actually adding new records to USB...

 

[Tarek Yag]

@Ripshod, I'm very grateful for your post edit, as it incidentally uncovered the endless possibilities and testing scenarios I had to deal with over the course of many hours of continuous troubleshooting having all these extra remnant files in my setup and other factors too (which must be the same case for all similar setups on similar older routers too).

This now definitely clarifies and even proves much -if not all- of my previous conclusions and recommendations, I guess all readers and especially OSR Team members will get their brains unleashed after finally reaching my last post here, as the big picture might not have been clear to them the whole time, especially they each have a very modern and cutting-edge Tech routers!

Words can't thank you enough right now!! To be honest, ever since I completed this bug report, I've always had this feeling that there's something missing to successfully deliver the big idea, but I never thought that this was it! It wouldn't have been possible without your post edit!

Readers and developers now be like:
- Oh! Config files in two locations!!
- Oh, no! Data in two locations, too!!
- Wow! How config file discovery should work now?!!
- Daym.. duplicate core files everywhere!!
- What if it reads config from USB and records data to JFFS!!!
- .... etc.

There you go! Imagine what is happening right now here in my router and in the older part of the world!
OSR Team, It's time to step in now! These files all remained after a straight forward clean install from within AMTM after a factory reset, and a formatted ext4 USB flash, and all settings changed from within the scripts!