spdMerlin spdMerlin - v4.x - speedtest and bandwidth monitoring

Sebastienbo

Occasional Visitor
for the ookla binary that's available to download, its typically invoked by:
Code:
/jffs/addons/spdmerlin.d/ookla/speedtest --interface="$IFACE" --format="human-readable" --unit="Mbps"
where IFACE is eth0,tun11 and so on

The help output doesn't suggest one way or the other that core configuration is an option
Code:
[email protected]:/tmp/home/root# /jffs/addons/spdmerlin.d/ookla/speedtest -h
Speedtest by Ookla is the official command line client for testing the speed and performance of your internet connection.

Version: speedtest 1.1.1.28

Usage: speedtest [<options>]
  -h, --help                        Print usage information
  -V, --version                     Print version number
  -L, --servers                     List nearest servers
  -s, --server-id=#                 Specify a server from the server list using its id
  -I, --interface=ARG               Attempt to bind to the specified interface when connecting to servers
  -i, --ip=ARG                      Attempt to bind to the specified IP address when connecting to servers
  -o, --host=ARG                    Specify a server, from the server list, using its host's fully qualified domain name
  -p, --progress=yes|no             Enable or disable progress bar (Note: only available for 'human-readable'
                                    or 'json' and defaults to yes when interactive)
  -P, --precision=#                 Number of decimals to use (0-8, default=2)
  -f, --format=ARG                  Output format (see below for valid formats)
      --progress-update-interval=#  Progress update interval (100-1000 milliseconds)
  -u, --unit[=ARG]                  Output unit for displaying speeds (Note: this is only applicable
                                    for ‘human-readable’ output format and the default unit is Mbps)
  -a                                Shortcut for [-u auto-decimal-bits]
  -A                                Shortcut for [-u auto-decimal-bytes]
  -b                                Shortcut for [-u auto-binary-bits]
  -B                                Shortcut for [-u auto-binary-bytes]
      --selection-details           Show server selection details
      --ca-certificate=ARG          CA Certificate bundle path
  -v                                Logging verbosity. Specify multiple times for higher verbosity
      --output-header               Show output header for CSV and TSV formats

I know nothing about kernel limits, that's a question for @RMerlin to help us out with (please :) )!


the built-in ookla binary on the other hand, mentions
Code:
threadnum = "8"
packetlength = "32000000"
in its config, but i must admit i don't see much difference between the built-in ookla at /usr/sbin/ookla and the /jffs/addons/spdmerlin.d/ookla/speedtest when run against the same servers
Jack, & @RMerlin ,
I'll passthrough your comments to ookla

From what I see in your code, ookla configuration clearly would allow multiple cores to be used.
So the restriction must come from higher
  1. the shell (very likely)
  2. the kernel (doubtfull)
  3. the parent process (likely)
  4. the scripting environment (very likely)

Using multicores requires Multicores to be NOT restricted (It could have been restricted by setting affinity of a process or thread on one of the 4 ways herabove;
Affinity is the action of choosing a specific processor to allow the process to run in; For example a quad core has 4 cpu's named 0,1,2 and 3
In linux affinity can be set using the command called "cpuset" or "taskset" (so if you find this command somewhere it is worth to check if it is not restricted)
for example "taskset -c 0 mycommand" to use the first core OR "taskset -c 0,1 mycommand" to use the first two cores
You can also look if the kernel is configured for multicores with the command uname -> with parameter -a : SMP should be part of the response of that command (symmetric multi-processing)

If your script is an actual script like python or perl the affinity can also have been set globally in the script config
Ways to set affinity in the shell or scripts are explained in this forum:
As soon as you set affinity somewhere then binaries will not use all the cores anymore, the same thing happens if a parent process was configured with affinity, then all child processes will have that same affinity, until the affinity is turned off or the affinity is changed to more cores by using the taskset command.
I think you can also use the taskset command to see the current affinity of a process for troubleshooting.
And when affinity was set somewhere you can also break out of it by setting a new affinity for the ookla binary (an affinity that uses more cores or reset the affinity to all cores)
 
Last edited:

Jack Yaz

Part of the Furniture
Jack,

I'll passthrough your comments to ookla

From what I see in your code, ookla configuration clearly would allow multiple cores to be used.
So the restriction must come from higher
  1. the shell (very likely)
  2. the kernel (doubtfull)
  3. the parent process (likely)
  4. the scripting environment (very likely)

Using multicores requires Multicores to be NOT restricted (It could have been restricted by setting affinity of a process or thread on one of the 4 ways herabove;
Affinity is the action of choosing a specific processor to allow the process to run in; For example a quad core has 4 cpu's named 0,1,2 and 3
In linux affinity can be set using the command called "cpuset" or "taskset" (so if you find this command somewhere it is worth to check if it is not restricted)
for example "taskset -c 0 mycommand" to use the first core OR "taskset -c 0,1 mycommand" to use the first two cores
You can also look if the kernel is configured for multicores with the command uname -> with parameter -a : SMP should be part of the response of that command (symmetric multi-processing)

If your script is an actual script like python or perl the affinity can also have been set globally in the script config
Ways to set affinity in the shell or scripts are explained in this forum:
As soon as you set affinity somewhere then binaries will not use all the cores anymore, the same thing happens if a parent process was configured with affinity, then all child processes will have that same affinity, until the affinity is turned off or the affinity is changed to more cores by using the taskset command.
I think you can also use the taskset command to see the current affinity of a process for troubleshooting.
And when affinity was set somewhere you can also break out of it by setting a new affinity for the ookla binary (an affinity that uses more cores or reset the affinity to all cores)
Would it be possible to put me in direct contact with Ookla please? I can PM you an email address if that's how they're happy to communicate
 

elorimer

Very Senior Member
Neat!! Is there a way to make the URL clickable? Rather than cut and paste into a browser?
 

elorimer

Very Senior Member
I'm a dope. I ran a manual speedtest to see if it worked, got a url in the resultsbox that wasn't clickable. But down in the table, those links are clickable.
 

Lord Lovaduck

Regular Contributor
Hi guys! Hope you all had great holidays! (My wife, my preteen son and myself, all fully vaccinated got infected with Omicron to start the year... probably at the supermarket... all three mostly recovered by now but other than that, it was great!)
So to kill the quarantine time I installed John's fork in my AC68U. After that I ran AMTM, installed EP, then created swap and then installed SPDMERLIN. It failed.... here's the installation log from the SSH session....

Thanks for the excellent work JackYaz, I am sure this is a simple thing....


Code:
Welcome to spdMerlin v4.4.3, a script by JackYaz

By installing spdMerlin you are agreeing to Ookla's license: https://raw.githubusercontent.com/jackyaz/spdMerlin/master/speedtest-cli-license


Do you wish to continue? (y/n)  y
Checking your router meets the requirements for spdMerlin

Installing required packages from Entware

Downloading https://bin.entware.net/armv7sf-k2.6/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Installing sqlite3-cli (3330000-2) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/sqlite3-cli_3330000-2_armv7-2.6.ipk
Installing zlib (1.2.11-3) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/zlib_1.2.11-3_armv7-2.6.ipk
Installing libsqlite3 (3330000-2) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/libsqlite3_3330000-2_armv7-2.6.ipk
Installing libncursesw (6.2-3) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/libncursesw_6.2-3_armv7-2.6.ipk
Installing libncurses (6.2-3) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/libncurses_6.2-3_armv7-2.6.ipk
Installing libedit (20210522-3.1-1) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/libedit_20210522-3.1-1_armv7-2.6.ipk
Configuring zlib.
Configuring libsqlite3.
Configuring libncursesw.
Configuring libncurses.
Configuring libedit.
Configuring sqlite3-cli.
Installing jq (1.6-2) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/jq_1.6-2_armv7-2.6.ipk
Configuring jq.
Installing p7zip (16.02-3) to root...
Downloading https://bin.entware.net/armv7sf-k2.6/p7zip_16.02-3_armv7-2.6.ipk
Configuring p7zip.
Package findutils (4.8.0-1) installed in root is up to date.
New version of spdstats_www.asp downloaded

Mounting WebUI tab for spdMerlin

Mounted spdMerlin WebUI page as user1.asp

New version of shared-jy.tar.gz downloaded

Upgrading database...

/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied

(repeats multiple times)

/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied
Error: near line 4: no such function: printf

(repeats multiple times)

Error: near line 4: no such function: printf
Database ready, continuing...

Starting speedtest using auto-selected server for WAN interface

Error running speedtest for WAN

Retrieving data for WebUI charts

Error: near line 4: no such function: printf

(repeats multiple times)

Error: near line 4: no such function: printf

####################################################################
##                       _  __  __              _  _              ##
##                      | ||  \/  |            | |(_)             ##
##       ___  _ __    __| || \  / |  ___  _ __ | | _  _ __        ##
##      / __|| '_ \  / _  || |\/| | / _ \| '__|| || || '_ \       ##
##      \__ \| |_) || (_| || |  | ||  __/| |   | || || | | |      ##
##      |___/| .__/  \__,_||_|  |_| \___||_|   |_||_||_| |_|      ##
##          | |                                                   ##
##          |_|                                                   ##
##                                                                ##
##                       v4.4.3 on RT-AC68U                       ##
##                                                                ##
##              https://github.com/jackyaz/spdMerlin              ##
##                                                                ##
####################################################################

sed: /tmp/menuTree.js: No such file or directory
WebUI for spdMerlin is available at:
WebUI page not found

1.    Run a speedtest now

2.    Choose a preferred server for an interface

3.    Toggle automatic speedtests
      Currently: Enabled

4.    Configure schedule for automatic speedtests
      Hours: *    -    Minutes: 12,42
      Days of week: All

5.    Toggle time output mode
      Currently unix time values will be used for CSV exports

6.    Toggle storage of speedtest result URLs
      Currently: Enabled

7.    Set number of speedtest results to show in WebUI
      Currently: 10 results will be shown

8.    Set number of days data to keep in database
      Currently: 30 days data will be kept

9.    Toggle between built-in Ookla speedtest and speedtest-cli
      Currently: external will be used for speedtests

c.    Customise list of interfaces for automatic speedtests
r.    Reset list of interfaces for automatic speedtests to default

s.    Toggle storage location for stats and config
      Current location is jffs

q.    Toggle exclusion of spdMerlin speedtests from QoS
      Currently spdMerlin speedtests are excluded from QoS

a.    AutoBW

u.    Check for updates
uf.   Update spdMerlin with latest version (force update)

rt.   Reset spdMerlin database / delete all data

e.    Exit spdMerlin

z.    Uninstall spdMerlin

####################################################################

Choose an option:
 

Lord Lovaduck

Regular Contributor
Also, if I reset the database using RT and then via SCP I change permissions for spdstats.db to allow for execution, I also get an error:


Code:
1.    All
2.    WAN

Choose an option:  1

What mode would you like to use?

1.    Auto-select
2.    Preferred server
3.    Choose a server

Choose an option:  1

Starting speedtest using auto-selected server for WAN interface

Error running speedtest for WAN

Retrieving data for WebUI charts


Press enter to continue...
 

pinkgrae

Regular Contributor
v4.4.3 is now available
Changelog

  • NEW: Add Result URL feature when using built-in speedtest
Thanks for this @Jack Yaz - love your work!

However, I seem to be experiencing an issue with the new functionality in v4.4.3 - I get "No result URL" in the results table for both manual and automatic tests, though I can see a URL being generated when after manual tests via both the web gui and cli. Both the table in the web gui and exported .csv files correctly record the server name and server ID but no URLs.

Uninstalling & reinstalling retaining existing data and configs doesn't result in any change. I suppose the next step would be to reinstall removing existing data? Don't really want to lose my precious data though...
 

Jack Yaz

Part of the Furniture
Thanks for this @Jack Yaz - love your work!

However, I seem to be experiencing an issue with the new functionality in v4.4.3 - I get "No result URL" in the results table for both manual and automatic tests, though I can see a URL being generated when after manual tests via both the web gui and cli. Both the table in the web gui and exported .csv files correctly record the server name and server ID but no URLs.

Uninstalling & reinstalling retaining existing data and configs doesn't result in any change. I suppose the next step would be to reinstall removing existing data? Don't really want to lose my precious data though...
just checking the obvious, result url is turned on?
 

pinkgrae

Regular Contributor
In the cli? Yes confirmed.
 

Lord Lovaduck

Regular Contributor
Back to this thread.... Since I have a spare AC68U, I tried spdMerlin 4.4.3 with the latest and greatest Merlin 386.4.
I get the same error when installing
Code:
Upgrading database...

/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied
(repeats multiple times)
/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied

Database ready, continuing...

Starting speedtest using auto-selected server for WAN interface

     Server: AirNet Internet - Puerto Montt (id = 38863)
        ISP: AVC ANGOSTURA VIDEO CABLE
    Latency:    27.30 ms   (2.72 ms jitter)
   Download:    66.19 Mbps (data used: 78.1 MB)
     Upload:    29.30 Mbps (data used: 36.6 MB)
Packet Loss:     0.0%
And after that SpdMerlin works just fine and I have the WebUi page working fine.

I will also mention that CONMON also fails on John's fork, in that case the WebUi page opens and quickly presents a screen with a work in progress message and loops there. Forgot the exact text but in any case CONMON wasn't my first priority.

So the issue is probably something else other than this spdstats.db permission denied. How can I get a more verbose execution to try and see where it fails?

Thanks in advance for any help!

Also, if I reset the database using RT and then via SCP I change permissions for spdstats.db to allow for execution, I also get the error when the "database upgrade" portion runs.


Code:
1.    All
2.    WAN

Choose an option:  1

What mode would you like to use?

1.    Auto-select
2.    Preferred server
3.    Choose a server

Choose an option:  1

Starting speedtest using auto-selected server for WAN interface

Error running speedtest for WAN

Retrieving data for WebUI charts


Press enter to continue...
 

Jack Yaz

Part of the Furniture
Back to this thread.... Since I have a spare AC68U, I tried spdMerlin 4.4.3 with the latest and greatest Merlin 386.4.
I get the same error when installing
Code:
Upgrading database...

/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied
(repeats multiple times)
/jffs/scripts/spdmerlin: line 1: /jffs/addons/spdmerlin.d/spdstats.db: Permission denied

Database ready, continuing...

Starting speedtest using auto-selected server for WAN interface

     Server: AirNet Internet - Puerto Montt (id = 38863)
        ISP: AVC ANGOSTURA VIDEO CABLE
    Latency:    27.30 ms   (2.72 ms jitter)
   Download:    66.19 Mbps (data used: 78.1 MB)
     Upload:    29.30 Mbps (data used: 36.6 MB)
Packet Loss:     0.0%
And after that SpdMerlin works just fine and I have the WebUi page working fine.

I will also mention that CONMON also fails on John's fork, in that case the WebUi page opens and quickly presents a screen with a work in progress message and loops there. Forgot the exact text but in any case CONMON wasn't my first priority.

So the issue is probably something else other than this spdstats.db permission denied. How can I get a more verbose execution to try and see where it fails?

Thanks in advance for any help!
when you're installing, is there a database already there or installing from scratch? you shouldn't ever need ot be changing permissions on the files
 

Lord Lovaduck

Regular Contributor
Hi there Jack! Thanks for responding.
All installations were from scratch, both Router firmware and your packages, SpdMerlin and Conmon.
I agree I shouldn't be changing permissions! Which leads me to believe somehow the permissions are lost or not there to start with. In any case in Merlin 386.4 the error doesn't seem to completely abort the install. In John's fork there are more messages that are not very informative (PrintF not working doesn't help!) and then the whole thing fails, no initial speed test is taken.
Take care!

when you're installing, is there a database already there or installing from scratch? you shouldn't ever need ot be changing permissions on the files
 

Jack Yaz

Part of the Furniture
Hi there Jack! Thanks for responding.
All installations were from scratch, both Router firmware and your packages, SpdMerlin and Conmon.
I agree I shouldn't be changing permissions! Which leads me to believe somehow the permissions are lost or not there to start with. In any case in Merlin 386.4 the error doesn't seem to completely abort the install. In John's fork there are more messages that are not very informative (PrintF not working doesn't help!) and then the whole thing fails, no initial speed test is taken.
Take care!
Ah f**k that's a rookie error on my part. When I cleaned up swallowing of any errors i deleted a vital part of some of the commands. I'll ship a fix first thing in the morning. 1am here, not a good time to start coding!
 

Lord Lovaduck

Regular Contributor
Ah f**k that's a rookie error on my part. When I cleaned up swallowing of any errors i deleted a vital part of some of the commands. I'll ship a fix first thing in the morning. 1am here, not a good time to start coding!
Great!!! Thanks!!! BTW I did my best coding after 1 AM.... Of course did mostly scripting but not in Unix Shells, I am a mainframe person. REXX was my language of choice.
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top