[Beta] VNSTAT on Merlin; CLI, UI and email data use monitoring (install script offline for now)

  • ATTENTION! You'll notice a Prefix dropdown when you create a thread. If your post applies to one of the topics listed, please use that Prefix for your post. When browsing the thread list you can use the Prefix to filter the view.
  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.
Status
Not open for further replies.

dev_null

Senior Member
Note: this version has been superseded by the beta 2 version: https://www.snbforums.com/threads/b...-monitoring-with-full-install-and-menu.70727/

I will be asking mods to close this thread (27-Feb-2021). Thanks to everyone who provided feedback on the alpha/beta versions!


Background
  • This is an implementation of vnstat/vnstati for use on AsusWRT-Merlin routers. This effort was created to enable accurate measurement of data use on my local network. There were two primary drivers:
  1. Supplement or replace the internal monitoring tool, Traffic Analyzer > Traffic Monitor, which suffers from (false) “17GB” data use bursts on some firmware (e.g., RT-AC68U 386.1). These ghost bursts interfered with tracking data usage (though monthly totals still appeared accurate).
  2. To monitor traffic locally (for privacy reasons).
  • This was created with support from @Jack Yaz, who provided scaffold and scripting assistance to create the AddOn "Vnstat on Merlin". Many thanks, Jack!
  • This concept and view was created for my personal use. I have not (yet) tried to automate the installation and set-up steps. I welcome your feedback - and more importantly your collaboration - to improve and/or automate installation.
    • A Day-2 alpha install script created by @Martineau - thanks so much!
    • A Day-3 single-line install script updated by @Martineau - really appreciate the collaboration!
      • See notes below about post-install steps required (location of db files, email configuration).
Features and benefits:
  • Once set up, the recurring refreshes and other operations are automated, controlled by a few cron jobs.
CLI:
  • The standard "vnstat" install from Entware will track data usage, with few additional configuration tweaks required (the configuration needs to be modified so the data isn't stored in a volatile location).
    • In my tests, the CLI version of vnstat totals are consistent with those collected by the built-in Traffic Monitor, without any 17GB ghosts.
    • Non-UI access to vnstat is via SSH.
    • The vnstat totals tend to be a bit higher (10-15%) than those currently reported by Comcast (NE region, 1Q2021).

      vnstat-cli-red.PNG
UI:
  • Vnstat has a corresponding application called vnstati which creates PNGs from the vnstat data totals. Thanks, Teemu Toivola!
  • The "Vnstat on Merlin" UI uses the scaffolding from @Jack Yaz 's connmon script (with his permission and assistance) to create a reasonably attractive visual representation of data usage on the AddOns tab:

    Screenshot_2021-02-Vnstat-xp2.png
    • Each section can be collapsed to simplify the view
    • Refreshes require a cron job, through which the refresh frequency is adjustable. There doesn't seem to be a discernible resource impact.
Email summaries:
  • Because I don't log on to the router every day, I wanted a "push" notification of data use. I had hacked together a script to email semi-weekly Skynet reports (as described here: https://www.snbforums.com/threads/r...urity-enhancements.16798/page-280#post-538221), so I repurposed and updated it into an aggregation and email script for vnstat statistics.
  • This report is an amalgamation of monthly, weekly and daily usage. The report content is customizable.

    vnstat-email-xp.png
    • The frequency and timing of this email is modifiable by adjusting the associated cron job.
    • This report is converted into an image (using the "imagemagick" package from Entware) and is located at the bottom of the UI page.
Requirements:
  • AsusWRT-Merlin; scripting support is required.
  • Entware, typically installed with Diversion.
    • To take advantage of the encrypted Diversion password, functioning Diversion "communications" (amtm > 1 (Diversion) > c (communication) > 5 (edit email settings, test email))
  • One or more @Jack Yaz scripts, particularly connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
  • The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
  • Experience with SSH, scripting and comfort modifying settings on your router.
Support:
  • I have reproduced the steps outlined in the repository and found that it works on the hardware available to me. Your hardware and settings may differ (most differences should be obvious and readily addressable; the community may be able to help).
  • I'm providing this as-is, for the potential benefit of the community. Use is at your own risk.
  • Any contributions from the community to optimize the install and performance are welcomed.
Location of the Tutorial - updated to Github:
New! Installer command (courtesy of @Martineau ) - see his post below.
  • Because each change in bitbucket updates the UUID, the link needs to be updated. If you get an error, please try again later (we'll get the link fixed as soon as practical). In the meantime, see this post:

15 Feb 2021: For now the installer is off-line. I am working with @Martineau to try to get it back. Please follow the manual steps outlined in the github repository.
 
Last edited:

dev_null

Senior Member
Notes, versions and errata:
 
Last edited:

dev_null

Senior Member
Reserved for FAQ, wishlist, and mea-culpas.

Wishlist/future development:
  • Encrypt/support 2FA for email. Perhaps leverage Diversion email process? (@elorimer 's idea - need to follow-up) Done.
  • Create an email notification when data use limits are being approached. Tried grepping vnstat monthly to grab the total column, but no success (thoughts anyone)? Edit: exploring exporting a json version of vnstat to see if that's easier to grep (comma delineated)
  • Install script checks for existence of Diversion email credentials and sets up the correct script (div-email vs send-vnstat). Done (per Martineau)
 
Last edited:

spanjap

Regular Contributor
@dev_null: Thanks for this very nice implementation.

Tried to set it up as your instructions on my RT-AX88U. There was one problem I had.
Got the following error when starting "vnstat -u && sh /jffs/scripts/vnstat-ww.sh" manually:
vnstati: error while loading shared libraries: libjpeg.so.62: cannot open shared object file: No such file or directory

Installed "opkg install libjpeg-turbo" and now it's seems to be up and running.

PS: You don't need to "/mnt/path_to/entware/bin/vnstat" vnstat or vnstati is enough.

vnstat.jpg
 
Last edited:

dev_null

Senior Member
@dev_null: Thanks for this very nice implementation.

Tried to set it up as your instructions on my RT-AX88U. There was one problem I had.
Got the following error when starting "vnstat -u && sh /jffs/scripts/vnstat-ww.sh" manually:
vnstati: error while loading shared libraries: libjpeg.so.62: cannot open shared object file: No such file or directory

Installed "opkg install libjpeg-turbo" and now it's seems to be up and running.

PS: You don't need to "/mnt/path_to/entware/bin/vnstat" vnstat or vnstati is enough.
Great, thanks for the feedback. I'll add the libijpeg note to the instructions... for me perhaps it was installed by another program or add-in.

W/r/t full path: I think if the path is added to the conf file it should be okay with just the command, but since the execution is generated by cron, I didn't want it to error out, unseen.
 

ColinTaylor

Part of the Furniture
Good stuff @dev_null :).

Two quick questions - 1) How are you determining the WAN interface to monitor? 2) Presumably on older models (like the RT-AC68U) this will produce no meaningful data if hardware acceleration is enabled?
 

dev_null

Senior Member
Good stuff @dev_null :).

Two quick questions - 1) How are you determining the WAN interface to monitor? 2) Presumably on older models (like the RT-AC68U) this will produce no meaningful data if hardware acceleration is enabled?
Thanks very much.

In the instructions on bitbucket, I recommend that users verify the interface by issuing "nvram get wan0_ifname" from the cli before editing the configuration.

On my set-up I don't use HW acceleration, so I don't know how that would affect vnstat. If you could report back, I can add that as a notation in the above and on the bitbucket page.
 

ColinTaylor

Part of the Furniture
I've just tried the command line installation of vnstat on my RT-AC68U using John's firmware. As suspected it appears to be using the regular interface stats and therefore suffers from the same issues regarding hardware acceleration (interface vlan2), namely no traffic is recorded for wired Ethernet clients.

I noticed a couple of differences in the install procedure. On my router the default Entware installation is in /opt rather the /entware shown in your instructions. Also, the startup script provided by Entware was called S32vnstat not s32vnstat.
 

dev_null

Senior Member
I've just tried the command line installation of vnstat on my RT-AC68U using John's firmware. As suspected it appears to be using the regular interface stats and therefore suffers from the same issues regarding hardware acceleration (interface vlan2), namely no traffic is recorded for wired Ethernet clients.

I noticed a couple of differences in the install procedure. On my router the default Entware installation is in /opt rather the /entware shown in your instructions. Also, the startup script provided by Entware was called S32vnstat not s32vnstat.
OK, I added your comments on HA above and to the bitbucket notes. I wonder if vlan2 can be added to the vnstat configuration (I'm not familiar enough with HA to know if this is simply another interface or if data tracking is completely bypassed) - I think vnstat can be configured for multiple interfaces.

A favor, if you're willing: install darkstat (it's available on Entware). I'm curious if that shows accurate traffic (it has it's own GUI - routerip:667 IIRC). I could never figure out how to get darkstat to tell me what the cumulative usage was, so I focused on vnstat, which could.
 

Martineau

Part of the Furniture
  • AsusWRT-Merlin; scripting support is required.
  • Entware, typically installed with Diversion.
  • One or more @Jack Yaz scripts, preferably connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
  • The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
  • Experience with SSH, scripting and comfort modifying settings on your router.
Support:
  • I have reproduced the steps outlined in the repository and found that it works on the hardware available to me. Your hardware and settings may differ (most differences should be obvious and readily addressable; the community may be able to help).
  • I'm providing this as-is, for the potential benefit of the community. Use is at your own risk.
  • Any contributions from the community to optimize the install and performance are welcomed.
Location of the Tutorial:
Crude install/uninstall script (uninstall isn't complete) that may assist in the take-up of this useful addition to the firmware or at least let non-techies trial vnstat.

/jffs/scripts/vnstat-install.sh

Beta vnstat-install.sh script
 
Last edited:

ColinTaylor

Part of the Furniture
I wonder if vlan2 can be added to the vnstat configuration...
Yes this is what I did, use vlan2. If I had left it as eth0 it would have produced even more inaccurate results.
 

ColinTaylor

Part of the Furniture
I also noticed that vnstat couldn't correctly identify the speed of my WAN interface and set it at 10Mb which produces warnings in the syslog:
Code:
Feb  7 17:25:16 vnstatd[9284]: vnStat daemon 1.18 started. (pid:9284 uid:0 gid:0)
Feb  7 17:25:16 vnstatd[9284]: Monitoring: vlan2 (10 Mbit) br0 (1000 Mbit)
Feb  7 17:25:16 admin: Started vnstatd from .
Feb  7 17:30:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r1030 t90), syncing.
Feb  7 17:31:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r961 t92), syncing.

Setting BandwidthDetection to 0 in vnstat.conf solved that problem:
Code:
Feb  7 17:39:35 vnstatd[10349]: vnStat daemon 1.18 started. (pid:10349 uid:0 gid:0)
Feb  7 17:39:35 vnstatd[10349]: Monitoring: vlan2 (1000 Mbit) br0 (1000 Mbit)
 

dev_null

Senior Member
Crude install/uninstall script (uninstall isn't complete) that may assist in the take-up of this useful addition to the firmware or at least let non-techies trial vnstat.
Feel free to host it on your bitbuckit repository.
@Martineau , thank you very much for this contribution. I've followed your work for a long time, so I'm grateful for your interest and assistance.

I'll add this to the repository as an "early alpha install" and dig through it so I can understand how you've constructed it.

At first glance it's principally installing the scripts and applications, then the user would still need to modify the actual settings to reflect their set-up, correct?

I also noticed that vnstat couldn't correctly identify the speed of my WAN interface and set it at 10Mb which produces warnings in the syslog:
Code:
Feb  7 17:25:16 vnstatd[9284]: vnStat daemon 1.18 started. (pid:9284 uid:0 gid:0)
Feb  7 17:25:16 vnstatd[9284]: Monitoring: vlan2 (10 Mbit) br0 (1000 Mbit)
Feb  7 17:25:16 admin: Started vnstatd from .
Feb  7 17:30:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r1030 t90), syncing.
Feb  7 17:31:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r961 t92), syncing.

Setting BandwidthDetection to 0 in vnstat.conf solved that problem:
Code:
Feb  7 17:39:35 vnstatd[10349]: vnStat daemon 1.18 started. (pid:10349 uid:0 gid:0)
Feb  7 17:39:35 vnstatd[10349]: Monitoring: vlan2 (1000 Mbit) br0 (1000 Mbit)
@ColinTaylor now that I see this pointed out, I think I had the same issue with my setup (my vnstat.conf has '0' in the bandwidth detection setting, which I don't think is the default) so I'll add that to the guidance.
 

Martineau

Part of the Furniture
At first glance it's principally installing the scripts and applications, then the user would still need to modify the actual settings to reflect their set-up, correct?
The script also modifies the base installation given most users may not be familiar with nano/WinSCP nor feel confident to manually edit configuration files,

e.g. I've edited my post to v1.02 and you can see that it now automatically applies the bug fix as identified below.

@ColinTaylor now that I see this pointed out, I think I had the same issue with my setup (my vnstat.conf has '0' in the bandwidth detection setting, which I don't think is the default) so I'll add that to the guidance.
NOTE: The script does not alter the Database location and clearly it cannot modify the email send-vnstatscript with the user credentials.

The script does still require minimal SSH experience to be able to execute a single command.
Requirements:
  • AsusWRT-Merlin; scripting support is required.
  • Entware, typically installed with Diversion.
  • One or more @Jack Yaz scripts, preferably connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
  • The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
  • Experience with SSH, scripting and comfort modifying settings on your router.
 
Last edited:

ColinTaylor

Part of the Furniture
@dev_null Can you check to contents of your /entware/etc/init.d/rc.unslung file. If it's the same as mine your startup script won't run unless the first letter is a capital S, e.g. S32vnstat.

I suggest that you also check to see if you have a symbolic link for /opt (ls -l /opt /tmp/opt) because if you don't it could cause all sort of problems with Entware packages in general.
 
Last edited:

dev_null

Senior Member
@dev_null Can you check to contents of your /entware/etc/init.d/rc.unslung file. If it's the same as mine your startup script won't run unless the first letter is a capital S, e.g. S32vnstat.

I suggest that you also check to see if you have a symbolic link for /opt (ls -l /opt /tmp/opt) because if you don't it could cause all sort of problems with Entware packages in general.
Indeed the init.d file is (capital) S32vnstat. I will correct the capitalization in my directions, but that file is created when Entware installs the application, so it should only be my reference that needs to be corrected.

And yes, there is a symbolic link. Which must be by default (I did not create that symlink).

Code:
[email protected]:/tmp/home/root# ls -l /opt /tmp/opt
lrwxrwxrwx    1 username root             7 Jan 30 16:00 /opt -> tmp/opt
lrwxrwxrwx    1 username root            26 May  5  2018 /tmp/opt -> /tmp/mnt/my_SSD/entware
 
Last edited:

ColinTaylor

Part of the Furniture
And yes, there is a symbolic link. Which must be by default (I did not create that symlink).

Code:
[email protected]:/tmp/home/root# ls -l /opt /tmp/opt
lrwxrwxrwx    1 username root             7 Jan 30 16:00 /opt -> tmp/opt
lrwxrwxrwx    1 username root            26 May  5  2018 /tmp/opt -> /tmp/mnt/AWMX_120G/entware
Then I suggest you amend your documentation to use /opt instead of /entware as that is what is hard coded into the standard Entware scripts.
 
Last edited:

dev_null

Senior Member
Then I suggest you amend your documentation to use /opt instead of /entware as that is what is hard coded into the standard Entware scripts.
Yes, I'm just realizing that with the symlink they are pretty much the same. I'll make those changes, thanks @ColinTaylor and @Martineau.

I'm learning here, folks!

UPDATE: DONE.
 
Last edited:

dev_null

Senior Member
The script also modifies the base installation given most users may not be familiar with nano/WinSCP nor feel confident to manually edit configuration files,

e.g. I've edited my post to v1.02 and you can see that it now automatically applies the bug fix as identified below.


NOTE: The script does not alter the Database location and clearly it cannot modify the email send-vnstatscript with the user credentials.

The script does still require minimal SSH experience to be able to execute a single command.
@Martineau I created a file called "vnstat-install.sh" in the bitbucket scripts folder. I don't think that having a different name makes a difference, but wanted to differentiate it from the other files called vnstat. Can you verify your script will work even with the name "vnstat-install.sh"?
 
Status
Not open for further replies.

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top