[Beta 3] Vnstat on Merlin - UI, CLI and email - data use monitoring - now with data limit monitoring

[Jack Yaz]

The output of --oneline, --json or --xml isn't intended to be imported back. If you need to move the version 1.x database around between architectures then --exportdb and --importdb are there for that. In version 2.x it's possible to move the database file directly as sqlite handles the format while version 1.x database stops working if the endianness or architecture bit width changes.

The situations when you'd want to use --oneline, --json or --xml is when you are trying to pull out some single measurement or a limited series from the database. These output allow you to get access to the values directly without having to parse the text formatted output. As an example, I found this from db-vnstat.sh:

$VNSTAT_COMMAND -m | tail -n 3 | head -n 1 | cut -d "|" -f3 | awk '{print $1}'

while it could be replaced with

$VNSTAT_COMMAND --oneline b | cut -d\; -f11

and you wouldn't even need to worry about running the command again since that drops the unit and provides the result directly in bytes. 2 commands executed instead of 10.

PS. When the daemon ("vnstatd") is running in the background, there shouldn't be a need to run "vnstat -u". When that combination is executed, the "vnstat -u" command ends up rewriting the database file based on the limit visibility it has on the current situation. Meanwhile, the daemon still stays running in the background with a cached copy of the database and will eventually overwrite the database with that cached and updated copy based on the longer duration changes it has seen. "vnstat -u" was removed from the 2.x versions to avoid this to even happen while in 1.x versions it can cause some confusing result as the database gets overwritten with cached data after a while.

I personally hope that the Entware team provide vnstat v2 as I can plug sqlite into chartjs and give this project some great interactive charts.

Thanks for clarification re use of -u and - -oneline, much cleaner. I wasn't familiar with how to use vnstat so was provided with the commands from others closer to the tool. Your input is very valued!

 

[Jack Yaz]

vnstat -u usage is being removed, and since I just love JSON, bandwidth usage is pulled using the below one-liner:

bandwidthused="$($VNSTAT_COMMAND --json m | jq -r --arg month "$(date +%m)" '.interfaces[].traffic.months[] | select(.date.month == ($month | tonumber)) | .rx + .tx')"

 

[maghuro]

@Jack Yaz

This is from another AC86U I use for testing purposes.

 

[Jack Yaz]

@Jack Yaz View attachment 32171View attachment 32172

This is from another AC86U I use for testing purposes.

Second screenshot is too small to see. Also i see you're back on the develop branch which isn't stable.

 

[maghuro]

Second screenshot is too small to see. Also i see you're back on the develop branch which isn't stable.

I know it isn't... But I like to test and and help debug!

Screenshot-20210319-012609440

Image Screenshot-20210319-012609440 hosted in ImgBB

ibb.co

 

[Jack Yaz]

I know it isn't... But I like to test and and help debug!

Screenshot-20210319-012609440

Image Screenshot-20210319-012609440 hosted in ImgBB

ibb.co

plese post the output of

vnstat --config /jffs/addons/dn-vnstat.d/vnstat.conf --json m

 

[maghuro]

plese post the output of

vnstat --config /jffs/addons/dn-vnstat.d/vnstat.conf --json m

vnstat --config /jffs/addons/dn-vnstat.d/vnstat.conf --json m                                                       {"vnstatversion":"1.18","jsonversion":"1","interfaces":[{"id":"imq2","nick":"imq2","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"eth0","nick":"eth0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":39904735,"tx":5304190},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":39904735,"tx":5304190}]}},{"id":"spu_us_dummy","nick":"spu_us_dummy","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"imq0","nick":"imq0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"spu_ds_dummy","nick":"spu_ds_dummy","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ifb1","nick":"ifb1","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"eth4","nick":"eth4","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ip6tnl0","nick":"ip6tnl0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ifb0","nick":"ifb0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"imq1","nick":"imq1","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"dpsta","nick":"dpsta","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}}]}

 

[Jack Yaz]

vnstat --config /jffs/addons/dn-vnstat.d/vnstat.conf --json m                                                       {"vnstatversion":"1.18","jsonversion":"1","interfaces":[{"id":"imq2","nick":"imq2","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"eth0","nick":"eth0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":39904735,"tx":5304190},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":39904735,"tx":5304190}]}},{"id":"spu_us_dummy","nick":"spu_us_dummy","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"imq0","nick":"imq0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"spu_ds_dummy","nick":"spu_ds_dummy","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ifb1","nick":"ifb1","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"eth4","nick":"eth4","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ip6tnl0","nick":"ip6tnl0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"ifb0","nick":"ifb0","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"imq1","nick":"imq1","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}},{"id":"dpsta","nick":"dpsta","created":{"date":{"year":2021,"month":3,"day":15}},"updated":{"date":{"year":2021,"month":3,"day":19},"time":{"hour":10,"minutes":41}},"traffic":{"total":{"rx":0,"tx":0},"months":[{"id":0,"date":{"year":2021,"month":3},"rx":0,"tx":0}]}}]}

what do you have set for the interface in /jffs/addons/dn-vnstat.d/vnstat.conf
it looks like you have multiple interfaces being tracked which is non-standard for this project

 

[maghuro]

what do you have set for the interface in /jffs/addons/dn-vnstat.d/vnstat.conf
it looks like you have multiple interfaces being tracked which is non-standard for this project

Only eth0, and I didn't have selected any other interfaces since day one.

But in fact I saw that some interfaces have some data.

 

[Jack Yaz]

Only eth0, and I didn't have selected any other interfaces since day one.

But in fact I saw that some interfaces have some data.

hm, may have been from the previous script iteration if the interface wasn't configured correctly then vnstat would log everything, I believe. this commit should fix things: https://github.com/de-vnull/vnstat-on-merlin/commit/065065a017a34c691a7452cb11a19f939bef626b

 

[maghuro]

hm, may have been from the previous script iteration if the interface wasn't configured correctly then vnstat would log everything, I believe. this commit should fix things: https://github.com/de-vnull/vnstat-on-merlin/commit/065065a017a34c691a7452cb11a19f939bef626b

Perfect as usual

Thanks pal

 

[Jack Yaz]

Perfect as usual

Thanks pal

just dont jump ahead to 2022 and get data for march 2022, cos that'll break too (I'll worry about year filtering later on)

 

[maghuro]

just dont jump ahead to 2022 and get data for march 2022, cos that'll break too (I'll worry about year filtering later on)

Ahahahah you know I'm the one with the odd problems, so that could be possible

 

[Vergo]

vnstat -u usage is being removed, and since I just love JSON, bandwidth usage is pulled using the below one-liner:

bandwidthused="$($VNSTAT_COMMAND --json m | jq -r --arg month "$(date +%m)" '.interfaces[].traffic.months[] | select(.date.month == ($month | tonumber)) | .rx + .tx')"

That "$(date +%m)" isn't going to work in all situations as you'd most likely want. If the "MonthRotate" configuration option is changed to have a value different than 1 then for the first <value> days of the month $(date +%m) will output something that's not yet visible in the database. You can instead directly query .months[0] as that is always the currently ongoing month the json output of version 1.18.

 

[Jack Yaz]

That "$(date +%m)" isn't going to work in all situations as you'd most likely want. If the "MonthRotate" configuration option is changed to have a value different than 1 then for the first <value> days of the month $(date +%m) will output something that's not yet visible in the database. You can instead directly query .months[0] as that is always the currently ongoing month the json output of version 1.18.

Good point. when we briefly tested monthrotate it seemed non functional (month didn't rotate on the chosen day) in the version we have available, unless there's a condition that wasn't obvious and thus the testing was flawed.

 

[Vergo]

When we briefly tested monthrotate it seemed non functional (month didn't rotate on the chosen day) in the version we have available, unless there's a condition that wasn't obvious and thus the testing was flawed.

If MonthRotate has a value of 1 then the month obviously changes on the first day of every month. If you give it, for example, a value of 10 then the month would change on the 10th day. However, from vnStat point of view, when you make that change the month can't already be the ongoing month or otherwise you aren't going to see any change until the next month.

As an example, if in January I'd set MonthRotate to 10 then vnStat would continue showing January until it's the 10th of February. If I'd change the value from 10 to 1 on the 3rd of February then I'd right away get vnStat to start counting for February. So increasing the MonthRotate value results in the change being visible only during the next month if the current day of month was equal or greater than the previous value. Decreasing the MonthRotate value results in the month to change if the new value is less or equal to the current day of month.

 

[Jack Yaz]

If MonthRotate has a value of 1 then the month obviously changes on the first day of every month. If you give it, for example, a value of 10 then the month would change on the 10th day. However, from vnStat point of view, when you make that change the month can't already be the ongoing month or otherwise you aren't going to see any change until the next month.

As an example, if in January I'd set MonthRotate to 10 then vnStat would continue showing January until it's the 10th of February. If I'd change the value from 10 to 1 on the 3rd of February then I'd right away get vnStat to start counting for February. So increasing the MonthRotate value results in the change being visible only during the next month if the current day of month was equal or greater than the previous value. Decreasing the MonthRotate value results in the month to change if the new value is less or equal to the current day of month.

i think that clears it up, the expectation was the month would do a hard reset at the date selected, e.g. if I set it to 20 today, I expected it would reset tomorrow. would resetting the database with the new month rotate achieve this? obviously a destructive action so if users needed to change it, it'll be surfaced at install time to minimise data loss

 

[Vergo]

Would resetting the database with the new month rotate achieve this? obviously a destructive action so if users needed to change it, it'll be surfaced at install time to minimise data loss

If you'd want to hack it around a little bit then (without testing) I'd try to do it something like this:

1. edit the configuration
2. stop the daemon
3. vnstat -i someinterface --export | sed '/^m;0;/d' >some_temp_file
4. import that back and overwrite the existing database
5. start the daemon

That discards only the current month from the database but keeps everything else intact. The next database write from the daemon will start the current month from zero but using to the configured month rotate. You'd however still need to have some check in place that the current month isn't the same as the previous month with the new configuration as that would otherwise result in another line with the same month visible (but with different data) in the output. The 1.x versions have the capability of incrementing only the index 0 of every time period that's being stored. Compared to the 2.x versions where the sql operation used to insert the data handles the placement regardless of possible past (or future) data.

 

[dev_null]

If MonthRotate has a value of 1 then the month obviously changes on the first day of every month. If you give it, for example, a value of 10 then the month would change on the 10th day. However, from vnStat point of view, when you make that change the month can't already be the ongoing month or otherwise you aren't going to see any change until the next month.

As an example, if in January I'd set MonthRotate to 10 then vnStat would continue showing January until it's the 10th of February. If I'd change the value from 10 to 1 on the 3rd of February then I'd right away get vnStat to start counting for February. So increasing the MonthRotate value results in the change being visible only during the next month if the current day of month was equal or greater than the previous value. Decreasing the MonthRotate value results in the month to change if the new value is less or equal to the current day of month.

Excellent, clarifies what I couldn't figure out.

If you'd want to hack it around a little bit then (without testing) I'd try to do it something like this:

1. edit the configuration
2. stop the daemon
3. vnstat -i someinterface --export | sed '/^m;0;/d' >some_temp_file
4. import that back and overwrite the existing database
5. start the daemon

That discards only the current month from the database but keeps everything else intact. The next database write from the daemon will start the current month from zero but using to the configured month rotate. You'd however still need to have some check in place that the current month isn't the same as the previous month with the new configuration as that would otherwise result in another line with the same month visible (but with different data) in the output. The 1.x versions have the capability of incrementing only the index 0 of every time period that's being stored. Compared to the 2.x versions where the sql operation used to insert the data handles the placement regardless of possible past (or future) data.

I did something very similar, but without the sed in my testing. Will try it with the above this weekend, just to see what happens. I keep various exported dbs around just for testing, and can export from my main (not test) router too (that testing is what I was asking about import function capability with 2.x).

 

[thelonelycoder]

These are two separate errors. The first one about deprecated key you can ignore. At some point that will be fixed in diversion but it will be a headache until @thelonelycoder figures out a smooth way of transitioning. Every confidence he will.

I have it all figured out and working locally. I'm just not ready for prime time with this Diversion update. I guess I packed too many new features into it to finish it all at the time it is needed the most