What's new

Restore Traffic Stats

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Now I wonder how I can merge my rstats data from multiple backups into one rstats database.
 
I am co
Now I wonder how I can merge my rstats data from multiple backups into one rstats database.

rstat is associated with Traffic Monitor, not Traffic Analyzer. I am confused now. From your last few posts, it seems to be traffic analyzer that you actually wanted.
 
My preference is to have Traffic Analyzer data for as long as I want to keep it. I am assuming since that's Trend Micro code there's no way to change its behavior. So, not being able to achieve my first preference, I would like to join my disjointed collection of rstats data from the past 2 years (I think that's the limit of RS01) and at least have that data available.
 
After examining your code and the rstats files, it looks to be fairly easy for me to manually "merge" my backups by manually creating my own rstats file in a hex editor. I wouldn't want to do it on a regular basis, but as a one-time exercise it shouldn't be too bad.

If I understand the format right, it just looks like the monthly data starts at offset 0x5F9 and is two bytes: the decimal month (e.g., 06 for July), and the year since 1900 (e.g., 2021 is 0x79). Then comes the downloaded byes in little endian (e.g., 1GB would be recorded as 0x00CA9A3800000000), followed by the upload bytes in the same format. So I would enter 1GB upload and 1GB download for July 2021 as:

Code:
00 06 79 00 00 00 00 00 00 CA 9A 38 00 00 00 00 00 CA 9A 38 00 00 00 00
 
Last edited:
Upon further examination, I think my rstats files had complete daily data, but the Excel script didn't pick up on them because it presumes them to be in chronological order in the rstats file. It doesn't appear that the daily data are in chronological order. For example, from my rstats backup:

Code:
1C 02 7A 00 00 00 00 00 52 D6 2B 91 02 00 00 00 4E AE 0E 1A 00 00 00 00
1D 02 7A 00 00 00 00 00 5F 6E D5 97 03 00 00 00 5E 63 C6 1D 00 00 00 00
1E 02 7A 00 00 00 00 00 60 7E 1C 0D 03 00 00 00 C9 10 15 B9 03 00 00 00
1F 02 7A 00 00 00 00 00 E8 B1 B0 9B 00 00 00 00 EF 95 AC A3 08 00 00 00
1D 00 7A 00 00 00 00 00 B5 36 FC EB 03 00 00 00 DC 79 FB 20 00 00 00 00
1E 00 7A 00 00 00 00 00 01 F3 62 A1 02 00 00 00 DF AD 08 5C 00 00 00 00
...

Translated to human-readable:

Code:
3/28/2022    11025503826        437169742
3/29/2022    15432248927        499540830
3/30/2022    13104873056        15990067401
3/31/2022    2612048360        37105735151
1/29/2022    16844076725        553351644
1/30/2022    11297551105        1544072671
...

However, the Excel sheet only picks up on the data from 3/28-3/31.

What's also interesting is the disparity between rstats and Traffic Analyzer. For example, rstats reports 13,104,873,056 bytes downloaded on 3/30. However, Traffic Analyzer reports 12,857,189,520 bytes downloaded on 3/30.
 
Last edited:
That is entirely possible. It has been a couple of years since I wrote that code. I wrote it with Excel 2007 Pro, which does not have an unsigned long data type hence why some of the code is the way it is (newer versions of Excel has a "long long" data type for an unsigned long variables. I probably did write it for sequential dates as it is the most logical to assume. For myself, I still use it to transfer my usage to an excel sheet monthly .

I am glad you were able to get the data out. I've attached the structure of the file.

Now, that said, is it the Traffic Monitor data you are after. From one of your posts above, I got the impression that the aggregate data is not want you wanted, but rather data by application, which would be traffic analyzer. Or is your interest in both now?

Code:
    # RStats log format:
        # First four bytes are magic string that indicate version
        # Next four bytes are empty, presumably for QWORD alignment
        # Next 24 bytes are empty, for future usage?
        # Next are 61 3-tuples for day-level stats,
        # contains date, download and upload totals (8 byte/value)
        # Next is one-byte counter that should match actual number of previous 3-tuples
        # Next seven bytes are empty, presumably for QWORD alignment
        # Next 24 bytes are empty, again for future usage?
        # Next are 12 3-tuples for month-level stats,
        # contains date, download and upload totals (8 byte/value)
        # Version 1 logfiles hold another 12 month-level stat entries
        # Last is one-byte counter that should match actual number of previous 3-tuples

year  = ((num >> 16) & 0xFF) + 1900
month = ((num >>  8) & 0xFF) + 1
day   = (num & 0xFF) or 1
 
Now that you have pointed that out, I will have to go back and look at it again. I believe that if you use the option 1, the script should just dump the rstat file as is, without looking at any chronological dates. That is option 2, which I done to keep a running year over year data tracking. Option 1 just simply reads an rstat file and dumps it to a pre built template from within the excel sheet.
 
Upon further examination, I think my rstats files had complete daily data, but the Excel script didn't pick up on them because it presumes them to be in chronological order in the rstats file. It doesn't appear that the daily data are in chronological order. For example, from my rstats backup:

Code:
1C 02 7A 00 00 00 00 00 52 D6 2B 91 02 00 00 00 4E AE 0E 1A 00 00 00 00
1D 02 7A 00 00 00 00 00 5F 6E D5 97 03 00 00 00 5E 63 C6 1D 00 00 00 00
1E 02 7A 00 00 00 00 00 60 7E 1C 0D 03 00 00 00 C9 10 15 B9 03 00 00 00
1F 02 7A 00 00 00 00 00 E8 B1 B0 9B 00 00 00 00 EF 95 AC A3 08 00 00 00
1D 00 7A 00 00 00 00 00 B5 36 FC EB 03 00 00 00 DC 79 FB 20 00 00 00 00
1E 00 7A 00 00 00 00 00 01 F3 62 A1 02 00 00 00 DF AD 08 5C 00 00 00 00
...

Translated to human-readable:

Code:
3/28/2022    11025503826        437169742
3/29/2022    15432248927        499540830
3/30/2022    13104873056        15990067401
3/31/2022    2612048360        37105735151
1/29/2022    16844076725        553351644
1/30/2022    11297551105        1544072671
...

However, the Excel sheet only picks up on the data from 3/28-3/31.

What's also interesting is the disparity between rstats and Traffic Analyzer. For example, rstats reports 13,104,873,056 bytes downloaded on 3/30. However, Traffic Analyzer reports 12,857,189,520 bytes downloaded on 3/30.

Would you mind PM me your rstat file? I would like to use it as a test file. Albeit, I may be a few weeks before I can look at it again.
 
I will PM the data.

It looks like once the rstats daily data reaches the limit it just rolls over and starts overwriting the oldest daily data and advances from there.

I was able to put everything together by cutting/pasting 24-byte blocks between rstats files in a hex editor.

The only hiccup I noticed was that if you killall -SIGTERM rstats it does write out all its files in /var/lib/misc, but when you start rstats again it has "lost" all the "Last 24 hours" data. It picks up all the other data from past days and months, but today's data up until the time the process was killed is lost.
 
I will PM the data.

It looks like once the rstats daily data reaches the limit it just rolls over and starts overwriting the oldest daily data and advances from there.

I was able to put everything together by cutting/pasting 24-byte blocks between rstats files in a hex editor.

The only hiccup I noticed was that if you killall -SIGTERM rstats it does write out all its files in /var/lib/misc, but when you start rstats again it has "lost" all the "Last 24 hours" data. It picks up all the other data from past days and months, but today's data up until the time the process was killed is lost.
LOL -> FAIL , "killall" wont work ;)
 
LOL -> FAIL , "killall" wont work ;)
It worked fine.

Code:
admin@RT-AX86U-AR28:/var/lib/misc# killall -SIGTERM rstats
admin@RT-AX86U-AR28:/var/lib/misc# pidof rstats
admin@RT-AX86U-AR28:/var/lib/misc#

I have combined the data from three different rstats backups into one rstats file and they were successfully loaded in the UI.

2022-04-02 13_37_24-Window.png


2022-04-02 14_10_57-Window.png


It's actually really easy once you understand the RS01 file format.

The issue with losing today's stats happens even if you don't do anything with the data files. Just look at your "Last 24 hours" data and then execute:

Code:
killall -SIGERM rstats
service start_rstats

Now go look at your "Last 24 hours" and see that it's empty.
 
LOL -> FAIL , "killall" wont work ;)
Yes it does. Maybe you don't know what the killall command is. LOL

The issue with losing today's stats happens even if you don't do anything with the data files. Just look at your "Last 24 hours" data and then execute:

Code:
killall -SIGERM rstats
service start_rstats

Now go look at your "Last 24 hours" and see that it's empty.
That's just cosmetic as the "last 24 hours" graph data is held in memory. The data is still recorded in the rstats file up until the last save - which is why you should set the save frequency to 1 hour.
 
Yes it does. Maybe you don't know what the killall command is. LOL
I thought it was to kill any processes matching the name given as the argument?

That's just cosmetic as the "last 24 hours" graph data is held in memory. The data is still recorded in the rstats file up until the last save - which is why you should set the save frequency to 1 hour.
I wish it would save that state, too. Of course it's not a big deal but it is a little annoying.

EDIT - Sorry, I didn't see that comment wasn't directed at me.
 
I will PM the data.

It looks like once the rstats daily data reaches the limit it just rolls over and starts overwriting the oldest daily data and advances from there.

I was able to put everything together by cutting/pasting 24-byte blocks between rstats files in a hex editor.

The only hiccup I noticed was that if you killall -SIGTERM rstats it does write out all its files in /var/lib/misc, but when you start rstats again it has "lost" all the "Last 24 hours" data. It picks up all the other data from past days and months, but today's data up until the time the process was killed is lost.

That is what I am thinking as well. I am interested in looking at your rstat file to see what value is written in the byte the specifies how many valid entries there actually are. I suspect it is 3, which is why my script only printed out the first three lines. The rest of the entries were considered stale and was a happy happenstance for you that they were there.
 
I found a python script that spits out comma-separated human-readable data from an rstats file in case anyone is interested. It can be handy if you want to keep data longer term than the 24 months that the rstats format will allow.


I've seen this script. I tried to port it at one time to Windows using a windows python compiler, but had no luck. It is what prompted me to write the Excel script for windows.
 
Last edited:
I've seen this script. I tried to port it at one time to Windows using a windows python compiler, but had luck. It is what prompted me to write the Excel script for windows.
I was able to run it using python under cygwin.
 
I was able to run it using python under cygwin.

I'll have to try again. I wrote the option 2 portion of my script to keep a month over month and day over day beyond the 61 day / 24 months for long term tracking of my usage to make sure I am getting bang for the buck.

@sbsnb, I looked at your rstat file, and the byte value representing the actual number of days of data available was actually 4. That is why my script stopped where it did. It considered the rest of the entries as stale/invalid.
 
@sbsnb, thanks for your indirect work on my excel sheet project. Fixed up the script this morning to read all data in the rstat file, regardless of the current counter.

Cheers
 
Judging on the rstats file format, it would only require about 88 kb to store a decade's worth of daily and monthly stats uncompressed and about half that or less gzipped. Changing the code to allow that is trivial, but what I don't know is if there are other considerations I may not be aware of. Is the graphing library used in the UI sophisticated enough to handle longer datasets, e.g., will it add scrollbars or have multiple pages instead of cramming it all in one page?
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top