What's new

Traffic history data

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Matt Storr

Occasional Visitor
Hi,

I was wondering if someone could tell me what I'm doing wrong. I've just upgraded my RT-N66U to 380.57 and am trying to save my traffic data to a USB stick. I've added the location to the Save History Location and initialized the file. I can see in putty that the USB stick is mounted and that the location is writable. I see a couple of files created but even though its been running all day and the save frequency is set to 1hr, the files dont grow :-

-rwxrwxrwx 1 admin root 62 Feb 6 17:42 tomato_rstats_08606e21d350.gz
-rwxrwxrwx 1 admin root 457 Feb 6 17:42 tomato_cstats_08606e21d350.gz
admin@RT-N66U:/tmp/mnt/ESD-USB#

Do I nee any special software to read the data in the files? All I want to do is prove to my ISP how much data I'm consuming and uploading per hour each day so I can win an argument I'm having with them ;)

Thanks in advance.
 

Attachments

  • save_location.PNG
    save_location.PNG
    186.4 KB · Views: 784
Login into your router. Goto: "Traffic Manager", "Traffic Monitor" tab, use the drop down. You should have 7 options to view the data files (e.g. daily, monthly, per device, etc). I have been trying to write a script so that I get daily emails with this data so I can monitor without having to log into the router.
 
Hi and thanks for responding. I've looked at those tabs but they dont seem to give me exportable data on a per hour basis. My ISP uses traffic management and they claim I'm uploading over 2.2Gb/hr and according to my basic analysis, I'm not doing anything like it. They then claim that it must be other devices on my LAN, which is why I started to take the data from the router. Now all I need is a way to download the data and then analyse it so I can prove them wrong. Any ideas?
 
The only options from the drop down are: Global/real-time, last 24 hours, daily, monthly. Per-Device/real-time, daily, monthly. There is not a way to export from the GUI.

As I mentioned I have begun looking around for a way to email a daily, per device traffic report. I just started and there are a few python scripts out there that help read these types of log files. For your situation in order to diagnose, I would think the Per-Device/real-time and daily reports would guide you to the 2.2Gb/hr offender.

If you want to try to find the offender via the log files, I would probably copy those log files from the USB stick and put on a linux box. Then grab a python script to get the data you need. You might be able to find a python script someone created already. As I mentioned, I just started looking myself. Perhaps someone on this forum could guide us to a usable script.
 
I may be misunderstanding, but I am trying to save the traffic data directly to the USB stick using a customised Traffic History Location (under Traffic Monitoring). I assumed that the data would be stored in a format that I could interpret, but currently the logs just dont seem to be updated (they are created, but not populated).
 
I may be misunderstanding, but I am trying to save the traffic data directly to the USB stick using a customised Traffic History Location (under Traffic Monitoring). I assumed that the data would be stored in a format that I could interpret, but currently the logs just dont seem to be updated (they are created, but not populated).
Select default location, enable reset and save.
Reboot router and then select your custom location, again enable reset and save.
I hope it works...

Search for tomato traffic log analyser. I've seen a script or two that decode the files.
 
I assumed that the data would be stored in a format that I could interpret,
The files aren't in a human readable format. I've only ever seen 1 pair of python scripts that could read the data and they only showed totals for day and month.
 
I may be misunderstanding, but I am trying to save the traffic data directly to the USB stick using a customised Traffic History Location (under Traffic Monitoring). I assumed that the data would be stored in a format that I could interpret, but currently the logs just dont seem to be updated (they are created, but not populated).

The file created *.gz is very compact. After accumulating two years of data my file is only 1.21 KB in size. Given the small size of the file there probably isn't any additional detail/ presentation that you can't be viewed using the router.

As previously mentioned it is not in a format that can be read/viewed outside of Tomato except using a script.
 
I really appreciate you guys taking the time to reply. Am I asking the impossible? Is it simply not viable to be able to mine the data to give me some proper usage stats down to an hourly resolution on how my broadband is being used? It seems to me that with per IP monitoring configured there is all the raw information available but just no way to get hold of it. Can anyone give me some hope that there is a solution?
 
OK. I've been having a look at the source code + various bits of information on the internet.
Am I asking the impossible? Is it simply not viable to be able to mine the data to give me some proper usage stats down to an hourly resolution on how my broadband is being used?
Nothing is impossible given enough time and effort. :)
It seems to me that with per IP monitoring configured there is all the raw information available but just no way to get hold of it.
"per IP monitoring" (the cstats file) doesn't help because it only records daily and monthly totals per IP. The same is true of the rstats file.

However, there is another file on the router (/var/spool/rstats-speed.js) that contains the global bandwidth figures for the last 24 hours at a resolution of 2 minutes.

So, how'd you fancy writing a little program to extract the data? :) Here's a clue to start with, search the file for the correct interface (called INTERNET) then look at the following tx and rx lines.
 
Thank you Colin (;)). I've looked for that file but cant find it. Is it related to a specific Tomato build? I'm running Asuswrt-Merlin. Happy to try other builds (but I like the way Asuswrt-Merlin still retains the original Asus format) :)
 
Last edited:
Steve?

My mistake. It looks like that file gets created when you go to the "Last 24 Hours" graph in Traffic Monitor.

You can also do the same thing from the command line as follows:
Code:
admin@RT-N66U:/# killall -USR1 rstats
admin@RT-N66U:/# ls -l /var/spool/rstats-speed.js
-rw-rw-rw-    1 admin    root        171565 Feb  7 15:43 /var/spool/rstats-speed.js

UPDATE: OK It looks like the GUI creates and then deletes the rstats-speed.js file when you're watching the graph. So you'll have to use the command line method.

UPDATE 2: Another correction :rolleyes: The sampling interval is every 30 seconds, so that's 2880 for 24 hours. Still haven't quite worked out the units yet. Doesn't appear to be in bytes. - This is on Merlin.

UPDATE 3: Oops. Yes it is in bytes per 30 second interval. I got my bytes and bits mixed up.:p
 
Last edited:
Thanks Colin ;) Is there a spec for these stats? i can see the TX section for the INTERNET interface, but the numbers look like kb/s with no cumulative figure i.e. its only recording the egress bandwidth at the point of taking the sample. Unfortunately I lost the logs as soon I logged back into the router and went to the monitoring page, so I've only got the sample I had in my CLI at the time (attached). Trying to work out how each number corresponds to a particular time. Any ideas?
 

Attachments

  • internet_tx.txt
    10.5 KB · Views: 565
ok, I missed the point you made about the data sample rate being every 2mins. Trying to work out if its a running 24hrs now from the point the router was initialised, or whether it resets to midnight each night. Guess I could grab the current time and then work backwards from that anyway :)
 
ok, I missed the point you made about the data sample rate being every 2mins. Trying to work out if its a running 24hrs now from the point the router was initialised, or whether it resets to midnight each night. Guess I could grab the current time and then work backwards from that anyway :)
Hi Matt,

Please note the correction I made to myself above (UPDATE 2) where the interval is 30 seconds not 2 minutes.

I believe the period is just the 24 hour period prior to when you issed the "killall -USR1 rstats" command.
 
Thanks for the update Colin :) So, if the sample rate is per 30secs, and the value recorded is the bytes/sec at the point of sample, then the total TX amount can only be an average of data sent over the period? if I'm correct, I can't see how I can get a true cumulative figure of data sent with this source, as the best they seem to be able to give me is an approximation. Am I understanding this correctly?
 
No that's not correct. The figures are the total number of bytes sent and the total number of bytes received during that 30 second period. It's not a point-in-time measurement.

So if you add up all the rx numbers you will get the total number of bytes received in the last 24 hours. Same applies to the tx numbers.

What are you trying to find out. The total number of bytes (MB) or the rate (MB/sec)? Over what period?

If you want to know the total number of bytes sent and received over the last 24 hours, then that is already calculated for you. :) Here's the file layout.
Code:
'INTERNET': {
rx: [11326538,2546358,10344513,10315894,3237319,9644564,...],
rx_avg: 12067687,
rx_max: 244953204,
rx_total: 34754939572,
tx: [168839,38551,177042,180537,71221,158879,174849,...],
tx_avg: 268476,
tx_max: 7178760,
tx_total: 773211443 },
 
fantastic! thank you Colin. I dont know how you worked that out tbh. Was it just pure logic or did the source give you an idea?
 
I wish I could say it was my own work but it wasn't. :)

I started out by looking at the source code for the rstats/cstats format. When it became apparent that those files only contained daily/monthly data I turned to Google. I then stumbled upon this post which talked about the rstats-speed.js file: http://www.linksysinfo.org/index.php?threads/is-the-rstats-file-format-documented.28245/

What confused me for a long time was that post's assertion that the data was 720 x 2 minutes! Hence my earlier updated post. Looking at my own data and cross referencing it with the routers 24 hour Traffic Monitor graph I realised that it was actually 2880 x 30 seconds. Importing it into Excel confirmed this ;). Perhaps that article was referring to an older version of the code.

So there you are, not clever, just lots of googling, guessing and calculator work.:D
 
its all looking very good! Each time I create the rstats file (killall -USR1 rstats) I get a snapshot of the previous 24hrs. What I'm trying to work out now though is the position of the data relative to time. For example, I want to select all data between 6pm-9pm. I'm assuming that when the router first starts, it populates the JSON array with zero's. New data is being recorded every 30 sec to (I think) the start of the relevant node and old data pushed off the other end (I'm visualising the 24hr graph in the interface) as there can only ever be 2880 elements. If correct, then if I took a snapshot of the data at 10pm, the first value in the array will be 10pm, the next 9:59:30, 9:59:00, 9:58:30 etc, etc. Does this make sense? I need to try it :)
 

Attachments

  • rstats-speed.js.2.txt
    166 KB · Views: 683

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top