What's new

Extracting Real-time Stats

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

No idea what darkstat is, but I assume it's using its own web server and not Asus's, so it will be fine authentication-wise.

Must have been about 10 years since I last looked at NTOP + Netflow - sorry...

Darkstat is an Entware package.

OK. I'm all set. Final thanks. :)
 
FYI, both cstats and /proc/net/ipt_account/lan worked out really well. I have more than enough of what I need. I really appreciate your help.
 
That good news Glenn. Would it be possible for you to explain how you extracted the information from cstats or what the record format is? Is it some sort of "standard" RRD format perhaps?

It's just that this question has been asked many times and as far as I know no one has worked out the format. Until now.:D
 
That good news Glenn. Would it be possible for you to explain how you extracted the information from cstats or what the record format is? Is it some sort of "standard" RRD format perhaps?

It's just that this question has been asked many times and as far as I know no one has worked out the format. Until now.:D

Sure, no problem. I started with a Python script that I found somewhere. Then I used cstats.[ch] from the latest release and a hex editor. I'm not sure why, but the format of the file doesn't exactly match the code.

The output format is customized for my use, so you'll have to modify it to get what you need.

Please let me know if you make any changes that I might be able to use.

https://drive.google.com/file/d/0B7AAdqGpSE8ZS1ZoX1lMZkpyNGc/view?usp=sharing
 
That's great. Thank you very much.

Personally I don't have a use for it at the moment but I'm sure it will be helpful to others.

I've not been able to run your test program myself because your gzip function relies on Python 2.7+ I believe and I only have 2.6 on my Centos box.

The only pedantic thing I could spot in your program was on line #19 (Count of loaded day records) where the indentation is wrong. But that's nitpicking. :)

Now I just need to think of a use for this.... Hmmm.... Maybe if I rewrote it in Perl.....
 
Sure, no problem. I started with a Python script that I found somewhere. Then I used cstats.[ch] from the latest release and a hex editor. I'm not sure why, but the format of the file doesn't exactly match the code.

The output format is customized for my use, so you'll have to modify it to get what you need.

Please let me know if you make any changes that I might be able to use.

https://drive.google.com/file/d/0B7AAdqGpSE8ZS1ZoX1lMZkpyNGc/view?usp=sharing

Python script might have been from here: http://pastebin.com/bZxLpmQL
 
Ah. That looks to be a slightly older version of this: https://github.com/emjelde/rstats/blob/master/RStatsHistory.py

(Which, together with RStatsOutput.py, were the only previously know programs that read the rstats file, but not the cstats)

You're right, they look very similar, but I still don't remember where the cstats version came from. Weird.

I'll let you know when I'm done with the script for /proc/net/ipt_account/lan. For now, I'm playing with fprobe --> ntop and PRTG.
 
Tried using cstats and...it fails:

python cstats.py /mnt/sda2/cstats/tomato_rstats_1cb72cc379c0.gz

>>> Warning!

Read 0 bytes.

Expected to read 2112 bytes.

Left to read 2112 bytes
 
You have to use it against a cstats file, not rstats. A cstats file is created when you enable "per IP monitoring" (IPTraffic).
 
You have to use it against a cstats file, not rstats. A cstats file is created when you enable "per IP monitoring" (IPTraffic).

Screenshot attached. And the tomato_rstats_1cb72cc379c0.gz is 65 bytes in size all the time.
Also, the 'TrafficAnalyzer_Statistic.asp' is able to display all the stats... so either something is wrong or I'm missing something...
 

Attachments

  • Screenshot 2018-10-18 at 10.40.23.png
    Screenshot 2018-10-18 at 10.40.23.png
    150.8 KB · Views: 419
Screenshot attached. And the tomato_rstats_1cb72cc379c0.gz is 65 bytes in size all the time.
Also, the 'TrafficAnalyzer_Statistic.asp' is able to display all the stats... so either something is wrong or I'm missing something...
Did you select "Create or reset IPTraffic data files"?
 
Yes I did. As you can see file is being created and I am able to reads the data, so it seems like this option creates rstat file with In/Out data only...just wondering where is the data stored with IPs and hosts etc...the one that is being used in the 'TrafficAnalyzer_Statistic.asp' section
 
Yes I did. As you can see file is being created and I am able to reads the data,
No I don't see that. You posted output using an rstats file, not a cstats file.

so it seems like this option creates rstat file with In/Out data only...just wondering where is the data stored with IPs and hosts etc...the one that is being used in the 'TrafficAnalyzer_Statistic.asp' section
The cstats file contains the per IP information, not rstats.

I don't use the Traffic Analyser but I think its data file is under /jffs. I believe it's some sort of SQL database file.
 
Oh, ok. So, there are rstats and cstats, only after I have checked both:
"Create or reset data file" and "Create or reset IPTraffic data files".

Summary:
rstats contains info about In/Out only.
cstats contains info about In/Out per IP.

There are multiple python scripts on github, but only one works properly (attached) for cstats (screenshot attached).
Still no data there..I'll wait couple hours, maybe it is being cached or sth.

But now I have also realised those stats are still missing destination IP/Hosts which are visible on the Traffic Analyser site.
So I have checked the /jffs and there is a file called 'traffic.db' which contains all the information I needed and it is in sql format :)

So I guess there is no point in generating rstats and cstats then...
 

Attachments

  • cstats.py.txt
    4.1 KB · Views: 469
  • Screenshot 2018-10-18 at 11.18.26.png
    Screenshot 2018-10-18 at 11.18.26.png
    98.2 KB · Views: 456
Traffic Analyzer is unrelated to rstats or cstats. It's based on Trend Micro's DPI engine, which is a completely separate thing.
 

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top