Extracting Real-time Stats

Glenn Freidet · Mar 28, 2016

Hi all.

I'm trying to get at the real-time IP stats on Main_TrafficMonitor_devrealtime.asp so i can view them outside of the web page. I can see the info I need when I use View Source in the browser, but I haven't been able to get the same info outside of the browser. I don't know much about ASP, but I'd say I'm running into an authentication problem. I'll bet someone has already done this. Is there some other way to get the stats or does anyone have a clue about this?

Thanks.
-G

RMerlin · Mar 29, 2016

Glenn Freidet said:
Hi all.

I'm trying to get at the real-time IP stats on Main_TrafficMonitor_devrealtime.asp so i can view them outside of the web page. I can see the info I need when I use View Source in the browser, but I haven't been able to get the same info outside of the browser. I don't know much about ASP, but I'd say I'm running into an authentication problem. I'll bet someone has already done this. Is there some other way to get the stats or does anyone have a clue about this?

Thanks.
-G

The stats are gathered and provided by the web server, which inserts it into the webpage before providing it to the web browsers. Most of it come from the kernel itself.

Sent from my Nexus 9 using Tapatalk

Glenn Freidet · Mar 29, 2016

RMerlin said:
The stats are gathered and provided by the web server, which inserts it into the webpage before providing it to the web browsers. Most of it come from the kernel itself.

Hi RMerlin. Right, but the problem is that whichever method I use, I only get back a short header. For example, when I'm already logged in, wget returns:

<HTML><HEAD><script>top.location.href='/Main_Login.asp?error_status=9&page=index.asp&lock_time=0';</script>
</HEAD></HTML>

If I'm not logged in, error_status=1. Either way, I don't get the HTML that was returned by ASP.

RMerlin · Mar 29, 2016

Glenn Freidet said:
Hi RMerlin. Right, but the problem is that whichever method I use, I only get back a short header. For example, when I'm already logged in, wget returns:

<HTML><HEAD><script>top.location.href='/Main_Login.asp?error_status=9&page=index.asp&lock_time=0';</script>
</HEAD></HTML>

If I'm not logged in, error_status=1. Either way, I don't get the HTML that was returned by ASP.

You cannot directly query the web server, you'd need to provide a session ticket to authenticate yourself. This is to protect against XSS exploits.

Zirescu · Mar 29, 2016

Why not use the ASUS iOS or Android app?

Glenn Freidet · Mar 29, 2016

RMerlin said:
You cannot directly query the web server, you'd need to provide a session ticket to authenticate yourself. This is to protect against XSS exploits.

Not sure what a session ticket is, so I guess I won't be going down that road.

Is it possible to explain why Chrome - View Source can get the HTML, but I can't?

I can live without live stats, but I need at least daily u/d by IP, and it has to be downloadable. rstats/cstats doesn't appear to provide a breakdown by IP. Tcpdump pipped into awk works fine with low network activity, but I'm guessing it will kill the CPU when the traffic gets heavier. Also, it's a pretty weak hack. Port-monitoring to Wireshark has activity/CPU issues, and it's massive overkill.

Any suggestions?

Glenn Freidet · Mar 29, 2016

Zirescu said:
Why not use the ASUS iOS or Android app?

I'm not sure what you mean by that. Can you explain?

Zirescu · Mar 29, 2016

If you were just looking for real-time data than the App would likely suit your needs, but based on your new info, probably won't. iOS app link: https://itunes.apple.com/us/app/asus-router/id1033794044?mt=8

Glenn Freidet · Mar 30, 2016

Zirescu said:
If you were just looking for real-time data than the App would likely suit your needs, but based on your new info, probably won't. iOS app link: https://itunes.apple.com/us/app/asus-router/id1033794044?mt=8

Thanks for the suggestion anyway.

RMerlin · Mar 30, 2016

Glenn Freidet said:
Not sure what a session ticket is, so I guess I won't be going down that road. Is it possible to explain why Chrome - View Source can get the HTML, but I can't?

When requesting a webpage, you must be authenticated with the web server to be authorized to retrieve the page. That is done through a session ticket. The ticket is generated by the router when you log in, and is exchanged between the browser and the server with every web request, through either a cookie or a POSTed var, I don't know (I've never studied the detail of this mechanism, which replaced the old insecure Basic Authentication mechanism of old). Your wget doesn't have a valid session ticket to provide to the router, so the router denies its request.

Glenn Freidet said:
I can live without live stats, but I need at least daily u/d by IP, and it has to be downloadable. rstats/cstats doesn't appear to provide a breakdown by IP. Tcpdump pipped into awk works fine with low network activity, but I'm guessing it will kill the CPU when the traffic gets heavier. Also, it's a pretty weak hack. Port-monitoring to Wireshark has activity/CPU issues, and it's massive overkill.

Any suggestions?

You would have to develop your own program that can process the database stored in rstats. Not impossible, but not trivial either (the database uses some fancy list-based format - no idea why its original author went with something so complicated instead of the simple method used by cstats, which stores the global traffic data).

What would be simpler is to develop your own parser instead of relying on rstats's own database. IPTraffic data is retrieved through the iptraffic netfilter module. The data is readable this way:

Code:

cat /proc/net/ipt_account/lan

For a (C) example of a parser (and also to see what each fields contains), see the parser Asuswrt uses:

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/httpd/web.c#L8254

You could have a script that reads the data there every hour, parse it, and store the resulting data into a text file (a CSV file for example). Check if the counter you read are higher than the last one - if they are, then add the data. If not, then it means the router was rebooted, and you will have to take that into account in that day's total.

Resulting CSV could be stored on an SMB share (through a plugged USB disk), or sent through Email using a cron job.

RMerlin · Mar 30, 2016

Glenn Freidet said:
rstats/cstats doesn't appear to provide a breakdown by IP.

rstats is by IP - it's what is used by IPTraffic.

If by IP you mean destination IP, then you will need a more advanced tool, such as NTop. Many years ago, I got ntop+RRD to work under DD-WRT. That was a lifetime ago however, and I've forgotten all the details - sorry.

ColinTaylor · Mar 30, 2016

RMerlin said:
rstats is by IP - it's what is used by IPTraffic.

Actually, rstats is the global file and cstats is the per IP file.

RMerlin · Mar 30, 2016

ColinTaylor said:
Actually, rstats is the global file and cstats is the per IP file.

Thanks, I always get these two mixed up

Glenn Freidet · Mar 30, 2016

RMerlin said:
When requesting a webpage, you must be authenticated with the web server to be authorized to retrieve the page. That is done through a session ticket. The ticket is generated by the router when you log in, and is exchanged between the browser and the server with every web request, through either a cookie or a POSTed var, I don't know (I've never studied the detail of this mechanism, which replaced the old insecure Basic Authentication mechanism of old). Your wget doesn't have a valid session ticket to provide to the router, so the router denies its request.

Amazing response! Thank you SO much.

So, even if the following worked, and it doesn't, I would have to be logged out of the GUI in order for it to work?

Code:

wget --keep-session-cookies --save-cookies cookies.txt --post-data 'user=admin&password=XXX' http://10.0.1.1/Main_Login.asp
wget --load-cookies cookies.txt -p http://10.0.1.1/Main_TrafficMonitor_devrealtime.asp

Correct?

Glenn Freidet · Mar 30, 2016

RMerlin said:
What would be simpler is to develop your own parser instead of relying on rstats's own database. IPTraffic data is retrieved through the iptraffic netfilter module. The data is readable this way:

Code:

cat /proc/net/ipt_account/lan

For a (C) example of a parser (and also to see what each fields contains), see the parser Asuswrt uses:

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/httpd/web.c#L8254

You could have a script that reads the data there every hour, parse it, and store the resulting data into a text file (a CSV file for example). Check if the counter you read are higher than the last one - if they are, then add the data. If not, then it means the router was rebooted, and you will have to take that into account in that day's total.

Resulting CSV could be stored on an SMB share (through a plugged USB disk), or sent through Email using a cron job.

Interesting idea. In the end, this is probably the way I'll go. Can you give me a clue about what would make iptraffic generate that file? It's empty on my machine.

Glenn Freidet · Mar 30, 2016

RMerlin said:
rstats is by IP - it's what is used by IPTraffic.

If by IP you mean destination IP, then you will need a more advanced tool, such as NTop. Many years ago, I got ntop+RRD to work under DD-WRT. That was a lifetime ago however, and I've forgotten all the details - sorry.

I don't see individual IP data in cstats, just totals and the 24-hour RRD. I must have missed something in my script. I'll look closer.

Glenn Freidet · Mar 30, 2016

Sorry about exploding this thread. I just have a couple more thoughts about individual IP stats:

I'm currently looking at scrapping HTML from darkstats, but I'm not sure it's stable enough. It won't have the session ticket problem, correct?

I'm also looking at fprobe->PRTG/Netflow. Am I wasting my time?

Thanks again for all your help!

RMerlin · Mar 30, 2016

Glenn Freidet said:
Interesting idea. In the end, this is probably the way I'll go. Can you give me a clue about what would make iptraffic generate that file? It's empty on my machine.

Make sure you enable IPTraffic under Tools -> Other Settings. This will load the ipt_account module, and set it up to monitor your LAN subnet.

RMerlin · Mar 30, 2016

Glenn Freidet said:
Sorry about exploding this thread. I just have a couple more thoughts about individual IP stats:

I'm currently looking at scrapping HTML from darkstats, but I'm not sure it's stable enough. It won't have the session ticket problem, correct?

No idea what darkstat is, but I assume it's using its own web server and not Asus's, so it will be fine authentication-wise.

Glenn Freidet said:
I'm also looking at fprobe->PRTG/Netflow. Am I wasting my time?

Must have been about 10 years since I last looked at NTOP + Netflow - sorry...

RMerlin · Mar 30, 2016

Glenn Freidet said:
Amazing response! Thank you SO much.

So, even if the following worked, and it doesn't, I would have to be logged out of the GUI in order for it to work?

Yes. Asuswrt's httpd only allows one concurrent user.

Extracting Real-time Stats

Occasional Visitor

Asuswrt-Merlin dev

Occasional Visitor

Asuswrt-Merlin dev

Very Senior Member

Occasional Visitor

Occasional Visitor

Very Senior Member

Occasional Visitor

Asuswrt-Merlin dev

Asuswrt-Merlin dev

Part of the Furniture

Asuswrt-Merlin dev

Occasional Visitor

Occasional Visitor

Occasional Visitor

Occasional Visitor

Asuswrt-Merlin dev

Asuswrt-Merlin dev

Asuswrt-Merlin dev

Similar threads

Sign Up For SNBForums Daily Digest