Web History Reporting and Management (Traffic Analyzer/Aiprotection Monitor)

[garett_09]

Hey there, I am using AX86U which is on the latest firmware of 388.2_2. It seems to not return an issue like that but what I found is the count of the total data usage is different from the GUI page.

 

[Martineau]

Hey there, I am using AX86U which is on the latest firmware of 388.2_2. It seems to not return an issue like that but what I found is the count of the total data usage is different from the GUI page.

View attachment 51381

You could try running the report (in debug mode) to calculate the usage for the 24hour period

sh -x ./TrafficAnalyzer_Report.sh  date=2023/06/27 time=

Then cut'n' paste say the last 50 lines of the output

e.g.

+ nvram get tmp_RX_TOTAL
+ RX_TOTAL=42209946
+ nvram unset tmp_RX_TOTAL
+ nvram get tmp_TX_TOTAL
+ TX_TOTAL=207622244
+ nvram unset tmp_TX_TOTAL
+ printf %b%-12s%-12s\n \e[93m\e[K ----------- ------------
----------- ------------
+ [ -z ]
+ Size_Human 42209946
+ local SIZE=42209946
+ [ -z 42209946 ]
+ [ 42209946 -gt 1073741824 ]
+ [ 42209946 -gt 1048576 ]
+ echo 42209946
+ awk {$1=$1/(1024^2);   print $1;}
+ printf %2.2f MB\n 40.2545
+ return 0
+ Size_Human 207622244
+ local SIZE=207622244
+ [ -z 207622244 ]
+ [ 207622244 -gt 1073741824 ]
+ [ 207622244 -gt 1048576 ]
+ echo+  207622244
awk {$1=$1/(1024^2);   print $1;}
+ printf %2.2f MB\n 198.004
+ return 0
+ printf %b%-12s%-12s\n \e[92m\e[K 40.25 MB 198.00 MB
40.25 MB    198.00 MB
+ printf %b%-12s%-12s\n\n \e[93m\e[K =========== ============
=========== ============

+ [ -n  ]
+ [ -n  ]
+ nvram get tmp_TA_TOTAL
+ RESULT_CNT=442
+ nvram unset tmp_TA_TOTAL
+ [ -z 442 ]
+ [ 442 -eq 0 ]
+ IND=\e[92m
+ [ -z  ]
+ [ -n  ]
+ [ 442 -le 20 ]
+ echo -e \n\e[0m\e[7mSummary: Result count = 442 \e[27m

Summary: Result count = 442
+ nvram get TM_EULA
+ [ 1 -eq 0 ]
+ nvram get bwdpi_db_enable
+ [ 1 -eq 0 ]
+ echo -e \e[0m

+ exit 0

in an attempt to diagnose if the conversion of the Tx/Rx totals is failing, or if the accumulated totals are incorrect perhaps due to the expr causing an arithmetic overflow.

i.e. having reset the router, 238 MB shown in the GUI imatches the report total debugged above 40.25 MB + 198.00 MB

 

[garett_09]

I tried your suggestion and this is what it came up with

+ [ -z ]
+ Size_Human 78625915
+ local SIZE=78625915
+ [ -z 78625915 ]
+ [ 78625915 -gt 1073741824 ]
+ [ 78625915 -gt 1048576 ]
+ echo 78625915
+ awk {$1=$1/(1024^2);   print $1;}
+ printf %2.2f MB\n 74.9835
+ return 0
+ Size_Human 5768398859
+ local SIZE=5768398859
+ [ -z 5768398859 ]
+ [ 5768398859 -gt 1073741824 ]
+ echo 5768398859
+ awk {$1=$1/(1024^3); print $1;}
+ printf %2.2f GB\n 5.37224
+ return 0
+ printf %b%-12s%-12s\n \e[92m\e[K 74.98 MB 5.37 GB
74.98 MB    5.37 GB    
+ printf %b%-12s%-12s\n\n \e[93m\e[K =========== ============
=========== ============

+ [ -n  ]
+ [ -n  ]
+ nvram get tmp_TA_TOTAL
+ RESULT_CNT=200
+ nvram unset tmp_TA_TOTAL
+ [ -z 200 ]
+ [ 200 -eq 0 ]
+ IND=\e[92m
+ [ -z  ]
+ [ -n  ]
+ [ 200 -le 20 ]
+ echo -e \n\e[0m\e[7mSummary: Result count = 200 \e[27m

Summary: Result count = 200
+ nvram get TM_EULA
+ [ 1 -eq 0 ]
+ nvram get bwdpi_db_enable
+ [ 1 -eq 0 ]
+ echo -e \e[0m

I did what you suggested and placed a time period of 23 so it means the period of 22H-23H yesterday.

So, 75MB plus 5.37GB returns me a correct result of 5.45GB.


The problem would be running the script like

sh /jffs/scripts/Traffic*.sh

returns me a time period in which I do not know where and does not add the whole timeperiod for the day.

Thanks!

EDIT: How do i run the script calculating for the whole day?

 

[garett_09]

I just used nofilter and returned the correct amount of data usage. Now, I want to send this to my email, So I will add a CRON Job to the script. How do I add the sendEmail function?

Thanks!!

 

[Martineau]

The problem would be running the script like

sh /jffs/scripts/Traffic*.sh

returns me a time period in which I do not know where and does not add the whole timeperiod for the day.

Thanks!

EDIT: How do i run the script calculating for the whole day?

Running the script with no arguments defaults to filtering records for the current hour for 'today'

e.g.

sh /jffs/scripts/Traffic*.sh

Processing '/jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db' database....please wait!
(TrafficAnalyzer_Report.sh): 28463 v1.15 Traffic Analyzer starting.....
    NOTE: Columns in white are eligible for filters; red text indicates a match on the filters requested. (URLs are Xshell5/MobaXterm hyperlinks)

    Filter by Today, AND by current hour ==> '2023-06-28|08:'

To get a report on a specified 24 hour period you can override the hour

e.g. 27th June 2023

sh /jffs/scripts/Traffic*.sh date=2023/06/27 time=

Processing '/jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db' database....please wait!
(TrafficAnalyzer_Report.sh): 32059 v1.15 Traffic Analyzer starting.....
    NOTE: Columns in white are eligible for filters; red text indicates a match on the filters requested. (URLs are Xshell5/MobaXterm hyperlinks)

    Filter by Date ==> '2023-06-27'

 

[garett_09]

Running the script with no arguments defaults to filtering records for the current hour for 'today'

e.g.

sh /jffs/scripts/Traffic*.sh

Processing '/jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db' database....please wait!
(TrafficAnalyzer_Report.sh): 28463 v1.15 Traffic Analyzer starting.....
    NOTE: Columns in white are eligible for filters; red text indicates a match on the filters requested. (URLs are Xshell5/MobaXterm hyperlinks)

    Filter by Today, AND by current hour ==> '2023-06-28|08:'

To get a report on a specified 24 hour period you can override the hour

e.g. 27th June 2023

sh /jffs/scripts/Traffic*.sh date=2023/06/27 time=

Processing '/jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db' database....please wait!
(TrafficAnalyzer_Report.sh): 32059 v1.15 Traffic Analyzer starting.....
    NOTE: Columns in white are eligible for filters; red text indicates a match on the filters requested. (URLs are Xshell5/MobaXterm hyperlinks)

    Filter by Date ==> '2023-06-27'

Thanks for that! I am curious how the email function works? Looking at the script, the sendEmail function does not have any code in it. I would like to send it to my email everyday by running a Cron job.

Thanks

 

[Martineau]

I just used nofilter and returned the correct amount of data usage. Now, I want to send this to my email, So I will add a CRON Job to the script. How do I add the sendEmail function?

View attachment 51396

Originally I hard-coded the in-line SendMail() function for a Gmail account and at the time was criticised for not supporting other providers.

Things have moved on in the four and a half years since I wrote the script, and external support for other email accounts is now available via the em feature of amtm.

So if you setup your personal email credentials in amtm you can simply copy'n'paste the following code, either as a stand-alone script for use by cron or edit TrafficAnalyzer.sh either to call the stand-alone script

i.e. change line

    Say "You need to edit this script and add the Sendmail function first!"

to

[ -f /jffs/scripts/AMTM_Gmail.sh ] && sh /jffs/scripts/AMTM_Gmail.sh "/tmp/TrafficAnalyzer.txt" ||  Say "You need to edit this script and add the Sendmail function first!"

or include the code in-line as the 'missing' SendMail() function

e.g. AMTM_GMail.sh

#!/bin/sh

# To manually set up sendmail, see https://github.com/RMerl/asuswrt-merlin.ng/wiki/Sending-Email

# or use option 'em' in amtm to set the email credentials for the supported Server i.e. GMail, Outlook, Yahoo! etc.

REPORT=$1
[ -z "$REPORT" ] && REPORT="/tmp/TrafficAnalyzer.txt"    # Default to Traffic Analyzer Report 

SENDMAIL="/tmp/mail.txt"

# Header
echo "Subject:Traffic Analyzer Report"  >$SENDMAIL         
# Body of email
echo "Date: $(date -R)"                >>$SENDMAIL
echo "Body: SSL/TLS"                   >>$SENDMAIL
echo "Tx Bytes      Rx Bytes       YYYY/MM/DD HH:MM:SS MAC address        Host Name         IP address       Category                         Application" >>$SENDMAIL
cat $REPORT                            >>$SENDMAIL         

# Use amtm email function
. /jffs/addons/amtm/mail/email.conf
EMAIL_DIR=/jffs/addons/amtm/mail

# Use curl (rather than sendmail)
/usr/sbin/curl $verbose --url $PROTOCOL://$SMTP:$PORT                         \
                        --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
                        --upload-file $SENDMAIL                               \
                        --ssl-reqd                                            \
                        --user "$USERNAME:$(/usr/sbin/openssl aes-256-cbc $emailPwEnc -d -in "${EMAIL_DIR}/emailpw.enc" -pass pass:ditbabot,isoi)" $SSL_FLAG

#rm $SENDMAIL

logger -st "($(basename $0))" "e-mail sent using curl smtps:// SSL/TLS (non-Certificate) ${SMTP}:${PORT}"

 

[garett_09]

Originally I hard-coded the in-line SendMail() function for a Gmail account and at the time was criticised for not supporting other providers.

Things have moved on in the four and a half years since I wrote the script, and external support for other email accounts is now available via the em function of amtm.

So if you setup your personal email credentials in amtm you can simply copy'n'paste the following code, either as a stand-alone script for use by cron or include it in-line as the 'missing' SendMail() function

#!/bin/sh

# To manually set up sendmail, see https://github.com/RMerl/asuswrt-merlin.ng/wiki/Sending-Email

# or use option 'em' in amtm to set the email credentials for the supported Server i.e. GMail, Outlook, Yahoo! etc.

REPORT=$1
[ -z "$REPORT" ] && REPORT="/tmp/TrafficAnalyzer.txt"    # Default to Traffic Analyzer Report  

SENDMAIL="/tmp/mail.txt"

# Header
echo "Subject:Traffic Analyzer Report"  >$SENDMAIL          
# Body of email
echo "Date: $(date -R)"                >>$SENDMAIL
echo "Body: SSL/TLS"                   >>$SENDMAIL
echo "Tx Bytes      Rx Bytes       YYYY/MM/DD HH:MM:SS MAC address        Host Name         IP address       Category                         Application" >>$SENDMAIL
cat $REPORT                            >>$SENDMAIL          

# Use amtm email function
. /jffs/addons/amtm/mail/email.conf
EMAIL_DIR=/jffs/addons/amtm/mail

# Use curl (rather than sendmail)
/usr/sbin/curl $verbose --url $PROTOCOL://$SMTP:$PORT                         \
                        --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
                        --upload-file $SENDMAIL                               \
                        --ssl-reqd                                            \
                        --user "$USERNAME:$(/usr/sbin/openssl aes-256-cbc $emailPwEnc -d -in "${EMAIL_DIR}/emailpw.enc" -pass pass:ditbabot,isoi)" $SSL_FLAG

#rm $SENDMAIL

logger -st "($(basename $0))" "e-mail sent using curl smtps:// SSL/TLS (non-Certificate) ${SMTP}:${PORT}"

Awesome!!

Just pasted this in the script then it worked out instantly. I will set it at a cron job which sends me the stats every 12 hours! thanks again!

 

[garett_09]

Hey, @Martineau! How can I send an email by just sending this only?

 

[Martineau]

Hey, @Martineau! How can I send an email by just sending this only? View attachment 51465

Until I get around to adding say the supplementary emailsummary switch....you would filter out the contents of $REPORT

i.e. replace the SendMail() function line

cat $REPORT >>SENDMAIL

with

tail -n3 $REPORT >>SENDMAIL

or for just the data line

tail -n2 $REPORT | head -n1 >>SENDMAIL

 

[garett_09]

Until I get around to adding say the supplementary emailsummary switch....you would filter out the contents of $REPORT

i.e. replace the SendMail() function line

cat $REPORT >>SENDMAIL

with

tail -n3 $REPORT >>SENDMAIL

or for just the data line

tail -n2 $REPORT | head -n1 >>SENDMAIL

thanks for this one, mate!