Per IP Traffic accounting

eightiescalling · Sep 5, 2021

Out of curiosity, what's behind the need for real time data? Beyond data points in influx and graphs what do you need the data real time for?

Just wondering whether there may be other ways to achieve the end goal.

mekabe remain · Sep 5, 2021

eightiescalling said:
Out of curiosity, what's behind the need for real time data? Beyond data points in influx and graphs what do you need the data real time for?

Just wondering whether there may be other ways to achieve the end goal.

ok. good approach.

I am testing integrations at home environment. Sometimes IPTV services , sometimes other services.
For example testing an IPTV service on several devices (Samsung TV, LG TV, Android TV , mobile device etc.) , I need to find which device used how much bandwidth.
With my old router (AC88U) , I was able to get iptables accounting data per 2 or 5 minutes and then calculate average bandwidth usage from there.
Now this is not possible.
Traffic Analyzer just saves the total used TX RX per session , at the end of the session. We can of course calculate a bits/sec data from that total session but start and end of the session (incline/decline) will make it not usable.
Of course for this specific reason, I can collect data using tcpdump and then post process.
But it is not as easy as recording all data with accounting.

For the fancy part; I really love telling my visitors "you are now using X of my bandwidth"

mekabe remain · Sep 7, 2021

so , there is no hope ?

Martineau · Sep 8, 2021

mekabe remain said:
so , there is no hope ?

You may be able to use this crude monitoring script

[EDIT]Real-time Quota Monitoring (Download/Upload/Data Used)

There was a query recently about how to detect DOWNLOAD throughput activity and BLOCK the device if deemed appropriate so I have revised my suggested solution. Using the Traffic Analyzer reports, you can review device activity, but real-time monitoring may be useful. So here is my shoddy...

www.snbforums.com

as-is, or extract/adapt the necessary portions of code to meet your requirements.

mekabe remain · Sep 11, 2021

Martineau said:
You may be able to use this crude monitoring script

[EDIT]Real-time Quota Monitoring (Download/Upload/Data Used)

There was a query recently about how to detect DOWNLOAD throughput activity and BLOCK the device if deemed appropriate so I have revised my suggested solution. Using the Traffic Analyzer reports, you can review device activity, but real-time monitoring may be useful. So here is my shoddy...

www.snbforums.com

as-is, or extract/adapt the necessary portions of code to meet your requirements.

thanks. but I could not figure out how to use this script.
in fact, I believe it is not even compatible with current firmware.

Because I get the following error:

Code:

admin@RT-AX88U-21C1:/jffs/scripts# ./QuotaMonitor.sh   init    ip=XEON-POWER

 v1.04 Quota Monitoring.....

        Quota Monitoring Setup complete.

./QuotaMonitor.sh: line 1: ExpandIPRange: not found

so it is looking for a "ExpandIPRange" command but that is not available.
any ideas ?

Martineau · Sep 12, 2021

mekabe remain said:
thanks. but I could not figure out how to use this script.

The help should give you the options/usage examples

Code:

QuotaMonitor.sh  -h

#======================================================================================= © 2018-2020 Martineau v1.06
#
# Monitor device traffic quotas and apply block if limit exceeded (Alternative to Traffic Analyzer i.e. eliminates need to accept TrendMicro's EULA)
#
#         NOTE: Requires H/W Acceleration OFF? - LAN->Switch Control->'NAT Acceleration=DISABLE' i.e. NAT traffic is processed by CPU.
#               For HND models e.g. RT-AC86U - 'Runner=ENABLED/Flow cache=ENABLED' is preferred for performance, so probably wise to stick to Traffic Analyzer.
#
#    QuotaMonitor     [help | -h] ['ip='{[ip_address[,...] | hostname[...] | auto | all ]} ['remove']] ['init'[qos]] ['reset' | 'resetdel'] ['unblock'] ['zero']
#                     ['monitor'] ['interval='{interval_seconds}] ['dlimit={interval_limit}'] ['ulimit={interval_limit}']  ['quota='['rx'|'tx'|'used']]
#                     ['cap='{cap_limit}] ['ignorezero'] ['actionrx='{script_name}] ['actiontx='{script_name}] ['actionused='{script_name}] ['nolog'] ['qos']
#                     ['report='{path_for_quota.csv}] ['once'] ['status'[ 'verbose']]
#
#    QuotaMonitor     init
#                     Create the two Quota Monitor Chains (MyQUOTAi/MyQUOTAo) for DOWNLOAD/UPLOAD
#                     Individual device Quota limits may be defined in '/jffs/configs/QuotaLimits' but will be ignored if 'dlimit='/'ulimit=' specified
#                     e.g.
#                         #Hostname      Rx     Tx     Cap
#                         HP-Envy13   300MB   20MB   100GB
#                     NOTE: Quota limits may only be applied to the same 'interval=' value
#    QuotaMonitor     status
#                     Show the contents of the two Chains. The USED var is derived from the RECV/XMIT byte count!
#    QuotaMonitor     ip=192.168.1.123,iphone,laptop
#                     Add three devices to be monitored
#    QuotaMonitor     ip=auto
#                     Attempt to auto discover all LAN devices and add them to the monitor list.
#    QuotaMonitor     monitor
#                     Start Rx/Tx and Total Used quota monitoring for the devices defined (default limits 50MB in 60 secs Capped @50GB)
#                     If the limits are exceeded, then the device is BLOCKED.
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices defined in the Chains (uses 5MB in 10 secs Capped @100MB)
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB ignorezero quota=
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices, but devices which have not received/transmitted data are not displayed.
#                     This mode is useful for identifying suitable quota limits, as no quota metric (Rx,Tx or Used) is enforced
#                     Useful for testing, to prove if the Quota limits can be ENFORCED in real-time
#    QuotaMonitor     monitor actionRx=Quota_Rx.sh
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded run script Quota_Rx.sh
#    QuotaMonitor     monitor qos
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded apply QoS rules
#                     NOTE: 'initqos' must have been previously specified.
#    QuotaMonitor     ip=laptop unblock
#                     Unblock a specified device.
#    QuotaMonitor     ip=all unblockqos
#                     Unblock ALL devices that are currently throttled by QoS.
#    QuotaMonitor     ip=laptop remove
#                     Remove the device 'laptop' from the Quota Monitoring.
#                     NOTE: This will require a second instance of the script to allow uninterrupted monitoring
#    QuotaMonitor     reset
#                     Removes all devices from monitoring but retains the Chains.
#    QuotaMonitor     resetdel
#                     Deletes both chains and all iptables rules
#    QuotaMonitor     zero
#                     Resets the Quota counts to 0. Could be scheduled by a cron job every day @00:00 etc.
#    QuotaMonitor     monitor nolog
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog
#    QuotaMonitor     monitor nolog report=/tmp/mnt/xxx/Quota.csv once quota=
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog, instead write the results to '/tmp/mnt/xxx/Quota.csv'
#                     NOTE: **WARNING" If you schedule it via cron with the 'once' option YOU MUST SPECIFY 'quota=' otherwise it may apply the Quota BLOCK!

mekabe remain said:
I believe it is not even compatible with current firmware.

Perhaps you are undoubtedly correct

, but 'line 1' errors can indicate that the script is not in Linux format.

i.e. Have you used

Code:

dos2unix QuotaMonitor.sh

to ensure the script is compatible by removing the Windows CRLF chars ?

mekabe remain · Sep 12, 2021

Martineau said:

The help should give you the options/usage examples

Code:

QuotaMonitor.sh  -h

#======================================================================================= © 2018-2020 Martineau v1.06
#
# Monitor device traffic quotas and apply block if limit exceeded (Alternative to Traffic Analyzer i.e. eliminates need to accept TrendMicro's EULA)
#
#         NOTE: Requires H/W Acceleration OFF? - LAN->Switch Control->'NAT Acceleration=DISABLE' i.e. NAT traffic is processed by CPU.
#               For HND models e.g. RT-AC86U - 'Runner=ENABLED/Flow cache=ENABLED' is preferred for performance, so probably wise to stick to Traffic Analyzer.
#
#    QuotaMonitor     [help | -h] ['ip='{[ip_address[,...] | hostname[...] | auto | all ]} ['remove']] ['init'[qos]] ['reset' | 'resetdel'] ['unblock'] ['zero']
#                     ['monitor'] ['interval='{interval_seconds}] ['dlimit={interval_limit}'] ['ulimit={interval_limit}']  ['quota='['rx'|'tx'|'used']]
#                     ['cap='{cap_limit}] ['ignorezero'] ['actionrx='{script_name}] ['actiontx='{script_name}] ['actionused='{script_name}] ['nolog'] ['qos']
#                     ['report='{path_for_quota.csv}] ['once'] ['status'[ 'verbose']]
#
#    QuotaMonitor     init
#                     Create the two Quota Monitor Chains (MyQUOTAi/MyQUOTAo) for DOWNLOAD/UPLOAD
#                     Individual device Quota limits may be defined in '/jffs/configs/QuotaLimits' but will be ignored if 'dlimit='/'ulimit=' specified
#                     e.g.
#                         #Hostname      Rx     Tx     Cap
#                         HP-Envy13   300MB   20MB   100GB
#                     NOTE: Quota limits may only be applied to the same 'interval=' value
#    QuotaMonitor     status
#                     Show the contents of the two Chains. The USED var is derived from the RECV/XMIT byte count!
#    QuotaMonitor     ip=192.168.1.123,iphone,laptop
#                     Add three devices to be monitored
#    QuotaMonitor     ip=auto
#                     Attempt to auto discover all LAN devices and add them to the monitor list.
#    QuotaMonitor     monitor
#                     Start Rx/Tx and Total Used quota monitoring for the devices defined (default limits 50MB in 60 secs Capped @50GB)
#                     If the limits are exceeded, then the device is BLOCKED.
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices defined in the Chains (uses 5MB in 10 secs Capped @100MB)
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB ignorezero quota=
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices, but devices which have not received/transmitted data are not displayed.
#                     This mode is useful for identifying suitable quota limits, as no quota metric (Rx,Tx or Used) is enforced
#                     Useful for testing, to prove if the Quota limits can be ENFORCED in real-time
#    QuotaMonitor     monitor actionRx=Quota_Rx.sh
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded run script Quota_Rx.sh
#    QuotaMonitor     monitor qos
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded apply QoS rules
#                     NOTE: 'initqos' must have been previously specified.
#    QuotaMonitor     ip=laptop unblock
#                     Unblock a specified device.
#    QuotaMonitor     ip=all unblockqos
#                     Unblock ALL devices that are currently throttled by QoS.
#    QuotaMonitor     ip=laptop remove
#                     Remove the device 'laptop' from the Quota Monitoring.
#                     NOTE: This will require a second instance of the script to allow uninterrupted monitoring
#    QuotaMonitor     reset
#                     Removes all devices from monitoring but retains the Chains.
#    QuotaMonitor     resetdel
#                     Deletes both chains and all iptables rules
#    QuotaMonitor     zero
#                     Resets the Quota counts to 0. Could be scheduled by a cron job every day @00:00 etc.
#    QuotaMonitor     monitor nolog
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog
#    QuotaMonitor     monitor nolog report=/tmp/mnt/xxx/Quota.csv once quota=
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog, instead write the results to '/tmp/mnt/xxx/Quota.csv'
#                     NOTE: **WARNING" If you schedule it via cron with the 'once' option YOU MUST SPECIFY 'quota=' otherwise it may apply the Quota BLOCK!

Perhaps you are undoubtedly correct

, but 'line 1' errors can indicate that the script is not in Linux format.

i.e. Have you used

Code:

dos2unix QuotaMonitor.sh

to ensure the script is compatible by removing the Windows CRLF chars ?

I just tried this as you recommended. However it did not change anything.
in any case, this "ExpandIPRange" seems to be a command. it is not available on my environment (ax88u with merlin firmware)

any other ideas ?

Martineau · Sep 12, 2021

mekabe remain said:
I just tried this as you recommended. However it did not change anything.
in any case, this "ExpandIPRange" seems to be a command. it is not available on my environment (ax88u with merlin firmware)

any other ideas?

If you use the '-h' arg, does the script correctly present the help text?

john9527 · Sep 12, 2021

mekabe remain said:
any other ideas ?

Also check that the first line is a proper shebang
#!/bin/sh

I've seen cases where it was inexplicably corrupted or had an extra leading character inserted...

ColinTaylor · Sep 12, 2021

mekabe remain said:
thanks. but I could not figure out how to use this script.
in fact, I believe it is not even compatible with current firmware.

Because I get the following error:

Code:

admin@RT-AX88U-21C1:/jffs/scripts# ./QuotaMonitor.sh init ip=XEON-POWER v1.04 Quota Monitoring..... Quota Monitoring Setup complete. ./QuotaMonitor.sh: line 1: ExpandIPRange: not found

so it is looking for a "ExpandIPRange" command but that is not available.
any ideas ?

The problem is because the script is trying check whether "XEON-POWER" is a range or a name and the ExpandIPRange function is missing.

Code:

# ./QuotaMonitor.sh init ip=zen

v1.04 Quota Monitoring.....

        Quota Monitoring Setup complete.


        Quota Monitoring clients: zen (zen ) Added


# ./QuotaMonitor.sh init ip=XEON-POWER

v1.04 Quota Monitoring.....

        Quota Monitoring Setup complete.

./QuotaMonitor.sh: line 1: ExpandIPRange: not found

mekabe remain · Sep 12, 2021

Martineau said:
If you use the '-h' arg, does the script correctly present the help text?

yes

mekabe remain · Sep 12, 2021

john9527 said:
Also check that the first line is a proper shebang
#!/bin/sh

I've seen cases where it was inexplicably corrupted or had an extra leading character inserted...

the script is OK. it runs. it just can't find that specific command.

mekabe remain · Sep 12, 2021

ColinTaylor said:
The problem is because the script is trying check whether "XEON-POWER" is a range or a name and the ExpandIPRange function is missing.

Code:

# ./QuotaMonitor.sh init ip=zen v1.04 Quota Monitoring..... Quota Monitoring Setup complete. Quota Monitoring clients: zen (zen ) Added # ./QuotaMonitor.sh init ip=XEON-POWER v1.04 Quota Monitoring..... Quota Monitoring Setup complete. ./QuotaMonitor.sh: line 1: ExpandIPRange: not found

so where is that function ?

Martineau · Sep 12, 2021

mekabe remain said:
so where is that function ?

Does the script work if you specify the actual IP address rather than the hostname that contains the hyphen?

dave14305 · Sep 12, 2021

Martineau said:
Does the script work if you specify the actual IP address rather than the hostname that contains the hyphen?

Curious why you keep ignoring the fact that you have an undefined function in the script? Where did that function come from?

mekabe remain · Sep 13, 2021

Martineau said:
Does the script work if you specify the actual IP address rather than the hostname that contains the hyphen?

no it does not work.

Jack Yaz · Sep 13, 2021

dave14305 said:
Curious why you keep ignoring the fact that you have an undefined function in the script? Where did that function come from?

probably from this script: https://github.com/MartineauUK/TrafficAnalyzer-Report/blob/master/TrafficAnalyzer_Report.sh#L302

Martineau · Sep 13, 2021

mekabe remain said:
no it does not work.

OK, good to know.

EDIT: Some of my early scripts accepted a range of IPs as arguments, but this script was never tested to accept a group range of IPs on the commandline denoted by '-' without the group file '/jffs/configs/IPGroups' existing.

e.g. you can see that normally on my router the initialisation request fails gracefully because '/jffs/configs/IPGroups' currently exists (for use by my other scripts), but does not have the perceived range XEON-POWER defined.

However if I delete '/jffs/configs/IPGroups' and request the script again then the script does indeed panic

....... (until I reinstate the file)

You can temporarily try manually creating an empty 'IPGroups' file

Code:

touch /jffs/configs/IPGroups

to see if the script will now initialise until I can post a patched/tested version of the script (that has been unaltered on GitHub for nearly 2 1/2 years!) that either removes the reference to function ExpandIPRange() or includes it.

NOTE: The script may still fail/show inaccurate metrics if Hardware Acceleration is ENABLED.

Martineau · Sep 14, 2021

dave14305 said:
Curious why you keep ignoring the fact that you have an undefined function in the script?

Process of elimination of the usual suspects as evidenced by another respected coder @john9527 chiming in.......

Per IP Traffic accounting

Out of curiosity, what's behind the need for real time data? Beyond data points in influx and graphs what do you need the data real time for? Just wondering whether there may be other ways to achieve the end goal.

www.snbforums.com

together with the fact that the script doesn't bail-out unceremoniously in my environment where the function doesn't exist either.....

Per IP Traffic accounting

Out of curiosity, what's behind the need for real time data? Beyond data points in influx and graphs what do you need the data real time for? Just wondering whether there may be other ways to achieve the end goal.

www.snbforums.com

dave14305 said:
Where did that function come from?

Derives from 2016/2017 see my script IPCamsBlock

mekabe remain · Sep 14, 2021

Martineau said:
OK, good to know.

EDIT: Some of my early scripts accepted a range of IPs as arguments, but this script was never tested to accept a group range of IPs on the commandline denoted by '-' without the group file '/jffs/configs/IPGroups' existing.

e.g. you can see that normally on my router the initialisation request fails gracefully because '/jffs/configs/IPGroups' currently exists (for use by my other scripts), but does not have the perceived range XEON-POWER defined.

However if I delete '/jffs/configs/IPGroups' and request the script again then the script does indeed panic ....... (until I reinstate the file)

You can temporarily try manually creating an empty 'IPGroups' file

Code:

touch /jffs/configs/IPGroups

to see if the script will now initialise until I can post a patched/tested version of the script (that has been unaltered on GitHub for nearly 2 1/2 years!) that either removes the reference to function ExpandIPRange() or includes it.

Thanks for trying to help. But even if I create the IPgroups file , it still does not work for me. Please see below:

Code:

admin@RT-AX88U-26C8:/jffs/scripts# touch /jffs/configs/IPGroups
admin@RT-AX88U-26C8:/jffs/scripts# ./QuotaMonitor.sh init ip=XEON-POWER

 v1.07 Quota Monitoring.....

        Quota Monitoring Setup complete.

nslookup: can't resolve 'XEON-POWER'

                ***ERROR Hostname 'ip=XEON-POWER' INVALID

admin@RT-AX88U-26C8:/jffs/scripts# ./QuotaMonitor.sh init ip=MEDIASERVER

 v1.07 Quota Monitoring.....

        Quota Monitoring Setup complete.


nslookup: can't resolve 'MEDIASERVER'

                ***ERROR Hostname 'ip=MEDIASERVER' INVALID

admin@RT-AX88U-26C8:/jffs/scripts# 
admin@RT-AX88U-26C8:/jffs/scripts# ./QuotaMonitor.sh init ip=192.168.254.20

 v1.07 Quota Monitoring.....

        Quota Monitoring Setup complete.


        ***ERROR '192.168.254.20' already in monitor list or (192.168.254.20) invalid?


admin@RT-AX88U-26C8:/jffs/scripts# cat /jffs/configs/IPGroups
admin@RT-AX88U-26C8:/jffs/scripts# ./QuotaMonitor.sh status

        v1.07 Quota/Rate Monitoring every 60 secs (Quota Rx/Tx Limit 50.00 MB/10.00 MB and Quota Data Usage Capped Limit is 50.00 GB)

                        There are 0 devices monitored

                        There are 0 devices BLOCKED
                        There are 0 devices throttled by QoS

admin@RT-AX88U-26C8:/jffs/scripts#

Thread starter	Title	Forum	Replies	Date
S	RT-AX58U/RT-AX3000 Per IP Traffic Monitor	Asuswrt-Merlin	2	Jan 24, 2024
P	Help with traffic filtering	Asuswrt-Merlin	0	Friday at 9:08 PM
M	GT-AX11000 - Traffic Analyzer - Statistic data not showing	Asuswrt-Merlin	5	Apr 3, 2024
S	VPN director and LAN traffic	Asuswrt-Merlin	2	Mar 7, 2024
G	VPN Client with Public IP, split tunnel web server over VPN and other traffic not on the VPN	Asuswrt-Merlin	4	Feb 13, 2024
	Traffic Monitoring - Save frequency - issue	Asuswrt-Merlin	7	Jan 31, 2024
T	A possible bug with QOS traffic classification?	Asuswrt-Merlin	14	Dec 9, 2023
P	Openvpn connected but no wan/lan traffic	Asuswrt-Merlin	4	Nov 4, 2023
	Traffic Analyzer - Closed Source?	Asuswrt-Merlin	1	Nov 1, 2023
H	WireGuard DNS Traffic From All Clients Sent Over First WireGuard Client Connection	Asuswrt-Merlin	7	Oct 28, 2023

Per IP Traffic accounting

Senior Member

Regular Contributor

Regular Contributor

Part of the Furniture

Regular Contributor

Part of the Furniture

Regular Contributor

Part of the Furniture

Part of the Furniture

Part of the Furniture

Regular Contributor

Regular Contributor

Regular Contributor

Part of the Furniture

Part of the Furniture

Regular Contributor

Part of the Furniture

Part of the Furniture

Part of the Furniture

Regular Contributor

Similar threads

Similar threads

Sign Up For SNBForums Daily Digest