uiDivStats uiDivStats - WebUI for Diversion statistics

[Jack Yaz]

Working on a toggle which might help those with spikes caused by lots of PTR queries flying around....

 

[JemTheWire]

Any time a machine wants to resolve an IP into a name it would perform a PTR query. Like a monitoring tool maybe wanting to show you things in a nice friendly format.

grep -E " query\[PTR]" /opt/var/log/dnsmasq.log | awk ' { print $NF } ' | sort | uniq -c | sort -rn | head -10

This will show you the top 10 machines issuing PTR queries since early this morning. If most are coming from one machine, check that machine.

Just FYI, my desktop PC Ethernet NIC accounted for 4500 PTR and its WiFi NIC , 3900 PTR. Even the router was only 55 PTR. Everything else was > 22 PTR.

Thanks for the insight @dave14305. very interesting. I guess GlassWire was the culprit

 

[Jack Yaz]

Just FYI, my desktop PC Ethernet NIC accounted for 4500 PTR and its WiFi NIC , 3900 PTR. Even the router was only 55 PTR. Everything else was > 22 PTR.

Thanks for the insight @dave14305. very interesting. I guess GlassWire was the culprit

FYI you can now run:

uiDivStats develop

followed by option uf in the menu. I've added an option to toggle between all query types and just A+AAAA. Thanks for testing!

Thanks to @dave14305 for the suggestion!

 

[JemTheWire]

I'm having a brain fart. I want to download the latest develop version as a clean install.

To that end I have uninstalled the current 'test' version and removed the db.

For the life of me i cant remember how to install from the develop branch. Typing uiDivStats develop at the command line fails miserably.

Sorry brain dead tonight!!

 

[dave14305]

Sorry brain dead tonight!!

Must have been a good dinner!

 

[Jack Yaz]

I'm having a brain fart. I want to download the latest develop version as a clean install.

To that end I have uninstalled the current 'test' version and removed the db.

For the life of me i cant remember how to install from the develop branch. Typing uiDivStats develop at the command line fails miserably.

Sorry brain dead tonight!!

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/jackyaz/uiDivStats/develop/uiDivStats.sh" -o "/jffs/scripts/uiDivStats" && chmod 0755 /jffs/scripts/uiDivStats && /jffs/scripts/uiDivStats install

Will do the trick. The command I gave is for when you have uiDivStats installed and want to hop between stable and develop branches

 

[JemTheWire]

It was probably a mistake to follow it with a large G&T. Hic

 

[JemTheWire]

Thanks all for today. I have had some fun playtime and learnt stuff on the way.

 

[joe scian]

Hi Jack

Domains currently on blocklist incorrectly shows 57376 blocked domains when Fast Switch selected in Diversion using Medium Blocking List with 94,380 Blocked Domains.
IE - it does not differentiate chosen blocklist in Diversion using FS. Cheers.

 

[Jack Yaz]

Hi Jack

Domains currently on blocklist incorrectly shows 57376 blocked domains when Fast Switch selected in Diversion using Medium Blocking List with 94,380 Blocked Domains.
IE - it does not differentiate chosen blocklist in Diversion using FS. Cheers.

Sorry which number is which? Afaik that counting code is the same as v1 (unless I broke it)

 

[joe scian]

Sorry which number is which? Afaik that counting code is the same as v1 (unless I broke it)

The higher number is medium lower is standard

 

[Jack Yaz]

The higher number is medium lower is standard

So the code I'm using is:

blockinglistfile="$DIVERSION_DIR/list/blockinglist"
blacklistfile="$DIVERSION_DIR/list/blacklist"
blacklistwcfile="$DIVERSION_DIR/list/wc_blacklist"
  
BLL="$(($(/opt/bin/grep "^[^#]" "$blockinglistfile" | wc -w)-$(/opt/bin/grep "^[^#]" "$blockinglistfile" | wc -l)))"
[ "$(nvram get ipv6_service)" != "disabled" ] && BLL="$((BLL/2))"
BL="$(/opt/bin/grep "^[^#]" "$blacklistfile" | wc -l)"
[ "$(nvram get ipv6_service)" != "disabled" ] && BL="$((BL/2))"
WCBL="$(/opt/bin/grep "^[^#]" "$blacklistwcfile" | wc -l)"
blocklistdomains="$((BLL+BL+WCBL))"

so it looks like I'm missing the FS lists

 

[Wishmaster1965]

Ctrl-C out then

uiDivStats uninstall
rm /opt/share/uiDivStats.d/dnsqueries.db

then install from scratch.

Thanks Jack, Sorted

 

[Jack Yaz]

So the code I'm using is:

blockinglistfile="$DIVERSION_DIR/list/blockinglist"
blacklistfile="$DIVERSION_DIR/list/blacklist"
blacklistwcfile="$DIVERSION_DIR/list/wc_blacklist"
 
BLL="$(($(/opt/bin/grep "^[^#]" "$blockinglistfile" | wc -w)-$(/opt/bin/grep "^[^#]" "$blockinglistfile" | wc -l)))"
[ "$(nvram get ipv6_service)" != "disabled" ] && BLL="$((BLL/2))"
BL="$(/opt/bin/grep "^[^#]" "$blacklistfile" | wc -l)"
[ "$(nvram get ipv6_service)" != "disabled" ] && BL="$((BL/2))"
WCBL="$(/opt/bin/grep "^[^#]" "$blacklistwcfile" | wc -l)"
blocklistdomains="$((BLL+BL+WCBL))"

so it looks like I'm missing the FS lists

@joe scian How would you expect to see the count? A single total depending on whether you have FS enabled or not, overall total, or both totals displayed?

 

[joe scian]

@joe scian How would you expect to see the count? A single total depending on whether you have FS enabled or not, overall total, or both totals displayed?

I think a single total would be preferable coinciding with FS option selected similar to Diversion tag. Eg I have medium blocking list chosen using FS in Diversion and Diversion shows:-

 94,089  blocked domains by  2  hosts file(s)

 

[root]

I'm getting stuck at "uiDivStats: Creating database table and enabling write-ahead logging...".

I've tried to uninstall and reinstall, but still stuck at the same step.

uiDivStats: Welcome to uiDivStats v2.0.0, a script by JackYaz

uiDivStats: Checking your router meets the requirements for uiDivStats

Downloading http://bin.entware.net/aarch64-k3.10/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: New version of uidivstats_www.asp downloaded

uiDivStats: Mounting uiDivStats WebUI page as user1.asp

uiDivStats: New version of taildns.tar.gz downloaded

Downloading http://bin.entware.net/aarch64-k3.10/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: Creating database table and enabling write-ahead logging...

When I uninstall uiDivStats and then try to delete dnsqueries.db it can't find the file in /opt/share/uiDivStats.d/

What should I do?

 

[Jack Yaz]

When I uninstall uiDivStats and then try to delete dnsqueries.db it can't find the file in /opt/share/uiDivStats.d/

What should I do?

That suggests that the database file was never created. Does it always get stuck on a fresh install? Which router and f/w?

 

[root]

That suggests that the database file was never created. Does it always get stuck on a fresh install? Which router and f/w?

Yes, it always get stuck at the same step. I had the old uiDivStats installed and working flawless before I tried to update to 2.0.

AC86u running 384.17

 

[Jack Yaz]

I think a single total would be preferable coinciding with FS option selected similar to Diversion tag. Eg I have medium blocking list chosen using FS in Diversion and Diversion shows:-

 94,089  blocked domains by  2  hosts file(s)

So if I count up the hosts files across these 2 files, and then sum up the domains?

admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist_fs
https://pastebin.com/raw/JpYStTs4

 

[Jack Yaz]

Yes, it always get stuck at the same step. I had the old uiDivStats installed and working flawless before I tried to update to 2.0.

AC86u running 384.17

What happens if you run:

sqlite3 /opt/share/uiDivStats.d/dnsqueries.db
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS [dnsqueries] ([QueryID] INTEGER PRIMARY KEY NOT NULL, [Timestamp] NUMERIC NOT NULL, [SrcIP] TEXT NOT NULL,[ReqDmn] TEXT NOT NULL,[QryType] Text NOT NULL,[Result] Text NOT NULL);

any errors?