uiDivStats uiDivStats - WebUI for Diversion statistics

root · Jun 10, 2020

What happens if you run:

Code:

sqlite3 /opt/share/uiDivStats.d/dnsqueries.db
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS [dnsqueries] ([QueryID] INTEGER PRIMARY KEY NOT NULL, [Timestamp] NUMERIC NOT NULL, [SrcIP] TEXT NOT NULL,[ReqDmn] TEXT NOT NULL,[QryType] Text NOT NULL,[Result] Text NOT NULL);

any errors?

When I run "sqlite3 /opt/share/uiDivStats.d/dnsqueries.db" this comes up:

SQLite header and source version mismatch
2018-11-05 20:37:38 89e099fbe5e13c33e683bef07361231ca525b88f7907be7092058007b75036f2
2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6

Jack Yaz · Jun 10, 2020

root said:
When I run "sqlite3 /opt/share/uiDivStats.d/dnsqueries.db" this comes up:

SQLite header and source version mismatch
2018-11-05 20:37:38 89e099fbe5e13c33e683bef07361231ca525b88f7907be7092058007b75036f2
2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6

Ah.

Code:

opkg update && opkg remove --autoremove sqlite3-cli && opkg install sqlite3-cli

root · Jun 10, 2020

Jack Yaz said:
Ah.

Code:

opkg update && opkg remove --autoremove sqlite3-cli && opkg install sqlite3-cli

Thank you very much, it works now!

joe scian · Jun 10, 2020

Jack Yaz said:
So if I count up the hosts files across these 2 files, and then sum up the domains?

Code:

admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist_fs https://pastebin.com/raw/JpYStTs4

Im sure that would be just fine Jack

ugandy · Jun 10, 2020

after updating to latest develop version, i get:

cromo@RT-AX88U-8158:/tmp/home/root# uiDivStats generate
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/hostslist_fs: No such file or directory
uiDivStats: Stats updated successfully

cromo@RT-AX88U-8158:/tmp/home/root#

andresmorago · Jun 10, 2020

ugandy said:
after updating to latest develop version, i get:

cromo@RT-AX88U-8158:/tmp/home/root# uiDivStats generate
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/hostslist_fs: No such file or directory
uiDivStats: Stats updated successfully

cromo@RT-AX88U-8158:/tmp/home/root#

hello
i just encountered the same issue. update from developer 2.0.1

joe scian · Jun 10, 2020

andresmorago said:
hello
i just encountered the same issue. update from developer 2.0.1

Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

thelonelycoder · Jun 11, 2020

@Jack Yaz could you not start taildns when logging is disabled?
I modified my files thus:
/jffs/scripts/uiDivStats, line 1288:

Code:

    dnsmasq)
       if grep 'log-facility' /etc/dnsmasq.conf; then
           Print_Output "true" "dnsmasq has restarted, restarting taildns" "$PASS"
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           sleep 5
           /opt/etc/init.d/S90taildns start >/dev/null 2>&1
       else
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           Print_Output "true" "dnsmasq has restarted, Dnsmasq logging is disabled, stopped taildns" "$PASS"
       fi
       exit 0
   ;;

And /opt/bin/taildns:

Code:

#!/bin/sh
#shellcheck disable=SC2039
trap '' SIGHUP

if [ ! -f /opt/bin/pkill ]; then
    opkg update
    opkg install procps-ng-pkill
fi

if grep 'log-facility' /etc/dnsmasq.conf; then
    /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &

    while true; do
        sleep 5
        if [ "$(pidof taildnstotal | wc -w)" -lt 2 ]; then
            logger -t "uiDivStats" "taildns dead, restarting..."
            killall -q taildnstotal
            /opt/bin/pkill -f "tail -F /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log3"
            sleep 5
            /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &
            logger -t "uiDivStats" "taildns restarted"
        fi
    done
else
    logger -t "uiDivStats" "taildns not started, Dnsmasq logging is disabled"
fi

Edit: You may have to add more code elsewhere, I just changed the obvious files to me

Jack Yaz · Jun 11, 2020

joe scian said:
Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

Blacklist files are included in the total

Jack Yaz · Jun 11, 2020

thelonelycoder said:

@Jack Yaz could you not start taildns when logging is disabled?
I modified my files thus:
/jffs/scripts/uiDivStats, line 1288:

Code:

    dnsmasq)
       if grep 'log-facility' /etc/dnsmasq.conf; then
           Print_Output "true" "dnsmasq has restarted, restarting taildns" "$PASS"
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           sleep 5
           /opt/etc/init.d/S90taildns start >/dev/null 2>&1
       else
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           Print_Output "true" "dnsmasq has restarted, Dnsmasq logging is disabled, stopped taildns" "$PASS"
       fi
       exit 0
   ;;

And /opt/bin/taildns:

Code:

#!/bin/sh
#shellcheck disable=SC2039
trap '' SIGHUP

if [ ! -f /opt/bin/pkill ]; then
    opkg update
    opkg install procps-ng-pkill
fi

if grep 'log-facility' /etc/dnsmasq.conf; then
    /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &

    while true; do
        sleep 5
        if [ "$(pidof taildnstotal | wc -w)" -lt 2 ]; then
            logger -t "uiDivStats" "taildns dead, restarting..."
            killall -q taildnstotal
            /opt/bin/pkill -f "tail -F /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log3"
            sleep 5
            /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &
            logger -t "uiDivStats" "taildns restarted"
        fi
    done
else
    logger -t "uiDivStats" "taildns not started, Dnsmasq logging is disabled"
fi

Edit: You may have to add more code elsewhere, I just changed the obvious files to me

Can do. Is it causing problems? As far as I know other than running a redundant tail process or 2, it should be benign (albeit untidy).

thelonelycoder · Jun 11, 2020

Jack Yaz said:
Can do. Is it causing problems? As far as I know other than running a redundant tail process or 2, it should be benign (albeit untidy).

Its the tidy thing that bothers me, also why run it if logging is disabled?

Jack Yaz · Jun 11, 2020

thelonelycoder said:
Its the tidy thing that bothers me, also why run it if logging is disabled?

It's one of those things on the ever-growing "to-do". I have to draw the line on how much time I spend on these projects somewhere

I'll include it for the next release

thelonelycoder · Jun 11, 2020

Jack Yaz said:
It's one of those things on the ever-growing "to-do". I have to draw the line on how much time I spend on these projects somewhere

I'll include it for the next release

Mine never seem to shrink either.

Jack Yaz · Jun 11, 2020

joe scian said:
Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

I suppose I also need to test for whether the alternate instance of dnsmasq is enabled, and include the blocklist_fs if it is enabled...

martinr · Jun 11, 2020

thelonelycoder said:
Mine never seem to shrink either.

I’m sure there’s a medical term for that.

Edit: I was right: it’s priapism (and it’s quite painful....)

thelonelycoder · Jun 11, 2020

Jack Yaz said:
I suppose I also need to test for whether the alternate instance of dnsmasq is enabled, and include the blocklist_fs if it is enabled...

You tail both .log and log3 anyway. If your strategy is to run taildns no matter what, then YOU'll be OK

thelonelycoder · Jun 11, 2020

martinr said:
I’m sure there’s a medical term for thar.

And now were back at the 'router in the bedroom' topic!

Jack Yaz · Jun 11, 2020

thelonelycoder said:
You tail both .log and log3 anyway. If your strategy is to run taildns no matter what, then YOU'll be OK

well my scripts did all start off with me as the target audience ;-)

thelonelycoder · Jun 11, 2020

Jack Yaz said:
well my scripts did all start off with me as the target audience ;-)

Oh, I always had the users in mind, starting with my first script post here: https://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/

Jack Yaz · Jun 11, 2020

thelonelycoder said:
Oh, I always had the users in mind, starting with my first script post here: https://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/

Once i started adding menus it became user-focused!

Thread starter	Title	Forum	Replies	Date
	uiDivStats uiDivStats development version compatible with Diversion 5.0	Asuswrt-Merlin AddOns	63	Jan 13, 2024
B	uiDivStats Lots of disk writes due to fully re-writing file ./entware/share/uiDivStats.d/csv/SQLQueryLog.htm	Asuswrt-Merlin AddOns	0	Sep 28, 2023
J	Diversion uiDivStats client name only showing as IP address	Asuswrt-Merlin AddOns	8	Jul 3, 2023
	vnStat WebUI page not found vnStat & connmon	Asuswrt-Merlin AddOns	3	Mar 24, 2024
	Skynet Skynet WebUI as scMerlin sitemap	Asuswrt-Merlin AddOns	7	Dec 21, 2023
M	FTP with WebUI	Asuswrt-Merlin AddOns	2	Nov 19, 2023
B	Diversion Diversion help	Asuswrt-Merlin AddOns	14	Apr 6, 2024
C	Diversion Pixelserv replacement	Asuswrt-Merlin AddOns	2	Mar 25, 2024
A	Diversion apparently blocks domains not included in any list	Asuswrt-Merlin AddOns	9	Mar 21, 2024
E	Diversion GUI not working	Asuswrt-Merlin AddOns	1	Feb 13, 2024

uiDivStats uiDivStats - WebUI for Diversion statistics

Occasional Visitor

Part of the Furniture

Occasional Visitor

Very Senior Member

Very Senior Member

Senior Member

Very Senior Member

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Part of the Furniture

Similar threads

Similar threads

Sign Up For SNBForums Daily Digest