uiDivStats uiDivStats - WebUI for Diversion statistics

[root]

What happens if you run:

sqlite3 /opt/share/uiDivStats.d/dnsqueries.db
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS [dnsqueries] ([QueryID] INTEGER PRIMARY KEY NOT NULL, [Timestamp] NUMERIC NOT NULL, [SrcIP] TEXT NOT NULL,[ReqDmn] TEXT NOT NULL,[QryType] Text NOT NULL,[Result] Text NOT NULL);

any errors?

When I run "sqlite3 /opt/share/uiDivStats.d/dnsqueries.db" this comes up:

SQLite header and source version mismatch
2018-11-05 20:37:38 89e099fbe5e13c33e683bef07361231ca525b88f7907be7092058007b75036f2
2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6

 

[Jack Yaz]

When I run "sqlite3 /opt/share/uiDivStats.d/dnsqueries.db" this comes up:

SQLite header and source version mismatch
2018-11-05 20:37:38 89e099fbe5e13c33e683bef07361231ca525b88f7907be7092058007b75036f2
2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6

Ah.

opkg update && opkg remove --autoremove sqlite3-cli && opkg install sqlite3-cli

 

[root]

Ah.

opkg update && opkg remove --autoremove sqlite3-cli && opkg install sqlite3-cli

Thank you very much, it works now!

 

[joe scian]

So if I count up the hosts files across these 2 files, and then sum up the domains?

admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
admin@RT-AC86U:/tmp/mnt/entware/entware/share/diversion/list# cat hostslist_fs
https://pastebin.com/raw/JpYStTs4

Im sure that would be just fine Jack

 

[ugandy]

after updating to latest develop version, i get:

cromo@RT-AX88U-8158:/tmp/home/root# uiDivStats generate
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/hostslist_fs: No such file or directory
uiDivStats: Stats updated successfully

cromo@RT-AX88U-8158:/tmp/home/root#

 

[andresmorago]

after updating to latest develop version, i get:

cromo@RT-AX88U-8158:/tmp/home/root# uiDivStats generate
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/blockinglist_fs: No such file or directory
/opt/bin/grep: /opt/share/diversion/list/hostslist_fs: No such file or directory
uiDivStats: Stats updated successfully

cromo@RT-AX88U-8158:/tmp/home/root#

hello
i just encountered the same issue. update from developer 2.0.1

 

[joe scian]

hello
i just encountered the same issue. update from developer 2.0.1

Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

 

[thelonelycoder]

@Jack Yaz could you not start taildns when logging is disabled?
I modified my files thus:
/jffs/scripts/uiDivStats, line 1288:

    dnsmasq)
       if grep 'log-facility' /etc/dnsmasq.conf; then
           Print_Output "true" "dnsmasq has restarted, restarting taildns" "$PASS"
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           sleep 5
           /opt/etc/init.d/S90taildns start >/dev/null 2>&1
       else
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           Print_Output "true" "dnsmasq has restarted, Dnsmasq logging is disabled, stopped taildns" "$PASS"
       fi
       exit 0
   ;;

And /opt/bin/taildns:

#!/bin/sh
#shellcheck disable=SC2039
trap '' SIGHUP

if [ ! -f /opt/bin/pkill ]; then
    opkg update
    opkg install procps-ng-pkill
fi

if grep 'log-facility' /etc/dnsmasq.conf; then
    /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &

    while true; do
        sleep 5
        if [ "$(pidof taildnstotal | wc -w)" -lt 2 ]; then
            logger -t "uiDivStats" "taildns dead, restarting..."
            killall -q taildnstotal
            /opt/bin/pkill -f "tail -F /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log3"
            sleep 5
            /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &
            logger -t "uiDivStats" "taildns restarted"
        fi
    done
else
    logger -t "uiDivStats" "taildns not started, Dnsmasq logging is disabled"
fi

Edit: You may have to add more code elsewhere, I just changed the obvious files to me

 

[Jack Yaz]

Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

Blacklist files are included in the total

 

[Jack Yaz]

@Jack Yaz could you not start taildns when logging is disabled?
I modified my files thus:
/jffs/scripts/uiDivStats, line 1288:

    dnsmasq)
       if grep 'log-facility' /etc/dnsmasq.conf; then
           Print_Output "true" "dnsmasq has restarted, restarting taildns" "$PASS"
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           sleep 5
           /opt/etc/init.d/S90taildns start >/dev/null 2>&1
       else
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           Print_Output "true" "dnsmasq has restarted, Dnsmasq logging is disabled, stopped taildns" "$PASS"
       fi
       exit 0
   ;;

And /opt/bin/taildns:

#!/bin/sh
#shellcheck disable=SC2039
trap '' SIGHUP

if [ ! -f /opt/bin/pkill ]; then
    opkg update
    opkg install procps-ng-pkill
fi

if grep 'log-facility' /etc/dnsmasq.conf; then
    /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &

    while true; do
        sleep 5
        if [ "$(pidof taildnstotal | wc -w)" -lt 2 ]; then
            logger -t "uiDivStats" "taildns dead, restarting..."
            killall -q taildnstotal
            /opt/bin/pkill -f "tail -F /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log3"
            sleep 5
            /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &
            logger -t "uiDivStats" "taildns restarted"
        fi
    done
else
    logger -t "uiDivStats" "taildns not started, Dnsmasq logging is disabled"
fi

Edit: You may have to add more code elsewhere, I just changed the obvious files to me

Can do. Is it causing problems? As far as I know other than running a redundant tail process or 2, it should be benign (albeit untidy).

 

[thelonelycoder]

Can do. Is it causing problems? As far as I know other than running a redundant tail process or 2, it should be benign (albeit untidy).

Its the tidy thing that bothers me, also why run it if logging is disabled?

 

[Jack Yaz]

Its the tidy thing that bothers me, also why run it if logging is disabled?

It's one of those things on the ever-growing "to-do". I have to draw the line on how much time I spend on these projects somewhere

I'll include it for the next release

 

[thelonelycoder]

It's one of those things on the ever-growing "to-do". I have to draw the line on how much time I spend on these projects somewhere

I'll include it for the next release

Mine never seem to shrink either.

 

[Jack Yaz]

Yes not everyone has Fast Switch enabled in Diversion. Hence the error - you can safely ignore it - or Jack could supress those messages in the actual script.

BTW Jack summing up the domains across the two hosts files would be the preferred solution when selecting Medium Blocking list in Diversion. And subsequently the blocked domains should be equal to the blocking file chosen when executing FS- eg if you had Large/ Medium - or Large/Standard or Medium/Standard. There are also Minimal and Small categories which could potentially be included in FS however the maximum number of hosts files under any category is two. FS just toggles the chosen blocking hosts file(s) -(either/or not both) and Diversion correctly outputs the number of blocked domains depending on what FS toggle is selected.

So the BEST solution would be to accurately output the number of blocked domains depending on the FS toggle chosen - just like Diversion. Can you use the same code and logic as used in Diversion script? Ie either blockinglist+blacklist+wc_blacklist OR blockinglist_fs+blacklist+wc_blacklist should work depending on the toggle chosen. Not an addition of both.

Currently I get 151,598 domains currently on blocklist in uiDivStats. Whereas I only have selected Standard 57,524 blocked domains by 1 hosts file in Diversion via FS. When I select Medium Blocklist via FS I have 94089 blocked domains by 2 hosts files in Diversion.

Therefore (57524+94089)=151613 AND (151613 -15 )=151,598. (You have not added Blacklist which in my case is an additional 15 Domains.) Not sure whether you wish to include wc_blacklist for those that have that implemented as well?

I suppose I also need to test for whether the alternate instance of dnsmasq is enabled, and include the blocklist_fs if it is enabled...

 

[martinr]

Mine never seem to shrink either.

I’m sure there’s a medical term for that.

Edit: I was right: it’s priapism (and it’s quite painful....)

 

[thelonelycoder]

I suppose I also need to test for whether the alternate instance of dnsmasq is enabled, and include the blocklist_fs if it is enabled...

You tail both .log and log3 anyway. If your strategy is to run taildns no matter what, then YOU'll be OK

 

[thelonelycoder]

I’m sure there’s a medical term for thar.

And now were back at the 'router in the bedroom' topic!

 

[Jack Yaz]

You tail both .log and log3 anyway. If your strategy is to run taildns no matter what, then YOU'll be OK

well my scripts did all start off with me as the target audience ;-)

 

[thelonelycoder]

well my scripts did all start off with me as the target audience ;-)

Oh, I always had the users in mind, starting with my first script post here: https://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/

 

[Jack Yaz]

Oh, I always had the users in mind, starting with my first script post here: https://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/

Once i started adding menus it became user-focused!