uiDivStats uiDivStats - WebUI for Diversion statistics

  • ATTENTION! You'll notice a Prefix dropdown when you create a thread. If your post applies to one of the topics listed, please use that Prefix for your post. When browsing the thread list you can use the Prefix to filter the view.
  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

SomeWhereOverTheRainBow

Part of the Furniture
Code:
Diversion 4.1.12                 by thelonelycoder

 RT-AC88U (armv7l) FW-384.17 @ 172.24.5.1

 965,296  blocked domains by  2  hosts file(s)
 356,841 t  14,528 w  10 n ads since Jun 17 08:26

 /mnt/entware | Size 14.7G | Used 1.3G (9%)   

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 172.24.5.254
 l   logging              dnsmasq.log 25.5M

 ep  pixelserv-tls        172.24.5.254 v2.3.1

 b   blocking list        Large Mon @ 1:00
 el  edit lists            210 w  22 b  0 wb  936 yt

 f   follow dnsmasq.log

 ac  update ads counter   Auto-update  au  off
 ds  dnsmasq settings     on
 sw  manage swap file     /mnt/entware 1.0G
 ct  change theme colors   green

 sh  shell history        on
 cj  show cron job(s)
 sf  show file content

 e   exit  u  update                less options  o
Code:
Dnsmasq settings:

 See the Dnsmasq man page what these options do:
 http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

 1. Disable Dnsmasq settings
 2. log-async         5
 3. cache-size        1500
 4. bogus-priv        on
 5. domain-needed     on
 6. log-queries=extra

 Enter selection [1-6 e=Exit]

Disabling Dnsmasq settings does not change anything. I'm using Sandisk Extreme 16GB USB3.0 in USB 2.0 port.


turn off the custom DNSMasq options inside diversion, your issues should go away. those options are default in dnsmasq.

ds dnsmasq settings on

turn the ds feature off.

Also: You could dump your dnsmasq logfile. see if that helps.

the size of your log file...
dnsmasq.log 25.5M
 
Last edited:

Jack Yaz

Part of the Furniture
Code:
[email protected]:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext2           14.7G      1.3G     12.6G   9% /tmp/mnt/entware

https://imgur.com/a/Iy8jmHd

atop is too complex for me :D but as I check htop I saw taildns and pixelserv-tls at the top most of the time, and this is when I try to access many pages in a couple of seconds.
atop screenshot is fine. htop shows that network map is using all of your CPU. try navigating to a static WebUI page (or logging out) and repeat the tests.
 

pattiri

Senior Member
atop screenshot is fine. htop shows that network map is using all of your CPU. try navigating to a static WebUI page (or logging out) and repeat the tests.

Logged out and opened many web pages in about 15-20 seconds, all loaded fine with no error and I got this;
1.jpg


Then I've uninstalled uiDivStats; flushed DNS of my PC and deleted databases of firefox and restarted.

Screenshot_6.jpg


this time (after uninstalling uiDivStats) I've tried to open 40 tabs as soon as possible and for your reference here my 9900K CPU usage during this;

Screenshot_7.jpg


I mean second test should be more intense for router but it didn't even sweat.
 

SomeWhereOverTheRainBow

Part of the Furniture
Logged out and opened many web pages in about 15-20 seconds, all loaded fine with no error and I got this;
View attachment 24100

Then I've uninstalled uiDivStats; flushed DNS of my PC and deleted databases of firefox and restarted.

View attachment 24101

this time (after uninstalling uiDivStats) I've tried to open 40 tabs as soon as possible and for your reference here my 9900K CPU usage during this;

View attachment 24102

I mean second test should be more intense for router but it didn't even sweat.
I experienced this same issue with the DS setting in diversion enabled along time ago before UiDivstats was around. I disabled it and rebooted the router and the problem went away.
 

pattiri

Senior Member
I experienced this same issue with the DS setting in diversion enabled along time ago before UiDivstats was around. I disabled it and rebooted the router and the problem went away.

Could you please send your current DS settings? Even I disable settings I think I can change bogus-priv and domain-needed.

Let me change the settings as same of yours reboot and check
 

pattiri

Senior Member
Mine are complety disabled meaning I do not have that diversion "experimental" feature enabled.

OK; disabled and rebooted the router, installed iuDivStats and waited for a while for a cool down.

Screenshot_1.jpg


And did the same test with 40 tabs and here :(

Screenshot_2.jpg


additional notes I would like to mention.

- When I did the test with 40 tabs when iuDivStats is not installed; the CPU Load average reached its max just after all tabs are loaded
- When I did the test with 40 tabs with iuDivStats is installed when all tabs are loaded CPU load average for 1 min was 0.78 after about 10-15 seconds it reached 1.45.

I guess this shows this is not because of iuDivStats it maybe because of database or something else being written into USB drive.

I've never seen good write speeds with my router, I've tested again. it changes between 2 MB/S and 45 MB/s but this is USB 2.0 and I'm not sure how it's 45 MB/S :D

Screenshot_3.jpg


Maybe it's time for me to reset router to factory defaults and start over :D I haven't done this for more than one year I guess :D
 

Makaveli

Very Senior Member
That's a point. What filesystem and number of partitions do people have on their USB drive?
Code:
[email protected]:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda3            ext4            6.3G      2.7G      3.2G  46% /tmp/mnt/entware
/dev/sda1            tfat            7.9G      1.3M      7.9G   0% /tmp/mnt/data

I've had no issue on this version aswell.

on a 512GB SSD attached via USB 3.0 port.

Code:
ASUSWRT-Merlin RT-AX88U 384.17_0 Sun Apr 26 02:29:04 UTC 2020
[email protected]:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext4          436.6G      1.3G    412.0G   0% /tmp/mnt/Scripts

Cpu usage normal.

 

doczenith1

Very Senior Member
Here we go.
Uninstalled 2.1.0
Deleted dnsqueries.db
Installed 2.1.0

Diversion and scMerlin are the the only scripts I currently have installed on 384.17. It has been at least a year since a factory reset and looking at an old screen shot I took of Diversion after first installing the version is 4.0.6.

Now seeing the cpu spiking to 100% and alternating between cores. It's been going at it for about 20 minutes now.

/dev/sda1 ext2 7.2G 2.1G 4.8G 30% /tmp/mnt/Diversion

upload_2020-6-17_11-21-56.png


upload_2020-6-17_11-22-9.png



upload_2020-6-17_11-22-19.png
 

dave14305

Part of the Furniture
Has anyone tested putting the database on a separate USB drive from the dnsmasq logfiles? Split the contention, so-to-speak.
 

Jack Yaz

Part of the Furniture
Here we go.
Uninstalled 2.1.0
Deleted dnsqueries.db
Installed 2.1.0

Diversion and scMerlin are the the only scripts I currently have installed on 384.17. It has been at least a year since a factory reset and looking at an old screen shot I took of Diversion after first installing the version is 4.0.6.

Now seeing the cpu spiking to 100% and alternating between cores. It's been going at it for about 20 minutes now.

/dev/sda1 ext2 7.2G 2.1G 4.8G 30% /tmp/mnt/Diversion

View attachment 24109

View attachment 24110


View attachment 24111
htop is showing hardly any cpu usage - do you see the 100% there or on network map? honestly, im starting to think network map is a steaming pile of s**t that causes the router to work overtime to show anything
 

doczenith1

Very Senior Member
htop is showing hardly any cpu usage - do you see the 100% there or on network map? honestly, im starting to think network map is a steaming pile of s**t that causes the router to work overtime to show anything

Good catch. htop is not showing the spikes when I monitor it and it is showing around 1-4% usage. But...the last log entry from uiDivStats is:
Jun 17 11:05:23 uiDivStats: Starting first run of stat generation...
Should there be a log entry stating that the generation has completed? It's been running for over three hours now.

I am seeing values in the uiDivStats tab. On the DNS Queries graph the x-axis jives with the time that I installed 2.1.0. It's showing roughly 300 total DNS queries and ~50 blocked.

Just in case it might make a difference my stats config in Diversion is:
upload_2020-6-17_14-23-42.png


My USB stick does have an activity light and it seems way more active that it used to be. Pretty much going nonstop with a pause here and there.
 

Jack Yaz

Part of the Furniture
Good catch. htop is not showing the spikes when I monitor it and it is showing around 1-4% usage. But...the last log entry from uiDivStats is:
Jun 17 11:05:23 uiDivStats: Starting first run of stat generation...
Should there be a log entry stating that the generation has completed? It's been running for over three hours now.

I am seeing values in the uiDivStats tab. On the DNS Queries graph the x-axis jives with the time that I installed 2.1.0. It's showing roughly 300 total DNS queries and ~50 blocked.

Just in case it might make a difference my stats config in Diversion is:
View attachment 24114

My USB stick does have an activity light and it seems way more active that it used to be. Pretty much going nonstop with a pause here and there.
Looks like the "complete" message only prints to the console at the moment, and wont appear in the log file. I assume that's where you're looking? I'll fix that for the next release
 

doczenith1

Very Senior Member
Yes, I am looking in the system log for the "complete" message.

My initial atop didn't show this but I'm seeing it now.
upload_2020-6-17_16-6-42.png

It's hard to see but it's showing the dsk busy 100%.

After uninstalling uiDivStats the GUI cpu graph settles down and my USB stick light stops flashing non stop. And now dsk only shows up intermittently in atop.
 

Jack Yaz

Part of the Furniture
Yes, I am looking in the system log for the "complete" message.

My initial atop didn't show this but I'm seeing it now.
View attachment 24115
It's hard to see but it's showing the dsk busy 100%.

After uninstalling uiDivStats the GUI cpu graph settles down and my USB stick light stops flashing non stop. And now dsk only shows up intermittently in atop.
Sit tight, I'm finalising testing on a caching based approach. Once I'm happy I'll add it to the develop branch.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.
 

TheMorpN

Regular Contributor
I re-installed my entire router again, and have this :

Code:
ATOP - RT-AC5300-5880 2020/06/17  21:03:0 2020/06/17  21:05:07                                 --                       10
PRC | sys    0.32s  | user   0.21s |  #proc    141 |  #trun      1 | #tslpi   134  | #tslpu     6 |  #zombie    1 |  no  procacct |
CPU | sys       6%  | user     12% |  irq       2% |  idle     88% | wait     92%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       2%  | user      9% |  irq       0% |  idle     41% | cpu001 w 48%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       4%  | user      3% |  irq       2% |  idle     46% | cpu000 w 45%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
CPL | avg1    2.35  | avg5    1.36 |  avg15   0.99 |               | csw    28010  | intr    8853 |               |  numcpu     2 |
MEM | tot   503.1M  | free  187.6M |  cache  30.5M |  buff    1.9M | slab  116.6M  | shmem   2.4M |  vmbal   0.0M |  hptot   0.0M |
SWP | tot     2.0G  | free    2.0G |               |               |               |              |  vmcom  63.2M |  vmlim   2.2G |
DSK |  sda  | busy     98% |  read      41 |  write     41e     88 | KiB/w      7  | MBr/s    1.2 |  MBw/s    0.0 |  avio  120 ms |
NET | transport     | tcpi     136 |  tcpo      93 |  udpi     372 | udpo     391  | tcpao      0 |  tcppo     20 |  tcprs      0 |
NET | network       | ipi      816 |  ipo      794 |  ipfrw    306 | deliv    508  |              |  icmpi      0 |  icmpo      6 |
NET | eth0     20%  | pcki    3118 |  pcko    3023 |  sp   10 Mbps | si 2023 Kbps  | so 2030 Kbps |  erri       0 |  erro       0 |
NET | vlan1     1%  | pcki    1192 |  pcko     189 |  sp   10 Mbps | si  109 Kbps  | so   48 Kbps |  erri       0 |  erro       0 |
NET | fwd1      1%  | pcki    1872 |  pcko    1134 |  sp   10 Mbps | si    0 Kbps  | so  102 Kbps |  erri       0 |  erro       0 |
NET | fwd0      0%  | pcki      85 |  pcko      58 |  sp   10 Mbps | si    0 Kbps  | so    7 Kbps |  erri       0 |  erro       0 |
NET | eth3    ----  | pcki       0 |  pcko    1873 |  sp    0 Mbps | si    0 Kbps  | so 1910 Kbps |  erri       0 |  erro       0 |
NET | lo      ----  | pcki     358 |  pcko     358 |  sp    0 Mbps | si   60 Kbps  | so   60 Kbps |  erri       0 |  erro       0 |
NET | br0     ----  | pcki     403 |  pcko     183 |  sp    0 Mbps | si   58 Kbps  | so   47 Kbps |  erri       0 |  erro       0 |
NET | eth2    ----  | pcki       0 |  pcko      69 |  sp    0 Mbps | si    0 Kbps  | so   19 Kbps |  erri       0 |  erro       0 |
NET | eth1    ----  | pcki       0 |  pcko      54 |  sp    0 Mbps | si    0 Kbps  | so   13 Kbps |  erri       0 |  erro       0 |
NET | wl1.2   ----  | pcki       0 |  pcko     100 |  sp    0 Mbps | si    0 Kbps  | so   12 Kbps |  erri       0 |  erro       0 |
NET | wl2.2   ----  | pcki       0 |  pcko      37 |  sp    0 Mbps | si    0 Kbps  | so    9 Kbps |  erri       0 |  erro       0 |
  PID    SYSCPU    USRCPU     VGROW     RGROW    RUID EUID ST    EXC     THR    S    CPUNR     CPU    CMD 1/3
  318     0.02s     0.09s 0K 0K    francop     francop     --      - 1    S 0      1%    nt_center

upload_2020-6-17_17-13-6.png


upload_2020-6-17_17-13-37.png
 

TheMorpN

Regular Contributor
I re-installed my entire router again, and have this :

Code:
ATOP - RT-AC5300-5880 2020/06/17  21:03:0 2020/06/17  21:05:07                                 --                       10
PRC | sys    0.32s  | user   0.21s |  #proc    141 |  #trun      1 | #tslpi   134  | #tslpu     6 |  #zombie    1 |  no  procacct |
CPU | sys       6%  | user     12% |  irq       2% |  idle     88% | wait     92%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       2%  | user      9% |  irq       0% |  idle     41% | cpu001 w 48%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       4%  | user      3% |  irq       2% |  idle     46% | cpu000 w 45%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
CPL | avg1    2.35  | avg5    1.36 |  avg15   0.99 |               | csw    28010  | intr    8853 |               |  numcpu     2 |
MEM | tot   503.1M  | free  187.6M |  cache  30.5M |  buff    1.9M | slab  116.6M  | shmem   2.4M |  vmbal   0.0M |  hptot   0.0M |
SWP | tot     2.0G  | free    2.0G |               |               |               |              |  vmcom  63.2M |  vmlim   2.2G |
DSK |  sda  | busy     98% |  read      41 |  write     41e     88 | KiB/w      7  | MBr/s    1.2 |  MBw/s    0.0 |  avio  120 ms |
NET | transport     | tcpi     136 |  tcpo      93 |  udpi     372 | udpo     391  | tcpao      0 |  tcppo     20 |  tcprs      0 |
NET | network       | ipi      816 |  ipo      794 |  ipfrw    306 | deliv    508  |              |  icmpi      0 |  icmpo      6 |
NET | eth0     20%  | pcki    3118 |  pcko    3023 |  sp   10 Mbps | si 2023 Kbps  | so 2030 Kbps |  erri       0 |  erro       0 |
NET | vlan1     1%  | pcki    1192 |  pcko     189 |  sp   10 Mbps | si  109 Kbps  | so   48 Kbps |  erri       0 |  erro       0 |
NET | fwd1      1%  | pcki    1872 |  pcko    1134 |  sp   10 Mbps | si    0 Kbps  | so  102 Kbps |  erri       0 |  erro       0 |
NET | fwd0      0%  | pcki      85 |  pcko      58 |  sp   10 Mbps | si    0 Kbps  | so    7 Kbps |  erri       0 |  erro       0 |
NET | eth3    ----  | pcki       0 |  pcko    1873 |  sp    0 Mbps | si    0 Kbps  | so 1910 Kbps |  erri       0 |  erro       0 |
NET | lo      ----  | pcki     358 |  pcko     358 |  sp    0 Mbps | si   60 Kbps  | so   60 Kbps |  erri       0 |  erro       0 |
NET | br0     ----  | pcki     403 |  pcko     183 |  sp    0 Mbps | si   58 Kbps  | so   47 Kbps |  erri       0 |  erro       0 |
NET | eth2    ----  | pcki       0 |  pcko      69 |  sp    0 Mbps | si    0 Kbps  | so   19 Kbps |  erri       0 |  erro       0 |
NET | eth1    ----  | pcki       0 |  pcko      54 |  sp    0 Mbps | si    0 Kbps  | so   13 Kbps |  erri       0 |  erro       0 |
NET | wl1.2   ----  | pcki       0 |  pcko     100 |  sp    0 Mbps | si    0 Kbps  | so   12 Kbps |  erri       0 |  erro       0 |
NET | wl2.2   ----  | pcki       0 |  pcko      37 |  sp    0 Mbps | si    0 Kbps  | so    9 Kbps |  erri       0 |  erro       0 |
  PID    SYSCPU    USRCPU     VGROW     RGROW    RUID EUID ST    EXC     THR    S    CPUNR     CPU    CMD 1/3
  318     0.02s     0.09s 0K 0K    francop     francop     --      - 1    S 0      1%    nt_center

View attachment 24117

View attachment 24118
To add, this is a new install, and don't have much traffic on my router:

upload_2020-6-17_17-17-8.png
 

Jack Yaz

Part of the Furniture
I re-installed my entire router again, and have this :

Code:
ATOP - RT-AC5300-5880 2020/06/17  21:03:0 2020/06/17  21:05:07                                 --                       10
PRC | sys    0.32s  | user   0.21s |  #proc    141 |  #trun      1 | #tslpi   134  | #tslpu     6 |  #zombie    1 |  no  procacct |
CPU | sys       6%  | user     12% |  irq       2% |  idle     88% | wait     92%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       2%  | user      9% |  irq       0% |  idle     41% | cpu001 w 48%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       4%  | user      3% |  irq       2% |  idle     46% | cpu000 w 45%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
CPL | avg1    2.35  | avg5    1.36 |  avg15   0.99 |               | csw    28010  | intr    8853 |               |  numcpu     2 |
MEM | tot   503.1M  | free  187.6M |  cache  30.5M |  buff    1.9M | slab  116.6M  | shmem   2.4M |  vmbal   0.0M |  hptot   0.0M |
SWP | tot     2.0G  | free    2.0G |               |               |               |              |  vmcom  63.2M |  vmlim   2.2G |
DSK |  sda  | busy     98% |  read      41 |  write     41e     88 | KiB/w      7  | MBr/s    1.2 |  MBw/s    0.0 |  avio  120 ms |
NET | transport     | tcpi     136 |  tcpo      93 |  udpi     372 | udpo     391  | tcpao      0 |  tcppo     20 |  tcprs      0 |
NET | network       | ipi      816 |  ipo      794 |  ipfrw    306 | deliv    508  |              |  icmpi      0 |  icmpo      6 |
NET | eth0     20%  | pcki    3118 |  pcko    3023 |  sp   10 Mbps | si 2023 Kbps  | so 2030 Kbps |  erri       0 |  erro       0 |
NET | vlan1     1%  | pcki    1192 |  pcko     189 |  sp   10 Mbps | si  109 Kbps  | so   48 Kbps |  erri       0 |  erro       0 |
NET | fwd1      1%  | pcki    1872 |  pcko    1134 |  sp   10 Mbps | si    0 Kbps  | so  102 Kbps |  erri       0 |  erro       0 |
NET | fwd0      0%  | pcki      85 |  pcko      58 |  sp   10 Mbps | si    0 Kbps  | so    7 Kbps |  erri       0 |  erro       0 |
NET | eth3    ----  | pcki       0 |  pcko    1873 |  sp    0 Mbps | si    0 Kbps  | so 1910 Kbps |  erri       0 |  erro       0 |
NET | lo      ----  | pcki     358 |  pcko     358 |  sp    0 Mbps | si   60 Kbps  | so   60 Kbps |  erri       0 |  erro       0 |
NET | br0     ----  | pcki     403 |  pcko     183 |  sp    0 Mbps | si   58 Kbps  | so   47 Kbps |  erri       0 |  erro       0 |
NET | eth2    ----  | pcki       0 |  pcko      69 |  sp    0 Mbps | si    0 Kbps  | so   19 Kbps |  erri       0 |  erro       0 |
NET | eth1    ----  | pcki       0 |  pcko      54 |  sp    0 Mbps | si    0 Kbps  | so   13 Kbps |  erri       0 |  erro       0 |
NET | wl1.2   ----  | pcki       0 |  pcko     100 |  sp    0 Mbps | si    0 Kbps  | so   12 Kbps |  erri       0 |  erro       0 |
NET | wl2.2   ----  | pcki       0 |  pcko      37 |  sp    0 Mbps | si    0 Kbps  | so    9 Kbps |  erri       0 |  erro       0 |
  PID    SYSCPU    USRCPU     VGROW     RGROW    RUID EUID ST    EXC     THR    S    CPUNR     CPU    CMD 1/3
  318     0.02s     0.09s 0K 0K    francop     francop     --      - 1    S 0      1%    nt_center

View attachment 24117

View attachment 24118
Do you have a different usb drive just to rule it out being a usb 3 issue? Or try it in the 2.0 port. Are you using samba, ftp server or dlna on the router at all?
 

doczenith1

Very Senior Member
@TheMorpN is showing similar DSK values and GUI cpu activity that I experienced. To add more info on my setup I am using an old USB 2 stick on the USB 3 port that is set to USB 2 mode. I'm not using samba, ftp server or dlna on the router nor Qos or AiProtection.
 

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top