uiDivStats uiDivStats - WebUI for Diversion statistics

[Jack Yaz]

Hello Jack
I’m running your developer version and works ok. I noticed that the client drop down menu under the query log is now gone

Shouldn't be! Can you send a screenshot please?

 

[JemTheWire]

Running v2.2.0.

No CPU spikes at long last. UI is as responsive as ever.

Looks like you have a winner on your hands.

Thanks for you hard work.

 

[QuikSilver]

That's weird. Stop taildns :

/opt/etc/init.d/S90taildns stop

Does disk usage drop? If so, start it again then check again. If disk usage increases again please send output of

ps | grep taildns

When stopping taildns this is what atop shows.

DSK |          sda  | busy     28%  | read   63202  |               | write 151010  | KiB/r     20  | KiB/w     23  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.84  | avio  311 ms  |

Starting back up....

DSK |          sda  | busy     28%  | read   63263  |               | write 151196  | KiB/r     20  | KiB/w     23  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.86  | avio  311 ms

 

[QuikSilver]

When stopping taildns this is what atop shows.

DSK |          sda  | busy     28%  | read   63202  |               | write 151010  | KiB/r     20  | KiB/w     23  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.84  | avio  311 ms  |

View attachment 24138

Starting back up....

DSK |          sda  | busy     28%  | read   63263  |               | write 151196  | KiB/r     20  | KiB/w     23  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.86  | avio  311 ms

View attachment 24139

Right after I posted that atop shows this....Guess it took some time to start back up.

DSK |  sda  | busy    102%  | read 0| read   63263  |               | write     11  | KiB/r      0  | KiB/w      5  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.39  | avio  910 ms

 

[Jack Yaz]

Right after I posted that atop shows this....Guess it took some time to start back up.

DSK |  sda  | busy    102%  | read 0| read   63263  |               | write     11  | KiB/r      0  | KiB/w      5  | MBr/s    0.0  | MBw/s    0.0  |               | avq     6.39  | avio  910 ms

you definitely have the cache mode enabled? though in this output it looks like your dsk is choking on reads

might be time to replace that USB!

 

[QuikSilver]

you definitely have the cache mode enabled? though in this output it looks like your dsk is choking on reads

might be time to replace that USB!

Pretty sure I do. Will toggle it again and see if message under "C" changes. I'd agree on the USB comment. Oddly enough I did change the usb recently to that one but Im pretty sure the USB being used there isn't an expensive one though.

 

[QuikSilver]

View attachment 24140
Pretty sure I do. Will toggle it again and see if message under "C" changes. I'd agree on the USB comment. Oddly enough I did change the usb recently to that one but Im pretty sure the USB being used there isn't an expensive one though.

So i noticed that I was logging "All" in the query mode and chose to change that. The next option seems to have helped settle it some. I've seen percentage go as high as 86% but nothing as high as 102% like before.

 

[doczenith1]

Running v2.2.0 with the toggled cache-option since 5 hours without any issues (RT-86U). And it seems to work! Thanks @Jack Yaz!

Nice to hear it's also working for you as we have almost the identical setup except I'm using ext2. old USB2-Stick, 8 GB capacity, 2GB SWAP (ext2), plugged in the USB3-Port but downgraded to USB2 in the GUI

 

[dave14305]

Another differentiator might be running DNSSEC which generates additional log entries that must be tailed and filtered out.

 

[CriticJay]

@Jack Yaz I've been away for a few days mostly focussed on benchmarking all of my USB drives and getting rid of slow ones (most of them are 7-12 years old so it makes sense they're so slow).

Finally found one which is half decent according to IOZONE benchmark, and so I re-installed your script ... and guess what, even your previous script version (v2.1.0) without the cache enabled, it is handling it perfectly fine.

This basically confirms my suspicion from earlier in this thread that slow USB drives are the culprit. I think anyone reporting performance issues should first post their IOZONE benchmark results before engaging in any further troubleshooting!

 

[CriticJay]

For those experiencing high DSK and/or CPU usage with 2.X

Please can you test the "cache" mode available on the develop branch? You can switch by running:

uiDivStats develop

and using option c in the CLI menu.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

Is there a benefit to using the CACHE mode if you have a high-performing USB stick which appears to work okay even without it?

Functionally, is one better than the other? Accuracy of statistics, etc.?

 

[Jack Yaz]

Is there a benefit to using the CACHE mode if you have a high-performing USB stick which appears to work okay even without it?

Functionally, is one better than the other? Accuracy of statistics, etc.?

Should be identical functionally!

 

[andresmorago]

Shouldn't be! Can you send a screenshot please?

sure.
let me know if i can provide more info

 

[Jack Yaz]

sure.
let me know if i can provide more info
View attachment 24144

ah right sorry, client wasn't a dropdown but a freeform text box. i should probably work on getting hostnames in there to be honest

 

[Jack Yaz]

Interesting... the new usb stick I just got seems to have done the trick.

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 20480 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: /jffs/bin/iozone -e -I -a -s 20M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frd
           20480       4    22482    28003    20559    20939    16365    13989
           20480      16    53438     1759    46531    56935    37743    20731
           20480     512   120908   124107   172335   175229   149034   109495
           20480    1024   117373   113702   181513   185488   170472   107876
           20480   16384   110677   119308   240721   238468   242302   117076
iozone test complete.

The specs are light years ahead of my other USB sticks.

So far, I am not seeing any CPU spikes or any activity in atop.

One thing that is weird, the previous USB sticks I had, IOZONE was still showing better performance that what @Jack Yaz had on his drives.

I could be possible that the USB drives I had were "lemons" even if they were only a few months old...weird.

someone much smarter than me would need to explain why your better benchmarking drive hit such a bottleneck.

 

[CriticJay]

Interesting... the new usb stick I just got seems to have done the trick.

please share which stick you got, and which port you are using (USB2 / USB3 in 2.0 mode / USB3 in 3.0 mode) --- and your router is the ax88u?

 

[andresmorago]

ah right sorry, client wasn't a dropdown but a freeform text box. i should probably work on getting hostnames in there to be honest

im sorry, my bad. i could swear i had seen a dropdown menu before.
thanks! it would be great to have it in addition to the freeform text box

 

[Jack Yaz]

Off topic, I am not sure if this the right place or even possible to do so in uiDivStats (a feature request).

Would it be possible, in the Query Log section, if you see a "blocked" domain, it would be nice to click on the domain, and put in whitelist. Or vice-versa, block a domain. This would mean that it needs to update the Diversion whitelist and blocklists and then allow diversion to process the lists.

As I said, this might be completely off topic, but would be pretty neat to have.

In any case, really appreciate all the work @Jack Yaz put into developing such a cool tool.

@thelonelycoder Is there a command line invocation that I could pass a domain to for Diversion to do this, or am I treading on Diversion's WebUI for this sort of thing?

 

[gattaca]

WRT the USB stick discussion.. I think I've said this before, though not sure here. The NAND used in USB sticks is some of the "cheapest" NAND available. It is not designed for repeated use and does not "wear well" over time. For those considering replacing the sticks, I highly advise getting a good UGREEN SATA 2.5" SSD enclosure and a smaller SATA SSD. Use those. Whole different ballgame. There's a thread on this around here somewhere where members benchmarked them - there's no comparison. You will have many less headaches in the future as the SATA SSD does wear well and is designed with much higher quality materials.

Some examples:
-> https://www.snbforums.com/threads/is-using-a-usb-stick-a-bad-idea.59248/#post-518348
-> https://www.snbforums.com/threads/diversion-the-router-ad-blocker.48538/page-156#post-498658"]Diversion - the Router Ad-Blocker
-> https://www.snbforums.com/threads/entware-which-usb-stick-performs-well.48188/page-7#post-446057

 

[Jack Yaz]

Just wanted to share this to close off the issue I had with my USB stick.

After a fresh install and brand new USB3.0 stick, I have my router running for almost 24 hr with the following stats:
View attachment 24169

There is no real activity on my CPU, in both the webui or htop and atop:

View attachment 24170

In the end, we need to make sure that our USB sticks/drives are actually good. Even brand new ones could be the issue.

Thanks for all the help from everyone.

While I'm perplexed that neither your previous USB or the caching mode helped, I'm pleased it's working for you now!