Diversion Diversion - the Router Ad-Blocker v4.2.x (see new thread for 4.3.x)

[elorimer]

My numbers show a different story:

slh 47418 # of accepted HTTPS requests
slm 936   # of rejected HTTPS requests (missing certificate)
sle 297   # of rejected HTTPS requests (certificate available but not usable)
slc 779   # of dropped HTTPS requests (client disconnect without sending any request)
slu 12841 # of dropped HTTPS requests (other TLS handshake errors)

impressive. Mine is 1 thousand out of one million when I just looked. Maybe something is wrong with my setup.

 

[SomeWhereOverTheRainBow]

That's partially correct. Without installing the certs Diversion will still work but without the benefits that pixelserv-tls brings.

While having the certs allows the processing of the pixel-serv cert, without it the provided blocking will be similar to 0.0.0.0 blocking or just will completely refuse the request like a firewall block from my understanding.

 

[SomeWhereOverTheRainBow]

impressive. Mine is 1 thousand out of one million when I just looked. Maybe something is wrong with my setup.

sometimes the certificate store becomes corrupt on the pixelserv-tls, especially if it has been recycled for a long time. I would consider purging your certificate cache to see if your outcome changes.

 

[minhgi]

@thelonelycoder
Is it possible to enable pixelservr filter only devices that have the certificate installed and the rest with 0.0.0.0? Would nice to have the function and GUI that select the device to filter with. At least could target the device that we could get the benefit of faster web browsing and 1 pixel feature.

 

[SomeWhereOverTheRainBow]

@thelonelycoder
Is it possible to enable pixelservr filter only devices that have the certificate installed and the rest with 0.0.0.0? Would nice to have the function and GUI that select the device to filter with. At least could target the device that we could get the benefit of faster web browsing and 1 pixel feature.

How do you propose this to be possible?

 

[Ubimo]

In Diversion, I see the dnsmasq.log ist 494.0M big. Is that 494 Megabytes?
Is that normal? Can I purge that somehow?

 

[chongnt]

In Diversion, I see the dnsmasq.log ist 494.0M big. Is that 494 Megabytes?
Is that normal? Can I purge that somehow?

View attachment 41988

That should be the size. You can see dnsmasq.log files in /opt/var/log directory.
It seems you have configured to update your blocklist twice a week. I used to set it to once a week and usually dnsmasq log files are reset/purged at that time. If you have a lot of chatty devices that frequently perform DNS query the file size will grow accordingly as these are logged.
This is from diversion.ch:
“Diversion counts the ads at 5:20 and 17:20 and rotates the log files at 5:20 every day. This is to ensure the active log file does not get too large, as there is no built in log rotation option in the Asuswrt-Merlin firmware. Three log files are kept in the /logs/ directory: dnsmasq.log, dnsmasq.log1 and dnsmasq.log2. They are used for the weekly and current stats functions when enabled.”
This is what happen when logging is enabled. All DNS activity is logged in dnsmasq.log. At 5:20 everyday, dnsmasq.log1 will be appended to dnsmasq.log2 file; then dnsmaq.log will replace dnsmasq.log1 and a new dnsmasq.log file is created.

 

[Ubimo]

Thanks, I examined one log file, one log file contains 97% of these:

Jun 18 22:45:27 dnsmasq[25305]: query[A] eupush1.nwsvr2.com from 192.168.1.151
Jun 18 22:45:27 dnsmasq[25305]: config eupush1.nwsvr2.com is 0.0.0.0
Jun 18 22:45:29 dnsmasq[25305]: query[A] ampush1.nwsvr2.com from 192.168.1.151
Jun 18 22:45:29 dnsmasq[25305]: config ampush1.nwsvr2.com is 0.0.0.0

That's a camera. I don't want the camera to access the internet. I don't see a setting in the camera options.
How can I prevent this from being logged?
Edit:
I have added these domains some months ago to the wc-blacklist in diversion.

 

[minhgi]

No to sure myself. I'm only asking if it could be done.

How do you propose this to be possible?

 

[elorimer]

sometimes the certificate store becomes corrupt on the pixelserv-tls, especially if it has been recycled for a long time. I would consider purging your certificate cache to see if your outcome changes.

I did this, and after 2 days my success rate is now 6 per million.

EDIT: After 6 days I'm up to 500 per million.

 

[elorimer]

Is it possible to enable pixelservr filter only devices that have the certificate installed and the rest with 0.0.0.0?

What useful thing would this accomplish? It would just slow the other devices down.

 

[SomeWhereOverTheRainBow]

I did this, and after 2 days my success rate is now 6 per million.

Assuming you mean 6000 per million, that is pretty good depending on the size of your block list. On my pihole I run about 4 mil domain block file, which can range between 10,000 to 11,000 request getting blocked per million.

If you mean 6 per million literally, I would consider updating the certificates stored on each client, also keep an eye on how many certs are being reused from cache.

 

[minhgi]

What useful thing would this accomplish? It would just slow the other devices down.

Why would this slow down the the other devices? Having only devices with certificate installed, then you get the benefit from the pixelservr and the rest get normal diversion filter. Not sure if you want pixelservr get apply to devices that don't have certificate and break the mobile apps.

 

[minhgi]

i'm asking if this doable and no not going to debate either. Your guys more knowledgeable =)

 

[iTyPsIDg]

impressive. Mine is 1 thousand out of one million when I just looked. Maybe something is wrong with my setup.

Thanks for asking this question. It prompted me to look into my own environment. I discovered that I could import the certificate to my FireTV (which I finally did), I forgot to enable trust explicitly on my iOS devices, and I had the wrong (old) certificate on a few devices.

I'll have to wait a while to see if things are better. This is my output immediately after those changes:

slh    5650    # of accepted HTTPS requests
slm    993     # of rejected HTTPS requests (missing certificate)
sle    0       # of rejected HTTPS requests (certificate available but not usable)
slc    2005    # of dropped HTTPS requests (client disconnect without sending any request)
slu    2966283 # of dropped HTTPS requests (other TLS handshake errors)

 

[elorimer]

Why would this slow down the the other devices? Having only devices with certificate installed, then you get the benefit from the pixelservr and the rest get normal diversion filter.

If you do just diversion without pixelserv, every request to a blocked site is directed to 0.0.0.0. That address never responds, so after a time, the request times out and the page continues to load. If you do diversion with pixelserv, every request is directed to the pixelserv IP. If it is an https request, pixelserv responds almost immediately with its certificate (from cache or newly generated) from the router's certificate authority . If the router's certificate authority has been imported into the device as a trusted authority, the web page treats the certificated response as valid, loads the single pixel, and continues along more or less immediately. If the certificate authority has not been imported, the response is treated as invalid, the load is rejected, and the web page continues along more or less immediately. Now, the good folks in advertising recognize that people are doing this, so they now include some other tests that result in rejecting pixelserv's one pixel load.

So the upshot is, in order:

1. No Diversion, ads plus the page load. NY Times is maybe 5 to 10 seconds to load.
2. Diversion without pixelserv, no ad, blank space is on the page, the load is affected by lots of time outs but no loading of ads. NY TImes is maybe 2-3 seconds to load.
3. Diversion with pixelserv but no imported certificate, no ad, blank space is on the page, no time out or ad load. NY times is under a second.
4. Diversion with pixelserv but a rejected certificate, same as #3.
5. Diversion with pixelserv and an accepted certificate, same as #3 but the space is replaced by a single pixel.

The adblocking itself isn't affected by the certificate.

At least, that's how I understand it.

 

[ComputerSteve]

Is Diversion not compatible with the new beta firmware of Merlin version 386.7 beta 2. Every time I upgrade to that firmware the iOS shortcuts stop working / Diversion doesn't update ads anymore. Meanwhile the script appears to be working. I get no errors with the previous firmware it looks like this and its updating the ad counter

AS soon as I update even with a full factory reset / Format JFFS partition at next boot I get a diversion that just isn't working it displays like this:

 

[elorimer]

If you mean 6 per million literally, I would consider updating the certificates stored on each client, also keep an eye on how many certs are being reused from cache.

Six accepted requests out of one million requests in two days, starting from purged certificates. Almost all of those requests are amazon or roku telemetry. An Amazon Echo is generating about 300 requests over a second or two every 10 minutes, rejecting the pixelserv certificate. Alexa must be very mad at me.

 

[ComputerSteve]

Diversion is at 4.2.2, so I would trash 1.3 and reinstall from the beginning.

thats the IOS / Mac Shortcut located here https://diversion.ch/ios-shortcut.html

 

[thelonelycoder]

Is Diversion not compatible with the new beta firmware of Merlin version 386.7 beta 2. Every time I upgrade to that firmware the iOS shortcuts stop working / Diversion doesn't update ads anymore. Meanwhile the script appears to be working. I get no errors with the previous firmware it looks like this and its updating the ad counter View attachment 42017

AS soon as I update even with a full factory reset / Format JFFS partition at next boot I get a diversion that just isn't working it displays like this:
View attachment 42019

See this post, there's something new how 386.7 treats SSH logins for certain clients which have yet to be (root) addressed by someone: https://www.snbforums.com/threads/ios-shortcut-for-diversion.55974/post-771557