pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[thelonelycoder]

AMAZING. And thanks for the input.

So. pinterest.com had successfully been removed from /blocking_file.

HOWEVER, www.pinterest.com was still in there. I didn't realize that with and without the www were two separate entries.
Strange that "follow the logfile" in ABS wasn't reflecting that however. It was just silently being blocked, although it was showing pinterest.com before I whitelisted it. :dunno:

pinterest.com is forwarded/redirected to your country specific TLD. Over here in Switzerland that would be pinterest.ch.
It's likely that this is cached somewhere.

As for the entries in the blocking file, StevenBlack's and adblock.mahakala.is both list pinterest.com while adblock.mahakala.is also lists www.pinterest.com.

FYI: foo.com and www.foo.com can be entirely different websites, just as ads.foo.com and fun.foo.com are.

 

[SMS786]

Stats update:

pixelserv-tls 2.1.3-test.4 (compiled: Jul 17 2018 01:44:19)

uts    2d 11:24    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    11    number of active service threads
kmx    40    maximum number of service threads
kvg    6.07    average number of requests per service thread
krq    584    max number of requests by one service thread
req    10547    total # of requests (HTTP, HTTPS, success, failure etc)
avg    699 bytes    average size of requests
rmx    128091 bytes    largest size of request(s)
tav    9 ms    average processing time (per request)
tmx    4802 ms    longest processing time (per request)
slh    6835    # of accepted HTTPS requests
slm    29    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    415    # of dropped HTTPS requests (client disconnect without sending any request)
slu    3058    # of dropped HTTPS requests (other TLS handshake errors)
uca    1745    slu break-down: # of unknown CA reported by clients
uce    653    slu break-down: # of unknown cert reported by clients
ush    263    slu break-down: # of shutdown by clients after ServerHello
sct    50    cert cache: # of certs in cache
sch    4105    cert cache: # of reuses of cached certs
scm    232    cert cache: # of misses to find a cert in cache
scp    219    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    486    sess cache: # of reuses of cached TLS sessions
ssm    280    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    2277    # of GET requests for server-side scripting
gif    46    # of GET requests for GIF
ico    5    # of GET requests for ICO
txt    460    # of GET requests for Javascripts
jpg    41    # of GET requests for JPG
png    6    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    14    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    366    # of GET requests /w unknown file extension
opt    2730    # of OPTIONS requests
pst    605    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    464    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    3    # of unknown HTTP requests (HTTP 501 response)
tmo    28    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    416    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[bengalih]

Some basic questions from someone new to this specific technology.

Based on the verbiage about pixelserv-tls from the ab-solution page I infer that ab-solution by itself is incapable of blocking any ads on pages that are secured by https://? IOW - if I am going to a website with an https:// URI (which is getting to be most places these days) ab-solver can't/won't do anything to block these ads unless I also install/integrate with pixelserv-tls?

I'm asking because I was pretty sure that I was getting ads blocked on https:// sites with ab-solution prior to installing pixelserv-tls.

Other than seeing the stats in pixelserv-tls, is there a good test I can use to see it in action - especially to distinguish what it is doing outside of standard ab-solution?

Finally - why do some ads not appear at all, whereas other times I see (in Chrome) what looks like a frowny page/broken link in a box? Is this normal, or is something not working optimally?

thanks

 

[Asad Ali]

Finally - why do some ads not appear at all, whereas other times I see (in Chrome) what looks like a frowny page/broken link in a box? Is this normal, or is something not working optimally?

One purpose of pixelserv-tls is to fix this problem.

 

[bengalih]

One purpose of pixelserv-tls is to fix this problem.

So if I'm running pixelserv and still seeing these does that mean something isn't right? Or is it just minimized, but can still happen?

 

[Asad Ali]

So if I'm running pixelserv and still seeing these does that mean something isn't right? Or is it just minimized, but can still happen?

Running pixelserv-tls is just one piece of the puzzle, you need to import the Root CA certificate to all your devices as well to complete the puzzle.

 

[bengalih]

A

Running pixelserv-tls is just one piece of the puzzle, you need to import the Root CA certificate to all your devices as well to complete the puzzle.

Ahhh...see I'm new to pixleserv - not just on this device. I didn't see instructions about this anywhere.
Is there a manual/tutorial that will describe this (and other pieces of the puzzle).

I have a self-signed certificate chain that I use on all my other web servers in my network that I have installed on my clients so that I don't get any SSL cert warnings, etc. I assume I can probably install this into the pixleserv implementation so that I don't have to use another self-signed?

Thanks!

 

[Asad Ali]

A


Ahhh...see I'm new to pixleserv - not just on this device. I didn't see instructions about this anywhere.
Is there a manual/tutorial that will describe this (and other pieces of the puzzle).

I have a self-signed certificate chain that I use on all my other web servers in my network that I have installed on my clients so that I don't get any SSL cert warnings, etc. I assume I can probably install this into the pixleserv implementation so that I don't have to use another self-signed?

Thanks!

https://github.com/kvic-z/pixelserv-tls/wiki/Create-and-Import-the-CA-Certificate

 

[bengalih]

https://github.com/kvic-z/pixelserv-tls/wiki/Create-and-Import-the-CA-Certificate

Thanks...I did find that. However it states:

Importing your CA cert on clients is not mandatory but recommended.

I'm just trying to figure out exactly what having the CA cert on my client is going to do.
What additional functionality is going to work that doesn't?
How can I tell if it is working vs. it not working?

I 100% understand how ab-solution is working (theoretically), but am unclear how pixelserv intercepts these additional ads and how having the certificate installed on the client is involved. I've searched for at least an hour around the web and short of trying to decipher source code I'm just not really sure architecturally how this application functions at all.

 

[thelonelycoder]

I'm asking because I was pretty sure that I was getting ads blocked on https:// sites with ab-solution prior to installing pixelserv-tls

That's easily possible for a simple reason. The https website you visit loads ads from non-https sites embedded in the https site source code.

 

[elorimer]

I'm just trying to figure out exactly what having the CA cert on my client is going to do.
What additional functionality is going to work that doesn't?
How can I tell if it is working vs. it not working?

I 100% understand how ab-solution is working (theoretically), but am unclear how pixelserv intercepts these additional ads and how having the certificate installed on the client is involved. I've searched for at least an hour around the web and short of trying to decipher source code I'm just not really sure architecturally how this application functions at all.

Spend the hour instead to read through this thread, which was highly educational to me on a number of levels.

Basically, when pixelserv is in the mix a call to a domain blocked by ab-s is forwarded to the pixelserv address. For a http request, pixelserv sends back a one pixel response instantly. For a https request, pixelserv sends back a one pixel response with a cert it generates, or pulls from one already generated, for that site. If it has to generate a cert, that takes a little bit of time. If it has already generated a cert, it pulls that cert from the stick and sends that back. If it has that cert in cache, then it sends it back more or less instantly. If the cert it sends back matches the imported root cert, then the pixel is served up and the web page goes on its merry way. If you havent' imported the root cert, then the web page sees a security violation and handles it that way. So either way, the web page moves on; but one way is faster and cleaner.

You can tell from the stats page if it is working, because it breaks out https requests as accepted or not, and if they fail, why. There is a script in the thread that will run a cron and email you a report that identifies which devices are generating denied requests, and those are where the import should be made. For some devices, like IP cameras, you may not be able to import a cert.

 

[M@rco]

Spend the hour instead to read through this thread, which was highly educational to me on a number of levels.

So was your post. Perfect way to explain it. I know how it works, just couldn't find the words to describe it in a way it made sense. You just did.

 

[thelonelycoder]

So was your post. Perfect way to explain it. I know how it works, just couldn't find the words to describe it in a way it made sense. You just did.

To quote @bengalih :

I 100% understand how ab-solution is working (theoretically)

So do I, and about 1% of the magic that pixelserv-tls does...

 

[bengalih]

Spend the hour instead to read through this thread, which was highly educational to me on a number of levels.

Basically, when pixelserv is in the mix a call to a domain blocked by ab-s is forwarded to the pixelserv address. For a http request, pixelserv sends back a one pixel response instantly. For a https request, pixelserv sends back a one pixel response with a cert it generates, or pulls from one already generated, for that site. If it has to generate a cert, that takes a little bit of time. If it has already generated a cert, it pulls that cert from the stick and sends that back. If it has that cert in cache, then it sends it back more or less instantly. If the cert it sends back matches the imported root cert, then the pixel is served up and the web page goes on its merry way. If you havent' imported the root cert, then the web page sees a security violation and handles it that way. So either way, the web page moves on; but one way is faster and cleaner.

You can tell from the stats page if it is working, because it breaks out https requests as accepted or not, and if they fail, why. There is a script in the thread that will run a cron and email you a report that identifies which devices are generating denied requests, and those are where the import should be made. For some devices, like IP cameras, you may not be able to import a cert.

Thanks @elorimer. I was continuing to search for information while I posted this. Weeding through 108 pages of thread posts to find relevant information was not really proving effective. I appreciate you having done it and sharing what you can. I was trying to focus on the documentation provided in the OP, which I have read most of, including:

• Project Blog https://kazoo.ga/pixelserv-tls/
• FAQ https://github.com/kvic-z/pixelserv-tls/wiki/FAQ

I also (just) imported the cert onto my clients. In all honesty I couldn't observe any tangible difference in behavior. Ads appeared to still be blocked, but my servstats definitely did change. Prior to doing this it looked like "# of accepted HTTPS requests" was 0 and after about 50% of all total requests are now accepted. That being said I still have 50% of them listed under slm/sle/slc/slu/uca/uce. Is this normal? Or is this something wrong with my configuration?

I still have some questions about architecture. Some of this may be re-iteration of what you described I'm just looking to clarify here as I'm a bit fuzzy. In the end, I'm looking to understand this both architecturally and from an end-user perspective so I can actually observe if things are not working optimally. While I can look at the servstats they only tell me numbers, not if those numbers are optimal or if something is misconfigured:

1) It sounds like what you are saying is that if using pixleserv-tls with ab-s, then pixleserv is actually doing *all* the blocking - both http & https. Is this correct? If so, then the main think ab-s continues to do is just generate the lists, but instead of using the ab-s behavior of sending them to 127.0.0.1, pixleserv-tls updates ALL the entires to the pixleserv-tls IP address to serve back the transparent pixel instead of serving back a time-out. Do I have that right?

2) Forgive my ignorance on the complete picture. With basic https traffic between a client and server the client obtains a copy of the server's public key. It trusts this public key either because this certificate (or it's issuing authority) is stored as a trusted root certificate. Using this public certificate the browser encrypts the data and sends it back to the server along with a (symmetric) session key. The rest of the secure exchange (on both ends) is done using this private session key. Now - this brief description is from my understanding 20 years ago and I don't believe the basic idea here has changed.

I'm trying to reconcile that basic behavior with the pixelserv-tls requirements/behavior.

1) As you state - pixleserv-tls is creating (or retrieving from cache) a certificate for each session. These appear to be stored here:

/tmp/mnt/xxxxx/entware/var/cache/pixelserv

Which certificates are these? As in within the basic SSL structure I describe above? Normally the only key a client/browser has to generate is the symmetric session key to return to the server. I was under the belief that this key was not something that was normally saved, but rather the client would generate a new session key for every new session and therefore not something we even needed to be aware of. Perhaps I was wrong and the keys stored in this location are the session keys? If so, that would make sense to me (though be surprising from a security standpoint that all session keys were saved). If not, then I don't understand why pixleserv is creating/needing these keys.
(I believe this to *not* be the case as I update below. I'm keeping it here though for others who may have the same misunderstanding)

2) When a client talks to a server and obtains it's public key it will give an error of some sort if that public key isn't trusted by a CA it knows about. Usually this is manifested in a browser with a security warning but I suspect in a non-interactive environment this will simply fail to establish any connection depending on the coding (kind of like wget with a --no-certificate override). Since we need to communicate to pixelserv-tls to get our transparent pixel we need to trust it just like any other https server or else we would get a security error.
Apart from a speed differential (which may be so nominal as to be unobservable) will the page actually appear different? It seems many pages still show frames/divs even when blocked so whether it was empty or a single pixel the blocked ad may still take up the same space. What I haven't seen (prior to importing the cert) is any blocked ads saying "certificate error" which I guess is what I would expect if we couldn't talk to the server?

Actually, in describing the scenarios myself and re-reading your description I have one more interpretation of what you describe regarding the certs generated. This may be what you were trying to describe and I simply couldn't grasp it in those words until re-hashing the above.
The pixelserv-tls server needs to communicate back to me as if it was the server I was talking to. So if the ad being redirected is from ads.sucky.com my browser needs to believe it is communicating with ads.sucky.com the whole time and not the pixelserv server at all. So when pixelserv intercepts this query from my browser it creates its own certificate for ads.sucky.com, caches it and returns it to me to make me believe it is that source. This makes more sense then my description of the session keys above since there is really no reason for pixelserv-tls to even speak to the real ads.sucky.com since the point is *not* to talk to it.

Once again, if I don't trust the pixelserv-tls CA, this communication would be blocked (I couldn't get my pixel) and instead I would get a certificate error. What I still don't understand on this is from observed behavior I wasn't seeing any "certificate errors" shown on my webpages. You state that without the certs installed, things still work - it is just "faster and cleaner." I'm trying to understand exactly what is cleaner about it from an end-user perspective. What will/might I see if this certificate transaction fails vs it working?

For instance, I am still seeing a lot of blocked ads showing with the gray frowny page face (Chrome) or the "Can’t reach this page" (IE) within the frame of the ad. I don't know if this is normal behavior or not. I would expect that the page *should* be found in the form of the transparent pixel. But, perhaps there are certain ads that pixleserv still can't deal with?

I realize these questions might be tedious for those of you who already know the answers, but I suspect the hard work put into this project wasn't just meant to be enjoyed by those who have immaculate understandings of all things SSL. I believe I'm far from a layperson when it comes to technology, but I'm not an SSL expert and I can only RTFM of which I'm not finding all the answers.

I appreciate your patience and assistance!

 

[thelonelycoder]

1) It sounds like what you are saying is that if using pixleserv-tls with ab-s, then pixleserv is actually doing *all* the blocking - both http & https. Is this correct? If so, then the main think ab-s continues to do is just generate the lists, but instead of using the ab-s behavior of sending them to 127.0.0.1, pixleserv-tls updates ALL the entires to the pixleserv-tls IP address to serve back the transparent pixel instead of serving back a time-out. Do I have that right?

Yes and no. Let me answer what I know.
AB-Solution does all the redirecting. If pixelserv-tls is disabled or not installed, all http requests for blocked domains in the blocking file and the blacklist are directed to the null-address 0.0.0.0:

0.0.0.0 blockeddomain.com

With pixelserv-tls installed and enabled, the blocking file and blacklist are auto-converted by AB-Solution to redirect blocked domains to the IP address pixelserv-tls is listening on, this is for http and https requests:

<pixelserv-tls IP> blockeddomain.com

Instead of the null-address 0.0.0.0 sending requests to never-land, pixelserv-tls answers the request and may send back a pixel-sized image to the browser.

 

[bengalih]

To quote @bengalih :

So do I, and about 1% of the magic that pixelserv-tls does...

Heh... I would hope so. I actually do have some professional experience with some of this technology, including SSL proxy product. This is party why I'm trying to understand it. But...I am admittedly a little rusty here and just looking to uncloud my head.

Yes and no. Let me answer what I know.
AB-Solution does all the redirecting. If pixelserv-tls is disabled or not installed, all http requests for blocked domains in the blocking file and the blacklist are directed to the null-address 0.0.0.0:

0.0.0.0 blockeddomain.com

Right. Well...I mean technically dnsmasq does all the redirecting, right? AB-Solution is responsible for properly forming the files which get appended into the hosts file. AB-solution never actually communicates with the client - it simply does all the grunt-work for the dns server to have the correct information to do so.

With pixelserv-tls installed and enabled, the blocking file and blacklist are auto-converted by AB-Solution to redirect blocked domains to the IP address pixelserv-tls is listening on, this is for http and https requests:

<pixelserv-tls IP> blockeddomain.com

Got that. I actually have some more questions on this...but will post in the ab-solution thread as they are more germane just there I think.

Instead of the null-address 0.0.0.0 sending requests to never-land, pixelserv-tls answers the request and may send back a pixel-sized image to the browser.

Yup got that. Although you say "May"? It doesn't always? I mean assuming it is configured correctly and doesn't sent a certificate error, it should send back the pixel right?

I'm wondering if there is a way for me to replace the pixel image and/or find in the source of a page what the pixel image is for testing purposes? If I put a bigger picture in I should definitely be able to tell that the ad is being blocked.

 

[thelonelycoder]

Right. Well...I mean technically dnsmasq does all the redirecting, right? AB-Solution is responsible for properly forming the files which get appended into the hosts file. AB-solution never actually communicates with the client - it simply does all the grunt-work for the dns server to have the correct information to do so.

Exactly and I wanted to add the remark but forgot to add it.
AB-Solution adds cron jobs, start scripts, compiles the hosts files, installs Entware and pixelserv and modifies some more files for it all to work as seamless as it does on the router. Dnsmasq is just one more binary involved along with many more.

Yup got that. Although you say "May"? It doesn't always? I mean assuming it is configured correctly and doesn't sent a certificate error, it should send back the pixel right?

A pixel is only served if there is a physical ad image to replace. Most of the ad content these days is loaded from javascript and is not replaced by anything at all. The space is just left blank.

 

[elorimer]

I also (just) imported the cert onto my clients. In all honesty I couldn't observe any tangible difference in behavior. Ads appeared to still be blocked, but my servstats definitely did change. Prior to doing this it looked like "# of accepted HTTPS requests" was 0 and after about 50% of all total requests are now accepted. That being said I still have 50% of them listed under slm/sle/slc/slu/uca/uce. Is this normal? Or is this something wrong with my configuration?

This is about what I get. Tivos, Rokus, IP cameras, thermostats, and for reasons I don't understand, two Android phones that have the root cert imported.

For all your other questions, I'll let @kvic respond. As @thelonelycoder said, 99% is magic. But the thread is entertaining as you see pixelserv develop: first the basic mod to work with tls, then the separate IP address, then the intermediate certs, then the way @thelonelycoder leveraged it into ab-s, then the speed improvements, then the memory improvements, then the email script. The way skynet, pixelserv-tls and ab-s came together to be a tightly integrated package that is waay more than the sum of the parts.

 

[bengalih]

This is about what I get. Tivos, Rokus, IP cameras, thermostats, and for reasons I don't understand, two Android phones that have the root cert imported.

Yes, that makes some sense. I have 2 Rokus online and at least one other phone and possibly some tablet use that don't have any certs on them. I plan to install them on what I can, but obviously you can't get them on the Rokus, etc.

I've actually got the certificates setup nicely now. I redid my PKI last night and created an Intermediate CA solely for pixelserv.
I did this because you have either remove the password from the signing cert or else place it in a "ca.key.passphrase" file on the system. While I probably have much greater problems if my router is compromised, I wouldn't want my main RootCA credentials to be taken.
But, now that I have it running off of my own CA, I don't need to distribute any certificates to my existing systems since they already are configured with my RootCA as trusted (or at least those devices that are capable).
I also created my own web server certificate for the /servstats page signed by my main Intermediate CA so that I don't get any warnings when accessing that.

For all your other questions, I'll let @kvic respond. As @thelonelycoder said, 99% is magic. But the thread is entertaining as you see pixelserv develop: first the basic mod to work with tls, then the separate IP address, then the intermediate certs, then the way @thelonelycoder leveraged it into ab-s, then the speed improvements, then the memory improvements, then the email script. The way skynet, pixelserv-tls and ab-s came together to be a tightly integrated package that is waay more than the sum of the parts.

Yeah...heh... I was doing some searching and looked through some older posts in this thread last night and found a post by @kvic explaining how to manually generate each wildcard cert for blocking and install it. I was really confused like "doesn't it do this automatically??" when I realized the thread was from 2-3 years ago and realized that development has come a long way

One other question for @kvic or @thelonelycoder regarding the initial PKI setup. I see in the FAQ/Wiki there are instructions for setting up the default self-signed pixelserv-tls certificates. However, contrary to instruction this is all done for you automatically. I'm just interested if that is old info now for pixelserv as a whole, or if ab-solution has just added scripts into its proprietary install routine to do those steps for you?

Thanks again guys - I'll be monitoring this thread moving forward so hopefully will keep up to date and be able to contribute something back!

The one last thing I'm interested in (for now ) is to get some more information on each of the servstats, particularly the sxx and uxx ones and how to interpret the log files. For instance, when debugging the logfile yesterday I was seeing some failures from my wife's phone (with no cert) that were logging her IP address. But, when I removed the cert from my laptop it looked like I was seeing some hits from my machine, but no IP was shown. If anyone knows of a good resource that explains these things in more detail I'd love to take a look!

 

[Martineau]

I'm interested in (for now ) is to get some more information on each of the servstats, particularly the sxx and uxx ones
<snip>
If anyone knows of a good resource that explains these things in more detail I'd love to take a look!

Did you check the home page/changelogs and pixelserv man page