pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[jtp10181]

I got it now, edited my post. I was missing the hash bang line

 

[jtp10181]

Ok, now that I have it running, some feedback.

I noticed on some sites they are trying to pull in a lot of .js (javascript) files for animated ads. It does block them but the site itself gets stuck in a loop and the site keeps requesting multiple files over and over and over. I guess getting a image back when it asked for a .js file makes it angry. Suggestion is that if the request is for a .js file just feed it a empty blank.js file, or even the same filename it asked for, but blank.

Unless it is doing that already?

 

[kvic]

I noticed on some sites they are trying to pull in a lot of .js (javascript) files for animated ads. It does block them but the site itself gets stuck in a loop and the site keeps requesting multiple files over and over and over.

You're very observant indeed.

When situation like this happens, it's either those websites are poorly programmed or they're programmed with much effort to ensure the integrity of page components are intact. E.g. if missing some js, major functionalities of a page won't work as designed. Some might even crash e.g. HTML based smartphone apps.

In either case, return an empty js is very unlikely to resolve the problem. On one occasion, I ended up simply removing a few domains from the block list because the website/phone app is important to me...

 

[mstombs]

Ok, now that I have it running, some feedback.

I noticed on some sites they are trying to pull in a lot of .js (javascript) files for animated ads. It does block them but the site itself gets stuck in a loop and the site keeps requesting multiple files over and over and over. I guess getting a image back when it asked for a .js file makes it angry. Suggestion is that if the request is for a .js file just feed it a empty blank.js file, or even the same filename it asked for, but blank.

Unless it is doing that already?

Should already be returning a null text response to ".js*" Internet explorer used to try to execute the gif as a script! But agree it doesn't always work, may be better to just block the request in some cases? To be expected as websites (like this one) rely on ad revenue - if you want to browse the site whitelist the relevant ads!

 

[jtp10181]

Yeah i just turned the whole thing off for now. Might be better to just try browser based stuff and only block the sites I want. Most sites I feel are fine but some have really disrupting ads such that you can't even find the content you want.

Sent from my XT1080 using Tapatalk

 

[kvic]

Turning the whole dnsmasq adblock off is sad. In fact, one of the main benefits of running pixelserv is to easily catch what hosts to white list!

To do that, simply run pixelserv with access log command line option. On your PC browser, click the page/website that fails to load.

Now stop pixelserv by killing it.

Look at the tail of the syslog, you will find a few attempted access to ad servers from your last action. From there on, quite obvious what hosts to try putting into white list..

 

[hammer]

Very nice port/improvement of pixelserv @kvic !
Found it a few days ago while looking at what forks that was made of Pixelserv.
I've been using HunterZ version until now on a TP-Link Archer C5 (V1) running OpenWrt. Unfortunately the pre-compiled mips binaries for pixelserv-tls does not run on my router so I've spent some time making them compile.
Compiling using "uClibc" works for the "dynamic" versions but not the "static" versions.
When compiling using "musl" the TEMP_FAILURE_RETRY macro is unknown and therefore linking fails.
This can be resolved be adding the follow to the top of "pixelserv.c"

#ifndef TEMP_FAILURE_RETRY
#define TEMP_FAILURE_RETRY(expression)            \
  (__extension__                                  \
    ({ long int __result;                         \
       do __result = (long int) (expression);     \
       while (__result == -1L && errno == EINTR); \
       __result; }))
#endif

After adding the code "static" builds work (these were tested on "chaos calmer" and "trunk" - I did not test the "dynamic" versions compiled using "musl" but I assume they work)

 

[kvic]

Very nice port/improvement of pixelserv @kvic !
Found it a few days ago while looking at what forks that was made of Pixelserv.
I've been using HunterZ version until now on a TP-Link Archer C5 (V1) running OpenWrt. Unfortunately the pre-compiled mips binaries for pixelserv-tls does not run on my router so I've spent some time making them compile.
Compiling using "uClibc" works for the "dynamic" versions but not the "static" versions.
When compiling using "musl" the TEMP_FAILURE_RETRY macro is unknown and therefore linking fails.
This can be resolved be adding the follow to the top of "pixelserv.c"

#ifndef TEMP_FAILURE_RETRY
#define TEMP_FAILURE_RETRY(expression)            \
  (__extension__                                  \
    ({ long int __result;                         \
       do __result = (long int) (expression);     \
       while (__result == -1L && errno == EINTR); \
       __result; }))
#endif

After adding the code "static" builds work (these were tested on "chaos calmer" and "trunk" - I did not test the "dynamic" versions compiled using "musl" but I assume they work)

Bravo on you! Thanks for letting me know. Happy pixelserv-ing!

Spoiler: my current servstat

pixelserv version: V35.HZ12.Kg compiled: Dec 24 2015 11:35:33 options: 192.168.1.1
uts: Uptime                      7 days 19:37
req: Total # of requests         39832
avg: Avg size of reqs            729 bytes
rmx: Max size of reqs            44687 bytes
tav: Avg process time            540 ms
tmx: Max process time            10120 ms
err: # of error reqs             0
tmo: # of client timeout         551
cls: # of client shutdown           683
nou: # of reqs w/o URL           0
pth: # of invalid URL            0
nfe: # of missing file ext       6612
ufe: # of unknown file ext       3374
gif: # of GIF reqs               236
bad: # of unknown HTTP methods   388
txt: # of TXT reqs               17481
jpg: # of JPG reqs               22
png: # of PNG reqs               10
swf: # of SWF reqs               2
ico: # of ICO reqs               1
slh: # of HTTPS /w a good cert   4566
slm: # of HTTPS w/o a cert       8
sle: # of HTTPS /w a bad cert    0
slu: # of unrecognized HTTPS     29
sta: # of HTML stats             1
stt: # of text stats             0
204: # of HTTP/204 (no content)  0
rdr: # of redirects              8747
pst: # of POST method            1724
hed: # of HEAD method            0

 

[peveks]

Proceed to setup auto run of pixelserv on startup by deploying the init.d script in post #1.

Could you please explain a little bit more detailed how to setup auto run?
I made by this path /opt/etc/init.d/ a file S81pixelserv-custom with your script from the post #1, but after reboot it doesn't start... What do I do wrong?

 

[kvic]

Could you please explain a little bit more detailed how to setup auto run?
I made by this path /opt/etc/init.d/ a file S81pixelserv-custom with your script from the post #1, but after reboot it doesn't start... What do I do wrong?

Sorry for the late response... I assume you have Entware up and running. If so, check permission on the script. You can add execute perm by "chmod a+x /opt/etc/init.d/S81pixelserv-custom"

The Entware rc.unslung won't pick up init scripts that lack execute perm.

 

[peveks]

Sorry for the late response... I assume you have Entware up and running. If so, check permission on the script. You can add execute perm by "chmod a+x /opt/etc/init.d/S81pixelserv-custom"

The Entware rc.unslung won't pick up init scripts that lack execute perm.

Thank you very much. Currently I do have the server up and running and can see server stats by the link:
http://192.168.1.1/servstats

But I'm still a bit confused. I have 2 questions left. Could you please help me to clear them?
1. How can I manage black and white lists for this server? I didn't find information about it.

2. Is it possible to run the server with possibility of full logging of all connections and web sites visited? Where this log is stored? I tried to run the server with "-l" option but only the records I got it's like:

Jan 20 17:01:36 dnsmasq-dhcp[446]: DHCPREQUEST(br0) 192.168.1.83 f4:09:d8:f4:22:29
Jan 20 17:01:36 dnsmasq-dhcp[446]: DHCPACK(br0) 192.168.1.83 f4:09:d8:f4:22:29 android-509a45de26a67851
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPDISCOVER(br0) 70:ee:50:02:69:f0
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPOFFER(br0) 192.168.1.132 70:ee:50:02:69:f0
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPDISCOVER(br0) 70:ee:50:02:69:f0
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPOFFER(br0) 192.168.1.132 70:ee:50:02:69:f0
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPREQUEST(br0) 192.168.1.132 70:ee:50:02:69:f0
Jan 20 17:01:42 dnsmasq-dhcp[446]: DHCPACK(br0) 192.168.1.132 70:ee:50:02:69:f0
Jan 20 17:01:49 dnsmasq-dhcp[446]: DHCPRELEASE(br0) 192.168.1.132 70:ee:50:02:69:f0

Thank you very much.

 

[kvic]

1. How can I manage black and white lists for this server? I didn't find information about it.

Pixelserv doesn't do whitelist/blacklist by itself. Those two functions are part of the process you generate config files for dnsmasq.

Think of Pixelserv as a tiny companion to dnsmasq.

One of the things pixelserv can help is to spot host names to whitelist for a broken site, through enabling logging and inspecting the log.

2. Is it possible to run the server with possibility of full logging of all connections and web sites visited?

Full web access log can be obtained by deploying a web proxy such as privacy. I think Adaptive QoS when enabled also provides limited web logging.

The logs u provided aren't from pixelserv. I believe u took it from dnsmasq.

When -l is used, pixelserv logging by default goes to syslog.log. It will easily flood your syslog. Hence -l only recommended when u are fixing a broken website caused by excessive blocking.

 

[peveks]

Pixelserv doesn't do whitelist/blacklist by itself. Those two functions are part of the process you generate config files for dnsmasq.

So with Pixelserv you can find and add required sites to dnsmaq to block it, right?

Full web access log can be obtained by deploying a web proxy such as privacy. I think Adaptive QoS when enabled also provides limited web logging.

Do you mean can not find it in pixelserv logs and need to install an additional service?

The logs u provided aren't from pixelserv. I believe u took it from dnsmasq.

Where by default pixelsrv store it's log? The logs I provided were from syslog.log

When -l is used, pixelserv logging by default goes to syslog.log. It will easily flood your syslog. Hence -l only recommended when u are fixing a broken website caused by excessive blocking.

What parameters can you recommend to use for pexelsrv to monitor hosts that were connected through the router?


I'm sorry for a lot of probably stupid questions but I really want to find out how does it works. Before I was thinking that this server is for blocking host, now I realise that it's like an instrument to find required hosts to block or unblock.

Actually why I started all of this. I would like to block youtube ads from my tv, so I tried many methods and it still shows. SO I would like to find finally required hosts and block it. Maybe you have some thoughts how to do it?

UPD.: I tried to run the server with -l and no additional information in syslog.log. The same as I mentioned before... what do I do wrong? . I disabled adaptive QoS...

 

[kvic]

Actually why I started all of this. I would like to block youtube ads from my tv, so I tried many methods and it still shows. SO I would like to find finally required hosts and block it. Maybe you have some thoughts how to do it?

I'm sorry to say but pixelserv will not block anything including youtube ads.

You need to work on the blacklist for your dnsmasq setup. Depends on how extensive blocking you applied there. You may or may not have luck blocking youtube ads.

UPD.: I tried to run the server with -l and no additional information in syslog.log. The same as I mentioned before... what do I do wrong? . I disabled adaptive QoS...

That's expected because I think you didn't change ip address in your blacklist to point to 192.168.1.1. So your ad requests go to 0.0.0.0, and not hitting 192.168.1.1 where pixelserv listens in your run.

That's the most probable explanation according to your report. I would recommend you not to pursue pixelserv further for blocking youtube ads.

 

[peveks]

That's expected because I think you didn't change ip address in your blacklist to point to 192.168.1.1. So your ad requests go to 0.0.0.0, and not hitting 192.168.1.1 where pixelserv listens in your run.
.

Where can I change it? Anyway as it installed so will get some statistics.

 

[kvic]

Where can I change it? Anyway as it installed so will get some statistics.

In your script that generate blacklist files for dnsmasq. Search 0.0.0.0 and replace with 192.168.1.1.

 

[terno]

Thanks, kvic, for your work on this project.
I'm having an issue whereby the CA doesn't show up in the certification path of the generated certificates, causing the browser to throw either a CERTIFICATION_AUTHORITY_INVALID (chrome) or unknown issuer (firefox).
The CA certificate has been imported on the client, in the OS and in the respective browsers.

Am I missing something here, or do the browsers have more stringent certificate checks than before?

 

[kvic]

Thanks, kvic, for your work on this project.

I'm having an issue whereby the CA doesn't show up in the certification path of the generated certificates, causing the browser to throw either a CERTIFICATION_AUTHORITY_INVALID (chrome) or unknown issuer (firefox).
The CA certificate has been imported on the client, in the OS and in the respective browsers.

I just tried in latest Chrome and couldn't reproduce the error...

Am I missing something here, or do the browsers have more stringent certificate checks than before?

For the auto-gen'ed certs, issuer name is automagically taken from your Root CA cert supplied to pixelserv (default is /opt/var/cache/pixelserv/ca.*).

I would check the way you generated the Root CA cert. Make sure Common Name is filled in at the minimum, and without funky characters. If I recall correctly, the CN of Root CA will be issuer name if not explicitly overriden. Then this CN will be also the issuer name for certs auto-gen'ed inside pixelserv..

 

[Goobi]

@kvic, I was wondering if you could give me some pointers on how to actually install your fork of pixelserv? I downloaded pixelserv.V35.HZ12.Kg.arm.zip and pixelserv-tls-master.zip. However, it is not clear to me exactly what file(s) I need to execute in order to install pixelserv. Any pointers would be much appreciated. BTW, I am running the latest merlin on a 68U if that matters. I see in your 1st post that the pixelserv binary must be placed in /opt/sbin. Is it pixelserv.c from the pixelserv-tls-master package that needs to be place in /opt/sbin?

Edit: I was able to get it installed by moving the pixelserv.arm.performance.static binary into /opt/sbin and but also renaming it to pixelserv so that the /opt/etc/init.d/S81pixelserv script picks it up.

 

[kvic]

@Goobi glad that you sorted it out already.

To avoid the same pitfall for future users, I've amended the init.d script in post #1. Added "/jffs/bin" to PATH variable. So that the binary can simply be saved in /jffs/bin as an alternative location, not touching /opt/sbin or any other Entware executable paths.

Here is my practice:

1. Save pixelserv.arm.performance.static in /jffs/bin
2. Make a symbolic link to the binary with "ln -s pixelserv.arm.performance.static pixelserv"