pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

@kvic
I'm just a little confused on how I could set it with pixelserv-tls. I would like to have both ip and hostname set on pixelserv-tls.

It's either ip or hostname. Can't be both. The purpose of hostname is a convenience to get the ip address. I'm sorry if it implies more than that and confused you..

E.g. you can start with "pixelserv-tls doubleclick.com" which is exactly same as "pixelserv-tls 192.168.1.3" where the hostname is one of the blocked hostnames which are all set to 192.168.1.3 in your adblock script.

Each adblock script does it different but if you reverse lookup on 192.168.1.3, I doubt that you'll get doubleclick.com. Most likely you'll get 192.168.1.1, assuming that's the ip address of your physical NIC.

Does it clarify.. or do you fancy a new feature to be added?

 

[kvic]

v2.0.0 is released

Perhaps already old news but here is an official post for v2 release.

In terms of new features and enhancement in speed, robustness & scalability, v2 is the best pixelserv-tls so far and one notch up from previous releases in the past two years.

We now also have e.g. a new frontpage and a FAQ on Github. All details are summarised on this blog post. Please do spend sometime on it.

We shall thank the following members in participation of testing & providing helpful feedback:

@jrmwvu04 @elorimer @tom- @thelonelycoder @Raphie @Protik @shooter40sw @baltosml @Correlor @hervon @DME @Butterfly Bones @pattiri

Some noticeable events I could recall:

@jrmwvu04's stress tests crashed & pushed v2 into a much more robust build. @tom- showed up so that now we had a test binary for mipsel. @shooter40sw's detailed problem description of an issue that everyone took it as granted but it actually revealed a non-trivial issue and the best part pushed v2 into much more efficient on SSL/HTTPS handling. @Raphie's request for a script to install beta. Wow..now everyone can easily participate. @Protik's first entry into this brave new world & reported bugs that made the script in better shape.

Accept my appology as I might have missed some events & names.

Please give these gents applause (*clap*^N). I think v2 is the most enjoyable dev cycle and with the most features added/changed in the past two months.

For the impatient...

You can get v2.0.0-rc1 with the beta script instruction on wiki. You do need to have Entware-ng & pixelserv-tls ver. KK installed first.

Update instructions v2.0.0-rc1 to v2.0.0

When it's available on Entware-ng...You don't have to revert back to prod version first. Simply go ahead with Entware upgrade.

thank you all.

 

[jrmwvu04]

In terms of new features and enhancement in speed, robustness & scalability, v2 is the best pixelserv-tls so far and one notch up from previous releases in the past two years.

That’s an understatement to be very honest. If you doubt that, you need only go back to Kk for a little while. There is a perceptible difference in how quickly general web browsing performs now.

 

[Makaveli]

When it's available on Entware-ng...You don't have to revert back to prod version first. Simply go ahead with Entware upgrade.

I prefer to upgrade pixel serv inside of AB-solution how long does it usually take for it to be available on Entware-NG?

 

[thelonelycoder]

I prefer to upgrade pixel serv inside of AB-solution how long does it usually take for it to be available on Entware-NG?

Usually about a week or two, or longer.
Depends on the workload of @zyxmon and/or @ryzhov_al . It'll be added to Entware-ng when it's ready.

 

[rromeroa]

@rromeroa Does a reboot resolve your issue?

Problem persists after reboot.
Pixelserv crash with no overload. Really, I dont know what is happening. I'll try to run the process with AB disabled, maybe the conflict is with some other service.

 

[kvic]

I prefer to upgrade pixel serv inside of AB-solution how long does it usually take for it to be available on Entware-NG?

Pls keep an eye on the ticket I submitted to Entware-ng & mentioned on my blog post.

When the ticket is changed to done/closed, it'll be soon. Last time version KK was within two weeks after the ticket changed to "closed"

 

[kvic]

Problem persists after reboot.
Pixelserv crash with no overload. Really, I dont know what is happening. I'll try to run the process with AB disabled, maybe the conflict is with some other service.

Okay. Pls keep me updated on your investigation..

You may also want to add '-l 5' to enable log. Be warned that very chatty & may swamp your syslog. So turn it on briefly during your test perhaps. It seems you can easily make pixelserv-tls crash. Or else your syslog may be swamped with tonnes of access log.

 

[rromeroa]

Okay. Pls keep me updated on your investigation..

You may also want to add '-l 5' to enable log. Be warned that very chatty & may swamp your syslog. So turn it on briefly during your test perhaps. It seems you can easily make pixelserv-tls crash. Or else your syslog may be swamped with tonnes of access log.

Hi @kvic,

I have detected pixelserv-tls crash when open tapatalk app in my Android. Occurs every time, this is level 5 traces:

Nov 30 17:59:53 pixelserv-tls[12683]: [-binary POST content not dumped-]
Nov 30 18:00:28 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /servstats HTTP/1.1
Nov 30 18:00:29 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /favicon.ico HTTP/1.1
Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 graph.facebook.com GET /v2.9/469687153111700?fields=supports_implicit_sdk_logging%2Cgdpv4_nux_content%2Cgdpv4_nux_enabled%2Cgdpv4_chrome_custom_tabs_enabled%2Candroid_dialog_configs%2Candroid_sdk_error_categories%2Capp_events_session_timeout%2Capp_events_feature_bitmask%2Cseamless_login%2Csmart_login_bookmark_icon_url%2Csmart_login_menu_icon_url&format=json&sdk=android HTTP/1.1 secure
Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Nov 30 18:00:38 pixelserv-tls[12683]: [-binary POST content not dumped-]

 

[tom-]

It's either ip or hostname. Can't be both. The purpose of hostname is a convenience to get the ip address. I'm sorry if it implies more than that and confused you..

E.g. you can start with "pixelserv-tls doubleclick.com" which is exactly same as "pixelserv-tls 192.168.1.3" where the hostname is one of the blocked hostnames which are all set to 192.168.1.3 in your adblock script.

Each adblock script does it different but if you reverse lookup on 192.168.1.3, I doubt that you'll get doubleclick.com. Most likely you'll get 192.168.1.1, assuming that's the ip address of your physical NIC.

Does it clarify.. or do you fancy a new feature to be added?

Okay, thanks for clearing that up for me. I found just setting ip on pixelserv-tls and giving it a hostname with dnsmasq to work fine. Just saves time typing the ip. Like you mentioned I could just enter a blocked domain with the /servstats on the end, but it having its own hostname would be cool I think. New features are always welcome.

 

[tom-]

I'm having one small issue with both test 8d and the 2.0 from the install script on mipsel. I'm not sure if previous test 8s have this issue. Test 6 runs fine. After a few hours. Doesn't matter if there's any heavy load. Pixelserv goes un responsive and I can't access its serv stats page which then leads to pages take awhile to load because of the ad domains forwarded to pixelserv not getting an answer from pixelserv. Checking it if its running using /opt/etc/init.d/S80pixelserv-tls check shows its running. Restarting it fixes it, but it does it again a few hrs later. I'm now testing test 7. Then the rest of the 8s. Maybe itll help pin point what was changed between the versions that could be causing the issue. A little late, but better than never and have it pushed out to entware-ngs list with the issue. Work just gets in the way.

EDIT 1: Test 7 still no issue just like test 6. So, something that was done in test 8 must be causing an issue. Moving on to test 8a. Will edit again with my results.
EDIT 2: TEST 8a also does the same as 8d and 2.0. Becomes un responsive.

 

[Protik]

I'm having one small issue with both test 8d and the 2.0 from the install script on mipsel. I'm not sure if previous test 8s have this issue. Test 6 runs fine. After a few hours. Doesn't matter if there's any heavy load. Pixelserv goes un responsive and I can't access its serv stats page which then leads to pages take awhile to load because of the ad domains forwarded to pixelserv not getting an answer from pixelserv. Checking it if its running using /opt/etc/init.d/S80pixelserv-tls check shows its running. Restarting it fixes it, but it does it again a few hrs later. I'm now testing test 7. Then the rest of the 8s. Maybe itll help pin point what was changed between the versions that could be causing the issue. A little late, but better than never and have it pushed out to entware-ngs list with the issue. Work just gets in the way.

EDIT 1: Test 7 still no issue just like test 6. So, something that was done in test 8 must be causing an issue. Moving on to test 8a. Will edit again with my results.

Had similar type of issue. It's really hard to figure out what causes it as the service is running when this happens. Restarting the service solved the issue for me.

 

[kvic]

Hi @kvic,

I have detected pixelserv-tls crash when open tapatalk app in my Android. Occurs every time, this is level 5 traces:

Nov 30 17:59:53 pixelserv-tls[12683]: [-binary POST content not dumped-]
Nov 30 18:00:28 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /servstats HTTP/1.1
Nov 30 18:00:29 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /favicon.ico HTTP/1.1
Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 graph.facebook.com GET /v2.9/469687153111700?fields=supports_implicit_sdk_logging%2Cgdpv4_nux_content%2Cgdpv4_nux_enabled%2Cgdpv4_chrome_custom_tabs_enabled%2Candroid_dialog_configs%2Candroid_sdk_error_categories%2Capp_events_session_timeout%2Capp_events_feature_bitmask%2Cseamless_login%2Csmart_login_bookmark_icon_url%2Csmart_login_menu_icon_url&format=json&sdk=android HTTP/1.1 secure
Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Nov 30 18:00:38 pixelserv-tls[12683]: [-binary POST content not dumped-]

What device do you run pixelserv-tls?

Your log seems not '-l 5' as some lines are missing in-between.

I installed Tapatalk on iOS and tried with it for a while. No crash. I believe using Tapatalk is just a coincidence.

EDIT:

It's easy to collect pixelserv-tls log from syslog. Use this command

grep pixelserv-tls /tmp/syslog.log

 

[kvic]

@tom- @Protik

When your pixelserv-tls get stucked, can you 'killall -SIGUSR1 pixelserv-tls' to dump servstats into syslog and post here?

Can you also try with a different value of 'select_timeout' e.g. 10s?

 

[tom-]

@tom- @Protik

When your pixelserv-tls get stucked, can you 'killall -SIGUSR1 pixelserv-tls' to dump servstats into syslog and post here?

Can you also try with a different value of 'select_timeout' e.g. 10s?

daemon.crit pixelserv-tls[19569]: 12373 uts, 1 log, 20 kcc, 20 kmx, 1.02 kvg, 3 krq, 2150 req, 405 avg, 2427 rmx, 81 tav, 284 tmx, 4 slh, 1 slm, 0 sle, 490 slc, 0 slu, 1 nfe, 0 gif, 2 ico, 6 txt, 0 jpg, 38 png, 0 swf, 19 sta, 0 stt, 42 ufe, 0 opt, 5 pst, 0 hed, 0 rdr, 0 nou, 0 pth, 1 204, 0 bad, 0 tmo, 502 cls, 1 cly, 0 clt, 0 err[CODE/]

Thats for KL TEST 8A Mipsel

I will try the timeout now

 

[rromeroa]

What device do you run pixelserv-tls?

It's running in RT-AC68U (arm device). I do not know the cause, but I can say that always with just opening tapatalk app then process crash.
This is a full syslog from beginning until crash (open tapatalk on android device), a few seconds up...

Dec  1 07:17:52 admin: Started pixelserv-tls (AB-Solution) from .
Dec  1 07:17:52 pixelserv-tls[4746]: pixelserv-tls: v2.0.0-rc1 compiled: Nov 26 2017 20:53:17 options: 192.168.1.3 -p 80 -k 4434 -l 5
Dec  1 07:17:52 pixelserv-tls[4746]: Listening on :192.168.1.3:80
Dec  1 07:17:52 pixelserv-tls[4746]: Listening on :192.168.1.3:4434
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 709
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 688
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 656
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 708
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 658
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 691
Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
Dec  1 07:18:03 pixelserv-tls[4746]: 10.8.0.2 192.168.1.3 GET /servstats HTTP/1.1
Dec  1 07:18:03 pixelserv-tls[4746]: 10.8.0.2 192.168.1.3 GET /favicon.ico HTTP/1.1
Dec  1 07:18:09 pixelserv-tls[4746]: no file extension /469687153111700 from path /v2.9/469687153111700
Dec  1 07:18:09 pixelserv-tls[4746]: 10.8.0.2 graph.facebook.com GET /v2.9/469687153111700?fields=supports_implicit_sdk_logging%2Cgdpv4_nux_content%2Cgdpv4_nux_enabled%2Cgdpv4_chrome_custom_tabs_enabled%2Candroid_dialog_configs%2Candroid_sdk_error_categories%2Capp_events_session_timeout%2Capp_events_feature_bitmask%2Cseamless_login%2Csmart_login_bookmark_icon_url%2Csmart_login_menu_icon_url&format=json&sdk=android HTTP/1.1 secure

Hope this can help with mysterious...

Thanks

 

[kvic]

@rromeroa

This full log looks more interesting for analysis. A couple of observations:

1. I believe after you launched Tapatalk, there were a burst of six POST requests sent over HTTPS
2. You launched pixelserv-tls with options "-p 80 -k 4434". "-p" for a HTTP port defaults to 80, so it's not required to specify from command line. This is trivial. The "-k 4434" looks a bit weird. "-k" specifies a HTTPS port. Websites commonly use 443 (which is specified by standard). Rarely websites will use 4434. So there are two issues here:
   • pixelserv-tls won't receive almost all of ad requests over HTTPS
   • pixelserv-tls somehow receives those six POST requests over HTTPS. So they must be sent directly to 4434 or you have some other complicated setup through iptables to redirect those requests to 4434.
3. pixelserv-tls listens on 192.168.1.3 which is LAN
4. your client accesses pixelserv-tls from 10.8.0.2 which looks like a VPN (or some kind of other setup)? pixelserv-tls shall work over VPN. So just want some clarification here about where your Tapatalk client sits. Is it on LAN or over VPN?

If my guess is correct, your setup looks unnecessarily complicated and perhaps wrong for pixelserv-tls' HTTPS port. Is the setup default by AB-Solution?

It's sufficiently good enough to start pixelserv-tls with "pixelserv-tls 192.168.1.3 -l 5"

All these might not explain the crash puzzle but let's take one step at a time.

 

[tom-]

So far so good after setting the select_timeout to 10 seconds. No un responsiveness. Its been almost 5 hrs since I done that. Before it would go un responsive already. Hopefully this solves the issue. I'll have to test that timeout with the more recent versions. Right now I have 8a running. Its just something that, on test 7 and earlier this issue didn't occur.

 

[kvic]

@tom-

Please keep it running as long as you can on 8a for this test. Touch wood if it get stuck again or you feel gd enough & bored, simply jump to rc1 & try with 10s select_timeout.

 

[rromeroa]

I believe after you launched Tapatalk, there were a burst of six POST requests sent over HTTPS

Yes, launch tapatalk wa thes only activity in my network

You launched pixelserv-tls with options "-p 80 -k 4434".

Ok, the -p 80 comes from the past when it was necessary to specify both ports. I'll remove it now...
Port https is 4434 because standard 443 is in use and I can not change it.
I use a rule that redirects traffic with destination 192.168.1.3:443 to 192.168.1.3:4434

your client accesses pixelserv-tls from 10.8.0.2

Now I'm not at home and I've tested through vpn but the same thing happens in the local network.


Thanks