pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[elorimer]

Over four days, this is what I collected:

log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 3 number of active service threads
kmx 55 maximum number of service threads
kvg 1.17 average number of requests per service thread
krq 167 max number of requests by one service thread

The average isn't so bad, but I wonder if a website could spawn 55 threads with 167 requests in each!

 

[kvic]

Over four days, this is what I collected:

log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 3 number of active service threads
kmx 55 maximum number of service threads
kvg 1.17 average number of requests per service thread
krq 167 max number of requests by one service thread

The average isn't so bad, but I wonder if a website could spawn 55 threads with 167 requests in each!

'kmx' and 'krq' are independent counters i.e two events may register themselves at different time.

In KL-test2, they register the corresponding maximum ever reached. In next test version, i'm going to change 'kmx', 'kvg' and 'krq" into exponential moving averages (similar to e.g. 'tav').

EMA's will be able to better represent the recent performance. Otherwise, if your pixelserv is up for 100 days. On the first day you happened to register the maximums, they'll stuck there forever.

With more clients on LAN/VPN hitting pixelserv, 'kmx' and 'krq' could reach very high values since service threads are more readily re-used.

Nevertheless, 167 beats mine and blows my mind away once more!

 

[jrmwvu04]

syslog full of "Failed to create conn_handler thread" errors, 346 of them in these four days. Usually they come in bunches, 5 to 10, even sometimes as many as 20 within one second.

 

[kvic]

syslog full of "Failed to create conn_handler thread" errors, 346 of them in these four days. Usually they come in bunches, 5 to 10, even sometimes as many as 20 within one second.

Assume you're on AC1900P and all Entware-ng libraries are up today . The error showing up often sounds very weird. it indicates the pixelserv-tls process fails to spawn new pthreads...usually due to lack of memory (but could be due to other reasons). I should have added better error codes...going to do it in next test version.

My AC56U has never seen this error. Its used RAM level usually at around 78MB mark (not counting cache+buffer). I can only think of your router is very busy or having tonnes of ad requests in burst and running short of RAM. Sound like a "hostile" environment good for testing.

 

[elorimer]

Something to do with having 347 active threads?

 

[kvic]

Something to do with having 347 active threads?

347 active threads are a LOT for the little device. Is the -O option changed from its default? Only consider increasing it significantly if running pixelserv-tls on a powerful machine..

Interestingly I didn't see the screenshot when I replied on pc. Now I see it on phone. Weird.

 

[jrmwvu04]

Assume you're on AC1900P and all Entware-ng libraries are up today

Yes.

I can only think of your router is very busy or having tonnes of ad requests in burst and running short of RAM

Nothing really going on most of the time. I run entware but only for pixelserv and the openssh sftp server. I use John's traditional QoS. I don't use VPN and have most of the servers (iTunes, minidlna, samba, etc.) disabled.

Sound like a "hostile" environment good for testing.

I'm game if I can help test something.

Is the -O option changed from its default?

I run it on a virtual ip with the following flags:

172.19.1.10 -p 80 -p 81 -p 88 -p 8080 -k 443 -k 8443 -o 1

 

[kvic]

There is relationship between -o and -O. The number of "wait_cnt" loops in case socket returns nothing in a service thread is equal to # (-O) divided by # (-o).

With default, 300/10=30 successive nothing from socket read. With 300/1=300 successive nothing. So that the time a HTTP/1.1 connection stays alive is always constant as specified in -O regardless of the value of -o in each wait for a socket read. It's to keep things simple.

@jrmwvu04 I'm running with -o 1 now. Let's see..

 

[jrmwvu04]

Loading dailymail.co.uk in safari kills Kl-test2. Have reproduced it a number of times. The process just dies entirely. Dropping down to Kk from entware solves it. Same settings from my earlier post for both Kl-test2 and Kk.

Edit: worth mentioning this is with the mvps and yoyo sources

 

[kvic]

Figured out what caused crash and >300 threads. Testing on my home LAN. If it runs smoothly, might put up for download in a couple of days.

 

[kvic]

KL-test3 is available: https://kazoo.ga/pixelserv-tls/

Here is a snapshot from my automated torture tests.

pls try as hard as you can to crash it..

 

[jrmwvu04]

pls try as hard as you can to crash it..

It's surviving what test2 would not, so far.

edit:

 

[elorimer]

A little over 2 hours on KL3, no issues:

uts 0d 02:43 process uptime
log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 1 number of active service threads
kmx 47 maximum number of service threads
kvg 1.23 average number of requests per service thread
krq 28 max number of requests by one service thread
req 2393 total # of requests (HTTP, HTTPS, success, failure etc)
avg 946 bytes average size of requests
rmx 3816 bytes largest size of request(s)
tav 100 ms average processing time (per request)
tmx 10000 ms longest processing time (per request)

469 active service threads! I'm curious what websites jrmwvu04 went to in those first 16 minutes. I only got to 47 threads visiting the most ad intensive sites I know.

 

[jrmwvu04]

469 active service threads! I'm curious what websites jrmwvu04 went to in those first 16 minutes. I only got to 47 threads visiting the most ad intensive sites I know.

Nothing too crazy or out of the ordinary. The thing is that the kcc counter never decreases. So after a few websites it will be maybe 20 or 30. 100 some later on. Then 300-400 eventually.

Edit: This is with intentionally aggressive browsing.

 

[dirtyreturn]

@kvic .

Can you ease describe how to get your package of pixelserv-tls up and running on a netgear r8000 running ddwrt?

I did manage to finally get the zip downloaded onto the router. Though, I'm not sure where or how to continue. opkg install does not work for me and the unzip command spits out an error.
I just don't know what to do. Would you help me out?

Sent from my Nexus 6P using Tapatalk

 

[kvic]

@kvic .

Can you ease describe how to get your package of pixelserv-tls up and running on a netgear r8000 running ddwrt?

I did manage to finally get the zip downloaded onto the router. Though, I'm not sure where or how to continue. opkg install does not work for me and the unzip command spits out an error.
I just don't know what to do. Would you help me out?

Sent from my Nexus 6P using Tapatalk

See if these steps work for you.

 

[kvic]

@jrmwvu04

After you stop the "intentional aggressive browsing", does the thread count gradually drop to a single digit?

 

[thelonelycoder]

I'm not much of a tester because I restart pixelserv often to test other things on my always-on router.
Kl test 3 runs without a hitch so far.

 

[jrmwvu04]

@jrmwvu04

After you stop the "intentional aggressive browsing", does the thread count gradually drop to a single digit?

No, it doesn't.

This is what it's looking like right now, there has been little to no traffic for several hours.

 

[jrmwvu04]

Also, I have noticed that at times when I’m testing it (especially during “intentionally aggressive browsing”) the web interface at pixelserv-ip-address/servstats is slow or unresponsive, sometimes for 10-15 seconds. Had never happened to me before the Kl test builds.