pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[Twiglets]

Seems like the TLS report script in #2067 fits your need! The script is under tests. It requires you edit a few lines to complete config. Good news is help should be available from a few ppl. We could add you the the testing group.

I have got 'shutdown by clients after ServerHello sent' and also would like to identify where it is coming from, could I have access to your script as well ?

 

[kvic]

I have got 'shutdown by clients after ServerHello sent' and also would like to identify where it is coming from, could I have access to your script as well ?

Of coz! More testing the better

 

[Asad Ali]

86U as always has newer stuff. If Openwrt/Entware doesn't change the default, I believe the feature is already available! I can check later to see if that's the case.

Alrighty! Thanks.

 

[Twiglets]

Of coz! More testing the better

Thank you muchly
Reading through the conversation now to catch up with everyone !!!

 

[kvic]

2.1.3-test.4 is available

Changes

• relax memory pool restriction for better concurrent thread performance
• increase number of retries in handshakes
• improve accuracy in processing time of requests recovered from initially failed handshakes

Install

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

Testing

This test version is expected to improve performance in medium to heavy workload. The chance of "reached max retries" failures is reduced (if not completely eliminated in your environment). Memory usage shall be similar to previous versions.

Also tav shall more accurately reflect the reality, not screwed to longer but false processing time by failed requests of "reached max retries". For such requests, time is mostly spent on waiting. In previous versions, it's accounted as processing time. That's not correct.

Would appreciate feedback on the following counters from your runs

slh 1385   # of accepted HTTPS requests
slm 3        # of rejected HTTPS requests (missing certificate)
sle 0        # of rejected HTTPS requests (certificate available but bad)
slc 14       # of dropped HTTPS requests (client disconnect without sending any request)
slu 30      # of dropped HTTPS requests (other TLS handshake errors)
uca 0       slu break-down: # of unknown CA reported by clients
uce 28     slu break-down: # of unknown cert reported by clients
ush 1       slu break-down: # of shutdown by clients after ServerHello

 

[Makaveli]

2.1.3-test.4 is available

Just updated to this will post my counters later after I allow it to run for a few hours.

 

[Clark Griswald]

@kvic up time 13hrs

 

[Twiglets]

Uptime 15hrs [Includes loss of DSL overnight ]

 

[jrmwvu04]

First time with a build with any of the changes since 2.1.1 was finalized. Lots of new info in the logs as I look through them.

 

[XIII]

slh: 305
slm: 42
sle: 0
slc: 1490
slu: 131
uca: 0
uce: 0
ush: 84

 

[Makaveli]

pixelserv-tls 2.1.3-test.4 (compiled: Jul 17 2018 01:44:19) options: 192.168.1.3 -l 2

uts 1d 02:02 process uptime
log 2 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 1 number of active service threads
kmx 23 maximum number of service threads
kvg 1.11 average number of requests per service thread
krq 14 max number of requests by one service thread

req 6382 total # of requests (HTTP, HTTPS, success, failure etc)
avg 754 bytes average size of requests
rmx 6119 bytes largest size of request(s)
tav 7 ms average processing time (per request)
tmx 1828 ms longest processing time (per request)

slh 785 # of accepted HTTPS requests
slm 4 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slc 4040 # of dropped HTTPS requests (client disconnect without sending any request)
slu 751 # of dropped HTTPS requests (other TLS handshake errors)
uca 0 slu break-down: # of unknown CA reported by clients
uce 330 slu break-down: # of unknown cert reported by clients
ush 7 slu break-down: # of shutdown by clients after ServerHello

sct 50 cert cache: # of certs in cache
sch 5093 cert cache: # of reuses of cached certs
scm 36 cert cache: # of misses to find a cert in cache
scp 23 cert cache: # of purges to give room for a new cert
sst 6 sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh 1010 sess cache: # of reuses of cached TLS sessions
ssm 563 sess cache: # of misses to find a TLS session in cache
ssp 0 sess cache: # of purges to give room for a new TLS session

 

[jrmwvu04]

Some thoughts. For the time being, I've modified syslog-ng to capture the entirety of pixelserv log output to a singular file, though it would be trivial to sort uca, uce, and ush into separate files. I've not been inclined to think it's worth doing yet. Seeing a few of the same domains in my logs as mentioned previously in this thread while I was away - is there any sort of benefit to sending for instance iCloud metrics to 0.0.0.0 versus letting pixelserv handle it? (Assuming you wouldn't prefer it plainly whitelisted instead.) I understand in a basic sense what shutdown by clients after ServerHello means, but not at all what the implications are.

 

[joe scian]

slh 6860 # of accepted HTTPS requests
slm 92 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slc 1056 # of dropped HTTPS requests (client disconnect without sending any request)
slu 887 # of dropped HTTPS requests (other TLS handshake errors)
uca 0 slu break-down: # of unknown CA reported by clients
uce 0 slu break-down: # of unknown cert reported by clients
ush 796 slu break-down: # of shutdown by clients after ServerHello

 

[Protik]

After 26 hours,

pixelserv-tls 2.1.3-test.4 (compiled: Jul 17 2018 01:44:19) options: 192.168.2.2 -c 150 -l 2

uts    1d 02:37    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    21    number of active service threads
kmx    34    maximum number of service threads
kvg    1.33    average number of requests per service thread
krq    55    max number of requests by one service thread
req    5085    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1023 bytes    average size of requests
rmx    9191 bytes    largest size of request(s)
tav    30 ms    average processing time (per request)
tmx    1330 ms    longest processing time (per request)
slh    1577    # of accepted HTTPS requests
slm    35    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1206    # of dropped HTTPS requests (client disconnect without sending any request)
slu    2069    # of dropped HTTPS requests (other TLS handshake errors)
uca    27    slu break-down: # of unknown CA reported by clients
uce    1161    slu break-down: # of unknown cert reported by clients
ush    760    slu break-down: # of shutdown by clients after ServerHello

 

[kvic]

Some thoughts. For the time being, I've modified syslog-ng to capture the entirety of pixelserv log output to a singular file, though it would be trivial to sort uca, uce, and ush into separate files. I've not been inclined to think it's worth doing yet.

My script can sift through one or more log files (e.g. from two instances of pixelserv-tls) and do the counting in a nice way. you'd perhaps shall give it a try instead. I won't recommend to use syslog-ng for such a filtering job.

Seeing a few of the same domains in my logs as mentioned previously in this thread while I was away - is there any sort of benefit to sending for instance

metrics.icloud.com to 0.0.0.0 versus letting pixelserv handle it? (Assuming you wouldn't prefer it plainly whitelisted instead.) I understand in a basic sense what shutdown by clients after ServerHello means, but not at all what the implications are.

Personally I've sent graph.instagram.com to 0.0.0.0 because when it happens it's like an avalanche...a burst of thousand requests.

metrics.icloud.com only a few per day is fine for me. I added it to "suppression list" in my script instead.

My latest stats from the past 38 hours:

 

[kvic]

Thanks for all your interest! I kind of get it working on my 56U. Really have to observe longer if it's stable. If it's unstable, it could cause lots of crashes.

BTW, the libc library will be usable on 56U/68U/87U/3100/88U/etc. All Asus routers that share the ARMv7-softfloat with kernel v2.6.x Entware environment. Basically all these devices share the same Entware build/repository.

I'll invite all ppl expressed interest with replies/likes. Let's expect something to test before next weekend.

I took the chance to re-think some of the optimizations done in v2.1. Perhaps we can relax a bit (as the main memory hog is proven to be the ssl library). So expect more little changes coming in next pixelserv-tls beta. Will have better performance under medium to heavy workload.

Some update on this

I've been trying to find some benchmarks to test the new feature. Luckily got one after a few days. Ported the bench test to 56U and run. Bad news: performance is worse than without the feature!

Run the same bench test on Debian Jessie (very outdated software components) vs Arch Linux (everything is latest and..greatest). Same hardware. The new feature (together with many other new features I assume) shows significant boost in the bench test!

A few obvious questions:
1) my porting of the feature to 56U has problem
2) the bench test isn't appropriate to this particular new feature
3) etc..

It requires more of my time to look at them. Perhaps another week (if not more)..

p.s. there is an early reply to this thread that's pending moderation. People can read here for the time being.

 

[Mutzli]

Stats from the last 24 hours:
12788 req, 508 avg, 49458 rmx, 1 tav, 621 tmx
30 slh, 23 slm, 0 sle, 3120 slc, 2986 slu, 0 uca, 2477 uce, 308 ush

 

[kvic]

Some update on this

I've been trying to find some benchmarks to test the new feature. Luckily got one after a few days. Ported the bench test to 56U and run. Bad news: performance is worse than without the feature!

Run the same bench test on Debian Jessie (very outdated software components) vs Arch Linux (everything is latest and..greatest). Same hardware. The new feature (together with many other new features I assume) shows significant boost in the bench test!

A few obvious questions:
1) my porting of the feature to 56U has problem
2) the bench test isn't appropriate to this particular new feature
3) etc..

It requires more of my time to look at them. Perhaps another week (if not more)..

p.s. there is an early reply to this thread that's pending moderation. People can read here for the time being.

Okay..the situation isn't that bad. Getting some progress.
But Houston, we got another problem!

I just found that this forum recently restricts to five participants in a conversation.

Bonkers!

Have to find another platform for small group discussions. That's what I would prefer to such tests initially.

Any suggestions?

 

[Asad Ali]

Okay..the situation isn't that bad. Getting some progress.
But Houston, we got another problem!

I just found that this forum recently restricts to five participants in a conversation.

Bonkers!

Have to find another platform for small group discussions. That's what I would prefer to such tests initially.

Any suggestions?

Create a discord channel and keep it invite only.

 

[JaimeZX]

Seems like the TLS report script in #2067 fits your need! The script is under tests. It requires you edit a few lines to complete config. Good news is help should be available from a few ppl. We could add you the the testing group.

Sure, sounds good! Would like to figure this out.... although I may already have cracked the nut. You can't have a user certificate if you don't have a screen lock password/pin/pattern, apparently. The tablets we just have at home for entertaining the two-year-old and misc browsing are just swipe-to-unlock; so they may be the source of most of these errors. What do you think?