1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

pixelserv - A Better One-pixel Webserver for Adblock

Discussion in 'Asuswrt-Merlin' started by kvic, Jul 28, 2015.

  1. Jumpstarter

    Jumpstarter Senior Member

    Joined:
    Apr 3, 2019
    Messages:
    290
    I am trying to compile this to a debian setup.. does anyone have good instructions to do so?
    These are my reads from compiling so far
    Code:
    $ make
    make  all-am
    make[1]: Entering directory '/home/pi/pixelserv-tls'
    gcc -DHAVE_CONFIG_H -I.    -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -MT pixelserv_tls-pixelserv.o -MD -MP -MF .deps/pixelserv_tls-pixelserv.Tpo -c -o pixelserv_tls-pixelserv.o `test -f 'pixelserv.c' || echo './'`pixelserv.c
    mv -f .deps/pixelserv_tls-pixelserv.Tpo .deps/pixelserv_tls-pixelserv.Po
    gcc -DHAVE_CONFIG_H -I.    -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -MT pixelserv_tls-socket_handler.o -MD -MP -MF .deps/pixelserv_tls-socket_handler.Tpo -c -o pixelserv_tls-socket_handler.o `test -f 'socket_handler.c' || echo './'`socket_handler.c
    mv -f .deps/pixelserv_tls-socket_handler.Tpo .deps/pixelserv_tls-socket_handler.Po
    gcc -DHAVE_CONFIG_H -I.    -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -MT pixelserv_tls-certs.o -MD -MP -MF .deps/pixelserv_tls-certs.Tpo -c -o pixelserv_tls-certs.o `test -f 'certs.c' || echo './'`certs.c
    mv -f .deps/pixelserv_tls-certs.Tpo .deps/pixelserv_tls-certs.Po
    gcc -DHAVE_CONFIG_H -I.    -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -MT pixelserv_tls-util.o -MD -MP -MF .deps/pixelserv_tls-util.Tpo -c -o pixelserv_tls-util.o `test -f 'util.c' || echo './'`util.c
    mv -f .deps/pixelserv_tls-util.Tpo .deps/pixelserv_tls-util.Po
    gcc -DHAVE_CONFIG_H -I.    -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -MT pixelserv_tls-logger.o -MD -MP -MF .deps/pixelserv_tls-logger.Tpo -c -o pixelserv_tls-logger.o `test -f 'logger.c' || echo './'`logger.c
    mv -f .deps/pixelserv_tls-logger.Tpo .deps/pixelserv_tls-logger.Po
    gcc -DDROP_ROOT -DIF_MODE -DDEFAULT_PEM_PATH=\"/var/cache/pixelserv\" -O3 -Wall -ffunction-sections -fdata-sections -fno-strict-aliasing  -g -O2 -Wl,--gc-sections -s  -o pixelserv-tls pixelserv_tls-pixelserv.o pixelserv_tls-socket_handler.o pixelserv_tls-certs.o pixelserv_tls-util.o pixelserv_tls-logger.o  -lssl -lcrypto -lpthread -lrt -ldl
    make[1]: Leaving directory '/home/pi/pixelserv-tls'
     $ sudo make install
    make[1]: Entering directory '/home/pi/pixelserv-tls'
     /bin/mkdir -p '/usr/local/bin'
      /usr/bin/install -c pixelserv-tls '/usr/local/bin'
     /bin/mkdir -p '/usr/local/share/man/man1'
     /usr/bin/install -c -m 644 pixelserv-tls.1 '/usr/local/share/man/man1'
    make[1]: Leaving directory '/home/pi/pixelserv-tls'
    
    I cannot figure out where i could be going wrong,or if i am successfully compiling I am not sure how to get it to start.
     
  2. Jumpstarter

    Jumpstarter Senior Member

    Joined:
    Apr 3, 2019
    Messages:
    290
    got a little bit further this time...

    Code:
    This package will be built according to these values:
    
    0 -  Maintainer: [ [email protected] ]
    1 -  Summary: [ Package created with checkinstall 1.6.2 ]
    2 -  Name:    [ pixelserv-tls ]
    3 -  Version: [ 2.3.0 ]
    4 -  Release: [ 1 ]
    5 -  License: [ GPL ]
    6 -  Group:   [ checkinstall ]
    7 -  Architecture: [ armhf ]
    8 -  Source location: [ pixelserv-tls ]
    9 -  Alternate source location: [  ]
    10 - Requires: [  ]
    11 - Provides: [ pixelserv ]
    12 - Conflicts: [  ]
    13 - Replaces: [  ]
    
    Enter a number to change any of them or press ENTER to continue:
    
    Installing with make install...
    
    ========================= Installation results ===========================
    make[1]: Entering directory '/home/pi/pixelserv-tls'
     /bin/mkdir -p '/usr/local/bin'
      /usr/bin/install -c pixelserv-tls '/usr/local/bin'
     /bin/mkdir -p '/usr/local/share/man/man1'
     /usr/bin/install -c -m 644 pixelserv-tls.1 '/usr/local/share/man/man1'
    make[1]: Leaving directory '/home/pi/pixelserv-tls'
    
    ======================== Installation successful ==========================
    
    Copying documentation directory...
    ./
    ./LICENSE
    ./ChangeLog
    ./INSTALL
    ./README.md
    
    Copying files to the temporary directory...OK
    
    Stripping ELF binaries and libraries...OK
    
    Compressing man pages...OK
    
    Building file list...OK
    
    Building Debian package...OK
    
    Installing Debian package...OK
    
    Erasing temporary files...OK
    
    Writing backup package...OK
    OK
    
    Deleting temp dir...OK
    
    
    **********************************************************************
    
     Done. The new package has been installed and saved to
    
     /home/pi/pixelserv-tls/pixelserv-tls_2.3.0-1_armhf.deb
    
     You can remove it from your system anytime using:
    
          dpkg -r pixelserv-tls
    
    **********************************************************************
    
    Code:
     $ sudo dpkg -i /home/pi/pixelserv-tls/pixelserv-tls_2.3.0-1_armhf.deb
    (Reading database ... 161435 files and directories currently installed.)
    Preparing to unpack .../pixelserv-tls_2.3.0-1_armhf.deb ...
    Unpacking pixelserv-tls (2.3.0-1) over (2.3.0-1) ...
    Setting up pixelserv-tls (2.3.0-1) ...
    Processing triggers for man-db (2.8.5-2) ...
    
     
  3. gattaca

    gattaca Senior Member

    Joined:
    Feb 18, 2012
    Messages:
    253
    It would be GREAT to have all this documented here or better still in the fork. I think Jack has a helluva time getting all the packages installed properly to compile. Most of these details need skills most of us do not have have. I consider "pixelserv is a "one-of-a-kind" community endeavor as evident in this thread and these AVS forums. Peace.
     
    rjpreston likes this.
  4. pdc

    pdc Occasional Visitor

    Joined:
    Nov 21, 2018
    Messages:
    25
    I have a newbie question on the stats. After 20 days, I have around 100K accepted HTTPS requests (slh), but 65K dropped (slu). Of those, the vast majority are "unknown cert reported by clients" (uce).

    At the bottom of the stats, there are also 30K "client disconnect before response sent" (cly).

    Everything else looks pretty good.

    The unknown cert sounds like I didn't install the cert correctly on at least one of my many clients with my pixelserv cert. Is that correct? And if so, is there some way I can figure out which client?

    Code:
    pixelserv-tls 2.2.1 (compiled: May 22 2019 13:44:50 flags: tfo tls1_3) 
    
    uts 20d 14:48 process uptime
    log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
    kcc 1 number of active service threads
    kmx 80 maximum number of service threads
    kvg 1.34 average number of requests per service thread
    krq 1873 max number of requests by one service thread
    
    req 178915 total # of requests (HTTP, HTTPS, success, failure etc)
    avg 1253 bytes average size of requests
    rmx 152991 bytes largest size of request(s)
    tav 16 ms average processing time (per request)
    tmx 5181 ms longest processing time (per request)
    
    slh    104309    # of accepted HTTPS requests
    slm    98    # of rejected HTTPS requests (missing certificate)
    sle    0    # of rejected HTTPS requests (certificate available but not usable)
    slc    8729    # of dropped HTTPS requests (client disconnect without sending any request)
    slu    63524    # of dropped HTTPS requests (other TLS handshake errors)
    
    v13    22912    slh/slc break-down: TLS 1.3
    v12    90056    slh/slc break-down: TLS 1.2
    v10    70    slh/slc break-down: TLS 1.0
    zrt    320    slh break-down: TLS 1.3 Early Data aka 0-RTT
    
    uca    2642    slu break-down: # of unknown CA reported by clients
    ucb    0    slu break-down: # of bad certificate reported by clients
    uce    60222    slu break-down: # of unknown cert reported by clients
    ush    597    slu break-down: # of shutdown by clients after ServerHello
    
    (etc...)
    
    cls    8761    # of dropped requests (client disconnect without sending any request)
    cly    28510    # of dropped requests (client disconnect before response sent)
    clt    0    # of dropped requests (reached maximum service threads)
    err    0    # of dropped requests (unknown reason)
     
    gattaca likes this.
  5. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    3,421
    Location:
    USA
    It's not a question of your own browser cert, but usually the device or app expecting a certain pinned certificate that it doesn't receive back.

    You can increase the log level temporarily by browsing http://192.168.1.2/log=2 and then watch the router's system log for pixelserv messages. Then you can reset to normal with http://192.168.1.2/log=1 (be sure to use your actual Pixelserv IP).

    Some of my log entries from my iPhone:
    Code:
    Oct 11 13:21:24 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60144 server e.crashlytics.com
    Oct 11 13:21:25 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60150 server e.crashlytics.com
    Oct 11 13:21:25 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60151 server settings.crashlytics.com
    Oct 11 13:21:27 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60156 server e.crashlytics.com
    Oct 11 13:21:27 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60160 server e.crashlytics.com
    Oct 11 13:21:31 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60161 server e.crashlytics.com
    Oct 11 13:21:31 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60162 server e.crashlytics.com
    EDIT: @elorimer is more accurate about the uce counts than I am. The extra logging will tell you which IPs report an unknown cert most likely because the CA isn't installed.
    Code:
    Oct 11 13:37:32 pixelserv-tls[1043]: handshake failed: unknown cert. client 192.168.1.129:60133 server csi.gstatic.com
     
    Last edited: Oct 11, 2019
    cvx01, gattaca and pdc like this.
  6. elorimer

    elorimer Very Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    1,200
    These are reasonable numbers, better than mine. The slu figures are usually things where you can't import a cert: Amazon gizmos, IP cameras, stuff like that.
     
    pdc, QuikSilver and dave14305 like this.
  7. pdc

    pdc Occasional Visitor

    Joined:
    Nov 21, 2018
    Messages:
    25
    Cool, I was not aware of the temporary log level trick, that's very handy!

    Thanks @elorimer as well, glad to know my pixelserv cert is working as well as it can.

    [Edit] Just got a few hits, turns out to also be some of my top-5 blocked domains, so makes sense the numbers are high.
    Code:
    Oct 11 13:56:10 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.82:37754 server ssl.google-analytics.com
    Oct 11 13:57:47 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53977 server p.typekit.net
    Oct 11 13:57:51 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53986 server p.typekit.net
    Oct 11 13:57:55 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53989 server p.typekit.net
    Oct 11 13:57:55 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53990 server p.typekit.net
    Oct 11 13:57:56 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53991 server p.typekit.net
    Oct 11 13:57:58 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53994 server p.typekit.net
    Oct 11 13:57:58 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53995 server p.typekit.net
    Oct 11 13:58:00 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53996 server p.typekit.net
    Oct 11 13:59:14 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54014 server p.typekit.net
    Oct 11 14:01:14 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54032 server p.typekit.net
    Oct 11 14:01:17 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54036 server p.typekit.net
    Oct 11 14:01:20 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54039 server p.typekit.net
    Oct 11 14:01:20 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54040 server p.typekit.net
    Oct 11 14:01:27 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54041 server p.typekit.net
    Oct 11 14:01:31 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.82:37774 server ssl.google-analytics.com
    Oct 11 14:01:31 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54042 server p.typekit.net
    Oct 11 14:01:47 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.81:45661 server ssl.google-analytics.com
    
     
    Last edited: Oct 11, 2019
  8. elorimer

    elorimer Very Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    1,200
    This reminds me. My thumb drive failed and I stupidly didn't have a backup. I lost @kvic's TLS reporting script shown in #2067. I can't seem to find it anywhere. Anyone have it?
     
  9. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    3,421
    Location:
    USA
    https://kazoo.ga/pixelserv-tls/tls-alert.sh
     
    Kingp1n, SMS786, Makaveli and 2 others like this.
  10. Kingp1n

    Kingp1n Very Senior Member

    Joined:
    Feb 27, 2018
    Messages:
    622
  11. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    3,421
    Location:
    USA
    I put mine in /jffs/scripts. To run it as-is, you also need to install bash from Entware and supply your own email script. I think it would be cool for one of our scripting gurus to replicate this in the built-in shell.
     
  12. elorimer

    elorimer Very Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    1,200
    I seem to recall it also needs dig, for the reverse lookup?

    Note that @thelonelycoder does it a different way in Diversion, doesn't use bind-dig.
     
    Last edited: Oct 13, 2019
  13. jrmwvu04

    jrmwvu04 Very Senior Member

    Joined:
    Mar 29, 2016
    Messages:
    621
    Location:
    United States
    bind-dig as I recall. Been a while though.
     
    dave14305 likes this.
  14. Kingp1n

    Kingp1n Very Senior Member

    Joined:
    Feb 27, 2018
    Messages:
    622
    Was
    Can you explain more on my own "email script"? Is this simple to accomplish?
     
  15. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    3,421
    Location:
    USA
    I hacked the original tls-alert.sh script with the following added near the end:
    Code:
    #${EMAIL_SCRIPT} "pixelserv-tls: Today's TLS handshake errors & new certs" /tmp/tmp.alert
    # email settings
    SMTP="smtp.gmail.com"
    PORT="465"
    USERNAME="[email protected]"
    PASSWORD="my gmail app password"
    
    # Mail Enveloppe
    FROM_NAME="My Name"
    FROM_ADDRESS="[email protected]"
    TO_NAME="My Name"
    TO_ADDRESS="[email protected]"
    
    echo "From: \"$FROM_NAME\" <$FROM_ADDRESS>" > /tmp/mail.txt
    echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >> /tmp/mail.txt
    echo "Subject: pixelserv-tls: Today's TLS handshake errors & new certs" >> /tmp/mail.txt
    echo "" >> /tmp/mail.txt
    cat /tmp/tmp.alert >> /tmp/mail.txt
    
    curl --url smtps://$SMTP:$PORT \
      --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
      --upload-file /tmp/mail.txt \
      --ssl-reqd \
      --user "$USERNAME:$PASSWORD"
    
    rm /tmp/tmp.alert /tmp/mail.txt
     
    Kingp1n and Butterfly Bones like this.
  16. elorimer

    elorimer Very Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    1,200
    I've often thought it could be a useful part of Diversion. If one attends to the suppression array, it helps to identify when a device is doing something new that is not associated with browser activity. So for some reason a device is trying to reach a site that Diversion has identified as blocked; not an ad, and not the 'normal' telemetrics, so something worthy of investigating. But maintaining the array is a fair amount of work.

    A lot of the Diversion coding concepts could be leveraged:
    --the email setup
    --the frequency of sending an email
    --the maintenance of the suppression array as a separate file, following the same treatment as the whitelist/blacklist maintenance
    --checking that the entware packages are installed and updated

    And it could be extended:
    --a means of automatically adding to the suppression array new combinations generated by the script
    --extending existing combinations to new devices

    But that would be a lot of work and I'm not sure there are enough paying attention to make it worth it, except that Diversion would draw attention to it.
     
    dave14305 likes this.
  17. elorimer

    elorimer Very Senior Member

    Joined:
    Dec 16, 2013
    Messages:
    1,200
    I did something similar yesterday (no where near as clean a job as yours) as a separate email.sh script passing the two parameters. On my list, though, is to pull all the specs from /opt/share/diversion/.conf/email.conf.

    EDIT: Yes, so leave the alert script alone and let Diversion do the heavy lifting (for all four email types) by creating /jffs/scripts/email.sh and make it executable:
    Code:
    #!/opt/bin/sh
    #Parameters passed#
    mailsubject=$1
    mailbody=$2
    # Email settings (mail envelope) #
    . /opt/share/diversion/.conf/email.conf
    #Build email
        echo "From: \"$FRIENDLY_ROUTER_NAME\" <$FROM_ADDRESS>" >/tmp/mail.txt
        echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >>/tmp/mail.txt
        echo "Subject: $mailsubject " >>/tmp/mail.txt
        echo "Date: $(date -R)" >>/tmp/mail.txt
        echo >>/tmp/mail.txt
        echo " $(cat $mailbody)" >>/tmp/mail.txt
     
    #Send Email
    /usr/sbin/curl --url $PROTOCOL://$SMTP:$PORT \
                   --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
                   --upload-file /tmp/mail.txt \
                   --ssl-reqd \
                   --user "$USERNAME:$PASSWORD" $SSL_FLAG
    #Log results
    if [ "$?" = "0" ]; then
                    logger -t pixelserv-tls "emailed compiled pixelserv breaking stats from $0"
                else
                    logger -t pixelserv-tls "failed to send pixelserv stats from $0"
    fi
    
    rm /tmp/mail.txt
    Props to @thelonelycoder .

    EDIT: As of Diversion 4.1.7 this no longer works, because the password is now encrypted in a separate file. Change the last line of the curl instruction to
    Code:
    --user "$USERNAME:$(openssl aes-256-cbc -d -in /opt/share/diversion/.conf/emailpw.enc -pass pass:ditbabot,isoi)
    Or, add a new line after the call to email.conf:
    Code:
    PASSWORD=$(openssl aes-256-cbc -d -in /opt/share/diversion/.conf/emailpw.enc -pass pass:ditbabot,isoi)
     
    Last edited: Dec 28, 2019
  18. Kingp1n

    Kingp1n Very Senior Member

    Joined:
    Feb 27, 2018
    Messages:
    622
  19. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    3,421
    Location:
    USA
    Check your inbox for the email. :)

    You can run it manually to see what happens. Make sure you adjust this line to point to default syslog.log if you aren’t using syslog-ng as kvic was fond of.

    Code:
    LOGFILE="/tmp/syslog.log-1 /tmp/syslog.log"
    Edit: Also make sure you schedule it in Cron:
    Code:
    cru a pixelserv-report "59 23 * * * /opt/bin/bash /jffs/scripts/tls-alert.sh"
     
    Last edited: Oct 13, 2019
    Kingp1n likes this.
  20. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    7,006
    Location:
    Switzerland
    Though it is there with Entware, as the shebang I use the default Almquist shell instead of Bash, none of my coding uses bash syntax.
    #!/opt/bin/sh
     
    gattaca and L&LD like this.