1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

pixelserv - A Better One-pixel Webserver for Adblock

Discussion in 'Asuswrt-Merlin' started by kvic, Jul 28, 2015.

  1. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    @rromeroa Does a reboot resolve your issue?

    I perhaps shall clarify a bit wrt my previous reply. I'm not the first go-to person for your issue (or any issue of interactions between ABS and pixelserv-tls) because I had the impression from previous two issues that were related to interactions between ABS and pixelserv-tls.

    Surely a robust pixelserv-tls as it's right now shall not crash at any rate (*cough*). But bad things happen with unknown reasons. I meant to say once there was some clue from ABS perspective, I would be readily available for help.
     
  2. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    dirtyreturn likes this.
  3. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    It's either ip or hostname. Can't be both. The purpose of hostname is a convenience to get the ip address. I'm sorry if it implies more than that and confused you..

    E.g. you can start with "pixelserv-tls doubleclick.com" which is exactly same as "pixelserv-tls 192.168.1.3" where the hostname is one of the blocked hostnames which are all set to 192.168.1.3 in your adblock script.

    Each adblock script does it different but if you reverse lookup on 192.168.1.3, I doubt that you'll get doubleclick.com. Most likely you'll get 192.168.1.1, assuming that's the ip address of your physical NIC.

    Does it clarify.. or do you fancy a new feature to be added?
     
  4. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    v2.0.0 is released

    Perhaps already old news but here is an official post for v2 release.

    In terms of new features and enhancement in speed, robustness & scalability, v2 is the best pixelserv-tls so far and one notch up from previous releases in the past two years.

    We now also have e.g. a new frontpage and a FAQ on Github. All details are summarised on this blog post. Please do spend sometime on it.

    We shall thank the following members in participation of testing & providing helpful feedback:

    @jrmwvu04 @elorimer @tom- @thelonelycoder @Raphie @Protik @shooter40sw @baltosml @Correlor @hervon @DME @Butterfly Bones @pattiri

    Some noticeable events I could recall:

    @jrmwvu04's stress tests crashed & pushed v2 into a much more robust build. @tom- showed up so that now we had a test binary for mipsel. @shooter40sw's detailed problem description of an issue that everyone took it as granted but it actually revealed a non-trivial issue and the best part pushed v2 into much more efficient on SSL/HTTPS handling. @Raphie's request for a script to install beta. Wow..now everyone can easily participate. @Protik's first entry into this brave new world & reported bugs that made the script in better shape.

    Accept my appology as I might have missed some events & names.

    Please give these gents applause (*clap*^N). I think v2 is the most enjoyable dev cycle and with the most features added/changed in the past two months.

    For the impatient...

    You can get v2.0.0-rc1 with the beta script instruction on wiki. You do need to have Entware-ng & pixelserv-tls ver. KK installed first.

    Update instructions v2.0.0-rc1 to v2.0.0

    When it's available on Entware-ng...You don't have to revert back to prod version first. Simply go ahead with Entware upgrade.

    thank you all.
     
    Xentrk, MacG32, tom- and 9 others like this.
  5. jrmwvu04

    jrmwvu04 Senior Member

    Joined:
    Mar 29, 2016
    Messages:
    495
    Location:
    United States
    That’s an understatement to be very honest. If you doubt that, you need only go back to Kk for a little while. There is a perceptible difference in how quickly general web browsing performs now.
     
    kvic and Protik like this.
  6. Makaveli

    Makaveli Senior Member

    Joined:
    Nov 4, 2016
    Messages:
    360
    Location:
    Canada
    I prefer to upgrade pixel serv inside of AB-solution how long does it usually take for it to be available on Entware-NG?
     
    eclp likes this.
  7. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,689
    Location:
    Switzerland
    Usually about a week or two, or longer.
    Depends on the workload of @zyxmon and/or @ryzhov_al . It'll be added to Entware-ng when it's ready.
     
    eclp, heysoundude and Makaveli like this.
  8. rromeroa

    rromeroa Regular Contributor

    Joined:
    Nov 13, 2015
    Messages:
    88
    Problem persists after reboot.
    Pixelserv crash with no overload. Really, I dont know what is happening. I'll try to run the process with AB disabled, maybe the conflict is with some other service.
     
  9. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    Pls keep an eye on the ticket I submitted to Entware-ng & mentioned on my blog post.

    When the ticket is changed to done/closed, it'll be soon. Last time version KK was within two weeks after the ticket changed to "closed"

    Screen Shot 2017-11-30 at 1.42.35 PM.png
     
    SMS786 and Makaveli like this.
  10. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    Okay. Pls keep me updated on your investigation..

    You may also want to add '-l 5' to enable log. Be warned that very chatty & may swamp your syslog. So turn it on briefly during your test perhaps. It seems you can easily make pixelserv-tls crash. Or else your syslog may be swamped with tonnes of access log.
     
  11. rromeroa

    rromeroa Regular Contributor

    Joined:
    Nov 13, 2015
    Messages:
    88
    Hi @kvic,

    I have detected pixelserv-tls crash when open tapatalk app in my Android. Occurs every time, this is level 5 traces:

    Nov 30 17:59:53 pixelserv-tls[12683]: [-binary POST content not dumped-]
    Nov 30 18:00:28 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /servstats HTTP/1.1
    Nov 30 18:00:29 pixelserv-tls[12683]: 10.8.0.2 192.168.1.3 GET /favicon.ico HTTP/1.1
    Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 graph.facebook.com GET /v2.9/469687153111700?fields=supports_implicit_sdk_logging%2Cgdpv4_nux_content%2Cgdpv4_nux_enabled%2Cgdpv4_chrome_custom_tabs_enabled%2Candroid_dialog_configs%2Candroid_sdk_error_categories%2Capp_events_session_timeout%2Capp_events_feature_bitmask%2Cseamless_login%2Csmart_login_bookmark_icon_url%2Csmart_login_menu_icon_url&format=json&sdk=android HTTP/1.1 secure
    Nov 30 18:00:38 pixelserv-tls[12683]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Nov 30 18:00:38 pixelserv-tls[12683]: [-binary POST content not dumped-]
     
  12. tom-

    tom- Occasional Visitor

    Joined:
    Oct 31, 2017
    Messages:
    25
    Okay, thanks for clearing that up for me. I found just setting ip on pixelserv-tls and giving it a hostname with dnsmasq to work fine. Just saves time typing the ip. Like you mentioned I could just enter a blocked domain with the /servstats on the end, but it having its own hostname would be cool I think. New features are always welcome. :)
     
  13. tom-

    tom- Occasional Visitor

    Joined:
    Oct 31, 2017
    Messages:
    25
    I'm having one small issue with both test 8d and the 2.0 from the install script on mipsel. I'm not sure if previous test 8s have this issue. Test 6 runs fine. After a few hours. Doesn't matter if there's any heavy load. Pixelserv goes un responsive and I can't access its serv stats page which then leads to pages take awhile to load because of the ad domains forwarded to pixelserv not getting an answer from pixelserv. Checking it if its running using /opt/etc/init.d/S80pixelserv-tls check shows its running. Restarting it fixes it, but it does it again a few hrs later. I'm now testing test 7. Then the rest of the 8s. Maybe itll help pin point what was changed between the versions that could be causing the issue. A little late, but better than never and have it pushed out to entware-ngs list with the issue. Work just gets in the way.

    EDIT 1: Test 7 still no issue just like test 6. So, something that was done in test 8 must be causing an issue. Moving on to test 8a. Will edit again with my results.
    EDIT 2: TEST 8a also does the same as 8d and 2.0. Becomes un responsive.
     
    Last edited: Nov 30, 2017
  14. Protik

    Protik Senior Member

    Joined:
    Oct 31, 2017
    Messages:
    237
    Location:
    /boot
    Had similar type of issue. It's really hard to figure out what causes it as the service is running when this happens. Restarting the service solved the issue for me.
     
  15. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    What device do you run pixelserv-tls?

    Your log seems not '-l 5' as some lines are missing in-between.

    I installed Tapatalk on iOS and tried with it for a while. No crash. I believe using Tapatalk is just a coincidence.

    EDIT:

    It's easy to collect pixelserv-tls log from syslog. Use this command
    Code:
    grep pixelserv-tls /tmp/syslog.log
    
     
    Last edited: Nov 30, 2017
  16. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    @tom- @Protik

    When your pixelserv-tls get stucked, can you 'killall -SIGUSR1 pixelserv-tls' to dump servstats into syslog and post here?

    Can you also try with a different value of 'select_timeout' e.g. 10s?
     
  17. tom-

    tom- Occasional Visitor

    Joined:
    Oct 31, 2017
    Messages:
    25
    Code:
    daemon.crit pixelserv-tls[19569]: 12373 uts, 1 log, 20 kcc, 20 kmx, 1.02 kvg, 3 krq, 2150 req, 405 avg, 2427 rmx, 81 tav, 284 tmx, 4 slh, 1 slm, 0 sle, 490 slc, 0 slu, 1 nfe, 0 gif, 2 ico, 6 txt, 0 jpg, 38 png, 0 swf, 19 sta, 0 stt, 42 ufe, 0 opt, 5 pst, 0 hed, 0 rdr, 0 nou, 0 pth, 1 204, 0 bad, 0 tmo, 502 cls, 1 cly, 0 clt, 0 err[CODE/]
    
    Thats for KL TEST 8A Mipsel
    I will try the timeout now
     
  18. rromeroa

    rromeroa Regular Contributor

    Joined:
    Nov 13, 2015
    Messages:
    88
    It's running in RT-AC68U (arm device). I do not know the cause, but I can say that always with just opening tapatalk app then process crash.
    This is a full syslog from beginning until crash (open tapatalk on android device), a few seconds up...

    Code:
    Dec  1 07:17:52 admin: Started pixelserv-tls (AB-Solution) from .
    Dec  1 07:17:52 pixelserv-tls[4746]: pixelserv-tls: v2.0.0-rc1 compiled: Nov 26 2017 20:53:17 options: 192.168.1.3 -p 80 -k 4434 -l 5
    Dec  1 07:17:52 pixelserv-tls[4746]: Listening on :192.168.1.3:80
    Dec  1 07:17:52 pixelserv-tls[4746]: Listening on :192.168.1.3:4434
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 709
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 688
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 656
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 708
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 658
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 Content-Length: 691
    Dec  1 07:18:00 pixelserv-tls[4746]: POST socket: 9 expect length: 0
    Dec  1 07:18:00 pixelserv-tls[4746]: 10.8.0.2 e.crashlytics.com POST /spi/v2/events HTTP/1.1 secure
    Dec  1 07:18:00 pixelserv-tls[4746]: [-binary POST content not dumped-]
    Dec  1 07:18:03 pixelserv-tls[4746]: 10.8.0.2 192.168.1.3 GET /servstats HTTP/1.1
    Dec  1 07:18:03 pixelserv-tls[4746]: 10.8.0.2 192.168.1.3 GET /favicon.ico HTTP/1.1
    Dec  1 07:18:09 pixelserv-tls[4746]: no file extension /469687153111700 from path /v2.9/469687153111700
    Dec  1 07:18:09 pixelserv-tls[4746]: 10.8.0.2 graph.facebook.com GET /v2.9/469687153111700?fields=supports_implicit_sdk_logging%2Cgdpv4_nux_content%2Cgdpv4_nux_enabled%2Cgdpv4_chrome_custom_tabs_enabled%2Candroid_dialog_configs%2Candroid_sdk_error_categories%2Capp_events_session_timeout%2Capp_events_feature_bitmask%2Cseamless_login%2Csmart_login_bookmark_icon_url%2Csmart_login_menu_icon_url&format=json&sdk=android HTTP/1.1 secure
    
    Hope this can help with mysterious...

    Thanks
     

    Attached Files:

  19. kvic

    kvic Part of the Furniture

    Joined:
    Aug 11, 2014
    Messages:
    2,444
    Location:
    22.4399N 114.2222E
    @rromeroa

    This full log looks more interesting for analysis. A couple of observations:
    1. I believe after you launched Tapatalk, there were a burst of six POST requests sent over HTTPS
    2. You launched pixelserv-tls with options "-p 80 -k 4434". "-p" for a HTTP port defaults to 80, so it's not required to specify from command line. This is trivial. The "-k 4434" looks a bit weird. "-k" specifies a HTTPS port. Websites commonly use 443 (which is specified by standard). Rarely websites will use 4434. So there are two issues here:
      • pixelserv-tls won't receive almost all of ad requests over HTTPS
      • pixelserv-tls somehow receives those six POST requests over HTTPS. So they must be sent directly to 4434 or you have some other complicated setup through iptables to redirect those requests to 4434.
    3. pixelserv-tls listens on 192.168.1.3 which is LAN
    4. your client accesses pixelserv-tls from 10.8.0.2 which looks like a VPN (or some kind of other setup)? pixelserv-tls shall work over VPN. So just want some clarification here about where your Tapatalk client sits. Is it on LAN or over VPN?
    If my guess is correct, your setup looks unnecessarily complicated and perhaps wrong for pixelserv-tls' HTTPS port. Is the setup default by AB-Solution?

    It's sufficiently good enough to start pixelserv-tls with "pixelserv-tls 192.168.1.3 -l 5"

    All these might not explain the crash puzzle but let's take one step at a time.
     
  20. tom-

    tom- Occasional Visitor

    Joined:
    Oct 31, 2017
    Messages:
    25
    So far so good after setting the select_timeout to 10 seconds. No un responsiveness. Its been almost 5 hrs since I done that. Before it would go un responsive already. Hopefully this solves the issue. I'll have to test that timeout with the more recent versions. Right now I have 8a running. Its just something that, on test 7 and earlier this issue didn't occur.