Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Privacy Filter (Another IPSET Script)

Discussion in 'Asuswrt-Merlin' started by swetoast, Jan 11, 2017.

  1. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    mind just giving me a print out of hostip from your distro ?

    hostip --help
     
  2. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    4,012
    Location:
    United States
    This is from V23E1...dnscrypt updated to 1.9.4
    Code:
    [email protected]:/tmp/home/root# hostip -V
    hostip v1.9.4
    [email protected]:/tmp/home/root# hostip -h
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
      -6, --ipv6: ask for AAAA records
      -h, --help: show usage
      -r, --resolver-address=<ip>: the resolver IP address
      -V, --version: show version number
    
    Example: hostip www.example.com
    
     
  3. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Code:
    # hostip -V
    hostip v1.9.1
    # hostip --help
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
      -6, --ipv6: ask for AAAA records
      -h, --help: show usage
      -r, --resolver-address=<ip>: the resolver IP address
      -V, --version: show version number
    
    Example: hostip www.example.com
    same but lesser version from entware so i think its just on or two domains that didnt resolve or hes using an older list that had alot of dead domains, @zyxmon mind updating hostip ?
     
  4. skeal

    skeal Regular Contributor

    Joined:
    Apr 30, 2016
    Messages:
    77
    Is there any way to make this script report to sys log the fact that it's started and running?
     
  5. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    sure thing ill add it in the next revision
     
  6. skeal

    skeal Regular Contributor

    Joined:
    Apr 30, 2016
    Messages:
    77
    Hey @swetoast send or post your PayPal.me/ link I would like to send a few bucks!


    Edit: Figured it out cash on the way!
     
    Last edited: Mar 10, 2017
    swetoast and wallyg8r like this.
  7. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    Yep John!, i figured it out and run an opkg remove hostip. The entware version was 1.9.1 and the V22 version you included is 1.9.0

    Still don't want to upgrade to v23 because of the OpenVPN 2.4 upgrade .... i should check the upgrade is not going to mess up with my ovpn clients

    Code:
    [email protected]:/jffs/scripts# hostip
    
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
    
      -6, --ipv6: ask for AAAA records
    
      -h, --help: show usage
    
      -r, --resolver-address=<ip>: the resolver IP address
    
      -V, --version: show version number
    
    
    Example: hostip www.example.com
    
    
    [email protected]:/jffs/scripts# hostip -V
    
    hostip v1.9.0
    
    
    Still the same problem [name does not exist]

    Probably something with xargs and hostip. The script only adds one ip and ignores the rest of the privacy-filter.list. If i hostip some random hostnames from the privacy-filter.list everything looks good. I'm going to see what happens more in detail.

    BTW thanks to John for the fork! and swetoast and contribs for the script :)
     
  8. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    Thrown some bucks to swetoast :). Also John send me your paypal so i can send some more bucks to your cause!
     
  9. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    Code:
    run_ipv4_block () {
    if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                     cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
            fi
    
    I think i found something wrong with xargs/traceroute arguments. If i execute

    Code:
     cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
    I get the error:

    'raceroute: bad address 'a.rad.msn.com
    'raceroute: bad address 'a-0001.a-msedge.net
    'raceroute: bad address 'a-0002.a-msedge.net
    'raceroute: bad address 'a-0003.a-msedge.net
    'raceroute: bad address 'a-0004.a-msedge.net
    'raceroute: bad address 'a-0005.a-msedge.net
    'raceroute: bad address 'a-0006.a-msedge.net
    'raceroute: bad address 'a-0007.a-msedge.net
    'raceroute: bad address 'a-0008.a-msedge.net
    'raceroute: bad address 'a-0009.a-msedge.net
    'raceroute: bad address 'ac3.msn.com
    'raceroute: bad address 'aidps.atdmt.com
    'raceroute: bad address 'aka-cdn-ns.adtech.de
    'raceroute: bad address 'b.ads1.msn.com
    'raceroute: bad address 'b.rad.msn.com
    'raceroute: bad address 'bs.serving-sys.com
    'raceroute: bad address 'c.atdmt.com
    'raceroute: bad address 'c.msn.com
    'raceroute: bad address 'choice.microsoft.com
    'raceroute: bad address 'choice.microsoft.com.nsatc.net

    If i do the same in my osx laptop i get good results.

    To put it simple:

    This
    Code:
    cat privacy-filter.list | xargs -n1 traceroute
    does not work in the router but works in osx

    I'm going to mess with the xargs params
     
  10. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    Found the problem!

    The privacy-filter.list file downloaded was not properly encoded (CRLF)

    Now i have the ipset privacy-filter_ipv4 populated correctly
     
  11. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    114
    It is based upon how you downloaded the file. Simply run "dos2unix filename" from the command line on the router.
     
  12. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    @visortgw yeah, i tried that but i don't know why it didn't work. The thing is that if i delete the privacy-filter.list, the script downloads it again without the unix CRLF encoding ... weird ...
     
  13. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Hey guys awesome and thx for the beers tonight you guys rock and ive thrown all the supporters into the scripts :D
     
    sfx2000 likes this.
  14. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    next version will fix that with dos2unix :D
     
  15. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    update to revision 13 with syslog notification upon load and more streamlined script
     
  16. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Revision 14 is up with ipv6 fixes simpler detection
     
  17. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    12
    Works flawlessly :)
     
    swetoast likes this.
  18. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Rev 15 is up :)
     
  19. mikelees2

    mikelees2 New Around Here

    Joined:
    Feb 20, 2017
    Messages:
    7
    Where is it located?
     
  20. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    opening post, will add to wiki also
     

Share This Page