Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Privacy Filter (Another IPSET Script)

Discussion in 'Asuswrt-Merlin' started by swetoast, Jan 11, 2017.

  1. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    4,633
    Location:
    United States
    This is from V23E1...dnscrypt updated to 1.9.4
    Code:
    admin@AC68P-06650:/tmp/home/root# hostip -V
    hostip v1.9.4
    admin@AC68P-06650:/tmp/home/root# hostip -h
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
      -6, --ipv6: ask for AAAA records
      -h, --help: show usage
      -r, --resolver-address=<ip>: the resolver IP address
      -V, --version: show version number
    
    Example: hostip www.example.com
    
     
  2. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Code:
    # hostip -V
    hostip v1.9.1
    # hostip --help
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
      -6, --ipv6: ask for AAAA records
      -h, --help: show usage
      -r, --resolver-address=<ip>: the resolver IP address
      -V, --version: show version number
    
    Example: hostip www.example.com
    same but lesser version from entware so i think its just on or two domains that didnt resolve or hes using an older list that had alot of dead domains, @zyxmon mind updating hostip ?
     
  3. skeal

    skeal Senior Member

    Joined:
    Apr 30, 2016
    Messages:
    284
    Location:
    Moose Jaw Saskatchewan Canada
    Is there any way to make this script report to sys log the fact that it's started and running?
     
  4. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    sure thing ill add it in the next revision
     
  5. skeal

    skeal Senior Member

    Joined:
    Apr 30, 2016
    Messages:
    284
    Location:
    Moose Jaw Saskatchewan Canada
    Hey @swetoast send or post your PayPal.me/ link I would like to send a few bucks!


    Edit: Figured it out cash on the way!
     
    Last edited: Mar 10, 2017
    swetoast and wallyg8r like this.
  6. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Yep John!, i figured it out and run an opkg remove hostip. The entware version was 1.9.1 and the V22 version you included is 1.9.0

    Still don't want to upgrade to v23 because of the OpenVPN 2.4 upgrade .... i should check the upgrade is not going to mess up with my ovpn clients

    Code:
    admin@RT-N66U:/jffs/scripts# hostip
    
    Usage: hostip [-6] [-r resolver_ip[:port]] host_name
    
      -6, --ipv6: ask for AAAA records
    
      -h, --help: show usage
    
      -r, --resolver-address=<ip>: the resolver IP address
    
      -V, --version: show version number
    
    
    Example: hostip www.example.com
    
    
    admin@RT-N66U:/jffs/scripts# hostip -V
    
    hostip v1.9.0
    
    
    Still the same problem [name does not exist]

    Probably something with xargs and hostip. The script only adds one ip and ignores the rest of the privacy-filter.list. If i hostip some random hostnames from the privacy-filter.list everything looks good. I'm going to see what happens more in detail.

    BTW thanks to John for the fork! and swetoast and contribs for the script :)
     
  7. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Thrown some bucks to swetoast :). Also John send me your paypal so i can send some more bucks to your cause!
     
  8. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Code:
    run_ipv4_block () {
    if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                     cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
            fi
    
    I think i found something wrong with xargs/traceroute arguments. If i execute

    Code:
     cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
    I get the error:

    'raceroute: bad address 'a.rad.msn.com
    'raceroute: bad address 'a-0001.a-msedge.net
    'raceroute: bad address 'a-0002.a-msedge.net
    'raceroute: bad address 'a-0003.a-msedge.net
    'raceroute: bad address 'a-0004.a-msedge.net
    'raceroute: bad address 'a-0005.a-msedge.net
    'raceroute: bad address 'a-0006.a-msedge.net
    'raceroute: bad address 'a-0007.a-msedge.net
    'raceroute: bad address 'a-0008.a-msedge.net
    'raceroute: bad address 'a-0009.a-msedge.net
    'raceroute: bad address 'ac3.msn.com
    'raceroute: bad address 'aidps.atdmt.com
    'raceroute: bad address 'aka-cdn-ns.adtech.de
    'raceroute: bad address 'b.ads1.msn.com
    'raceroute: bad address 'b.rad.msn.com
    'raceroute: bad address 'bs.serving-sys.com
    'raceroute: bad address 'c.atdmt.com
    'raceroute: bad address 'c.msn.com
    'raceroute: bad address 'choice.microsoft.com
    'raceroute: bad address 'choice.microsoft.com.nsatc.net

    If i do the same in my osx laptop i get good results.

    To put it simple:

    This
    Code:
    cat privacy-filter.list | xargs -n1 traceroute
    does not work in the router but works in osx

    I'm going to mess with the xargs params
     
  9. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Found the problem!

    The privacy-filter.list file downloaded was not properly encoded (CRLF)

    Now i have the ipset privacy-filter_ipv4 populated correctly
     
  10. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    128
    It is based upon how you downloaded the file. Simply run "dos2unix filename" from the command line on the router.
     
  11. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    @visortgw yeah, i tried that but i don't know why it didn't work. The thing is that if i delete the privacy-filter.list, the script downloads it again without the unix CRLF encoding ... weird ...
     
  12. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Hey guys awesome and thx for the beers tonight you guys rock and ive thrown all the supporters into the scripts :D
     
    sfx2000 likes this.
  13. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    next version will fix that with dos2unix :D
     
  14. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    update to revision 13 with syslog notification upon load and more streamlined script
     
  15. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Revision 14 is up with ipv6 fixes simpler detection
     
  16. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Works flawlessly :)
     
    swetoast likes this.
  17. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Rev 15 is up :)
     
  18. mikelees2

    mikelees2 Regular Contributor

    Joined:
    Feb 20, 2017
    Messages:
    55
    Location:
    U.S.A.
    Where is it located?
     
  19. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    opening post, will add to wiki also
     
    lesandie likes this.
  20. spanjap

    spanjap New Around Here

    Joined:
    Apr 26, 2016
    Messages:
    4
    I had the same problem as bayern1975 that the blocking didn't seem to work.

    I got this information from the following page:
    https://github.com/RMerl/asuswrt-merlin/wiki/Using-ipset


    Note that every time you do something on the web UI or through your [android app] (https://play.google.com/store/apps/details?id=com.asus.aihome) to control your router that affects reloading the firewall rules, /jffs/scripts/firewall-start will be called, so the iptables rules that are defined outside will be wiped out. To reinstate the rules as defined by this script, you'd need to add this to your existing /jffs/scripts/firewall-start:

    Code:
    # Reinstate the ipset rules if they have been created already
    [ "$(uname -m)" = "mips" ] && MATCH_SET='--set' || MATCH_SET='--match-set'
    for ipSet in $(ipset -L | sed -n '/^Name:/s/^.* //p'); do
      case $ipSet in
        AcceptList) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j ACCEPT;;
        BruteForceLogins|TorNodes|BlockedCountries|CustomBlock) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j DROP;;
        MicrosoftSpyServers) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet dst -j DROP;;
        *) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;
      esac
    done


    I also use this script and I changed the MicroSpyServers line to :
    privacy-filter_ipv4) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;
    I could use this line because it is already in the privacy-filter so I deleted it in the IPSET list from above


    Code:
    # Reinitiate the ipset rules if they have been created already
    [ "$(uname -m)" = "mips" ] && MATCH_SET='--set' || MATCH_SET='--match-set'
    for ipSet in $(ipset -L | sed -n '/^Name:/s/^.* //p'); do
      case $ipSet in
        AcceptList) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j ACCEPT;;
        BruteForceLogins|TorNodes|BlockedCountries|CustomBlock) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j DROP;;
        privacy-filter_ipv4) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;
        *) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;
      esac
    done
    Now the privacy-filter_ipv4 always works.

    I hope this helps and is correct.
     
    Last edited: Mar 25, 2017

Share This Page