Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Privacy Filter (Another IPSET Script)

Discussion in 'Asuswrt-Merlin' started by swetoast, Jan 11, 2017.

  1. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    485
    Cleanup is a bit dangerous if the user doesn't know what he/she is doing when setting the path. If the path isn't unique the cleanup will wipeout everything in the directory
     
  2. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    yeah it can be but im hoping that noone is stupid enough to set it in a dir with other stuff thats important, im banking on that users set the filter to its own dir.
     
  3. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    @tomsk my plan is to do everything in tmp in the future cause im tired of getting the opt question so it would be safe to create dir in tmp where it wipes everytime, agree with that approach ?
     
  4. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    485
    well temp will only wipe on restart.. but you might want to consider a small modification to your cleanup which would be safer
    Code:
    cleanup () {
    find $path -type f -name partialnameforyourtempfiles* -delete
    }
    And then call all your temp files partialnameforyourtempfilesXXX.XXX
     
    Last edited: Feb 16, 2017
  5. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    fixed revision 11 in OP did it Tomsk way instead for cleanup
     
  6. Goobi

    Goobi Regular Contributor

    Joined:
    Dec 3, 2015
    Messages:
    53
    I went ahead and updated the script to use revision 11... I can't recall what revision I was using previously. I am running 380.64 on a 68U. When I attempt to run the new script, I get the following errors:

    Code:
    # ./privacy-filter.sh
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [[name does not existname does not exist]
    ]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [[name does not existname does not exist]
    ]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    [name does not exist]
    find: unrecognized: -type
    BusyBox v1.20.2 (2016-12-16 12:24:33 EST) multi-call binary.
    
    Usage: find [PATH]... [OPTIONS] [ACTIONS]
    
    Search for files and perform actions on them.
    First failed action stops processing of current file.
    Defaults: PATH is current directory, action is '-print'
    
            -follow         Follow symlinks
    
    Actions:
            ! ACT           Invert ACT's success/failure
            ACT1 [-a] ACT2  If ACT1 fails, stop, else do ACT2
            ACT1 -o ACT2    If ACT1 succeeds, stop, else do ACT2
                            Note: -a has higher priority than -o
            -name PATTERN   Match file name (w/o directory name) to PATTERN
            -iname PATTERN  Case insensitive -name
            -mtime DAYS     mtime is greater than (+N), less than (-N),
                            or exactly N days in the past
    If none of the following actions is specified, -print is assumed
            -print          Print file name
            -print0         Print file name, NUL terminated
            -exec CMD ARG ; Run CMD with all instances of {} replaced by
                            file name. Fails if CMD exits with nonzero
    If I check to see if the ipset got created I only see 1 IP:

    # ipset -L privacy-filter_ipv4
    Name: privacy-filter_ipv4
    Type: hash:ip
    Revision: 0
    Header: family inet hashsize 1024 maxelem 65536
    Size in memory: 8264
    References: 1
    Members:
    104.131.0.69
    **************************************************************************

    I have included the script in the event I may have missed something.


    Any pointers in the right direction would be greatly appreciated. Thanks in advance.
     

    Attached Files:

  7. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    using entware ? think i need to do a TLDR on what to do if it not working that way i dont have to review all the code.
     
  8. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    485
    The [name does not exist] entries mean that hostip is looking up the domain and not finding a match. You could try deleting your existing privacy-filter.list in case it has become corrupted and let the script download the list again. For info the entware version of hostip has been updated (hostip - 1.7.0-1 - 1.9.1-1), so you might want to try the newer version. if hostip is not resolving the domains, then it won't create a list of IP addresses to put in an ipset.

    For the "find: unrecognized: -type" problem, that may be due to the compiled version of busybox in that firmware... will have to check.
     
    Last edited: Feb 24, 2017
    swetoast likes this.
  9. Goobi

    Goobi Regular Contributor

    Joined:
    Dec 3, 2015
    Messages:
    53
    I do not have entware installed.
     
  10. Goobi

    Goobi Regular Contributor

    Joined:
    Dec 3, 2015
    Messages:
    53
    I am running version 1.6.0 of hostip but not running entware.

    I removed the privacy.list file and the script downloaded it again but produced the same errors. I then had a look at the privacy.list file and noted a bunch of ^M at the end of each line. I removed those and the script ran fine with the exception of the busybox error.

    Thanks for pointing me in the right direction. @swetoast thanks for the script as well!
     
  11. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    136
    The "^M"s are embedded carriage returns injected by editing on a non-UNIX (e.g., Windows) host. You can use the dos2unix command on a UNIX host to fix the file.
     
  12. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Revision 12

    there might be some breakage but feel free to test and see if it works for you :)

    Code:
    #!/bin/sh
    # Author: Toast
    # Contributers: Tomsk
    # Revision 12
    
    blocklist=/jffs/privacy-filter.list                     # Set your path here
    retries=3                                               # Set number of tries here
    failover=eth0                                           # Change only if WAN interface is not detected.
    
    # Dont change this value
    regexp_v4=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`
    local_v4=`echo "!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/"`
    regexp_v6=`echo "^(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4})"`
    local_v6=`echo "!(^(fc00::)"`
    # Dont change this value
    
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) # Value for ARM Routers
       MATCH_SET='--match-set'
       HASH='hash:ip'
       SYNTAX='add'
       SWAPPED='swap'
       DESTROYED='destroy'
       INET6='family inet6'
       ipsetv=6
        lsmod | grep "xt_set" > /dev/null 2>&1 || \
        for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
        do
             insmod $module
        done
    ;;
    *v4) # Value for Mips Routers
       MATCH_SET='--set'
       HASH='iphash'
       SYNTAX='-q -A'
       SWAPPED='-W'
       DESTROYED='--destroy'
       IPV6=''
        ipsetv=4
        lsmod | grep "ipt_set" > /dev/null 2>&1 || \
        for module in ip_set ip_set_nethash ip_set_iphash ipt_set
        do
             insmod $module
        done
    ;;
    esac
    
    check_online () {
    if [ -z "$(which nvram)" ]; then
    iface=`grep "$failover" /proc/net/dev`
    if   [ -n "$iface" ]; then
         if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
         then get_list; fi
         else exit 1; fi
    else iface=`nvram get wan0_ifname`
    if   [ -n "$iface" ]; then
         if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
         then get_list; fi
         else exit 1; fi
    fi }
    
    get_list () {
    url=https://gitlab.com/swe_toast/privacy-filter/raw/master/privacy-filter.list
    if [ ! -f $blocklist ]
    then wget -q --tries=$retries --show-progress $url -O $blocklist; fi }
    
    run_ipv4_block () {
    if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                     cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
            fi
            
        if [ -f /tmp/privacy-filter_ipv4_presort.part ]; then
            awk $local_v4 /tmp/privacy-filter_ipv4_presort.part > /tmp/privacy-filter_ipv4.prelist; fi
            if [ -f /tmp/privacy-filter_ipv4.prelist ]; then sort -u /tmp/privacy-filter_ipv4.prelist > /tmp/privacy-filter_ipv4_sorted.part; fi
    }
            
    run_ipv6_block () {
    if [ -f /tmp/privacy-filter_ipv6_sorted.part ]; then rm /tmp/privacy-filter_ipv6_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""; fi
                     cat /tmp/privacy-filter_ipv6_raw.part | grep -oE "$regexp_v6" >> /tmp/privacy-filter_ipv6_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""; fi
            fi
            
        if [ -f /tmp/privacy-filter_ipv6_presort.part ]; then
            awk $local_v6 /tmp/privacy-filter_ipv6_presort.part > /tmp/privacy-filter_ipv6.prelist; fi
            if [ -f /tmp/privacy-filter_ipv6.prelist ]; then sort -u /tmp/privacy-filter_ipv6.prelist > /tmp/privacy-filter_ipv6_sorted.part; fi
    }
            
    run_ipset_4 () {
    ipset -L privacy-filter_ipv4 >/dev/null 2>&1
    if [ $? -ne 0 ]; then
       if [ "$(ipset --swap privacy-filter_ipv4 privacy-filter_ipv4 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
       nice ipset -N privacy-filter_ipv4 $HASH
       cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv4 {}
    fi
    else
       nice -n 2 ipset -N privacy-update_ipv4 $HASH
       cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv4 {}
       nice -n 2 ipset $SWAPPED privacy-update_ipv4 privacy-filter_ipv4
       nice -n 2 ipset $DESTROYED privacy-update_ipv4
    fi
    iptables -L | grep privacy-filter_ipv4 > /dev/null 2>&1
    if [ $? -ne 0 ]; then
       nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
    else
       nice -n 2 iptables -D FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
       nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
    fi }
    
    run_ipset_6 () {
    ipset -L privacy-filter_ipv6 >/dev/null 2>&1
    if [ $? -ne 0 ]; then
       if [ "$(ipset --swap privacy-filter_ipv6 privacy-filter_ipv6 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
       nice ipset -N privacy-filter_ipv6 $HASH $INET6
       cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv6 {}
    fi
    else
       nice -n 2 ipset -N privacy-update_ipv6 $HASH $INET6
       cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv6 {}
       nice -n 2 ipset $SWAPPED privacy-update_ipv6 privacy-filter_ipv6
       nice -n 2 ipset $DESTROYED privacy-update_ipv6
    fi
    iptables -L | grep privacy-filter_ipv6 > /dev/null 2>&1
    if [ $? -ne 0 ]; then
       nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
    else
       nice -n 2 ip6tables -D FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
       nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
    fi }
    
    run_blocklists () {
    run_ipv4_block
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipv6_block; fi ;;
    esac }
    
    run_ipset () {
    run_ipset_4
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipset_6; fi  ;;
    esac }
    
    cleanup () {
    find /tmp -type f -name 'privacy-filter_ipv*.part' -delete
    }
    
    check_online
    run_blocklists
    run_ipset
    cleanup
    
    exit $?
     
  13. wallyg8r

    wallyg8r Occasional Visitor

    Joined:
    Feb 10, 2017
    Messages:
    19
    Revision 12 is working ok for me as was Revision 11.
     
  14. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Thinking of adding dos2unix to the script since i noticed that people copy pastes the list in windows then uploadings it via SFTP thus giving the list windows endings instead of unix endings a simple command can fix that and hopefully save me of a headache so thats something thats upcoming in the next version, anyone else got any suggestions ?
     
  15. lesandie

    lesandie Occasional Visitor

    Joined:
    Jan 9, 2015
    Messages:
    15
    Hi,

    I'm in the
    374.43_2-22E4j9527 fork release.

    Still having problems with the entware hostip (1.9.1) and [name does not exist] output.

    I'll try to debug and change the code if i find what's wrong.

    Nice work guys! :)
     
  16. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    Name does not exist can happen sometimes i you have adblocker or the domain doesnt have an ip, no need for debuging there is nothing wrong.
     
  17. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    4,818
    Location:
    United States
    You may be running into a conflict.....since I include native dnscrypt support, hostip is also installed as part of the base firmware (in /usr/sbin).
     
  18. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    But @john9527 the dns server should still resolve the ip that should still work as intended on your fork right ?
     
  19. john9527

    john9527 Part of the Furniture

    Joined:
    Mar 28, 2014
    Messages:
    4,818
    Location:
    United States
    Yes, unless the hostip that's included with dnscrypt has a different syntax from the entware version (probably not) or there is a mismatch somehow in any linked libraries.
     
  20. swetoast

    swetoast Guest

    Joined:
    Apr 12, 2016
    Messages:
    804
    mind just giving me a print out of hostip from your distro ?

    hostip --help
     

Share This Page