Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Privacy Filter (Another IPSET Script)

Discussion in 'Asuswrt-Merlin' started by swetoast, Jan 11, 2017.

  1. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Hi Everyone, I'm back with another IPSET for your firewall this time around its for blocking Telemetry and some Android Rootkit along with Shodan.io Scanners. There are two ways to go about it either store the script in firewall_start or in services_start as a cronjob, for more information about this consult the wiki about the information.

    Note: Dont worry if there are messages like if you have Entware installed.
    • [no records in the reply]
    • [server failed]
    • [name does not exist]
    These are just domains that either have been deactivated or that AB-Solutions or uBlockr is redirecting traffic towards its own domain. It also blocks ipv6 traffic if an ip6 address exists but only if the router is running ipset version 6.x.

    Too see if the firewall rule is working use the following command:
    Code:
    watch iptables -vnL FORWARD
    should read give an output
    Code:
    Chain FORWARD (policy DROP 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination
      856 40504 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set privacy-filter_ipv4 src,dst reject-with icmp-port-unreachable
    Code:
    #!/bin/sh
    # Author: Toast
    # Contributers: Tomsk
    # Supporters: lesandie
    # Revision 15
    
    blocklist=/jffs/privacy-filter.list                     # Set your path here 
    retries=3                                               # Set number of tries here
    fwoption=REJECT                                         # DROP/REJECT    (Default Value: REJECT)
    
    # Dont change this value
    regexp_v4=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`
    local_v4=`echo "!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/"`
    regexp_v6=`echo "^(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4})"`
    local_v6=`echo "!(^(fc00::)"`
    # Dont change this value
    
    case $(ipset -v | grep -o "v[4,6]") in
      v6)
        MATCH_SET='--match-set'; CREATE='create'; ADD='add'; SWAP='swap'; IPHASH='hash:ip'; NETHASH='hash:net family inet'; FLUSH='flush'; DESTROY='destroy'; INET6='family inet6';
        lsmod | grep -q "xt_set" || \
        for module in ip_set ip_set_nethash ip_set_iphash xt_set; do
          insmod $module
        done;;
      v4)
        MATCH_SET='--set'; CREATE='--create'; ADD='--add'; SWAP='--swap'; IPHASH='iphash'; NETHASH='nethash'; FLUSH='--flush'; DESTROY='--destroy' INET6=''
        lsmod | grep -q "ipt_set" || \
        for module in ip_set ip_set_nethash ip_set_iphash ipt_set; do
          insmod $module
        done;;
      *) echo "unsupported version"; exit 1 ;;
    esac
    
    check_online () {
    while ! ping -q -c 1 google.com >/dev/null 2>&1; do
      sleep 1
      WaitSeconds=$((WaitSeconds+1))
      [ $WaitSeconds -gt 300 ] && logger -t system "$0: Warning: Router not online! Aborting after a wait of 5 minutes..." && exit 1
    done
    }
    
    check_ipv6 () {
      ping6 -q -c 1 google.com >/dev/null 2>&1 && $1 || echo
    }
    
    get_list () {
    url=https://gitlab.com/swe_toast/privacy-filter/raw/master/privacy-filter.list
    if [ ! -f $blocklist ]
    then wget -q --tries=$retries --show-progress $url -O $blocklist; fi }
    
    fix_list () {
    if [ -f $blocklist ]
    then dos2unix $blocklist; fi
    }
    
    run_ipv4_block () {
    if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                     cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
            fi
           
        if [ -f /tmp/privacy-filter_ipv4_presort.part ]; then
            awk $local_v4 /tmp/privacy-filter_ipv4_presort.part > /tmp/privacy-filter_ipv4.prelist; fi
            if [ -f /tmp/privacy-filter_ipv4.prelist ]; then sort -u /tmp/privacy-filter_ipv4.prelist > /tmp/privacy-filter_ipv4_sorted.part; fi
    }
           
    run_ipv6_block () {
    if [ -f /tmp/privacy-filter_ipv6_sorted.part ]; then rm /tmp/privacy-filter_ipv6_sorted.part; fi
        if [ -z "$(which hostip)" ]; then
            if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""; fi
                     cat /tmp/privacy-filter_ipv6_raw.part | grep -oE "$regexp_v6" >> /tmp/privacy-filter_ipv6_presort.part
    else    if [ -z "$(which /opt/bin/xargs)" ]
                then cat $blocklist | xargs -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""
                else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""; fi
            fi
           
        if [ -f /tmp/privacy-filter_ipv6_presort.part ]; then
            awk $local_v6 /tmp/privacy-filter_ipv6_presort.part > /tmp/privacy-filter_ipv6.prelist; fi
            if [ -f /tmp/privacy-filter_ipv6.prelist ]; then sort -u /tmp/privacy-filter_ipv6.prelist > /tmp/privacy-filter_ipv6_sorted.part; fi
    }
           
    run_ipset_4 () {
    ipset -L privacy-filter_ipv4 >/dev/null 2>&1
    if [ $? -ne 0 ]; then
       if [ "$(ipset $SWAP privacy-filter_ipv4 privacy-filter_ipv4 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
       nice ipset $CREATE privacy-filter_ipv4 $IPHASH
       cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $ADD privacy-filter_ipv4 {}
    fi
    else
       nice -n 2 ipset $CREATE privacy-update_ipv4 $IPHASH
       cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $ADD privacy-update_ipv4 {}
       nice -n 2 ipset $SWAP privacy-update_ipv4 privacy-filter_ipv4
       nice -n 2 ipset $DESTROY privacy-update_ipv4
    fi
    iptables -L | grep privacy-filter_ipv4 > /dev/null 2>&1
    if [ $? -ne 0 ]; then
       nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j $fwoption
    else
       nice -n 2 iptables -D FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j $fwoption
       nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j $fwoption
    fi }
    
    run_ipset_6 () {
    ipset -L privacy-filter_ipv6 >/dev/null 2>&1
    if [ $? -ne 0 ]; then
       if [ "$(ipset $SWAP privacy-filter_ipv6 privacy-filter_ipv6 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
       nice ipset $CREATE privacy-filter_ipv6 $IPHASH $INET6
       cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $ADD privacy-filter_ipv6 {}
    fi
    else
       nice -n 2 ipset -N privacy-update_ipv6 $IPHASH $INET6
       cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $ADD privacy-update_ipv6 {}
       nice -n 2 ipset $SWAP privacy-update_ipv6 privacy-filter_ipv6
       nice -n 2 ipset $DESTROY privacy-update_ipv6
    fi
    iptables -L | grep privacy-filter_ipv6 > /dev/null 2>&1
    if [ $? -ne 0 ]; then
       nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j $fwoption
    else
       nice -n 2 ip6tables -D FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j $fwoption
       nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j $fwoption
    fi }
    
    run_blocklists () {
    run_ipv4_block
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) check_ipv6 run_ipv6_block ;;
    esac }
    
    run_ipset () {
    run_ipset_4
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) check_ipv6 run_ipset_6 ;;
    esac }
    
    logipv6 () {
    logger -s -t system "Privacy Filter (ipv6) loaded $(ipset -L  privacy-filter_ipv6 | wc -l | awk '{print $1-7}') unique ip addresses."
    }
    
    cleanup () {
    find /tmp -name 'privacy-filter_ipv*.part' -exec rm {} +
    logger -s -t system "Privacy Filter (ipv4) loaded $(ipset -L  privacy-filter_ipv4 | wc -l | awk '{print $1-7}') unique ip addresses."
    check_ipv6 logipv6
    }
    
    check_online
    fix_list
    run_blocklists
    run_ipset
    cleanup
    
    exit $?
    
    Note: save this list as privacy-filter.list in your path on the router, if you set this file in the wrong place the script will automatically download a new copy and set it at either path or failover path.
     
    Last edited: Mar 20, 2017 at 3:19 PM
  2. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    470
    I don't quite get this line ..... maybe I'm reading it wrong. For me it says
    " if the dnsmasq.conf.add file doesn't exist OR privacy-filter doesn't turn up in a search of dnsmasq.conf.add" then... the next bit. Is that what is meant to test...if so isn't it going to be true all the time?
     
  3. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    it checks if the file is there then it adds to the file thats basically what it does :) then it adds the content to the file and sets it to 127.0.0.1
     
  4. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    470
    I dont understand why you do the file check and in the next line delete it anyway.
    Then you create it again when you append all the server entries.
    Code:
    rm -f $dnsmasq_cfg
        for i in `cat $path/privacy-filter.list`;
    do
        echo "server=/$i/127.0.0.1#1919" >> $dnsmasq_cfg
    done
    Dont get me wrong...not being critical at all...just trying to understand the code.. told you i was dumb :)
     
  5. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    it might be tricky but im sure you will figure it out :)
     
  6. Nutz2U2

    Nutz2U2 Occasional Visitor

    Joined:
    Nov 5, 2016
    Messages:
    14
    Hi to Everyone,

    Wish to thank ALL the boys and girls who made this script workable.

    Works like a charm ;).
     
    swetoast likes this.
  7. Xentrk

    Xentrk Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    215
    Location:
    Chiang Mai, Thailand
    Hi @swetoast,

    For preliminary testing purposes, I copied your script to a file called privacy-filter. When running it, I get two error messages:

    Code:
    [email protected]:/jffs/scripts# ./privacy-filter
    cat: can't open '/opt/var/cache/privacy-filter/privacy-filter.list': No such file or directory
    
    Done.
    cat: can't open '/opt/var/cache/privacy-filter/privacy-filter.txt': No such file or directory
    If I create the privacy-filter directory and privacy.list and .txt files, the error message goes away. But I have no values in the .list or .txt file. A dnsmasq.conf.add file does not get created. I need your help in getting this to work. Thanks again for your help!

    EDIT:
    Code:
    lsmod | grep "xt_set"
    xt_set                  2964  4
    ip_set                 18113  3 xt_set,ip_set_hash_ip,ip_set_hash_net
    The Done. response appears to come from this command:
    Code:
    service restart_dnsmasq
     
    Last edited: Jan 13, 2017
  8. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    114
    Post #1 shows you how to initially populate the privacy.list and .txt files.
     
  9. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    470
    The dnsmasq.conf.add file gets deleted every time you run the script and it only gets recreated by the cat command if there is actually something to append.
    Code:
    rm -f $dnsmasq_cfg
        for i in `cat $path/privacy-filter.list`;
    do
        echo "server=/$i/127.0.0.1#1919" >> $dnsmasq_cfg
    done
    as @visortgw mentions, the privacy-filter.list file will have to be populated with at least one domain.
     
  10. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    yes so that the firmware knows that its supposed to add it to then config, thats why.
     
  11. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    470
    I know I'm not the sharpest tool in the shed when it comes to code...but this bit of the script just doesn't make sense to me.
    The dnsmasq.conf.add file will be checked for the string "privacy-filter" with grep and the statement will be true if nothing is found. The dnsmasq.conf.add file should contain a load of lines with server=/some.host.com/127.0.0.1#1919 type entries..... there will never be a time where grep will find "privacy-filter" so it will always be true...... what am i missing? is "" not interpreted in the way i think?
     
  12. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    yepp thats a typo gonna fix it that should read "server=.*" then it should be -e for regexp still its not a major flaw :p sloppy on my part hence why i have the community to correct me :D

    been awhile since i wrote that script just updated with my ipset checking system think my plan was to have a commented line in the dnsmasq.add file
     
  13. Xentrk

    Xentrk Senior Member

    Joined:
    Jul 21, 2016
    Messages:
    215
    Location:
    Chiang Mai, Thailand
    Doh! I re-read it and see I have to create the files and save it with the data as you state. For some reason, I thought it was similar to the malware-filter script and was going to populate those files for me.

    Many Thanks! Sending you my gratitude from the Land of Smiles.
     
  14. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    yeah this one doesn't populate on its own
     
  15. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    470
    would it make sense to check for each entry in the dnsmasq.conf.add file and only append it if it doesn't exist... would save you having to delete and recreate it every time to avoid duplicates.. perhaps throw something like this in your "for i in" loop
    Code:
    [ "$(grep $i $dnsmasq_cfg)" = "" ]
    Would make it safer if some other script had written a dnsmasq.conf.add too (play nice with others policy)
     
    Last edited: Jan 14, 2017
  16. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Revision 2 posted in opening post

    Changes are proper detection if privacy filter is already loaded in dnsmasq

    So what it does now is to check if there is an add file then it checks if there is a server= present in the real dnsmasq if there isnt then it adds privacy filter to the dnsmasq file.

    Edit: seems i found a bug with grep with this revision
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=617183

    also getting this message so ill see if there is another way to do it.
     
    Last edited: Jan 26, 2017
  17. martinr

    martinr Very Senior Member

    Joined:
    Nov 27, 2014
    Messages:
    792
    Location:
    United Kingdom
    You seem pretty sharp to me.
     
  18. thelonelycoder

    thelonelycoder Very Senior Member

    Joined:
    Jan 23, 2014
    Messages:
    1,952
    Location:
    Confœderatio Helvetica
    And persistent, always gets what he wants...
     
    visortgw likes this.
  19. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    Well i dont mind tomsk input, keeps me on my game :)
     
    Last edited: Jan 29, 2017
  20. swetoast

    swetoast Very Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    512
    bump to revision 3 to get around the grep bug added a bell sign to dnsmasq with the title #privacy-filter that the script checks for then adds if its not added
     

Share This Page