Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Privacy Filter (Another IPSET Script)

Discussion in 'Asuswrt-Merlin' started by swetoast, Jan 11, 2017.

  1. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    Hey ho im back with another version of the telemetry script posted on the wiki but in my flavor rather then the old script thats a bit broken well enough preamble stuff

    So this script tries to block telemetry and some additional google server and some chinese data collection centers for android rootkits

    Code:
    #!/bin/sh
    # Original script by swetoast
    # Revision 1
    
    path=/opt/var/cache/privacy-filter                      # Set your path here
    dnsmasq_cfg=/jffs/configs/dnsmasq.conf.add
    regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value
    
    if [ ! -f $dnsmasq_cfg ] || [ "$(grep privacy-filter $dnsmasq_cfg)" = "" ];
    then
        rm -f $dnsmasq_cfg
        for i in `cat $path/privacy-filter.list`;
    do
        echo "server=/$i/127.0.0.1#1919" >> $dnsmasq_cfg
    done
        service restart_dnsmasq
    fi
    
    case $(ipset -v | grep -oE "ipset v[0-9]") in
    *v6) # Value for ARM Routers
    
        MATCH_SET='--match-set'
        HASH='hash:ip'
        SYNTAX='add'
        SWAPPED='swap'
        DESTROYED='destroy'
        OPTIONAL='family inet hashsize 2048 maxelem 65536'
    
         ipsetv=6
         lsmod | grep "xt_set" > /dev/null 2>&1 || \
         for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
         do
              insmod $module
         done
    ;;
    
    *v4) # Value for Mips Routers
    
        MATCH_SET='--set'
        HASH='iphash'
        SYNTAX='-q -A'
        SWAPPED='-W'
        DESTROYED='--destroy'
        OPTIONAL=''
    
         ipsetv=4
         lsmod | grep "ipt_set" > /dev/null 2>&1 || \
         for module in ip_set ip_set_nethash ip_set_iphash ipt_set
         do
              insmod $module
         done
    ;;
    esac
    
    
    run_ipset () {
    
    ipset -L privacy-filter >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        if [ "$(ipset --swap privacy-filter privacy-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
        nice ipset -N privacy-filter $HASH $OPTIONAL
        for i in `cat $path/privacy-filter.txt`; do nice -n 2 ipset $SYNTAX privacy-filter $i ; done
    fi
    else
        nice -n 2 ipset -N privacy-update $HASH $OPTIONAL
        for i in `cat $path/privacy-filter.txt`; do nice -n 2 ipset $SYNTAX privacy-update $i ; done
        nice -n 2 ipset $SWAPPED privacy-update privacy-filter
        nice -n 2 ipset $DESTROYED privacy-update
    fi
    
    iptables -L | grep privacy-filter > /dev/null 2>&1
    if [ $? -ne 0 ]; then
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter src,dst -j REJECT
    else
        nice -n 2 iptables -D FORWARD -m set $MATCH_SET privacy-filter src,dst -j REJECT
        nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter src,dst -j REJECT
    fi
    }
    
    run_ipset
    exit $?
    and here is the iplist of the telemertry server taken of the original script save as privacy-filter.txt in your path

    Code:
    23.99.10.11
    63.85.36.35
    63.85.36.50
    64.4.6.100
    64.4.54.22
    64.4.54.32
    64.4.54.254
    65.52.100.7
    65.52.100.9
    65.52.100.11
    65.52.100.91
    65.52.100.92
    65.52.100.93
    65.52.100.94
    65.55.29.238
    65.55.39.10
    65.55.44.108
    65.55.163.222
    65.55.252.43
    65.55.252.63
    65.55.252.71
    65.55.252.92
    65.55.252.93
    66.119.144.157
    93.184.215.200
    104.76.146.123
    111.221.29.177
    131.107.113.238
    131.253.40.37
    134.170.52.151
    134.170.58.190
    134.170.115.60
    134.170.115.62
    134.170.188.248
    157.55.129.21
    157.55.133.204
    157.56.91.77
    168.62.187.13
    191.234.72.183
    191.234.72.186
    191.234.72.188
    191.234.72.190
    204.79.197.200
    207.46.223.94
    207.68.166.254
    and last but not least the list save as privacy-filter.list in your path

    Code:
    googleadservices.com
    www.google-analytics.com
    google-analytics.com
    ssl.google-analytics.com
    secure.adnxs.com
    secure.flashtalking.com
    services.wes.df.telemetry.microsoft.com
    settings-sandbox.data.microsoft.com
    settings-win.data.microsoft.com
    sls.update.microsoft.com.akadns.net
    sqm.df.telemetry.microsoft.com
    sqm.telemetry.microsoft.com
    sqm.telemetry.microsoft.com.nsatc.net
    static.2mdn.net
    statsfe1.ws.microsoft.com
    statsfe2.update.microsoft.com.akadns.net
    statsfe2.ws.microsoft.com
    survey.watson.microsoft.com
    telecommand.telemetry.microsoft.com
    telecommand.telemetry.microsoft.com.nsatc.net
    telemetry.appex.bing.net
    telemetry.microsoft.com
    telemetry.urs.microsoft.com
    view.atdmt.com
    vortex.data.microsoft.com
    vortex-bn2.metron.live.com.nsatc.net
    vortex-cy2.metron.live.com.nsatc.net
    vortex-sandbox.data.microsoft.com
    vortex-win.data.microsoft.com
    watson.live.com
    watson.microsoft.com
    watson.ppe.telemetry.microsoft.com
    watson.telemetry.microsoft.com
    watson.telemetry.microsoft.com.nsatc.net
    wes.df.telemetry.microsoft.com
    www.msftncsi.com
    nametests.com
    oyag.lhzbdvm.com
    oyag.prugskh.net
    oyag.prugskh.com
    again this is not a fool proof solution its an additional layer of security with that said enjoy

    ps. this one is not as big as my other script malware-filter so this can be stored in firewall-start
     
    Last edited: Jan 12, 2017
    thelonelycoder likes this.
  2. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    306
    I don't quite get this line ..... maybe I'm reading it wrong. For me it says
    " if the dnsmasq.conf.add file doesn't exist OR privacy-filter doesn't turn up in a search of dnsmasq.conf.add" then... the next bit. Is that what is meant to test...if so isn't it going to be true all the time?
     
  3. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    it checks if the file is there then it adds to the file thats basically what it does :) then it adds the content to the file and sets it to 127.0.0.1
     
  4. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    306
    I dont understand why you do the file check and in the next line delete it anyway.
    Then you create it again when you append all the server entries.
    Code:
    rm -f $dnsmasq_cfg
        for i in `cat $path/privacy-filter.list`;
    do
        echo "server=/$i/127.0.0.1#1919" >> $dnsmasq_cfg
    done
    Dont get me wrong...not being critical at all...just trying to understand the code.. told you i was dumb :)
     
  5. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    it might be tricky but im sure you will figure it out :)
     
  6. Nutz2U2

    Nutz2U2 Occasional Visitor

    Joined:
    Nov 5, 2016
    Messages:
    14
    Hi to Everyone,

    Wish to thank ALL the boys and girls who made this script workable.

    Works like a charm ;).
     
    swetoast likes this.
  7. Xentrk

    Xentrk Regular Contributor

    Joined:
    Jul 21, 2016
    Messages:
    131
    Location:
    Chiang Mai, Thailand
    Hi @swetoast,

    For preliminary testing purposes, I copied your script to a file called privacy-filter. When running it, I get two error messages:

    Code:
    [email protected]:/jffs/scripts# ./privacy-filter
    cat: can't open '/opt/var/cache/privacy-filter/privacy-filter.list': No such file or directory
    
    Done.
    cat: can't open '/opt/var/cache/privacy-filter/privacy-filter.txt': No such file or directory
    If I create the privacy-filter directory and privacy.list and .txt files, the error message goes away. But I have no values in the .list or .txt file. A dnsmasq.conf.add file does not get created. I need your help in getting this to work. Thanks again for your help!

    EDIT:
    Code:
    lsmod | grep "xt_set"
    xt_set                  2964  4
    ip_set                 18113  3 xt_set,ip_set_hash_ip,ip_set_hash_net
    The Done. response appears to come from this command:
    Code:
    service restart_dnsmasq
     
    Last edited: Jan 13, 2017
  8. visortgw

    visortgw Regular Contributor

    Joined:
    Jun 18, 2015
    Messages:
    72
    Post #1 shows you how to initially populate the privacy.list and .txt files.
     
  9. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    306
    The dnsmasq.conf.add file gets deleted every time you run the script and it only gets recreated by the cat command if there is actually something to append.
    Code:
    rm -f $dnsmasq_cfg
        for i in `cat $path/privacy-filter.list`;
    do
        echo "server=/$i/127.0.0.1#1919" >> $dnsmasq_cfg
    done
    as @visortgw mentions, the privacy-filter.list file will have to be populated with at least one domain.
     
  10. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    yes so that the firmware knows that its supposed to add it to then config, thats why.
     
  11. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    306
    I know I'm not the sharpest tool in the shed when it comes to code...but this bit of the script just doesn't make sense to me.
    The dnsmasq.conf.add file will be checked for the string "privacy-filter" with grep and the statement will be true if nothing is found. The dnsmasq.conf.add file should contain a load of lines with server=/some.host.com/127.0.0.1#1919 type entries..... there will never be a time where grep will find "privacy-filter" so it will always be true...... what am i missing? is "" not interpreted in the way i think?
     
  12. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    yepp thats a typo gonna fix it that should read "server=.*" then it should be -e for regexp still its not a major flaw :p sloppy on my part hence why i have the community to correct me :D

    been awhile since i wrote that script just updated with my ipset checking system think my plan was to have a commented line in the dnsmasq.add file
     
  13. Xentrk

    Xentrk Regular Contributor

    Joined:
    Jul 21, 2016
    Messages:
    131
    Location:
    Chiang Mai, Thailand
    Doh! I re-read it and see I have to create the files and save it with the data as you state. For some reason, I thought it was similar to the malware-filter script and was going to populate those files for me.

    Many Thanks! Sending you my gratitude from the Land of Smiles.
     
  14. swetoast

    swetoast Senior Member

    Joined:
    Apr 12, 2016
    Messages:
    252
    yeah this one doesn't populate on its own
     
  15. tomsk

    tomsk Senior Member

    Joined:
    Sep 3, 2016
    Messages:
    306
    would it make sense to check for each entry in the dnsmasq.conf.add file and only append it if it doesn't exist... would save you having to delete and recreate it every time to avoid duplicates.. perhaps throw something like this in your "for i in" loop
    Code:
    [ "$(grep $i $dnsmasq_cfg)" = "" ]
    Would make it safer if some other script had written a dnsmasq.conf.add too (play nice with others policy)
     
    Last edited: Jan 14, 2017

Share This Page