Privacy Filter (Another IPSET Script)

[Coipu]

ahh ok sorry didn't realise had to wait, just started trying to ping sites from privacy list straight away. thank you for your help.

 

[swetoast]

for i in `cat $path/privacy-filter.list`; do traceroute $i | head -1 | grep -oE "$regexp" >> $path/privacy_block.pre; done
awk '!/^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/'  $path/privacy_block.pre > $path/privacy_block.list

think i made it safe

@tomsk mind reviewing ?

 

[swetoast]

test version this is without entware run it and check that it hasnt loaded any local lan ips afterwards with the following command

ipset -L privacy-filter | grep -oE '(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)'

that line should return nothing

 

[Martineau]

ipset -L privacy-filter | grep -oE '^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)'

Is it missing '(' ?

'(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)'

 

[swetoast]

yes it was! nice catch tnx

scripts are fixed above

 

[tomsk]

for i in `cat $path/privacy-filter.list`; do traceroute $i | head -1 | grep -oE "$regexp" >> $path/privacy_block.pre; done
awk '!/^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/'  $path/privacy_block.pre > $path/privacy_block.list

think i made it safe

@tomsk mind reviewing ?

According to ARIN private IP ranges are

• 10.0.0.0/8 IP addresses: 10.0.0.0 -- 10.255.255.255
• 172.16.0.0/12 IP addresses: 172.16.0.0 -- 172.31.255.255
• 192.168.0.0/16 IP addresses: 192.168.0.0 – 192.168.255.255

So with that and the 127 loopback, i think you got them all

Is it missing '(' ?

Nice one !

 

[swetoast]

cool ill update the wiki with both scripts

 

[swetoast]

@dandle latest revision is entware package is optional, so feel free to update since it doesnt require it anymore

 

[swetoast]

bump to revision 7

• cleanup less messy script

 

[swetoast]

so future plans here for privacy-filter i plan to support ipv6 pretty soon so that those with ipv6 can also benefit from this script although this can only benefit ipset 6.x users.

 

[tomsk]

so future plans here for privacy-filter i plan to support ipv6 pretty soon so that those with ipv6 can also benefit from this script although this can only benefit ipset 6.x users.

Nice plan...... local address info for IPv6 https://tools.ietf.org/search/rfc4193

Unofficial ULA register https://www.sixxs.net/tools/grh/ula/

 

[swetoast]

so i put it up on my personal gitlab

https://gitlab.com/swe_toast/privacy-filter/blob/master/privacy-filter

this is just for review not for actual usage since im unsure on how it would perform gonna need todo alot of testing.

USE AT OWN RISK

 

[tomsk]

so i put it up on my personal gitlab

https://gitlab.com/swe_toast/privacy-filter/blob/master/privacy-filter

this is just for review not for actual usage since im unsure on how it would perform gonna need todo alot of testing.

I gave it quick look over...saw a couple of small mistakes, but looks like it will work ok with those corrected.

You got the link-local IP and the loopback... dunno if you need to include the ULA as i posted just now.... wiki says

Local addresses[edit]

• ::1/128 — The loopback address is a unicast localhost address. If an application in a host sends packets to this address, the IPv6 stack will loop these packets back on the same virtual interface (corresponding to 127.0.0.1/8 in IPv4).
• fe80::/10 — Addresses in the link-local prefix are only valid and unique on a single link. Within this prefix only one subnet is allocated (54 zero bits), yielding an effective format of fe80::/64. The least significant 64 bits are usually chosen as the interface hardware address constructed in modified EUI-64 format. A link-local address is required on every IPv6-enabled interface—in other words, applications may rely on the existence of a link-local address even when there is no IPv6 routing. These addresses are comparable to the auto-configuration addresses 169.254.0.0/16 of IPv4.

Unique local addresses[edit]
Main article: Unique local address

• fc00::/7 — Unique local addresses (ULAs) are intended for local communication. They are routable only within a set of cooperating sites.[22] The block is split into two halves, the upper half (fd00::/8) is used for "probabilistically unique" addresses in which a 40-bit pseudorandom number is used to obtain a /48 allocation. This means that there is only a small chance that two sites that wish to merge or communicate with each other will have conflicting addresses. No allocation method for the lower half of the block (fc00::/8) is currently defined. These addresses are comparable to IPv4 private addresses (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16)

 

[swetoast]

so basically update to fc00:: in the regexp and voila the loopback is overrated since the regexp is never gonna match that anyways

 

[tomsk]

Line 76: awk $local_v4 $path/privacy-filter_v6.tmp > $path/privacy-filter_v6.pre; fi
Should Be:awk $local_v6 $path/privacy-filter_v6.tmp > $path/privacy-filter_v6.pre; fi

i see you got the nice -n 2 ipset -N privacy-update_ipv6 $HASH $INET6 already

 

[swetoast]

fixed tnx found a bunch of small errors with filenames and endings all fixed now.

ran an initial test and it was good for ipv4

 

[swetoast]

ipset -L privacy-filter_ipv6
Name: privacy-filter_ipv6
Type: hash:ip
Revision: 0
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 9080
References: 1
Members:
2400:cb00:2048:1::6810:335d
2606:2800:133:206e:1315:22a5:2006:24fd
2400:cb00:2048:1::6810:345d
2400:cb00:2048:1::6810:355d
2606:2800:234:2294:b59:11e7:560:10db
2a03:2880:f00a:2:face:b00c:0:8c
2a01:111:f30e:1790::f001:dcc1
2604:a880:800:10::99:7001
2400:cb00:2048:1::6810:315d
2a01:111:f303:1791::b01
2a00:1450:400f:808::2006
2400:cb00:2048:1::6810:325d
2a01:111:f330:1790::a01

 

[swetoast]

Revision 8 is up

changelog:

• massive rewrite
• ipv6 support (tnx tomsk for reviewing)

 

[Wolfclaw]

Nice one, you should team up with thelonelycoder and merge it with ABS3 until we get a one stop privacy package.

 

[Routy]

Hi, I am new to custom firmware on router (just flashed asuswrt-merlin on my rt-ac68). A major reason I do this is because I want to do adblock on my router, so I'm intrigued by this privacy filter script and I have an elementary question.

I've looked into AB-solution and uBlockr, and both scripts seem to support installation of custom adblock lists. Currently I use ublock origin on my laptop, and there I can subscribe to several privacy filter lists. If I add such lists to AB-solution or uBlockr, would I obtain the desired privacy filtering? Put it in another way, what additional/alternative features/functions this privacy filter provides that are not available via AB-solution or uBlockr?

This is a newbie question and not a criticism of your work. MANY THANKS for your help!