AdBlocking with combined hosts file with Lighttpd

[thelonelycoder]

I understand https blocking is an unresolved problem at the moment for the reason you stated. It is also bad practice as it serves mixed content. http and https on the same page ought to be blocked. Who am I to trust with https served ads?
Fck this. We'll get to that when things get worse.

One area I am looking into is to force the iPad to always show desktop pages. That involves dynamically rewriting URI / URL or UserAgent on the router for a set of IPs or MAC-Adresses. I hope to solve this as I see no reason to be served a mobile site an a device as the iPad. Some Apps allow you to set the UserAgent. However, this does not work with the in-App browser provided by Apple. I have made some progress with privoxy. If I get this to work this could also be a candidate for a better adblocking.

 

[Chrysalis]

yeah blocking mixed http/https is a way to deal with this, IE can do this, can firefox? cannot remember.

 

[prp2]

Easiest way to whitelist with this? I tried following these steps (http://www.smallnetbuilder.com/forums/showpost.php?p=118361&postcount=65), which is for the non-lighthttpd version, but it didn't seem to work correctly.

Thanks!

 

[thelonelycoder]

Easiest way to whitelist with this? I tried following these steps (http://www.smallnetbuilder.com/forums/showpost.php?p=118361&postcount=65), which is for the non-lighthttpd version, but it didn't seem to work correctly.

Thanks!

Can you post your scripts, I believe whitelisting should work with this version as well.

 

[baltosml]

I created a simple list of white listed sites (one per line), and added this after the line that creates the hosts.clean file:

cat /mnt/sda1/whitelist | while read line; do sed -i /${line}/d /mnt/sda1/hosts.clean ; done


---
baltosml@RT-N66U

Look at this regarding whitelisting - it has worked for me, added to the two scripts that create hosts.clean. It should take each whitelisted URL and remove that line from hosts.clean.


---
baltosml@RT-N66U

 

[Chrysalis]

still no solution to the insanely slow router https gui? it is very annoying but I want to keep the lighttpd running.

 

[thelonelycoder]

still no solution to the insanely slow router https gui?

With the latest Asuswrt-Merlin 376.44 it runs a lot better.

 

[Chrysalis]

well due to the wireless changes I wont be using 376.44, is it as fast in 376.43?

 

[thelonelycoder]

No, I have not checked it in 376.43.

 

[Chrysalis]

some additions

opkg install lighttpd-mod-redirect

is now required as the module is no longer in the base package.

the 'address=/0.0.0.0/0.0.0.0 in dnsmasq.conf is not only not required but actually breaks dnsmasq, dont add it.

rest of guide is fine tho.

also to confirm the https gui in 374_43 is snappy as I asked but didnt update.

 

[thelonelycoder]

the 'address=/0.0.0.0/0.0.0.0 in dnsmasq.conf is not only not required but actually breaks dnsmasq, dont add it.

OP says in the first post:
1. Login to the route via WinSCP > browse to /jffs/configs/ > right-click, edit dnsmasq.conf.add with changes:
remove this line as modifying it doesn't do anything:
Code:
address=/0.0.0.0/0.0.0.0

 

[Scaner]

Combined hosts for adblocking with lighttpd works correctly for me except one thing Google Chrome and Firefox ask for download "blank.gif" on blocked hosts instead opening it like Internet Explorer. Is there some solution (may be in config of lighttpd)?

 

[kvic]

Is there some solution (may be in config of lighttpd)?

I would search the forum for pixelserv..

 

[mstombs]

Thee are now Tomato pixelserv binaries on an easily accessed site

http://tomato-adblock.weebly.com/

This pixelserv is more selective about replies, only sends a gif if a gif requested etc, and tries to refuse an https request quickly

I don't know about other routers, but the small dynamic compiled one works on my asuswrt-merlin N66U

Most ads blocked now attempt to use https

/mnt/usb4gb/pixelserv version: V35.HZ12 compiled: Sep 13 2015 08:46:58 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -p 443 -o 2
311259 uts, 6797 req, 826 avg, 12486 rmx, 1512 tav, 3864 tmx, 0 err, 727 tmo, 43 cls, 0 nou, 0 pth, 263 nfe, 130 ufe, 160 gif, 0 bad, 1019 txt, 0 jpg, 11 png, 0 swf, 0 ico, 4214 ssl, 1 sta, 0 stt, 0 204, 229 rdr, 0 pst, 0 hed

 

[Scaner]

Thanks for advices, I`ll try to use pixelserv instead lighttpd.

 

[kvic]

pixelserv version: V35.HZ12 compiled: Sep 18 2015 18:54:52 options: 192.168.1.1
443298 uts, 28387 req, 489 avg, 3721 rmx, 605 tav, 10147 tmx, 0 err, 235 tmo, 113 cls, 0 nou, 0 pth, 2413 nfe, 1731 ufe, 272 gif, 25 bad, 7546 txt, 6 jpg,<p> 10 png, 5 swf, 4 ico, 13348 ssl, 0 sta, 7 stt, 0 204, 2574 rdr, 97 pst, 1 hed

In the past five days, a total of 28,387 ads were dutifully served and blocked. Of which 13,348 are https. That's about half of all my ad. Stunning!

@mstombs What are the additional benefits if pixelserv is enhanced to handle https requests properly?

 

[mstombs]

The c pixelserv will not be enhanced to talk https - you might be able to configure a real web browser as https, but you should not be able to convince a browser that it is the remote site requested - certificate signing etc - or this would be a massive security breach! Having written that I do remember getting a router login prompt up when using the default https port for router gui and using dns posoning to divert ad site to the router.

By the way the pixelserv ssl count is a bit inflated, browsers may try 2 or 3 times per ad, with different levels of security. It may be better to use iptables to 'REJECT' (not DROP), but what is best may be a function of your specific browser and web site.

 

[kvic]

It's quite easy with users' consent to trust an unsigned CA. I've done that with my PC browsers when I was still operating WebUI over https. Browsers will stop complaining I'm connecting to an insure web site. It's my router WebUI. I knew it's secure. I explicitly told my browsers once to trust it, and they have to remember what I said. lol.

Something similar can be done with pixelserv. One thing that I'm not certain is the benefit. As you rightly point out, we could use REJECT. If that serves the purpose well enough, little incentive to work on https for pixelserv.

Also it seems going back to square one. If REJECT works okay for https, why can't we do the same for http? Take another step back, why would we need a dummy adserver to begin with if redirecting to 0.0.0.0 works okay?

I haven't performed thorough tests in PC & mobile browsers to compare the difference between a dummy adserver and simply redirection to 0.0.0.0. Modern browsers on PC & mobile devices seem to cope with 0.0.0.0 well. I would love to hear feedbacks. That'll help to justify if we need to spend effort adding https into pixelerv..

 

[mstombs]

The original benefit of pixelserv was to supply a 1x1 gif in place of banner image which would allow the browser to collapse the whitespace rather than show a broken image symbol. Then we found IE was trying to parse the gif when it had requested javascript - leading to script errors in browser and slowdowns. The current trend to https changes the game, one motivation was an attempt by ISPs to inject their own banner ads in place of originals, as well as defeat adblockers. I do remember using host files and 127.0.0.1 or 0.0.0.0 - some OSes seem not to like a lot of them. Firefox, Chrome and Android can now all use adblockplus in the browser, so IE, Apple and locked mobile browsers are most interesting targets now.

 

[kvic]

I briefly tried 0.0.0.0 long time ago. I saw error messages loading ad in Chrome. But I didn't recall seeing that in Safari (...could be just that I didn't flush the cache). However, if people with adblock plugins turned on too, they will hardly notice. Could be the reason few ppl complain and move to next step i.e. adding a dummy adserver.

Android has adblock for a while. iOS 9 just comes with this capability. I believe more people will realise the benefit of adblock on saving mobile data. I've seen a flood of articles doing comparison of data consumption with and without adblock in mobile Safari. The numbers are stunning! For a tiny bit of information, so much junk are fetched. Before iOS 9, some ppl justifies using VPN tunnelling back to home router for simply adblock. The saving in mobile data outweighs VPN overheads.

What are people's experience using dnsmasq for adblock but without a dummy adserver? Do you see error messages in PC/mobile browsers when adblock plugins are off?