1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

AdBlocking with combined hosts file

Discussion in 'Asuswrt-Merlin' started by thelonelycoder, Feb 6, 2014.

  1. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,527
    Location:
    Switzerland
    AB-Solution is now Diversion
    Please check the Diversion website for latest version and discussion link.


    These snippets of code posted here are now a automated install script with control center character under the name of AB-Solution.

    This is a simple but effective way to block ads on all your devices
    ------------------------------------------------------------
    Changelog
    ------------------------------------------------------------
    September 06, 2015: Added Quick Enable / Disable Adblocking, TOC added
    August 31, 2015: Removed requirement for Entware, it was never really needed.
    April 10, 2015: Typo, changed addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked.txt to [...]/hosts.blocked in script dnsmasq.conf.add, thanks tijaune.
    April 08, 2015: Added Advanced method
    March 31, 2015: Added preparatory steps for jffs
    March 21, 2015: Simplified install steps.
    March 20, 2015: Adding the cron job with /jffs/scripts/services-start instead of /jffs/scripts/init-start for compatibility with latest firmware.
    March 17, 2015: If DNS-Filtering is enabled in your routers interface AdBlocking will not work. The DNS Filter bypasses Dnsmasq settings. See this thread for details.
    April 30, 2014: If you want to remove some domains appearing in the hosts.clean you can do this with a whitelist file. See post 65 in this thread.
    If you want to go even further with my ad-blocking solution, e2301010 has posted AdBlocking with combined hosts file with Lighttpd.
    March 14, 2014: wget scripts are modified to reflect the much better and smarter findings of decedion.

    ------------------------------------------------------------
    Table of content
    ------------------------------------------------------------
    Indroduction
    1. Standard method (the original)
    2. Advanced method
    3. Notes

    4. Quick Enable / Disable Adblocking
    ------------------------------------------------------------

    Introduction


    This combines well maintained hosts files into one, with no duplicate entries.
    The resulting file is about 728 KB (1450 KB Advanced method) containing more than 26000 (52600) lines of blocked domains.

    Preparatory steps for both methods:
    In Administration / System set to Yes:
    - Enable JFFS partition*
    - Enable JFFS custom scripts and configs
    *) If this has not been set before set 'Format JFFS partition at next boot' to Yes and Reboot.

    In this post two ways to use the combined hosts file are explained, the Standard and Advanced method. The Standard method is the original method, updated as per the Changelog. On April 04, 2015 the Advanced method was added with additional hosts files and support for a black- and whitelist of domains.
    Note: The two methods are not compatible, use only one in your router.

    1. Standard method (the original)

    The files are stored on a USB disk, /tmp/mnt/sda1/ is the path. If you use /jffs/ edit accordingly.

    Using a USB disk we need to delay the startscript.
    This will create an up-to-date hosts file after every reboot.
    Add to /jffs/scripts/post-mount
    Code:
    #!/bin/sh
    sleep 12
    wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
    sleep 5
    service restart_dnsmasq
    Tell dnsmasq to add the new hosts file and where to send the unwanted requests to (sending them to 0.0.0.0)
    Create or add to /jffs/configs/dnsmasq.conf.add
    Code:
    address=/0.0.0.0/0.0.0.0
    addn-hosts=/tmp/mnt/sda1/hosts.clean
    To update the hosts file (manually or as a cron job) create /jffs/scripts/update-hosts.sh
    Code:
    #!/bin/sh
    wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
    sleep 5
    service restart_dnsmasq
    To run update-hosts.sh every Friday midnight as a cron job create or add to /jffs/scripts/init-start /jffs/scripts/services-start
    Code:
    #!/bin/sh
    
    cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"
    Set all files in /jffs/scripts/ as executable
    Code:
    chmod a+rx /jffs/scripts/*
    That's it, reboot and enjoy!
    Tested on a RT-AC66U.

    2. Advanced method


    This method supports a black- and a whitelist for domains you want to include/exclude in the hosts file. It uses 5 well maintained hosts files (Standard method uses 3). Thank you Weble for the input in post #143.
    Note: The code below is NOT compatible with the Standard method!

    This will create an up-to-date hosts file after every reboot and update the hosts file with cron every Friday at midnight.
    Note that the files are stored in /tmp/mnt/sda1/hosts/. Edit all references in the code if you use a different path.
    Add this to /jffs/scripts/services-start
    Code:
    #!/bin/sh
    
    sleep 10
    cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"
    
    sleep 5
    source  /jffs/scripts/update-hosts.sh
    Create or add to /jffs/configs/dnsmasq.conf.add
    Code:
    # AdBlocking
    address=/0.0.0.0/0.0.0.0
    ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
    addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
    addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt
    To update the hosts file (to run it manually, and automatically as a cron job and when rebooting the rooter) create /jffs/scripts/update-hosts.sh
    Code:
    #!/bin/sh
    
    # set directory
    dir=/tmp/mnt/sda1/hosts
    
    # get hosts files and combine and sort, write to temp file
    wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > $dir/temp
    
    #get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
    wget -qO- "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/ad_servers.txt" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp
    
    # remove whitelisted entries in temp and write final file, remove temp file
    cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked
    
    #remove temp file
    rm $dir/temp
    
    #restart dnsmasq to apply changes
    sleep 1
    service restart_dnsmasq
    Set all files in /jffs/scripts/ as executable
    Code:
    chmod a+rx /jffs/scripts/*
    Create /tmp/mnt/sda1/hosts/whitelist.txt and fill it with domains you want to whitelist, one domain per line. An example would look like so:
    Code:
    clients2.google.com
    something.other.com
    anotherdomain.com
    Create /tmp/mnt/sda1/hosts/blacklist.txt and fill it with domains you want to blacklist that are not in the combined hosts file, one domain per line. Add 0.0.0.0 to every line as in this example:
    Code:
    0.0.0.0 www.rivalo.com
    0.0.0.0 rivalo.com
    0.0.0.0 madpandatv.net
    0.0.0.0 www.snapdo.com
    These two files (whitelist.txt and blacklist.txt) need to be present for the scripts to work. Leave the files empty if you have no white/blacklist domains.

    Reboot and enjoy!

    3. Notes

    A note for Windows users: You need to flush the DNS Cache when you tinker with your Router.
    In a command prompt enter ipconfig/flushdns and hit Enter.

    On Windows 7 or 8 Systems this generally needs Administrator privileges (hit Windowskey and type cmd. In the list, right-click on cmd (.exe) and choose 'run as Administrator').
    Also, clear your browsers cache after this.

    4. Quick Disable / Enable Adblocking

    To temporarily disable the hosts files remove the following, depending on which method you use:

    1. Standard method (the original)
    in /jffs/configs/dnsmasq.conf.add remove this:
    Code:
    address=/0.0.0.0/0.0.0.0
    addn-hosts=/tmp/mnt/sda1/hosts.clean
    2. Advanced method
    in /jffs/configs/dnsmasq.conf.add remove this:
    Code:
    # AdBlocking
    address=/0.0.0.0/0.0.0.0
    ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
    addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
    addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt
    And then restart dnsmasq (or reboot the router):
    Code:
    service restart_dnsmasq
     
    Last edited: Sep 4, 2018
    golgi42, propergol, zekesdad and 18 others like this.
  2. r00t4rd3d

    r00t4rd3d Regular Contributor

    Joined:
    Jan 29, 2014
    Messages:
    146
    Location:
    Toilet
    So this blocks for all clients connected to the router? Wouldnt it slow everything down considerably?
     
  3. sinshiva

    sinshiva Very Senior Member

    Joined:
    Nov 8, 2013
    Messages:
    1,067
    Location:
    FL
    just tried out the scripts, everything works great. thank you!
     
  4. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,527
    Location:
    Switzerland
    As this is a simple hosts file and dnsmasq handles it, yes it affects all clients connected to the router. This is exactly what I wanted it for. Almost free of ads on all devices in my household. I use a varied mix of Linux, Windows, Mac, iOS and Android devices. No need to install AdBlocker on any of them as it conveniently blocks almost everything, even InApp Ads. I love it!

    As for the performance I am happy as it is. Woud be great to have a free performance tool to stress the routers dns resolving. I am not aware of one.

    You are welcome. Glad it works for you.
     
  5. dancemonkey

    dancemonkey Occasional Visitor

    Joined:
    Feb 1, 2014
    Messages:
    14
    Working well on N66U here. Thanks!
     
  6. L&LD

    L&LD Part of the Furniture

    Joined:
    Dec 9, 2013
    Messages:
    9,369
    I would love to try this myself, but is this as step by step as can be?

    I'm not usually as dumb as I feel when I look at post #1, but I really don't know where to begin to get this going on my RT-N66U with the RMerlin 374.39_0-em firmware installed.

    Any baby steps walkthrough for a coding newbie?
     
  7. sinshiva

    sinshiva Very Senior Member

    Joined:
    Nov 8, 2013
    Messages:
    1,067
    Location:
    FL
    sure.

    it'll be easiest to do this from an ssh client, so be sure you have ssh enabled and are using something such as PuTTY.

    setting up entware is very simple, once logged in, just run;
    Code:
    entware-setup.sh
    i always used nano when i was first using linux, but i've since forced myself to learn vi. to keep it simple; after installing entware, you use entware's packagemanager 'opkg' to install it.
    Code:
    opkg install nano
    you can also check out what other things are available through the package manager by using 'opkg list' or 'opkg search blah'. you can also do something like 'opkg list | grep whatever' as a means of searching. to make it easier to read the whole list, you can use 'opkg list | less'

    to use nano, you run the command like thus to create/edit a file;
    Code:
    nano /jffs/scripts/init-start
    to exit and save, you first use CTRL + X and confirm by using SHIFT + Y

    anyway, you can copy/paste the scripts into nano. if you get any whitespace at the beginning of any of the lines in the script, remove it, so that they look more or less exactly like the examples above.

    entware creates some scripts in /jffs/scripts when it's installed, so when you are adding to the /jffs/scripts/post-mount script, for example, either replace the whole thing with thelonelycoder's script, or be sure to remove the following if you see it in the script twice; this is what's generated by the entware install:

    Code:
    if [ $1 = "/tmp/mnt/sda1" ]
    then
      ln -sf $1/entware /tmp/opt
    fi
    i think that covers all the important stuff.

    oh, and not sure if you know about tab completion, but when you're at the command line, you can type 'ent' and hit TAB and it'll replace it with entware-setup.sh
     
    Last edited: Feb 7, 2014
    ScottW likes this.
  8. L&LD

    L&LD Part of the Furniture

    Joined:
    Dec 9, 2013
    Messages:
    9,369
    sinshiva,

    I still feel just as 'duh' as ever looking at this stuff - but now, I have enough power to be dangerous! :)

    I hope to get a chance to try this in the next few days.

    Thanks again.
     
  9. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,527
    Location:
    Switzerland
    Relax, there is a human oriented way just for the lazy ass types. Forget about the nano stuff and PuTTY.

    Lean back and read first.

    Instead of PuTTY use mRemoteNG, an Open Source SSH, VNC ,Telnet, what have you client. It even stores ssh passwords and has a clean usable Interface. The PuTTY dev's never get that these are a must have these days. I will not budge. Key files are supported as well as Pageant (for the hard core guys).

    To transfer and edit files on the router use WinSCP so you don't have to use vi or nano on a ssh terminal. WinSCP runs over ssh but needs SFTP Server on the router installed. So install it after logging in with ssh:
    Code:
    opkg install openssh-sftp-server 
    Then connect with WinSCP and suddenly you feel at home.
    Now you can create and edit files more easily.

    While we're at it, use Notepad++ as a replacement of the standard Editor in Windows. In WinSCP under Preferences / Editor add Notepad++.

    Hope this helps.
     
    Shrmn likes this.
  10. Martineau

    Martineau Part of the Furniture

    Joined:
    Jul 8, 2012
    Messages:
    2,197
    Location:
    UK

    Sinshiva's post is excellent, but if you don't want to initially go anywhere near the arcane unix editors and you are on a Wintel box....

    Code:
    1. Install WINSCP and configure it for an SCP connection to the router and save the admin/password credentials (unless you want to type them for every connection!)
    
    2. Navigate in the right-hand pane to /jff/scripts
    
    3. If the file you want to edit already exists then simply double-click to open the file in edit mode
    
       otherwise SHIFT+F4 will ask you for a new name for your script to edit.
    
    4. Paste (or tediously manually type) in the new scipt lines as required.
    
    5. Simply close the file 'intuitively' none of this weird CTRL and SHIFT key combo nonsense! :-)
    
    6. If this is a new file, then right click and select 'properties',and make sure all 9 R/W/X boxes are ticked - should show octal=0777.
    
    7. Test script by right-clicking file, select 'custom commands' then click  'execute'
    
        or to see the results in a window select 'custom commands' then click 'enter....'
            and tick the box 'show results in terminal'  then type name of the script you want to run prefixed with the two characters  './' 

    Learning vi/nano and the unix shell is rewarding, but if you prefer point and click (and use Windows) then this is the way to get started! :)

    Regards,
     
  11. r00t4rd3d

    r00t4rd3d Regular Contributor

    Joined:
    Jan 29, 2014
    Messages:
    146
    Location:
    Toilet

    You can use WinSCP with the SCP protocol and not have to install anything to use it.
     
  12. jakey

    jakey Occasional Visitor

    Joined:
    Oct 19, 2012
    Messages:
    24
    I'd like try this but I can't install entware, I have put a small SD card in the internal slot of my RT-N66U and it auto mounts as sda1, the problem is it's formatted as fat32 and I know it needs to be ext2/3.

    I umount the card and try to format it with mkfs.ext3 /mnt/sda1 but it doesn't work.

    Where am I messing up ?

    thanks
     
    Last edited: Feb 7, 2014
  13. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,527
    Location:
    Switzerland
    We are not all bearded *nix Gurus here. Some want to work in a more relaxed environment. But we still code.

    Looking through the generated hosts.clean file I noticed some of Dan Pollock's (of soneonewhocares.org fame) comments in his hosts file have a different formatting. Some comments are not stripped out.

    There's no need to but if you want to replace
    Code:
    sed -i '/::1/d;/^#/d;/127.0.0.1    .*local/d;/255./d;/fe80/d;s/# .*//;s/127.0.0.1/0.0.0.0/' /tmp/mnt/sda1/whocares.org.hosts
    with
    Code:
    sed -i '/::1/d;/^#/d;/127.0.0.1    .*local/d;/255./d;/fe80/d;s/# .*//;s/#.*//;s/127.0.0.1/0.0.0.0/' /tmp/mnt/sda1/whocares.org.hosts
    The original post in this thread (#1) is updated.
     
  14. thelonelycoder

    thelonelycoder Part of the Furniture

    Joined:
    Jan 23, 2014
    Messages:
    5,527
    Location:
    Switzerland
    You're very close to success. mkfs.ext2 will work.
    It will only work with ext2 file system.
     
  15. jakey

    jakey Occasional Visitor

    Joined:
    Oct 19, 2012
    Messages:
    24
    thanks for reply m8 but the result is the same

    Code:
    [email protected]:/tmp/home/root# mkfs.ext2 /mnt/sda1
    mke2fs 1.42.8 (20-Jun-2013)
    /mnt/sda1 is not a block special device.
    Proceed anyway? (y,n) y
    mkfs.ext2: Device size reported to be zero.  Invalid partition specified, or
            partition table wasn't reread after running fdisk, due to
            a modified partition being busy and in use.  You may need to reboot
            to re-read your partition table.
    
    Should I being doing something first with fdisk ?

    The problem seems to be that once sda1 is unmounted sda1 just becomes a folder and not a device.
     
    Last edited: Feb 7, 2014
  16. sinshiva

    sinshiva Very Senior Member

    Joined:
    Nov 8, 2013
    Messages:
    1,067
    Location:
    FL
    though ext2 is probably better for an sdcard, you can use;

    Code:
    mke2fs -t ext3 /dev/sda1
    for ext3.
     
  17. sinshiva

    sinshiva Very Senior Member

    Joined:
    Nov 8, 2013
    Messages:
    1,067
    Location:
    FL
    ah, use /dev/sda1 rather than /mnt/sda1

    [edit1] forgot to say; only for formatting the partition. when it's mounted, it'll be accessed via /mnt/sda1 [/edit1]
    [edit2/]
    you may need to repartition to set the partition to linux type 83, etc.

    fdisk /dev/sda (no number)

    and just use the help menu, it's pretty easy if you are only creating a single partition on the drive. delete the existing partition, create a new one with type 83 using all the available space and be sure to write the partition table when you're done
     
    Last edited: Feb 7, 2014
  18. jakey

    jakey Occasional Visitor

    Joined:
    Oct 19, 2012
    Messages:
    24
    Thanks both for your help, got there in the end and entware installed.

    Will finish the adblocking part tomorrow, need some shuteye now :)
     
  19. Zirescu

    Zirescu Very Senior Member

    Joined:
    Jul 16, 2013
    Messages:
    728
    Location:
    Kelowna, BC
    Just finished deploying this to my RT-N66U but using the SD card and some minor changes to the mount points and works like a charm. No more annoying ads showing up when I run Skype. And most of the web sites I visit are now ad free on all of the mobile devices. Thank you for sharing theses instructions!
     
  20. jakey

    jakey Occasional Visitor

    Joined:
    Oct 19, 2012
    Messages:
    24
    I'd also like to thank thelonelycoder for doing the work and sharing this, all up and running now on my RT-N66, all my browsing seems to ad free now but the 'in app' ads on my Lumia920 Windows phone 8 remain.

    Thanks also to sinshiva for helping me sort out the SD card issue I had, perhaps you could help me with one thing more

    I always have to power off and on the router to reboot otherwise if using the CLI or GUI to reboot the SD card always mounts as sdb1, now with my Linux STB I would use the fstab to make sure devices mount the same every time.

    As the fstab is created dynamically on boot I'm not sure how to go about this so that the SD card always mounts as sda1 regardless of how the Asus is rebooted.

    BTW I always use WinSCP for creating files and use it's internal editor which is quite adequate for simple jobs like this, WinSCP can also be configured to automatically run and login with Putty so with a double click i'm into the file system with Putty loaded and logged in.