Pages

2010-12-22

Block ads on PC & Android with an uber HOSTS file!

Modifying your HOSTS file to get rid of ads should not be news to you, however, you can also do it on your Android device! and it will get rid of ads in the mobile browser, as well as ads in apps!

There even seems to be specific domains for mobile ad servers (like admob.com)...

You might even be aware of MVPS' famous, frequently updated hosts file & update utility for the PC, that protects your from 16-odd thousand ad servers and malware hosts.

There also seems to be a number of sources for potential hosts files:
http://www.mvps.org/winhelp2002/hosts.htm
http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts;showintro=0
http://androidcommunity.com/forums/132380-post3.html
http://forum.xda-developers.com/showpost.php?s=319920cb2a66122ee042ce09891a12ae&p=6195765&postcount=1
etc. (some individual ad servers are merely mentioned in forums here and there)

Having a read in the Android forums, I've discovered that there is an app for updating your hosts file on your mobile - AdFree - however it seems fraught with issues and slow on updates.

http://forum.xda-developers.com/showthread.php?t=664532
http://forum.cyanogenmod.com/topic/11409-adfree-error-on-61-rc2/page__p__98825__hl__hosts__fromsearch__1#entry98825
http://forum.cyanogenmod.com/topic/10474-ad-free-does-not-work/page__p__91976__hl__hosts__fromsearch__1#entry91976

...and who knows where AdFree is getting it's hosts from... does it provide the best possible coverage?

There also seems to be an issue when you use the wrong type of linebreaks in the hosts file on an Android device...


The only solution is to do it yourself - make an uber hosts file! :)


I quickly threw together a small Java app that can take a number of hosts files, and merge them, eliminates duplicates, eliminates comments, eliminate unnecessary spaces, ensures correct linebreaks, and validate line entries.


Download: requires Java 1.5 RTE or higher
Binary
Source
Uber hosts file already merged from the above sources and ready to use, dated 22 Dec 2010


How it works:
Collect an up-to-date set of hosts files from your favourite providers, extract the individual hosts files into one folder (e.g.: c:\temp\hosts\), and rename each file to "hosts.?", e.g.: hosts.1, hosts.2, hosts.3, etc.

Then run the MergeHosts application, supplying the location of the folder containing all your hosts files:
java -cp "c:\temp\MergeHosts.jar" com.ri.hosts.MergeHosts "c:\temp\hosts"

Output looks like this:
MergeHosts - looking for "hosts.*" files in: c:\temp\hosts

Found hosts files:
   hosts.1
   hosts.2
   hosts.3
   hosts.4
   hosts.5

Processing: hosts.1
   14896 domains successfully extracted, with 0 collisions

Processing: hosts.2
   20809 domains successfully extracted, with 13478 collisions

Processing: hosts.3
   1302 domains successfully extracted, with 571 collisions

Processing: hosts.4
   2748 domains successfully extracted, with 2555 collisions

Processing: hosts.5
   3 domains successfully extracted, with 3 collisions

DONE!
Final hosts file: c:\temp\hosts\hosts.final
Size: 667KB

You can now locate the new uber hosts file: hosts.final, which you can use on your Android device or on your PC!


On Android: requires rooted phone
The hosts file is located on a part of the file system which is read-only, and you will have to mount it as read-write to be able to copy the uber hosts file into place.
Copy the hosts.final file into the root of your SD card
Open your Android shell / console / terminal emulator (you can find them in the market if you don't have one, I'm using Terminal Emulator 1.0.16)
Enter the following commands:
su
mount -o remount,rw /dev/block/mtdblock3 /system
mv /etc/hosts /etc/hosts.bak
cp /mnt/sdcard/hosts.final /etc/hosts
chmod 644 /etc/hosts
mount -o remount,ro /dev/block/mtdblock3 /system
ls -l /etc/hosts

Depending on what root-access mechanism you have on your Android distro, you might be prompted to grant Terminal Emulator super user privileges.

Reboot if you are paranoid.


On PC:
Open your C:\WINDOWS\system32\drivers\etc folder
If you already have a hosts file, rename it to hosts.bak
Copy the new hosts.final uber file into your C:\WINDOWS\system32\drivers\etc\ folder, and rename it to hosts (no extension).


Test it: you might have to restart your browser after putting the uber hosts file in place before this will work
Locate any entry in the hosts.final file and open it in your PC's browser / Android browser. E.g.: http://00fun.com
If you get "unable to connect", then its working!


Disclaimer:
I'm sure AdFree is an awesome app, and the most recent version thereof might not be fraught with anything – give it a try if you want... but doing it yourself is always more fun :P

6 comments:

  1. THIS IS FRICKING AWESOME!!! I had found other tutorials about replacing the hosts file in android but this was easy to follow and took care of all the barriers! I specifically needed to add a humongous hosts file that I had found to block all pRon sites (1.2 million of them).

    I do work on phones and computers for people and had a request to block the pron for kids and for others with addiction problems. I knew the line breaks were screwing with it so your .jar fixed everything up nicely.

    BTW here is the site I used for the hosts file
    http://rlwpx.free.fr/WPFF/hosts.htm
    He has several lists all targeted for specific reasons. Awesome.

    The final hosts merge is like 37MB but it runs fine on my Evo!
    Merry Christmas!

    ReplyDelete
  2. So, I'm working on a project that could use your host-file merging code. I'm a bit curious as to what license you've released the source code under, if any?

    ReplyDelete
  3. Ah – thank you for noting the omission Dave.
    Henceforth, let it be known that the source code for the above MergeHosts app has been released on the prestigious WTFPL software license ;)

    ReplyDelete
  4. I feel like an idiot, but I can't figure this out. I downloaded the source and Binary, but dont know how/what to open. Do i need to unzip the MergeHosts folder? When i click the Mergehosts java file, a hosts.final file is created on my desktop, but i havent directed any files into it yet so its useless, right? I know there is just one thing im missing here, but have no idea what it is. any help would be appreciated!

    ReplyDelete
  5. Hey Oliver

    Assuming you have a JRE installed on your PC and you have root on your Android phone, try this:

    1) make directory C:\temp\
    2) make directory C:\temp\hosts\
    3) download the hosts files you would like to use as a source (to merge into the final hosts file)
    4) copy all these source hosts files into C:\temp\hosts\ and rename them hosts.1, hosts.2, hotst.3, etc
    5) download MergeHosts.jar (binary) from this blog post, and copy it into C:\temp\
    6) open command prompt window, and enter commands:
    cd \temp
    java -cp "c:\temp\MergeHosts.jar" com.ri.hosts.MergeHosts "c:\temp\hosts"
    7) output should be as per blog post above, and the hosts.final file will be located in c:\temp\hosts\

    Follow the instructions above for getting the hosts file onto your Android device.

    Post back with your email adr if you are still stuck.

    ReplyDelete
  6. for extra awesomeness I use your tool to create blacklists from URLs AdAway resists

    AdAway is a MUCH better hosts manager compared to AdFree


    thanks for all the fish

    ReplyDelete