ghosts
ghosts is a utility to evaluate, compare, and format hosts files. It's written in Go.
Here is what ghosts does:
- Summarize any hosts file retrieved over HTTP, or from a local file.
- Compare two hosts files, and determine their intersection.
- Compare a reference hosts file with a list of hosts presently in your system clipboard.
- List the tally of TLDs in the hosts file.
- Output the hosts as a plain list of domains, or with IP4 pefix.
- Sort the hosts coherently by domain, TLD, subdomain, subsubdomain, and so on.
Getting started
Get help just as you might expect
$ ghosts -h
Usage of ghosts:
-c string
Hosts list to compare.
A shortcut code, full URL, or a local file.
Use the -m option for the main comparison list.
Use the -clip option to use what is on the system clipboard.
Shortcut codes
==============
The following shortcut codes can be used to select among preset main lists.
Amalgamated lists' shortcuts:
-c b or -m base // use Steven Black's base amalgamated list.
-c f // use alternates/fakenews/hosts
-c fg // use alternates/fakenews-gambling/hosts
-c fgp // use alternates/fakenews-gambling-porn/hosts
-c fgps // use alternates/fakenews-gambling-porn-social/hosts
-c fgs // use alternates/fakenews-gambling-social/hosts
-c fp // use alternates/fakenews-porn/hosts
-c fps // use alternates/fakenews-porn-social/hosts
-c fs // use alternates/fakenews-social/hosts
-c g // use alternates/gambling/hosts
-c gp // use alternates/gambling-porn/hosts
-c gps // use alternates/gambling-porn-social/hosts
-c gs // use alternates/gambling-social/hosts
-c p // use alternates/porn/hosts
-c ps // use alternates/porn-social/hosts
-c s // use alternates/social/hosts
Source lists' shortcuts:
-c adaway // adaway.github.io
-c add2o7net // FadeMind add.2o7Net hosts
-c adddead // FadeMind add.Dead hosts
-c addrisk // FadeMind add.Risk hosts
-c addspam // FadeMind add.Spam hosts
-c adguard // AdguardTeam cname-trackers
-c baddboyz // mitchellkrogza Badd-Boyz-Hosts
-c clefspear // Clefspeare13 pornhosts
-c digitalside // davidonzo Threat-Intel
-c fakenews // marktron/fakenews
-c hostsvn // bigdargon hostsVN
-c kadhosts // PolishFiltersTeam
-c metamask // MetaMask eth-phishing hosts
-c mvps // winhelp2002.mvps.or
-c orca // orca.pet notonmyshift hosts
-c shady // hreyasminocha shady hosts
-c sinfonietta-gambling
-c sinfonietta-porn
-c sinfonietta-snuff
-c sinfonietta-social
-c someonewhocares // Sam Pollock someonewhocares.org
-c stevenblack // Steven Black ad-hoc list
-c tiuxo-porn
-c tiuxo-social
-c tiuxo // tiuxo list.
-c uncheckyads // FadeMind UncheckyAds
-c urlhaus // urlhaus.abuse.ch
-c yoyo // Peter Lowe yoyo.org
-clip
The comparison hosts are in the system clipboard
-d Include default hosts at the top of file.
-intersection
Return the list of intersection hosts? (default false)
-ip string
Localhost IP address (default "0.0.0.0")
-m string
The main list of hosts to analyze, or serve as a basis for comparison.
A shortcut code, a full URL, or a local file.
See the -c flag for the list of shortcut codes. (default "base")
-noheader
Remove the file header from output? (default false)
-o Return the list of hosts? (default false)
-p Return a plain output list of hosts, with no IP address prefix? (default false)
-root
Return the list of root domains and their tally (default false)
-s Sort the hosts? (default false)
-stats
display stats? (default true)
-tld
Return the list of TLD and their tally (default false)
-unique
List the unique domains in the comparison list
-v Return the current version
Summarize statistics from any hosts file
If you specify no hosts file, by default a summary of StevenBlack/hosts is produced.
$ ./ghosts
--------------------------------------------------------------------------------
Base hosts file summary:
--------------------------------------------------------------------------------
Location: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
Domains: 54,702
Bytes: 1.7 MB
--------------------------------------------------------------------------------
Specify any hosts file to summarize by using the -m option, like this:
$ ./ghosts -m https://someonewhocares.org/hosts/zero/hosts
--------------------------------------------------------------------------------
Base hosts file summary:
--------------------------------------------------------------------------------
Location: https://someonewhocares.org/hosts/zero/hosts
Domains: 14,401
Bytes: 417 kB
--------------------------------------------------------------------------------
Additionally produce a top-level-domain (TLD) report by using the -tld option, like this:
$ ./ghosts -m https://someonewhocares.org/hosts/zero/hosts -tld
----------------------------------------
Base hosts file summary:
----------------------------------------
Location: https://someonewhocares.org/hosts/zero/hosts
Domains: 16,933
Bytes: 483 kB
TLD tally: (177 unique TLD)
com: 10,093
net: 2,634
info: 563
ru: 296
de: 263
org: 241
pl: 186
nl: 184
uk: 158
skipping many lines for brevity
bo: 1
rw: 1
guru: 1
ae: 1
men: 1
ga: 1
watch: 1
ac: 1
Additionally produce a root domain report by using the -root option, like this:
Warning: the -root option can produce thousands of lines of output. I recommend piping this to a file.
$ ghosts -m https://someonewhocares.org/hosts/zero/hosts -root
----------------------------------------
Base hosts file summary:
----------------------------------------
Location: https://someonewhocares.org/hosts/zero/hosts
Domains: 16,933
Bytes: 483 kB
Root domain tally: (9,723 unique root domais)
2o7.net: 460
hitbox.com: 362
2mdn.net: 213
p2l.info: 198
co.uk: 152
oewabox.at: 125
am15.net: 120
intellitxt.com: 107
doubleclick.net: 103
fastclick.net: 102
checkm8.com: 102
adtech.de: 69
focalink.com: 65
adtech.us: 65
adtech.fr: 65
esomniture.com: 64
msn.com: 63
cjt1.net: 63
thruport.com: 59
imrworldwide.com: 55
bravenet.com: 54
plus.com: 52
skipping many (many!) lines for brevity
Compare two hosts files, local or remote, and assess their intersection by specifying -m <location> option for the main hosts file and -c <location> option for the second comparison file.
Let's compare the someonewhocares.org hosts file (15,474 domains) to the one at mvps.org (8,730 domains). The basic report tells us there are 1,354 domains in the interseation of the two.
Here we use shortcut presets to specify the two lists to compare, but we could have specified full URLs for either source.
$ ./ghosts -m someonewhocares -c mvps
----------------------------------------
Base hosts file summary:
----------------------------------------
Location: https://someonewhocares.org/hosts/zero/hosts
Domains: 15,474
Bytes: 445 kB
----------------------------------------
----------------------------------------
Compared hosts file summary:
----------------------------------------
Location: https://winhelp2002.mvps.org/hosts.txt
Domains: 8,730
Bytes: 335 kB
----------------------------------------
Intersection: 1,354 domains
Compare two hosts files, local or remote, and LIST their intersection by specifying -m <location> option for the main hosts file, -c <location> option for the second comparison file, and add the --intersection flag to get the detailed list of the intersecting domains.
Let's compare the someonewhocares.org hosts file (14,401 domains) to the one at mvps.org (10,473 domains). The basic report shows us all 1,548 domains in the interseation of the two.
Here we use shortcut presets to specify the two lists to compare, but we could have specified full URLs for either source.
$ ./ghosts -m someonewhocares -c mvps --intersection
--------------------------------------------------------------------------------
Base hosts file summary:
--------------------------------------------------------------------------------
Location: https://someonewhocares.org/hosts/zero/hosts
Domains: 15,474
Bytes: 445 kB
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Compared hosts file summary:
--------------------------------------------------------------------------------
Location: http://winhelp2002.mvps.org/hosts.txt
Domains: 8,730
Bytes: 335 kB
--------------------------------------------------------------------------------
intersection: [006.free-counter.co.uk 102.112.2o7.net 102.122.2o7.net 122.2o7.net 192.168.112.2o7.net
1ca.cqcounter.com 1uk.cqcounter.com 1up.us.intellitxt.com 1us.cqcounter.com .... long list ]
Intersection: 1,354 domains
Compare two hosts files, local or remote, and list what's unique in the second file by specifying -m <location> option for the main hosts file, -c <location> option for the second comparison file, and add the --unique flag to get the list of domains in the comparison file that are not in the main hoss file.
Output a list of domains in hosts format, or as a plaintext list
To list domains, use the -o [optional file] option. If you provide no file mame, the list goes to stdout.
To sort the domains, use the -s flag.
Hosts format output
The output is in hosts format by default. The default IP address is 0.0.0.0, and you can change that with the -ip option. for example, -ip 127.0.0.1 will list the hosts with a 127.0.0.1 prefix.
The hosts output will include the header of the original source file. This header typically includes information about the author, and sometimes includes a copyright statement. To not include the header, use the -noheader flag.
To include the list of default hosts at the top of the hosts output, use the d flag. By default the list of hosts does not include the loopback hosts at the top of the list.
These are the default hosts that will be included with the -d flag.
127.0.0.1 localhost
127.0.0.1 localhost.localdomain
127.0.0.1 local
255.255.255.255 broadcasthost
::1 localhost
::1 ip6-localhost
::1 ip6-loopback
fe80::1%lo0 localhost
ff00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
0.0.0.0 0.0.0.0
Plaintext domains output
To get a plaintext list of domains, use the -p flag.
Running the tests
$ go test runs the test suite.
$ gotest runs colorized tests.
Contributing
TBA.
License
MIT.
Related repositories
- StevenBlack/hosts is my amalgamated hosts file, with custom variants, from various curated sources.
- StevenBlack/rhosts hosts tools, written in Rust, just getting started on that.
