STATUS
Twitter is no longer maintaining this project or responding to issues or PRs.
FlockDB
FlockDB is a distributed graph database for storing adjancency lists, with
goals of supporting:
- a high rate of add/update/remove operations
- potientially complex set arithmetic queries
- paging through query result sets containing millions of entries
- ability to "archive" and later restore archived edges
- horizontal scaling including replication
- online data migration
Non-goals include:
- multi-hop queries (or graph-walking queries)
- automatic shard migrations
FlockDB is much simpler than other graph databases such as neo4j because it
tries to solve fewer problems. It scales horizontally and is designed for
on-line, low-latency, high throughput environments such as web-sites.
Twitter uses FlockDB to store social graphs (who follows whom, who blocks
whom) and secondary indices. As of April 2010, the Twitter FlockDB cluster
stores 13+ billion edges and sustains peak traffic of 20k writes/second and
100k reads/second.
It does what?
If, for example, you're storing a social graph (user A follows user B), and
it's not necessarily symmetrical (A can follow B without B following A), then
FlockDB can store that relationship as an edge: node A points to node B. It
stores this edge with a sort position, and in both directions, so that it can
answer the question "Who follows A?" as well as "Whom is A following?"
This is called a directed graph. (Technically, FlockDB stores the adjacency
lists of a directed graph.) Each edge has a 64-bit source ID, a 64-bit
destination ID, a state (normal, removed, archived), and a 32-bit position
used for sorting. The edges are stored in both a forward and backward
direction, meaning that an edge can be queried based on either the source or
destination ID.
For example, if node 134 points to node 90, and its sort position is 5, then
there are two rows written into the backing store:
forward: 134 -> 90 at position 5
backward: 90 <- 134 at position 5
If you're storing a social graph, the graph might be called "following", and
you might use the current time as the position, so that a listing of followers
is in recency order. In that case, if user 134 is Nick, and user 90 is Robey,
then FlockDB can store:
forward: Nick follows Robey at 9:54 today
backward: Robey is followed by Nick at 9:54 today
The (source, destination) must be unique: only one edge can point from node A
to node B, but the position and state may be modified at any time. Position is
used only for sorting the results of queries, and state is used to mark edges
that have been removed or archived (placed into cold sleep).
Building
In theory, building is as simple as
$ sbt clean update package-dist
but there are some pre-requisites. You need:
- java 1.6
- sbt 0.7.4
- thrift 0.5.0
If you haven't used sbt before, this page has a quick setup:
http://code.google.com/p/simple-build-tool/wiki/Setup.
My ~/bin/sbt looks like this:
#!/bin/bash
java -server -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx1024m -jar `dirname $0`/sbt-launch-0.7.4.jar "$@"
Apache Thrift 0.5.0 is pre-requisite for building java stubs of the thrift
IDL. It can't be installed via jar, so you'll need to install it separately
before you build. It can be found on the apache thrift site:
http://thrift.apache.org/.
You can find the download for 0.5.0 here:
http://archive.apache.org/dist/incubator/thrift/0.5.0-incubating/.
In addition, the tests require a local mysql instance to be running, and for
DB_USERNAME and DB_PASSWORD env vars to contain login info for it. You can
skip the tests if you want (but you should feel a pang of guilt):
$ NO_TESTS=1 sbt package-dist
Running
Check out
the demo
for instructions on how to start up a local development instance of FlockDB.
It also shows how to add edges, query them, etc.
Community
- Twitter: #flockdb
- IRC: #twinfra on freenode (irc.freenode.net)
- Mailing list: [email protected] subscribe
Contributors
- Nick Kallen @nk
- Robey Pointer @robey
- John Kalucki @jkalucki
- Ed Ceaser @asdf
