X's Recommendation Algorithm: Open Source Feed Ranking
/the-algorithm
X's Recommendation Algorithm is a real look at how a production feed pipeline is wired, not a tutorial dressed up as one. It's worth studying for the architecture — candidate sourcing into a heavy ranker into visibility filtering — but it's a frozen snapshot with no top-level build file, so treat it as a reference to read, not a system to run. Come for the architecture, not for a working X clone.
What the repository contains
X's Recommendation Algorithm is the codebase, mostly in Scala with Python and Rust components, that X released covering the services and jobs behind the For You Timeline and Recommended Notifications. It bundles data services like tweetypie and unified-user-actions, embedding models like SimClusters and TwHIN, and frameworks — product-mixer, navi, twml — that stitch candidate sourcing, ranking, and filtering into a served feed.
Core recommendation pipeline components
- ✓SimClusters builds sparse embeddings by detecting communities of accounts, then represents both users and posts as vectors in those communities.
- ✓TwHIN produces dense knowledge-graph embeddings for users and posts — it lives in the separate the-algorithm-ml repo, not this one.
- ✓Candidate sourcing splits two ways: search-index (Earlybird) supplies in-network posts, about half of all candidates by the README's own account, while tweet-mixer coordinates out-of-network fetches.
- ✓UTEG (user-tweet-entity-graph) keeps an in-memory graph of user-to-post interactions, built on the GraphJet framework, and surfaces candidates by walking that graph.
- ✓The heavy ranker is a neural network that re-scores candidates after sourcing — the main signal home-mixer uses to decide what actually makes the timeline.
- ✓home-mixer, built on product-mixer, assembles and serves the Home Timeline; visibility-filters run alongside it for legal and trust-and-safety filtering.
- ✓navi is a separate Rust service for serving ML models at low latency; twml is the legacy TensorFlow v1 framework the newer pieces are replacing.
Strengths
- ✓Real production code, not a toy reimplementation — you're reading the same heavy-ranker and home-mixer logic X actually ran.
- ✓The architecture table maps every component to its actual source path, so you're never guessing which folder does what.
- ✓navi and product-mixer are genuinely reusable references for anyone building their own model-serving or feed-assembly layer.
What you can and cannot learn from this code
- △No top-level BUILD or WORKSPACE file — the README says a full build and test system is still future work, so you can't compile and run the whole thing.
- △It's a snapshot, not a live mirror — the README describes still building the tooling to sync community suggestions back into X's internal repo.
- △Core pieces like TwHIN and the heavy ranker live in a separate the-algorithm-ml repo, so the recommendation logic is split across two codebases.
- △twml is explicitly labeled a legacy TensorFlow v1 framework, so part of what you'll read is already being phased out inside X.
Other open source recommendation systems
Frequently asked questions
X's Recommendation Algorithm is a partial release, not the complete system — it covers the For You Timeline and Recommended Notifications, includes Bazel BUILD files for most components, but ships no top-level BUILD or WORKSPACE file, and pieces like the heavy ranker and TwHIN live in a separate the-algorithm-ml repo.
X's Recommendation Algorithm is written primarily in Scala, with additional components in Python and Rust — navi, the model-serving framework, is written in Rust. The legacy twml framework is built on TensorFlow v1.
Not as a complete system. X's Recommendation Algorithm has no top-level BUILD or WORKSPACE file, and the README says a full build and test system is still future work. You can build or read individual components, like navi or product-mixer, with their own Bazel BUILD files, but not the whole pipeline end to end.
X's Recommendation Algorithm is released under the AGPL-3.0 license, which requires anyone who modifies it and runs it as a network service to share their source code changes.
SimClusters detects communities of accounts and builds sparse embeddings that place both users and posts into those communities. X's Recommendation Algorithm uses these embeddings as one signal feeding candidate sourcing and ranking on the For You Timeline.
The heavy ranker is a neural network that re-scores candidate posts after they've been sourced, and it's one of the main signals home-mixer uses to decide what actually lands in the For You Timeline. Its code lives in the separate the-algorithm-ml repo, not in this one.
The problem it solves
Before this release, anyone trying to explain why a given post showed up in someone's For You Timeline was reverse-engineering it from outside — screenshots, leaked slide decks, and guesswork about what "the algorithm" even meant. X's Recommendation Algorithm answers that with actual service boundaries: which job pulls in-network candidates, which model scores them, and which filter can still drop a post before it's served. It closes the gap between "the algorithm decided" and a specific function call you can point to.
Best use cases
- •Studying how a real company structures a multi-stage recommendation pipeline — candidate sourcing, light ranking, heavy ranking, filtering — instead of reading about it secondhand.
- •Pulling navi as a standalone reference for a Rust-based model-serving layer, since it's decoupled enough to read on its own.
- •Using product-mixer's structure as a design reference when sketching your own feed-assembly framework.
- •Preparing system-design interview answers with a concrete, named example instead of a hand-wavy diagram.
How to install / try
There's no top-level BUILD or WORKSPACE file, so you can't clone this and build the whole recommendation stack in one shot — the README says as much and calls a complete build and test system future work. Individual components ship their own Bazel BUILD files, so the realistic path is cloning the repo and building or reading one component, say navi or product-mixer, rather than trying to stand up the full pipeline.
How to use
Beyond the per-component READMEs linked from the architecture table, this repo doesn't document a single command to run the recommendation pipeline end to end — expected, since it's excerpted from X's internal monorepo rather than packaged as a deployable service. In practice, usage means opening the component you care about and reading its own README and BUILD file instead of invoking anything at the top level.
Who should try it — and who should skip
Backend and ML engineers who want to see a real, multi-service ranking pipeline — candidate sourcing, embeddings, a neural heavy ranker, and post-filtering — should spend time here; the component READMEs are detailed enough to actually learn from. Skip it if you're hoping to clone, build, and run your own X-like feed: there's no top-level build system, and the heavy ranker and TwHIN both live in a separate repo you'd also need to pull in.
Curious whether the-algorithm is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about the-algorithm.