xai-org/x-algorithm

X For You Feed Algorithm

This repository contains the core recommendation system powering the "For You" feed on X. It combines in-network content (from accounts you follow) with out-of-network content (discovered through ML-based retrieval) and ranks everything using a Grok-based transformer model.

Note: The transformer implementation is ported from the Grok-1 open source release by xAI, adapted for recommendation system use cases.

Table of Contents

• Updates — May 15th, 2026
• Overview
• System Architecture
• Components
  • Home Mixer
  • Thunder
  • Phoenix
  • Candidate Pipeline
• How It Works
  • Pipeline Stages
  • Scoring and Ranking
  • Filtering
• Key Design Decisions
• License

---

Updates — May 15th, 2026

This release updates the For You algorithm code, including a runnable end-to-end inference pipeline alongside new components for content understanding, ads, and candidate sourcing.

1. End-to-end inference pipeline: A new phoenix/run_pipeline.py replaces the separate run_ranker.py and run_retrieval.py scripts with a single entry point that runs retrieval → ranking from exported checkpoints, mirroring how the two stages are composed in production.

2. Pre-trained model artifacts: A pre-trained mini Phoenix model (256-dim embeddings, 4 attention heads, 2 transformer layers) is now packaged as a ~3 GB archive distributed via Git LFS, enabling out-of-the-box inference without training your own model first.

3. Grox content-understanding pipeline: A new grox/ service is included, providing classifiers, embedders, and a task-execution engine for content understanding workloads such as spam detection, post-category classification, and PTOS policy enforcement.

4. Ads blending system: Includes a new home-mixer/ads/ module that handles ad injection and positioning within the feed, including brand-safety tracking that respects sensitive content boundaries.

5. Query hydrators: Home mixer now hydrates user context including followed topics, starter packs, impression bloom filters, IP, mutual follow graphs, and served history.

6. Candidate hydrators: Additional hydrators for engagement counts, brand safety signals, language codes, media detection, quote post expansion, mutual follow scores, and more.

7. Candidate sources: Adds sources for ads, who to follow, Phoenix MoE, Phoenix topics, prompts, and updates Thunder/Phoenix ones.

---

Overview

The For You feed algorithm retrieves, ranks, and filters posts from two sources:

1. In-Network (Thunder): Posts from accounts you follow
2. Out-of-Network (Phoenix Retrieval): Posts discovered from a global corpus

Both sources are combined and ranked together using Phoenix, a Grok-based transformer model that predicts engagement probabilities for each post. The final score is a weighted combination of these predicted engagements.

We have eliminated every single hand-engineered feature and most heuristics from the system. The Grok-based transformer does all the heavy lifting by understanding your engagement history (what you liked, replied to, shared, etc.) and using that to determine what content is relevant to you.

---

System Architecture

┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│                                    FOR YOU FEED REQUEST                                     │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│                                         HOME MIXER                                          │
│                                    (Orchestration Layer)                                    │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                             │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                   QUERY HYDRATION                                   │   │
│   │  ┌──────────────────────────┐    ┌──────────────────────────────────────────────┐   │   │
│   │  │ User Action Sequence     │    │ User Features                                │   │   │
│   │  │ (engagement history)     │    │ (following list, preferences, etc.)          │   │   │
│   │  └──────────────────────────┘    └──────────────────────────────────────────────┘   │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                  CANDIDATE SOURCES                                  │   │
│   │         ┌─────────────────────────────┐    ┌────────────────────────────────┐       │   │
│   │         │        THUNDER              │    │     PHOENIX RETRIEVAL          │       │   │
│   │         │    (In-Network Posts)       │    │   (Out-of-Network Posts)       │       │   │
│   │         │                             │    │                                │       │   │
│   │         │  Posts from accounts        │    │  ML-based similarity search    │       │   │
│   │         │  you follow                 │    │  across global corpus          │       │   │
│   │         └─────────────────────────────┘    └────────────────────────────────┘       │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                      HYDRATION                                      │   │
│   │  Fetch additional data: core post metadata, author info, media entities, etc.       │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                      FILTERING                                      │   │
│   │  Remove: duplicates, old posts, self-posts, blocked authors, muted keywords, etc.   │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                       SCORING                                       │   │
│   │  ┌──────────────────────────┐                                                       │   │
│   │  │  Phoenix Scorer          │    Grok-based Transformer predicts:                   │   │
│   │  │  (ML Predictions)        │    P(like), P(reply), P(repost), P(click)...          │   │
│   │  └──────────────────────────┘                                                       │   │
│   │               │                                                                     │   │
│   │               ▼                                                                     │   │
│   │  ┌──────────────────────────┐                                                       │   │
│   │  │  Weighted Scorer         │    Weighted Score = Σ (weight × P(action))            │   │
│   │  │  (Combine predictions)   │                                                       │   │
│   │  └──────────────────────────┘                                                       │   │
│   │               │                                                                     │   │
│   │               ▼                                                                     │   │
│   │  ┌──────────────────────────┐                                                       │   │
│   │  │  Author Diversity        │    Attenuate repeated author scores                   │   │
│   │  │  Scorer                  │    to ensure feed diversity                           │   │
│   │  └──────────────────────────┘                                                       │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                                      SELECTION                                      │   │
│   │                    Sort by final score, select top K candidates                     │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                              │                                              │
│                                              ▼                                              │
│   ┌─────────────────────────────────────────────────────────────────────────────────────┐   │
│   │                              FILTERING (Post-Selection)                             │   │
│   │                 Visibility filtering (deleted/spam/violence/gore etc)               │   │
│   └─────────────────────────────────────────────────────────────────────────────────────┘   │
│                                                                                             │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│                                     RANKED FEED RESPONSE                                    │
└─────────────────────────────────────────────────────────────────────────────────────────────┘

---

Components

Home Mixer

Location: home-mixer/

The orchestration layer that assembles the For You feed. It leverages the CandidatePipeline framework with the following stages:

The server exposes a gRPC endpoint (ScoredPostsService) that returns ranked posts for a given user.

---

Thunder

Location: thunder/

An in-memory post store and realtime ingestion pipeline that tracks recent posts from all users. It:

• Consumes post create/delete events from Kafka
• Maintains per-user stores for original posts, replies/reposts, and video posts
• Serves "in-network" post candidates from accounts the requesting user follows
• Automatically trims posts older than the retention period

Thunder enables sub-millisecond lookups for in-network content without hitting an external database.

---

Phoenix

Location: phoenix/

The ML component with two main functions:

1. Retrieval (Two-Tower Model)

Finds relevant out-of-network posts:

• User Tower: Encodes user features and engagement history into an embedding
• Candidate Tower: Encodes all posts into embeddings
• Similarity Search: Retrieves top-K posts via dot product similarity

2. Ranking (Transformer with Candidate Isolation)

Predicts engagement probabilities for each candidate:

• Takes user context (engagement history) and candidate posts as input
• Uses special attention masking so candidates cannot attend to each other
• Outputs probabilities for each action type (like, reply, repost, click, etc.)

See phoenix/README.md for detailed architecture documentation.

---

Candidate Pipeline

Location: candidate-pipeline/

A reusable framework for building recommendation pipelines. Defines traits for:

The framework runs sources and hydrators in parallel where possible, with configurable error handling and logging.

---

How It Works

Pipeline Stages

1. Query Hydration: Fetch the user's recent engagements history and metadata (eg. following list)

2. Candidate Sourcing: Retrieve candidates from:

   • Thunder: Recent posts from followed accounts (in-network)
   • Phoenix Retrieval: ML-discovered posts from the global corpus (out-of-network)

3. Candidate Hydration: Enrich candidates with:

   • Core post data (text, media, etc.)
   • Author information (username, verification status)
   • Video duration (for video posts)
   • Subscription status

4. Pre-Scoring Filters: Remove posts that are:

   • Duplicates
   • Too old
   • From the viewer themselves
   • From blocked/muted accounts
   • Containing muted keywords
   • Previously seen or recently served
   • Ineligible subscription content

5. Scoring: Apply multiple scorers sequentially:

   • Phoenix Scorer: Get ML predictions from the Phoenix transformer model
   • Weighted Scorer: Combine predictions into a final relevance score
   • Author Diversity Scorer: Attenuate repeated author scores for diversity
   • OON Scorer: Adjust scores for out-of-network content

6. Selection: Sort by score and select the top K candidates

7. Post-Selection Processing: Final validation of post candidates to be served

---

Scoring and Ranking

The Phoenix Grok-based transformer model predicts probabilities for multiple engagement types:

Predictions:
├── P(favorite)
├── P(reply)
├── P(repost)
├── P(quote)
├── P(click)
├── P(profile_click)
├── P(video_view)
├── P(photo_expand)
├── P(share)
├── P(dwell)
├── P(follow_author)
├── P(not_interested)
├── P(block_author)
├── P(mute_author)
└── P(report)

The Weighted Scorer combines these into a final score:

Final Score = Σ (weight_i × P(action_i))

Positive actions (like, repost, share) have positive weights. Negative actions (block, mute, report) have negative weights, pushing down content the user would likely dislike.

---

Filtering

Filters run at two stages:

Pre-Scoring Filters:

Post-Selection Filters:

---

Key Design Decisions

1. No Hand-Engineered Features

The system relies entirely on the Grok-based transformer to learn relevance from user engagement sequences. No manual feature engineering for content relevance. This significantly reduces the complexity in our data pipelines and serving infrastructure.

2. Candidate Isolation in Ranking

During transformer inference, candidates cannot attend to each other—only to the user context. This ensures the score for a post doesn't depend on which other posts are in the batch, making scores consistent and cacheable.

3. Hash-Based Embeddings

Both retrieval and ranking use multiple hash functions for embedding lookup

4. Multi-Action Prediction

Rather than predicting a single "relevance" score, the model predicts probabilities for many actions.

5. Composable Pipeline Architecture

The candidate-pipeline crate provides a flexible framework for building recommendation pipelines with:

• Separation of pipeline execution and monitoring from business logic
• Parallel execution of independent stages and graceful error handling
• Easy addition of new sources, hydrations, filters, and scorers

---

License

This project is licensed under the Apache License 2.0. See LICENSE for details.