NVIDIA/Megatron-LM

About

This repository contains two components: Megatron-LM and Megatron Core.

Megatron-LM is a reference example that includes Megatron Core plus pre-configured training scripts, ideal for research teams, learning distributed training, and quick experimentation.

Megatron Core is a composable library with GPU-optimized building blocks for custom training frameworks. It provides transformer building blocks, advanced parallelism strategies (TP, PP, DP, EP, and CP), mixed precision support (FP16, BF16, FP8, and FP4), and model architectures, ideal for framework developers and ML engineers building custom training pipelines.

Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready recipes.

Getting Started

Install from PyPI:

uv pip install megatron-core

Or clone and install from source:

git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
uv pip install -e .

Note: Building from source can use a lot of memory. If the build runs out of memory, limit parallel compilation jobs by setting MAX_JOBS (for example, MAX_JOBS=4 uv pip install -e .).

For NVIDIA GPU Cloud (NGC) container setup and all installation options, review the Installation Guide.

• Your First Training Run - End-to-end training examples with data preparation
• Parallelism Strategies - Scale training across GPUs with TP, PP, DP, EP, and CP
• Contribution Guide - How to contribute to Megatron Core

Latest News

• [2026/05] DeepSeek-V4 initial support - Megatron Core's dev branch includes the initial DeepSeek-V4 implementation; Megatron Bridge provides conversion, inference, and pretraining recipes.
• [2026/04] Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron - Muon and other emerging optimizers are now supported in Megatron Core via the new Emerging-Optimizers library.
• [2026/03] Scalable Training of Mixture-of-Experts Models with Megatron Core - Technical report on scaling MoE training with integrated optimizations for memory, communication, and computation.
• [2026/03] Implementing Falcon-H1 Hybrid Architecture in Megatron Core - Technology Innovation Institute (TII) contributes Falcon-H1 hybrid transformer-Mamba architecture and BitNet ternary quantization support to Megatron Core.
• [2026/03] Megatron Core Roadmap - Roadmap for upcoming Megatron Core features and improvements.
• [2026/03] Deprecating Python 3.10 support: The upcoming 0.17.0 release drops Python 3.10 support. Downstream applications must raise their lower boundary to 3.12 to stay compatible with Megatron Core.
• [2026/01] Dynamic Context Parallelism - Up to 1.48x speedup for variable-length sequence training with adaptive CP sizing.
• [2025/12] Megatron Core development has moved to GitHub. All development and CI now happen in the open, and community contributions are welcome.
• [2025/10] Megatron Dev Branch - Early access branch with experimental features.
• [2025/10] Megatron Bridge - Bidirectional converter for interoperability between Hugging Face and Megatron checkpoints, featuring production-ready recipes for popular models.
• [2025/08] Mixture of Experts (MoE) Q3–Q4 2025 Roadmap - Comprehensive roadmap for MoE features including DeepSeek-V3, Qwen3, advanced parallelism strategies, FP8 optimizations, and Blackwell performance enhancements.
• [2025/08] GPT-OSS Model - Megatron Core integrates advanced features including YaRN RoPE scaling, attention sinks, and custom activation functions.
• [2025/06] Megatron MoE Model Zoo - Best practices and optimized configurations for training DeepSeek-V3, Mixtral, and Qwen3 MoE models with performance benchmarking and checkpoint conversion tools.

Previous News

Project Structure

Megatron-LM/
├── megatron/
│   ├── core/                    # Megatron Core (kernels, parallelism, building blocks)
│   │   ├── models/              # Transformer models
│   │   ├── transformer/         # Transformer building blocks
│   │   ├── tensor_parallel/     # Tensor parallelism
│   │   ├── pipeline_parallel/   # Pipeline parallelism
│   │   ├── distributed/         # Distributed training (FSDP, DDP)
│   │   ├── optimizer/           # Optimizers
│   │   ├── datasets/            # Dataset loaders
│   │   ├── inference/           # Inference engines and server
│   │   └── export/              # Model export (example: TensorRT-LLM)
│   ├── training/                # Training scripts
│   ├── legacy/                  # Legacy components
│   ├── post_training/           # Post-training (quantization, distillation, pruning, etc.)
│   └── rl/                      # Reinforcement learning (including RLHF)
├── examples/                    # Ready-to-use training examples
├── tools/                       # Utility tools
├── tests/                       # Comprehensive test suite
└── docs/                        # Documentation

Performance Benchmarking

For the latest performance benchmarking results, refer to NVIDIA Megatron Bridge Performance Summary.

The codebase efficiently trains models from 2B to 462B parameters across thousands of GPUs, achieving up to 47% Model FLOP Utilization (MFU) on H100 clusters.

Benchmark Configuration:

• Vocabulary size: 131,072 tokens
• Sequence length: 4,096 tokens
• Model scaling: Varied hidden size, attention heads, and layers to achieve target parameter counts
• Communication optimizations: Fine-grained overlapping with DP (--overlap-grad-reduce, --overlap-param-gather), TP (--tp-comm-overlap), and PP (enabled by default)

Key Results:

• 6,144 H100 GPUs: Successfully benchmarked 462B parameter model training.
• Superlinear scaling: MFU increases from 41% to 47–48% with model size.
• End-to-end measurement: Throughputs include all operations (data loading, optimizer steps, communication, and logging).
• Production ready: Full training pipeline with checkpointing and fault tolerance.
• Note: Performance results measured without training to convergence

Weak Scaling Results

The weak scaled results show superlinear scaling (MFU increases from 41% for the smallest model considered to 47–48% for the largest models); this is because larger GEMMs have higher arithmetic intensity and are consequently more efficient to execute.

Strong Scaling Results

This test strong scales the standard GPT-3 model (slightly more than 175 billion parameters due to larger vocabulary size) from 96 H100 GPUs to 4,608 GPUs, using the same batch size of 1,152 sequences throughout. Communication becomes more exposed at larger scale, leading to a reduction in MFU from 47% to 42%.

Roadmaps

• 2026 Q2 Roadmap
• 2026 Q2 MoE-Specific Roadmap [dev branch first developments]

Resources

Getting Help

• 📖 Documentation - Official guides and API reference
• 🐛 Issues - Bug reports and feature requests

Contributing

Contributions are welcome. Ways to contribute:

• 🐛 Report bugs - Help improve reliability
• 💡 Suggest features - Shape the future of Megatron Core
• 📝 Improve docs - Make Megatron Core more accessible
• 🔧 Submit PRs - Contribute code improvements

→ Contributing Guide

Citation

If you use Megatron in your research or project, use the following citation:

@article{megatron-lm,
  title={Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism},
  author={Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan},
  journal={arXiv preprint arXiv:1909.08053},
  year={2019}
}