Là một dự án mã nguồn mở, pytorch/torchtitan đã đạt 5.9k sao trên GitHub, ngôn ngữ Python. A PyTorch native platform for training generative AI models
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
torchtitan is a clean-room, PyTorch-native implementation of large-scale
training. One readable codebase takes a model from pretraining through
supervised fine-tuning to reinforcement learning, composing FSDP, tensor,
pipeline, context, and expert parallelism without rewriting the model.
Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP,
and EP stack into N-D layouts through DeviceMesh and DTensor.
Built for research. Minimal, hackable code; every run is a Python
recipe you can read, copy, and modify.
Pretraining to RL in one stack. TitanRL runs the
same model definitions and kernels in the trainer and in vLLM generation, with
a bitwise-reproducible batch-invariant mode.
Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and
FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.
[!NOTE]
torchtitan is under active development. To use the latest features, use a
recent PyTorch nightly.
Latest news
[2026/10] Zephon, a data loader by DatologyAI, released its TorchTitan integration. Zephon keeps the same training batches in the same order even when your GPU count changes (elastic determinism). Check out the blog post on pytorch.org.
[2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
[2025/11] AMD released an optimized fork of torchtitan for AMD GPUs.
[2025/10] We released torchtitan v0.2.0.
[2025/10] SkyPilot now supports torchtitan! See the tutorial.
Older news
[2025/07] We published instructions on how to add a model to torchtitan.
[2025/04] Our paper was accepted by ICLR 2025.
[2024/12] GPU MODE lecture on torchtitan.
[2024/07] Presentation at PyTorch Conference 2024.
Quick start
1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.
See the configuration guide for the full model,
and torchtitan_recipes for the verified recipes.
Supported models
Model
Sizes
Modality
Verified recipes
RL
Llama 3
1B – 405B
Text
llama3_8b, llama3_70b
✅
Qwen3
0.6B – 32B dense; 30B-A3B, 235B-A22B MoE
Text
qwen3_14b
✅
Qwen3.5 / 3.6 / 3.8
0.8B – 397B-A17B; 2.4T-A95B (3.8)
Text, vision
✅ text
DeepSeek V3
16B, 236B, 671B
Text
deepseek_v3_671b (+ Dist-MoE, NVFP4)
DeepSeek V4
Flash, Pro
Text
Soon
gpt-oss
20B, 120B
Text
✅
Kimi K2.7
Moonlight-16B-A3B, Kimi-VL-A3B, K2.5
Text, vision
Kimi K3
K3
Text, vision
Soon
Muse Glimmer
30B
Text, vision
muse_glimmer_30b
✅
Muse-Spark-Open
816B-A42B
Text, vision, audio
muse_spark_816b_a42b_dist_moe_bf16_spmd_256
✅ text
FLUX
dev, schnell
Text-to-image
flux_dev, flux_schnell
Every model has debug configurations exercised in CI. Verified recipes have
been run in training, convergence, or performance workflows. To add a model, see
the model guide.
Features
Parallelism
FSDP2, HSDP, and DDP
Tensor parallel, including async TP
Pipeline parallel: 1F1B, Interleaved 1F1B, GPipe, and other torch.distributed.pipelining schedules
Context parallel: all-gather KV and Ulysses
Expert parallel with DeepEP / HybridEP dispatch
Dist-MoE: fused dispatch, expert compute, and combine on Blackwell
Memory and performance
Activation checkpointing: full, selective, and regional
Regional torch.compile and CUDA graphs
MXFP8 and NVFP4 training
FlashAttention 3/4, FlexAttention, and varlen attention
BF16 optimizer states and FSDP CPU offload
Training
Grain data pipeline with packing, mixing, and checkpointable state
Pretraining, SFT on chat datasets, and LoRA
Muon (DistMuon), EMA, multi-token prediction
Warmup-stable-decay LR schedule and validation
Post-training with TitanRL
One model definition shared by the trainer and vLLM
Bitwise trainer/generator parity in batch-invariant mode
Async rollouts orchestrated with Monarch; trainer-to-generator model state dict transfer over TorchStore
GRPO and DAPO losses; DAPO Math, Search-R1, and Verifiers examples
Checkpointing
Distributed checkpointing, including async saves
Load and save Hugging Face safetensors directly
Conversion scripts between HF and DCP
Debugging and observability
Metrics: loss, memory, throughput, TFLOPs, and MFU in TensorBoard or W&B
Profiling and memory snapshots
Flight Recorder and other debugging tools
Fake-backend dry runs of large layouts on one GPU
Deterministic mode and loss comparison across commits
SDC replay and structured logging
Extensibility
Python recipes selected with --module and --config
Component overrides for custom kernels and modules
Replace cu132 with another CUDA version or a ROCm build (e.g. rocm10.0).
Stable releases
pip install torchtitan
# or
conda install conda-forge::torchtitan
Each stable release pins the nightly versions of torch and torchao it was
validated against; see RELEASE.md.
Optional dependencies
Low precision (MXFP8, NVFP4): install the torchao nightly CI pins, from
the index matching your torch build. It is not in requirements.txt so it
cannot be resolved against a different torch:
RL:scripts/install_rl_env.sh installs the vLLM, Monarch, and TorchStore
nightlies and the FlashAttention build for your GPU. See the
RL quick start.
Extras for vision, FLUX, RL, and development are also available as
pip install -e ".[vlm,flux,rl,dev]".
Importing from elsewhere: the source tree runs as is; to import
torchtitan from another directory, install it in editable mode without
re-resolving dependencies: pip install -e . --no-deps.
Multi-node training with Slurm
Submit scripts/multinode_trainer.slurm with
sbatch from the repository root. Set the node count in both the #SBATCH
header and the torchrun --nnodes argument:
If a node does not have 8 GPUs, also adjust --nproc_per_node and
#SBATCH --gpus-per-task.
Documentation
Topic
Where
Configuring runs and writing recipes
torchtitan/config
Adding a model
torchtitan/models
Parallelisms and distributed techniques
torchtitan/distributed
Datasets and data loading
torchtitan/components/data
Checkpointing and HF conversion
torchtitan/components/checkpointer
Reinforcement learning
torchtitan/rl
Debugging, reproducibility, and validating numerics
docs/debugging.md
Validation and evaluation
docs/evaluation.md
Benchmarks
docs/benchmarks
Tests and CI
tests
Contributing
We look forward to your contributions!
New ideas should start in the experiments folder;
follow the experiments guidelines.
For fixes and contributions to core, follow the contributing guidelines.
Citation
Our paper gives a detailed look at the parallelisms and optimizations in
torchtitan, with advice on when to use each technique:
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.
@inproceedings{
liang2025torchtitan,
title={TorchTitan: One-stop PyTorch native solution for production ready {LLM} pretraining},
author={Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=SFN6Wm7YBI}
}
License
Source code is made available under a BSD 3 license. You may have
other legal obligations that govern your use of other content linked in this
repository, such as the license or terms of service for third-party data and
models.
Trang TopGit này là một snapshot — tab "Readme" hiển thị nguyên văn README của repo (đã bỏ link, giữ ảnh). Repo GitHub ở github.com/pytorch/torchtitan là nguồn chính thức.
pytorch/torchtitan có bao nhiêu sao?
pytorch/torchtitan có 5.9k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/pytorch/torchtitan. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
pytorch/torchtitan có phải mã nguồn mở không?
Có — pytorch/torchtitan phát hành theo license BSD-3-Clause, nghĩa là mã nguồn mở để đọc, fork và (tùy license) tái sử dụng. Mã: github.com/pytorch/torchtitan.
pytorch/torchtitan còn đang phát triển không?
Commit gần nhất trên pytorch/torchtitan là 5 ngày trước (theo timestamp GitHub). Repo có 1.1k fork — một chỉ báo về mức độ quan tâm của cộng đồng.
pytorch/torchtitan là gì?
pytorch/torchtitan (pytorch/torchtitan) là dự án Python trên GitHub. Theo mô tả gốc: A PyTorch native platform for training generative AI models
Đọc đầy đủ README ở tab phía trên.
Chưa chắc torchtitan có hợp với bạn?
Để ChatGPT, Claude hoặc Perplexity tìm hiểu giúp — bấm bên dưới và xem AI nói gì về torchtitan.