Fairseq: PyTorch Toolkit for Sequence Modeling

What is fairseq?

Fairseq is a PyTorch sequence-to-sequence toolkit built by Facebook AI Research for training translation, language modeling, and text generation models. It bundles reference implementations of published papers — LSTM and CNN seq2seq, plus Transformer variants like RoBERTa, BART, and wav2vec — in one training and generation pipeline. You extend it by registering new models, tasks, and optimizers, not by rewriting the training loop.

Implemented papers and model architectures

• ✓Reference implementations of dozens of papers, including RoBERTa, BART, mBART, XLM-R, CamemBERT, wav2vec, and wav2vec 2.0, plus the original Transformer and LSTM sequence-to-sequence architectures.
• ✓Non-autoregressive translation models — Levenshtein Transformer, Mask-Predict — alongside the standard autoregressive Transformer and CNN seq2seq models.
• ✓Multiple decoding strategies at generation time: beam search, Diverse Beam Search, top-k/top-p sampling, and lexically constrained decoding.
• ✓Mixed precision training on NVIDIA tensor cores, plus gradient accumulation for large effective batch sizes on a single GPU.
• ✓Full parameter and optimizer state sharding with CPU offloading, for models too large to fit on one GPU.
• ✓Multi-GPU training across one machine or many, with both data-parallel and model-parallel modes.
• ✓Hydra-based configuration that combines code defaults, command-line flags, and config files instead of one flat flags file.
• ✓A registry pattern for models, criterions, tasks, optimizers, and learning rate schedulers, so new components plug in without touching the core training loop.

What you can train with fairseq

• •Neural machine translation between language pairs, starting from the pre-trained WMT19 models or training from scratch.
• •Language modeling with Transformer or convolutional architectures, including large-scale unidirectional models.
• •Speech recognition and self-supervised speech representation learning through the wav2vec and wav2vec 2.0 examples.
• •Text summarization and other generation tasks using the BART and pointer-generator implementations.
• •Fine-tuning pre-trained models like RoBERTa and XLM-R for downstream tasks with the R3F/R4F fine-tuning code.

Installing fairseq

Fairseq needs PyTorch 1.10.0 or newer and Python 3.8 or newer; training new models also assumes an NVIDIA GPU with NCCL, per the README. To develop locally: `git clone https://github.com/pytorch/fairseq`, `cd fairseq`, then `pip install --editable ./` (on macOS, the README adds a CFLAGS=-stdlib=libc++ environment variable before that last command). The latest stable release installs instead with `pip install fairseq`. For faster training, the README recommends building NVIDIA's apex library from source; for large datasets, it recommends installing PyArrow. Running Fairseq in Docker means increasing shared memory with `--ipc=host` or `--shm-size`, per the README.

Using a pre-trained model for translation

The README's fastest path to a working translation runs through `torch.hub`: `en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model')`, then `en2de.translate('Hello world', beam=5)`. That's the whole call. The README's own example returns 'Hallo Welt'. Generation defaults to beam search, but you can switch to Diverse Beam Search, top-k/top-p sampling, or lexically constrained decoding depending on the task. For training instead of inference, the README points to the full documentation at fairseq.readthedocs.io for per-example training and evaluation commands.

Alternatives to fairseq

Who should use fairseq

Researchers reproducing or building on a specific sequence modeling paper are the clearest fit — Fairseq hands you the actual reference code instead of a from-scratch reimplementation. Teams fine-tuning a pre-trained Fairseq model like RoBERTa or wav2vec 2.0 for their own translation, language modeling, or speech task get a documented starting point too. If you just need a pre-trained model for inference in a product, check whether the same checkpoint is available through Hugging Face's transformers first — it can save you learning Hydra's config system for a use case that doesn't need it.

Frequently asked questions

The problem it solves

Reproducing a sequence modeling paper from scratch usually means re-reading the paper, guessing at hyperparameters the authors left out, then hoping your reimplementation matches their reported numbers. Fairseq sidesteps that by shipping the code Facebook AI Research actually used for papers like RoBERTa, BART, and wav2vec in one repo, along with the training commands to reproduce their results.

Related repositories