facebookresearch/xformers sits at 10.6k stars on GitHub, written primarily in Python. Hackable and optimized Transformers building blocks, supporting a composable construction.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
xFormers - Toolbox to Accelerate Research on Transformers
xFormers is:
Customizable building blocks: Independent/customizable building blocks that can be used without boilerplate code. The components are domain-agnostic and xFormers is used by researchers in vision, NLP and more.
Research first: xFormers contains bleeding-edge components, that are not yet available in mainstream libraries like PyTorch.
Built with efficiency in mind: Because speed of iteration matters, components are as fast and memory-efficient as possible. xFormers relies on optimized kernels from MSLK, FlashAttention, PyTorch and Triton.
Installing xFormers
xFormers is pure Python, and runs on any PyTorch 2.10+. Its attention kernels come from the mslk package.
(RECOMMENDED) Install latest stable from the PyTorch index matching your PyTorch build:
# Replace cu132 with the CUDA (e.g. cu130) or ROCm (e.g. rocm7.14) version of your PyTorch.
# flash-attn-3 is optional: it speeds up attention on Ampere and Hopper GPUs. It is only
# available with CUDA (on Windows: only with CUDA 12.x!), so leave it out otherwise.
pip install -U xformers mslk flash-attn-3 --index-url https://download.pytorch.org/whl/cu132
Each mslk release targets a specific PyTorch version: for mslk's native kernels to work (some ROCm backends need them), install versions of torch and mslk from the same row of its compatibility table.
Install from source:
# Install mslk from the PyTorch index matching your PyTorch build, as above
pip install mslk --index-url https://download.pytorch.org/whl/cu132
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
Benchmarks
Memory-efficient MHASetup: A100 on f16, measured total time for a forward+backward pass
Note that this is exact attention, not an approximation, just by calling xformers.ops.memory_efficient_attention
More benchmarks
xFormers provides many components, and more benchmarks are available in BENCHMARKS.md.
(Optional) Testing the installation
This command will provide information on an xFormers installation, and what kernels are built/available:
python -m xformers.info
Using xFormers
Key Features
Optimized building blocks, beyond PyTorch primitives
Memory-efficient exact attention - up to 10x faster
sparse attention
block-sparse attention
fused softmax
fused linear layer
fused layer norm
fused dropout(activation(x+bias))
fused SwiGLU
Install troubleshooting
If importing xformers.ops warns that mslk could not be imported, or that its native library could not be loaded, install mslk from the PyTorch index matching your PyTorch build (see above). If mslk's native library can't be loaded, xFormers falls back to mslk's python-only mode, in which the default attention backends still work on CUDA.
On Windows with CUDA 12.x, there are no mslk builds, but you can install mslk's python-only build from source instead (nothing is compiled). The default attention backends work with it:
:: Replace cu126 with the CUDA version of your PyTorch. In PowerShell, use: $env:MSLK_PYTHON_ONLY = "1"
pip install torch flash-attn-3 --index-url https://download.pytorch.org/whl/cu126
set MSLK_PYTHON_ONLY=1
pip install https://github.com/meta-pytorch/MSLK/archive/refs/heads/main.zip
pip install -U xformers --index-url https://download.pytorch.org/whl/cu126
License
xFormers has a BSD-style license, as found in the LICENSE file.
It includes code from the triton-lang/kernels repo.
Citing xFormers
If you use xFormers in your publication, please cite it by using the following BibTeX entry.
@Misc{xFormers2022,
author = {Benjamin Lefaudeux and Francisco Massa and Diana Liskovich and Wenhan Xiong and Vittorio Caggiano and Sean Naren and Min Xu and Jieru Hu and Marta Tintore and Susan Zhang and Patrick Labatut and Daniel Haziza and Luca Wehrstedt and Jeremy Reizenstein and Grigory Sizov},
title = {xFormers: A modular and hackable Transformer modelling library},
howpublished = {\url{https://github.com/facebookresearch/xformers}},
year = {2022}
}
Credits
The following repositories are used in xFormers, either in close to original form or as an inspiration:
TopGit's last sync did not record any GitHub topics for facebookresearch/xformers. GitHub topics appear in the right sidebar of a repository page; that's the authoritative place to check.
How active is development on facebookresearch/xformers?
The most recent commit recorded on facebookresearch/xformers was 7 days ago, based on the GitHub push timestamp. The repository has 798 forks — one of the better signals of community interest.
How many stars does facebookresearch/xformers have?
facebookresearch/xformers has 10.6k GitHub stars — refresh the page for the live number, or check github.com/facebookresearch/xformers. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What is facebookresearch/xformers?
facebookresearch/xformers (facebookresearch/xformers) is a Python project on GitHub. From the project's own README: Hackable and optimized Transformers building blocks, supporting a composable construction.
What language is facebookresearch/xformers written in?
facebookresearch/xformers is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
Read full README in the tab above.
Curious whether xformers is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about xformers.