facebookresearch/xformers là dự án Python với 10.6k sao. Hackable and optimized Transformers building blocks, supporting a composable construction.
Tóm tắt dựng từ metadata GitHub của chính dự án — chưa có bài review TopGit. Trang sẽ tự động cập nhật khi bài review đầy đủ được xuất bản.
VÌ SAO CHƯA CÓ REVIEW
TopGit viết bài đầy đủ cho repo có nhiều sao nhất và được yêu cầu nhiều nhất. Trang này là snapshot trong thời gian chờ — xem README gốc ở tab READ ME.
xFormers - Toolbox to Accelerate Research on Transformers
xFormers is:
Customizable building blocks: Independent/customizable building blocks that can be used without boilerplate code. The components are domain-agnostic and xFormers is used by researchers in vision, NLP and more.
Research first: xFormers contains bleeding-edge components, that are not yet available in mainstream libraries like PyTorch.
Built with efficiency in mind: Because speed of iteration matters, components are as fast and memory-efficient as possible. xFormers relies on optimized kernels from MSLK, FlashAttention, PyTorch and Triton.
Installing xFormers
xFormers is pure Python, and runs on any PyTorch 2.10+. Its attention kernels come from the mslk package.
(RECOMMENDED) Install latest stable from the PyTorch index matching your PyTorch build:
# Replace cu132 with the CUDA (e.g. cu130) or ROCm (e.g. rocm7.14) version of your PyTorch.
# flash-attn-3 is optional: it speeds up attention on Ampere and Hopper GPUs. It is only
# available with CUDA (on Windows: only with CUDA 12.x!), so leave it out otherwise.
pip install -U xformers mslk flash-attn-3 --index-url https://download.pytorch.org/whl/cu132
Each mslk release targets a specific PyTorch version: for mslk's native kernels to work (some ROCm backends need them), install versions of torch and mslk from the same row of its compatibility table.
Install from source:
# Install mslk from the PyTorch index matching your PyTorch build, as above
pip install mslk --index-url https://download.pytorch.org/whl/cu132
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
Benchmarks
Memory-efficient MHASetup: A100 on f16, measured total time for a forward+backward pass
Note that this is exact attention, not an approximation, just by calling xformers.ops.memory_efficient_attention
More benchmarks
xFormers provides many components, and more benchmarks are available in BENCHMARKS.md.
(Optional) Testing the installation
This command will provide information on an xFormers installation, and what kernels are built/available:
python -m xformers.info
Using xFormers
Key Features
Optimized building blocks, beyond PyTorch primitives
Memory-efficient exact attention - up to 10x faster
sparse attention
block-sparse attention
fused softmax
fused linear layer
fused layer norm
fused dropout(activation(x+bias))
fused SwiGLU
Install troubleshooting
If importing xformers.ops warns that mslk could not be imported, or that its native library could not be loaded, install mslk from the PyTorch index matching your PyTorch build (see above). If mslk's native library can't be loaded, xFormers falls back to mslk's python-only mode, in which the default attention backends still work on CUDA.
On Windows with CUDA 12.x, there are no mslk builds, but you can install mslk's python-only build from source instead (nothing is compiled). The default attention backends work with it:
:: Replace cu126 with the CUDA version of your PyTorch. In PowerShell, use: $env:MSLK_PYTHON_ONLY = "1"
pip install torch flash-attn-3 --index-url https://download.pytorch.org/whl/cu126
set MSLK_PYTHON_ONLY=1
pip install https://github.com/meta-pytorch/MSLK/archive/refs/heads/main.zip
pip install -U xformers --index-url https://download.pytorch.org/whl/cu126
License
xFormers has a BSD-style license, as found in the LICENSE file.
It includes code from the triton-lang/kernels repo.
Citing xFormers
If you use xFormers in your publication, please cite it by using the following BibTeX entry.
@Misc{xFormers2022,
author = {Benjamin Lefaudeux and Francisco Massa and Diana Liskovich and Wenhan Xiong and Vittorio Caggiano and Sean Naren and Min Xu and Jieru Hu and Marta Tintore and Susan Zhang and Patrick Labatut and Daniel Haziza and Luca Wehrstedt and Jeremy Reizenstein and Grigory Sizov},
title = {xFormers: A modular and hackable Transformer modelling library},
howpublished = {\url{https://github.com/facebookresearch/xformers}},
year = {2022}
}
Credits
The following repositories are used in xFormers, either in close to original form or as an inspiration:
facebookresearch/xformers có 10.6k sao GitHub — tải lại trang để xem số mới nhất, hoặc xem trực tiếp github.com/facebookresearch/xformers. TopGit phản chiếu số sao của GitHub nhưng không cam kết đến từng phút.
facebookresearch/xformers có tag gì không?
Bản đồng bộ chưa ghi nhận topic GitHub nào cho facebookresearch/xformers. GitHub topics hiển thị ở thanh bên phải trang repo — đó là nơi đáng kiểm tra nhất.
facebookresearch/xformers còn đang phát triển không?
Commit gần nhất trên facebookresearch/xformers là 7 ngày trước (theo timestamp GitHub). Repo có 798 fork — một chỉ báo về mức độ quan tâm của cộng đồng.
facebookresearch/xformers là gì?
facebookresearch/xformers (facebookresearch/xformers) là dự án Python trên GitHub. Theo mô tả gốc: Hackable and optimized Transformers building blocks, supporting a composable construction.
facebookresearch/xformers viết bằng ngôn ngữ gì?
facebookresearch/xformers chủ yếu viết bằng Python. Trường "language" của GitHub dựa trên phần lớn byte ở nhánh mặc định.
Đọc đầy đủ README ở tab phía trên.
Muốn nghe thêm một ý kiến về xformers?
Hỏi một AI đọc được trang này — một cú bấm là có ngay nhận định về xformers.