linkedin/Liger-Kernel

Liger Kernel: Efficient Triton Kernels for LLM Training

Stable		Nightly		Discord

Installation | Getting Started | Examples | High-level APIs | Low-level APIs | Cite our work

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come. The kernel works out of the box with Flash Attention, PyTorch FSDP, and Microsoft DeepSpeed. We welcome contributions from the community to gather the best kernels for LLM training.

We've also added optimized Post-Training kernels that deliver up to 80% memory savings for alignment and distillation tasks. We support losses like DPO, CPO, ORPO, SimPO, KTO, JSD, and many more. Check out how we optimize the memory.

You can view the documentation site for additional installation, usage examples, and API references:https://linkedin.github.io/Liger-Kernel/

You can view the Liger Kernel Technical Report: https://openreview.net/forum?id=36SjAIT42G

Supercharge Your Model with Liger Kernel

With one line of code, Liger Kernel can increase throughput by more than 20% and reduce memory usage by 60%, thereby enabling longer context lengths, larger batch sizes, and massive vocabularies.

Speed Up	Memory Reduction

Note:

• Benchmark conditions: LLaMA 3-8B, Batch Size = 8, Data Type = bf16, Optimizer = AdamW, Gradient Checkpointing = True, Distributed Strategy = FSDP1 on 8 A100s.
• Hugging Face models start to OOM at a 4K context length, whereas Hugging Face + Liger Kernel scales up to 16K.

Optimize Post Training with Liger Kernel

We provide optimized post training kernels like DPO, ORPO, SimPO, and more which can reduce memory usage by up to 80%. You can easily use them as python modules.

from liger_kernel.chunked_loss import LigerFusedLinearORPOLoss
orpo_loss = LigerFusedLinearORPOLoss()
y = orpo_loss(lm_head.weight, x, target)

Examples

Key Features

• Ease of use: Simply patch your Hugging Face model with one line of code, or compose your own model using our Liger Kernel modules.
• Time and memory efficient: In the same spirit as Flash-Attn, but for layers like RMSNorm, RoPE, SwiGLU, and CrossEntropy! Increases multi-GPU training throughput by 20% and reduces memory usage by 60% with kernel fusion, in-place replacement, and chunking techniques.
• Exact: Computation is exact—no approximations! Both forward and backward passes are implemented with rigorous unit tests and undergo convergence testing against training runs without Liger Kernel to ensure accuracy.
• Lightweight: Liger Kernel has minimal dependencies, requiring only Torch and Triton—no extra libraries needed! Say goodbye to dependency headaches!
• Multi-GPU supported: Compatible with multi-GPU setups (PyTorch FSDP, DeepSpeed, DDP, etc.).
• Trainer Framework Integration: Axolotl, LLaMa-Factory, SFTTrainer, Hugging Face Trainer, SWIFT, oumi

Installation

Dependencies

CUDA

• torch >= 2.1.2
• triton >= 2.3.0

ROCm

• torch >= 2.5.0 Install according to the instruction in Pytorch official webpage.
• triton >= 3.0.0 Install from pypi. (e.g. pip install triton==3.0.0)

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.2

Ascend NPU

• torch == 2.7.1
• torch_npu == 2.7.1
• triton-ascend == 3.2.1 Install from the Ascend PyPI mirror (not on default PyPI).

pip install -e ".[dev]" --extra-index-url https://triton-ascend.osinfra.cn/pypi/simple

Optional Dependencies

• transformers >= 4.x: Required if you plan to use the transformers models patching APIs. The specific model you are working will dictate the minimum version of transformers.
• cuda-tile: Required when enabling the optional cuTile backend on CUDA. Use this when your environment already provides CUDA Toolkit 13.1 or newer, or an existing tileiras compiler installation.
• cuda-tile[tileiras]: Required when enabling the optional cuTile backend with the tileiras compiler installed directly into your Python environment.
• nvidia-cutlass-dsl >= 4.5.2: Required when enabling the optional CuTe DSL backend on CUDA (the CUDA-only Python DSL shipped with NVIDIA CUTLASS, import cutlass.cute). Targets Hopper (SM90) and Blackwell (SM100/SM110).

Note: Our kernels inherit the full spectrum of hardware compatibility offered by Triton.

To install the stable version:

$ pip install liger-kernel

To install the nightly version:

$ pip install liger-kernel-nightly

To install from source:

git clone https://github.com/linkedin/Liger-Kernel.git
cd Liger-Kernel

# Install Default Dependencies
# Setup.py will detect the local backend and select default dependencies.
# On ROCm, install ROCm PyTorch first from the PyTorch ROCm index.
pip install -e .

# Setup Development Dependencies
pip install -e ".[dev]"

# ROCm source installs
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.2
# Then choose one:
pip install -e .
pip install -e ".[dev]"

# Setup cuTile Dependencies
pip install -e ".[cutile]"

# Or install cuTile with the optional tileiras compiler
pip install -e ".[cutile-tileiras]"

# Setup CuTe DSL (NVIDIA CUTLASS Python DSL) Dependencies
pip install -e ".[cutedsl]"

Enable cuTile Backend

cuTile is an optional CUDA-only DSL implementation. After installing the cutile or cutile-tileiras extra, enable it explicitly:

LIGER_KERNEL_IMPL=cutile python your_script.py

LIGER_KERNEL_IMPL selects an opt-in implementation registered with Liger (currently cutile and cutedsl). Selecting one on an unsupported device, or without the required dependencies installed, raises an error.

Enable CuTe DSL Backend

CuTe DSL is the optional, CUDA-only Python DSL shipped with NVIDIA CUTLASS (import cutlass.cute), targeting Hopper (SM90) and Blackwell (SM100/SM110). After installing the cutedsl extra, enable it explicitly:

pip install "liger-kernel[cutedsl]"
LIGER_KERNEL_IMPL=cutedsl python your_script.py

It currently provides genuine cutlass.cute implementations of RMSNorm, cross entropy, and fused scaled cross entropy (LigerFusedLinearScaledCrossEntropyFunction). The frontend dispatches from the input device: Hopper compute capability 9.0 uses LigerFusedScaledCrossEntropySM90Function, while other devices and NVIDIA compute capabilities use a 512-token chunked PyTorch fallback adapted from Verl's fused linear PPO implementation. The scaled operator is an additional operator, not a replacement for the Triton LigerFusedLinearCrossEntropyFunction: it fuses x @ weight.T with the softmax, takes matching floating-point input/weight (BF16 on SM90) plus ignore_index, and returns the per-token negative log-likelihood [M] plus optional vocabulary entropy — reductions are composed in PyTorch. Backward therefore receives the per-token upstream gradient [M], which is used verbatim as the row scale of dZ; a scalar grad_output is rejected rather than silently reduced. temperature defaults to 1.0 and applies the same logits / temperature semantics as Verl. Set return_entropy=True to additionally return differentiable per-token vocabulary entropy in the input dtype; its backward contribution follows Verl's PPO formula. Both NLL and entropy are zeroed for ignore_index rows. On SM90, m_tiles_per_cluster (>= 1, default 1) selects the forward schedule: 1 uses the fastest M-outer self-TMA forward, values > 1 use the M-fast forward with that many live M tiles per cluster. Backward executes dZ, dX = dZ @ W, and dW = dZ.T @ X inside one persistent cluster-2 CUDA kernel. Its device-side wave loop aliases phase-specific shared memory, uses W multicast for dX, transposed TMA for dW, and HBM atomics between waves. Backward uses a fixed reusable BF16 dZ workspace covering 1024 tokens; wave zero plain-stores BF16 dW, and later waves use Hopper BF16 TMA reduce-add. Ops without a CuTe DSL kernel transparently fall back to the default Triton kernel. Selecting the backend on a non-CUDA device, or without nvidia-cutlass-dsl installed, raises an error.

from liger_kernel.ops import LigerFusedLinearScaledCrossEntropyFunction

nll, entropy = LigerFusedLinearScaledCrossEntropyFunction.apply(
    x, weight, target, 1.0, -100, 1, True
)  # [M], [M]
loss = nll.sum() / (target != -100).sum().clamp_min(1)

Getting Started

There are a couple of ways to apply Liger kernels, depending on the level of customization required.

1. Use AutoLigerKernelForCausalLM

Using the AutoLigerKernelForCausalLM is the simplest approach, as you don't have to import a model-specific patching API. If the model type is supported, the modeling code will be automatically patched using the default settings.

from liger_kernel.transformers import AutoLigerKernelForCausalLM

# This AutoModel wrapper class automatically monkey-patches the
# model with the optimized Liger kernels if the model is supported.
model = AutoLigerKernelForCausalLM.from_pretrained("path/to/some/model")

2. Apply Model-Specific Patching APIs

Using the patching APIs, you can swap Hugging Face models with optimized Liger Kernels.

import transformers
from liger_kernel.transformers import apply_liger_kernel_to_llama

# 1a. Adding this line automatically monkey-patches the model with the optimized Liger kernels
apply_liger_kernel_to_llama()

# 1b. You could alternatively specify exactly which kernels are applied
apply_liger_kernel_to_llama(
  rope=True,
  swiglu=True,
  cross_entropy=True,
  fused_linear_cross_entropy=False,
  rms_norm=False
)

# 2. Instantiate patched model
model = transformers.AutoModelForCausalLM("path/to/llama/model")

3. Compose Your Own Model

You can take individual kernels to compose your models.

from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss
import torch.nn as nn
import torch

model = nn.Linear(128, 256).cuda()

# fuses linear + cross entropy layers together and performs chunk-by-chunk computation to reduce memory
loss_fn = LigerFusedLinearCrossEntropyLoss()

input = torch.randn(4, 128, requires_grad=True, device="cuda")
target = torch.randint(256, (4, ), device="cuda")

loss = loss_fn(model.weight, input, target)
loss.backward()

High-level APIs

AutoModel

Patching

Low-level APIs

• Fused Linear kernels combine linear layers with losses, reducing memory usage by up to 80% - ideal for HBM-constrained workloads.
• Other kernels use fusion and in-place techniques for memory and performance optimization.

Model Kernels

Alignment Kernels

Distillation Kernels

Experimental Kernels

Contributing, Acknowledgements, and License

• Contributing Guidelines
• Acknowledgements
• License Information

Sponsorship and Collaboration

• Glows.ai: Sponsoring NVIDIA GPUs for our open source developers.
• AMD: Providing AMD GPUs for our AMD CI.
• Intel: Providing Intel GPUs for our Intel CI.
• Modal: Free 3000 credits from GPU MODE IRL for our NVIDIA CI.
• EmbeddedLLM: Making Liger Kernel run fast and stable on AMD.
• HuggingFace: Integrating Liger Kernel into Hugging Face Transformers and TRL.
• Lightning AI: Integrating Liger Kernel into Lightning Thunder.
• Axolotl: Integrating Liger Kernel into Axolotl.
• Llama-Factory: Integrating Liger Kernel into Llama-Factory.

CI status

Build

Contact

• For issues, create a Github ticket in this repository
• For open discussion, join our discord channel on GPUMode
• For formal collaboration, send an email to Yanning Chen([email protected]) and Zhipeng Wang([email protected])

Cite this work

Biblatex entry:

@inproceedings{
hsu2025ligerkernel,
title={Liger-Kernel: Efficient Triton Kernels for {LLM} Training},
author={Pin-Lun Hsu and Yun Dai and Vignesh Kothapalli and Qingquan Song and Shao Tang and Siyu Zhu and Steven Shimizu and Shivam Sahni and Haowen Ning and Yanning Chen and Zhipeng Wang},
booktitle={Championing Open-source DEvelopment in ML Workshop @ ICML25},
year={2025},
url={https://openreview.net/forum?id=36SjAIT42G}
}

Star History

↑ Back to Top ↑