AirLLM: Run 70B LLMs on a 4GB GPU
AirLLM lets you run a 70B Llama model on a single 4GB GPU by loading one transformer layer onto the card at a time instead of the whole model. Reach for it when you have a beefy disk and patience for slow, layer-by-layer decoding; skip it if you need fast interactive throughput, since streaming weights off disk every token is the trade-off, not a free lunch.
What is AirLLM?
AirLLM is a Python library that cuts GPU memory needs for large language model inference by decomposing a model into layers and loading them onto the GPU one at a time instead of all at once. It ships as the airllm pip package and exposes an AutoModel.from_pretrained() call modeled on Hugging Face Transformers, so swapping in a Hugging Face repo ID is close to the extent of the integration work for LLM inference.
How AirLLM Optimizes VRAM
- ✓Layer-by-layer streaming: only one transformer layer sits on the GPU at any moment, so VRAM use tracks a single layer's size, not the model's total parameter count.
- ✓Instead of streaming complete layers, Sparse MoE models process a single expert at a time, which is how AirLLM fits Kimi K3 (2.8T parameters) into under 4GB — measured at 3.72GB on an RTX 6000 Ada.
- ✓Optional block-wise quantization compression ('4bit' or '8bit') shrinks the on-disk model further and, per the README, speeds inference up to 3x with what the README calls 'almost ignorable accuracy loss.'
- ✓Prefetching overlaps model loading with compute, which the changelog credits with a 10% speed improvement.
- ✓A one-time layer-wise decomposition step splits and safetensors-saves the original checkpoint to disk before the first inference call, so the initial run needs real free disk space.
- ✓delete_original lets you drop the original downloaded Hugging Face checkpoint after conversion, keeping only the layered copy to save disk space.
Supported Models and GPU Requirements
- •Running Llama (2, 3, 3.1, 3.3, 4) family models on a single consumer GPU instead of a multi-GPU rig.
- •Loading Qwen (1, 2, 2.5, 3, including MoE and FP8 variants) checkpoints for local inference.
- •Running DeepSeek V2, V3, and R1, including DeepSeek-V3 (671B parameters) at roughly 12GB VRAM per the README's table.
- •Trying Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, or Yi models through the same AutoModel call.
- •Fitting Llama 3.1 405B on around 8GB VRAM, or a Llama 3.x 70B model at full precision on about 4GB, per the README's VRAM table.
Getting Started with AirLLM
Install is documented as a single pip command: `pip install airllm`. For the model-compression speed-up, the README adds a second dependency: `pip install -U bitsandbytes`, plus making sure the installed airllm version is later than 2.0.0 (`pip install -U airllm`). MacOS support needs mlx and torch installed separately, and only runs on Apple silicon; the README also flags that some MacOS setups need a native Python install rather than the Homebrew build. Kimi K3 support pulls in further packages — `pip install compressed-tensors flash-attn`, a CUDA 12 build of torch, and transformers pinned to the 4.56.x line, since the model's remote code doesn't load on transformers 5.x.
Performing LLM Inference
Usage is documented with runnable Python: `from airllm import AutoModel`, then `model = AutoModel.from_pretrained("Qwen/Qwen3-32B")` — any Hugging Face repo ID or a local path works the same way. Tokenize input with the model's own `model.tokenizer(...)`, call `model.generate(input_tokens['input_ids'].cuda(), max_new_tokens=20, use_cache=True, return_dict_in_generate=True)`, then decode with `model.tokenizer.decode(...)`. The README notes that the first call decomposes and saves the checkpoint layer-by-layer to disk before generation starts, so budget disk space equal to roughly the model's full size on that first run. Gated Hugging Face models take an `hf_token` argument at load time.
Strengths
- ✓One `AutoModel.from_pretrained()` call covers Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi — no per-family model class to pick.
- ✓The API mirrors Hugging Face Transformers closely enough that existing tokenizer and generate() calls need almost no rewriting.
- ✓GPU requirements scale with a model's layer size instead of its total parameter count, which is why a 671B DeepSeek-V3 checkpoint fits in roughly 12GB per the README's table.
- ✓Optional 4-bit/8-bit block-wise compression is opt-in, not required — you can run the uncompressed path first and add compression only if disk or speed becomes a problem.
Considerations and Requirements
- △The first run re-saves the entire checkpoint layer-by-layer to disk before any generation happens, so disk space — not VRAM — is the real bottleneck the README calls out.
- △Layer streaming trades speed for memory: the README's own numbers describe it as disk-bound, and the FAQ's #1 listed error (MetadataIncompleteBuffer) is caused by running out of that disk space mid-split.
- △MacOS support is Apple-silicon only and needs mlx installed alongside torch, with the README warning some setups also need a native (not Homebrew) Python install.
- △The newest model support isn't free of extra setup: Kimi K3 needs compressed-tensors, flash-attn, a CUDA 12 torch build, and transformers pinned to 4.56.x — none of which apply to the base install.
Other LLM Memory Optimization Methods
Common Questions
AirLLM's error `safetensors_rust.SafetensorError: ... MetadataIncompleteBuffer` almost always means the disk ran out of space during the layer-splitting step, since that process is disk-heavy. The README's fix is to free up disk space, clear the Hugging Face cache, and rerun.
This AirLLM error usually means you loaded a QWen or ChatGLM checkpoint with the AirLLMLlama2 class. Switch to `from airllm import AutoModel` and call `AutoModel.from_pretrained(...)` instead, which the README lists as the fix for both model families.
Pass your Hugging Face access token as the `hf_token` argument when calling `AutoModel.from_pretrained(...)`, for example `AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", hf_token='HF_API_TOKEN')`. Without it, AirLLM raises a 401 error on gated repos like that one.
AirLLM can raise a `ValueError` about a missing padding token on some models. The README fix is to pass `padding=False` in the `model.tokenizer(...)` call rather than setting a padding token.
The problem it solves
Running a 70B-parameter model normally means owning or renting multiple high-VRAM GPUs, since the full set of weights has to sit in GPU memory at once. AirLLM's specific angle is that most of that memory pressure comes from holding every layer resident simultaneously, not from the compute itself, so its GPU memory optimization works by changing what's resident at once rather than shrinking the weights — which is what turns a 4GB card into something that can finish a 70B forward pass at all.
Who should try it — and who should skip
Try AirLLM if you're experimenting with a 70B+ model on a single consumer GPU and have fast local disk plus patience for slower, disk-bound token generation — students, hobbyists, and researchers testing whether a model even fits before renting cloud GPUs. Skip it if you're serving production traffic where throughput and latency matter, or if you're already running quantized GGUF models through Ollama or llama.cpp and don't need full-precision weights.