nanochat: train your own GPT-2 for about $48
nanochat is Andrej Karpathy's minimal harness for training a GPT-2-class chat model on a single GPU node, and it earns its reputation by shipping the full pipeline in one readable codebase instead of a sprawling framework. There's no config-object maze here: one `--depth` knob resizes the whole model. That focus makes nanochat a genuinely good teaching and tinkering tool, not a production serving stack.
What is nanochat
nanochat is a from-scratch codebase for training a small ChatGPT-style language model on a single GPU node, covering tokenization, pretraining, finetuning, evaluation, and inference in one place. One dial, `--depth`, sets the transformer's size and derives the rest of the hyperparameters automatically. The reference `runs/speedrun.sh` script runs that pipeline on an 8xH100 node and hands you a CLI to chat with when it's done.
What nanochat covers end-to-end
- ✓Full pipeline in one repo: tokenizer training (BPE, GPT-4 style), pretraining, supervised finetuning, RL, evaluation, and inference all live under `nanochat/` and `scripts/`.
- ✓One complexity dial: setting `--depth` on the transformer automatically derives width, attention heads, learning rate schedule, training horizon, and weight decay for a compute-optimal model at that size.
- ✓Explicit precision control instead of `torch.amp.autocast` — a single `COMPUTE_DTYPE` global (bfloat16, float32, or float16) that auto-detects your hardware and can be overridden with the `NANOCHAT_DTYPE` environment variable.
- ✓Built-in evaluation tasks — ARC, GSM8K, HumanEval, MMLU, and SmolTalk — wired up in `tasks/` for scoring the chat model, plus a DCLM CORE score evaluator for the base model.
- ✓A KV-cache inference engine (`nanochat/engine.py`) and a tool-execution module (`nanochat/execution.py`) that lets the trained model run Python code as a tool.
- ✓A public GPT-2 speedrun leaderboard tracking wall-clock time and DCLM CORE score for reproducing GPT-2-grade capability, with commits and contributors listed per entry.
Getting set up with uv
nanochat uses uv for dependency management rather than a plain requirements.txt or Conda environment. Run `uv sync --extra gpu` on a CUDA box (A100, H100, etc.) or `uv sync --extra cpu` for CPU-only or Apple Silicon, then `source .venv/bin/activate`. If you're going to touch the code, add `--group dev` to the gpu sync command, which pulls in pytest, matplotlib, ipykernel, and transformers for development work. There's no separate pip package to install — you clone the repo and work inside it.
Running the GPT-2 speedrun script
The whole reproduce-a-GPT-2 workflow lives in one file, `runs/speedrun.sh`, meant to run on an 8xH100 GPU node — boot a box from a GPU cloud provider, run `bash runs/speedrun.sh` inside a screen session, and wait roughly 1.5 hours. Once it finishes, activate the venv again and run `python -m scripts.chat_cli` to talk to your model over a simple command-line chat interface. The same code runs on a single GPU by dropping `torchrun`, falling back to gradient accumulation, but then you're waiting about eight times longer. If your GPU has less than 80GB of VRAM, you'll need to shrink `--device-batch-size` from its default of 32 down to 16, 8, 4, 2, or 1 until it stops running out of memory.
Strengths
- ✓One file (`runs/speedrun.sh`) takes you from a blank GPU box to a chat model you can talk to — no separate data pipeline, tokenizer training, or eval harness to wire up yourself.
- ✓The `--depth` dial removes an entire category of hyperparameter guesswork: pick a model size, and width, heads, learning rate, and weight decay are derived for you.
- ✓Precision is explicit and inspectable (`COMPUTE_DTYPE` in `nanochat/common.py`) instead of hidden inside `autocast`, which matters when you're debugging numerics on unusual hardware.
- ✓The codebase stays small enough to read end to end, which is the whole point if your goal is understanding how pretraining, SFT, and RL actually work rather than treating them as a black box.
What nanochat doesn't do (yet)
- △The README itself says most of the code hasn't been exercised on hardware outside CUDA — xpu and other non-NVIDIA/non-Apple backends are untested and may have sharp edges.
- △float16 training gets automatic GradScaler support for the SFT stage, but the README states RL training currently does not — you're limited to bfloat16 or float32 there.
- △Running on CPU or Apple Silicon (`runs/runcpu.sh`) shrinks the model so drastically that the README warns you won't get strong results. It's a smoke test, not a real training path.
- △This is a single-person research codebase with an explicit anti-framework design: no configuration objects, no model factory, so extending it means editing the scripts directly rather than passing in a config.
Alternatives for small-scale LLM training
Frequently asked questions
nanochat's reference workflow, `runs/speedrun.sh`, is built for an 8xH100 GPU node rented from a cloud provider; the README notes it also runs on an 8xA100 node, just slower. You can run it on a single GPU too, but expect training to take roughly eight times longer since torchrun's parallelism is what speeds up the 8-GPU run.
Training a GPT-2-class model with nanochat's speedrun script takes about 1.5 hours on an 8xH100 node, according to the README, which also notes that figure was down from roughly 3 hours in earlier versions of the script. At current cloud GPU pricing, the README puts that run's compute cost at around $48.
nanochat is released under the MIT license, so you're free to use, modify, and redistribute it, including commercially. The README does ask that pull requests disclose any part written with substantial help from an LLM, but that's a contribution policy, not a restriction on using the code.
Yes — the README states all the code runs on a single GPU by simply omitting `torchrun`, producing nearly identical results through gradient accumulation instead of parallelism, though you'll wait several times longer. There's also a dedicated `runs/runcpu.sh` script for CPU or Apple Silicon, but the README warns it shrinks the model so much that results won't be strong.
nanochat is Andrej Karpathy's follow-up to nanoGPT, and the README describes nanoGPT as covering pretraining only. nanochat extends that scope to the full pipeline — tokenization, pretraining, supervised finetuning, reinforcement learning, evaluation, and inference — so you end up with a chat model you can talk to, not just a base language model checkpoint.
Yes — nanochat's `scripts/` directory includes `chat_sft.py` for supervised finetuning and `chat_rl.py` for reinforcement learning on top of the pretrained base model. The README notes one precision caveat: float16 training gets automatic GradScaler support for SFT, but not yet for RL training.
The problem it solves
Most public LLM training code is either a toy script that trains a tiny model in a notebook, or a production framework like the ones behind real ChatGPT-scale systems, wrapped in config layers built for teams, not one person on a rented GPU box. nanochat sits in the gap: it runs the entire tokenize-to-chat pipeline that produces a real, talkable GPT-2-class model, on hardware one person can rent for a couple of hours, without hiding any of the stages behind abstraction.
Best use cases
- •Learning how a ChatGPT-style model is actually built, stage by stage, by reading and running real training code instead of a diagram or a blog post.
- •Running a fast local experiment loop — the README's example trains a 12-layer, GPT-1-sized model in about 5 minutes to test a code change before committing to a full GPT-2-class run.
- •Reproducing a specific point on the compute-optimal scaling curve for research, using `runs/scaling_laws.sh` or `runs/miniseries.sh` to sweep `--depth` across a family of model sizes.
- •Getting a real, talkable small language model for about $48 in cloud GPU time, without licensing a hosted model or negotiating API access.
Who should try it — and who should skip
nanochat is for developers and researchers who want to actually run every stage of LLM training themselves — tokenizer, pretraining, SFT, RL, and inference — and are comfortable renting an 8xH100 node or waiting out a slower single-GPU run. It's a good fit if you learn by reading dense, working code rather than documentation, and you don't mind that there's no config system to lean on. Skip it if you want a model in production tomorrow, need multi-node or non-NVIDIA hardware support the README doesn't claim, or would rather fine-tune an existing checkpoint through a library like transformers or unsloth than train one from scratch.
Related repositories
Curious whether nanochat is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about nanochat.