nanoGPT: Minimal GPT Training in PyTorch

What is nanoGPT for language model development?

nanoGPT is a compact PyTorch repository for training and finetuning medium-sized GPT models, built as a rewrite of Karpathy's earlier minGPT project that trades some of minGPT's teaching clarity for raw training speed. The entire model lives in a roughly 300-line model.py and the training loop in a roughly 300-line train.py, and both can optionally load pretrained GPT-2 weights released by OpenAI.

Core features of nanoGPT

• ✓train.py is a boilerplate training loop at roughly 300 lines, and model.py defines the GPT architecture at about the same length, per the README.
• ✓Reproduces GPT-2 (124M) on the OpenWebText dataset, reaching a loss of about 2.85 after roughly 4 days of training on one 8xA100 40GB GPU node, per the README's own numbers.
• ✓Loads pretrained OpenAI GPT-2 checkpoints directly, including the four public sizes 'gpt2', 'gpt2-medium', 'gpt2-large', and 'gpt2-xl', so finetuning doesn't require training from a blank model.
• ✓Ships ready-made config files for different scenarios, from a CPU-only character-level Shakespeare run to a full multi-node GPT-2 reproduction via torchrun.
• ✓Uses PyTorch 2.0's torch.compile by default, which the README says cut iteration time from about 250ms to 135ms per step on its reference setup.
• ✓bench.py isolates the core training-loop math for benchmarking and profiling without the surrounding training complexity.
• ✓sample.py generates text from either a self-trained checkpoint or an OpenAI-released GPT-2 model, including gpt2-xl, from the command line.

Setting up nanoGPT for local use

Install the dependencies with a single pip command: `pip install torch numpy transformers datasets tiktoken wandb tqdm`. Per the README, transformers is only needed to load GPT-2 checkpoints from Hugging Face, datasets is only needed if you want to download and preprocess OpenWebText, wandb is optional logging, and tqdm just drives the progress bars. There's no separate setup.py step described in the README beyond that pip install line; you clone the repo and start running scripts like train.py directly.

Training a character-level GPT example

The fastest path in, per the README, is training a compact character-level GPT using text from Shakespeare's plays. Run `python data/shakespeare_char/prepare.py` to download a 1MB text file and turn it into train.bin and val.bin. With a GPU, `python train.py config/train_shakespeare_char.py` trains a 6-layer, 6-head Transformer with a 256-character context and 384 feature channels; on one A100 the README reports this finishes in about 3 minutes with a best validation loss of 1.4697. Sample from the result with `python sample.py --out_dir=out-shakespeare-char`. On a MacBook or other CPU-only machine, the README gives a scaled-down command (`--device=cpu --compile=False --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000`) that also finishes in about 3 minutes but lands at a loss of 1.88; Apple Silicon users can add `--device=mps` for a reported 2-3x speedup. To reproduce GPT-2 (124M) on OpenWebText instead, run `python data/openwebtext/prepare.py` to tokenize the dataset, then launch `torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py` on an 8xA100 40GB node; the README says this takes about 4 days and reaches a loss near 2.85. Finetuning a pretrained checkpoint reuses the same train.py entry point with a smaller learning rate, for example `python train.py config/finetune_shakespeare.py`, which the README says can finish in just a few minutes on a single GPU because the tiny Shakespeare dataset tokenizes in seconds rather than requiring the OpenWebText download.

Recommended alternatives to nanoGPT

Common questions about nanoGPT

The problem it solves

Most repos aiming to teach GPT internals sacrifice speed for readability, and most repos built for real training speed bury the model behind abstraction layers you have to peel back to see what's happening. nanoGPT's README frames its own reason for existing around that gap: it's a rewrite of Karpathy's earlier minGPT project that trades some of minGPT's teaching clarity for raw training speed, keeping the model and training loop each around 300 lines while still being able to reproduce a full GPT-2 training run.

Best use cases

• •Learning how a GPT training loop actually works end to end, since model.py and train.py are each small enough per the README to read start to finish in one sitting.
• •Training a small character-level language model on your own text corpus using the Shakespeare workflow as a template, on a GPU, a CPU, or an Apple Silicon Mac.
• •Finetuning an existing OpenAI GPT-2 checkpoint (up to the 1.3B gpt2-xl size, per the README) on a narrow dataset with just a few minutes of GPU time.
• •Studying nanoGPT's design as historical context before moving to Karpathy's newer nanochat project, since the README itself frames nanoGPT as superseded.

Who should try it — and who should skip

Try nanoGPT if you want a self-contained codebase to read, modify, or fork for your own GPT training experiments, especially if you value being able to trace every line rather than importing a black-box library. Skip it if you're starting a new project today: the README itself says nanoGPT is deprecated and to look at nanochat instead, and skip it too if you don't have GPU access, since the character-level demo is the only path that comfortably runs on a CPU or Apple Silicon Mac.