autoresearch: Autonomous AI Research for LLMs

How AI Agents Drive Research

autoresearch pairs a coding agent such as Claude or Codex with a simplified, single-GPU version of Karpathy's nanochat training code, so the agent can experiment without a person touching any Python file directly. You point it at program.md, a lightweight instructions file, and it edits train.py, runs a fixed 5-minute pass, checks val_bpb (validation bits per byte), and decides whether to keep or drop the change before starting the next one.

The Challenge of Manual LLM Tuning

Iterating on a training setup usually means a person babysitting one run at a time, deciding what to try next, and staying at the keyboard for hours to get through a handful of experiments. autoresearch hands that decision loop to a coding agent instead: give it a small, real single-GPU nanochat setup, a hard 5-minute time budget per run, and a single metric, val_bpb, to judge itself against, so it can work through roughly a dozen experiments an hour without anyone watching.

Key Design Principles

• ✓Three files with fixed roles: prepare.py handles data prep, tokenizer training, and evaluation utilities and is never touched; train.py holds the GPT model, the optimizer (Muon plus AdamW), and the training loop, and is the only file the agent edits.
• ✓A fixed 5-minute wall-clock budget per training run, excluding startup and compilation, which the README says works out to roughly a dozen experiments an hour and close to 100 overnight.
• ✓val_bpb (validation bits per byte) as the single scoring metric, picked because it stays comparable across architecture changes regardless of vocabulary size.
• ✓program.md is the file a human edits to steer the agent; the README calls it a lightweight skill rather than a config file.
• ✓Self-contained by design: dependencies are limited to PyTorch and a handful of small packages, with no distributed training or extra config system.
• ✓Training code is a simplified, single-GPU cut of Karpathy's own nanochat project, not a separate reimplementation.

Quick Start: Setting Up Autoresearch

The README's requirements: one NVIDIA GPU (the maintainer's own testing was on an H100), Python 3.10+, and the uv project manager. Four commands, start to finish. Install uv (`curl -LsSf https://astral.sh/uv/install.sh | sh`), install dependencies (`uv sync`), run the one-time data and tokenizer prep (`uv run prepare.py`, about 2 minutes), then run one training pass by hand to confirm it works (`uv run train.py`, about 5 minutes). Once that succeeds, open the repo in Claude, Codex, or a similar coding agent, turn off its permission prompts, and ask it to read program.md and start an experiment.

Community-Maintained Forks

Frequently Asked Questions

Best use cases

• •Running an unattended tuning session overnight on a spare GPU to see what a coding agent converges on when it's scored only by val_bpb.
• •Studying how an agent behaves inside a narrow, well-defined loop (edit train.py, run 5 minutes, check one number, repeat) instead of an open-ended coding task.
• •Starting from this repo to build a platform-specific fork, the way the MacOS, Windows, and AMD forks already listed in the README have done.
• •Looking at a stripped-down, single-GPU version of the nanochat training loop without the broader platform-support code the parent project carries.

Who should try it — and who should skip

Try autoresearch if you have a spare NVIDIA GPU and want to watch, or steer, a coding agent run its own overnight tuning loop against a real, small nanochat-based setup. It's also a fair weekend project to fork for your own hardware, since people already have for MacOS, Windows, and AMD. Skip it if you need a general-purpose training framework, don't have GPU access, or want something more finished than a bare-bones baseline you're expected to extend yourself.

Related repositories