Happy-LLM: A Free Tutorial to Build an LLM From Scratch
/happy-llm
Happy-LLM is worth starting if you want to understand how LLMs are actually trained, not just how to call one through an API. It beats blog-post tutorials because every chapter pairs theory with a runnable notebook, down to training a small LLaMA2 checkpoint yourself. This is not a quick read. Skip it for a weekend project — the eight-chapter arc assumes you'll sit with pretraining and fine-tuning code, not skim slides.
What is Happy-LLM?
Happy-LLM is a Datawhale-maintained curriculum for learning how large language models work by building one yourself. Its eight chapters move from NLP fundamentals and the Transformer architecture through pretraining, supervised fine-tuning with LoRA and QLoRA, RAG, Agent workflows, and Agentic RL methods like GRPO and Search-R1. The main chapters are written in Chinese, with an English README available.
Why Happy-LLM reached 30k+ stars
Happy-LLM's growth traces back to a gap Datawhale spotted in its own community: readers of the earlier self-llm project, which teaches how to run existing open models, kept asking how those models are actually trained. Happy-LLM answers that by pairing every chapter's theory with code you run yourself, down to training a small LLaMA2 checkpoint and publishing it on ModelScope so learners have a working baseline to compare against. That baseline matters. The project also ships a free PDF export and a companion PPT slide deck, which makes it easy to reference offline or reuse in a classroom — a real driver of how it spreads through Chinese AI-learning communities.
What Happy-LLM covers
- ✓Eight chapters moving from NLP basics and Transformer internals through pretraining, fine-tuning, RAG, Agents, and Agentic RL methods (GRPO, OPD, Search-R1, ReTool).
- ✓A hands-on LLaMA2 build in Chapter 5 — you implement and train the tokenizer and a small model yourself in PyTorch rather than importing one.
- ✓Chapter 6 switches to the Hugging Face Transformers framework and walks through supervised fine-tuning plus LoRA and QLoRA.
- ✓Two trained checkpoints (Base and SFT, both 215M parameters) published on ModelScope as a baseline to compare your own training run against.
- ✓A free PDF export, watermarked only to deter resellers, plus a companion PPT slide deck for offline study or teaching.
- ✓An Extra Chapter section of community-contributed posts, covering topics like small-model fine-tuning and Qwen3 multimodal work, still marked in progress.
Who is Happy-LLM for?
Happy-LLM suits university students, researchers, and LLM enthusiasts who already have some programming experience — the README specifically calls out Python fluency as a prerequisite. A background in deep learning and familiarity with NLP terminology help too, since the early chapters move through those ideas quickly. If you can't read Chinese, expect to lean on translation for anything past the README, since only the top-level README ships in English.
What you will build
- •Implement and pretrain a small LLaMA2-architecture model from raw code instead of importing a pre-built library.
- •Fine-tune a language model with LoRA or QLoRA and understand what each step in the pipeline does.
- •Build a RAG pipeline and a basic Agent following the Chapter 7 walkthrough.
- •Get a first pass on GRPO, OPD, Search-R1, and ReTool before going deeper in the author's separate agentic-rl-lab repo.
Strengths
- ✓Pairs theory chapters with code you actually run, down to training a small model rather than just reading about the process.
- ✓Published checkpoints on ModelScope give you a concrete baseline to check your own training results against.
- ✓Free PDF and PPT downloads make it usable offline or as classroom material without extra setup.
- ✓Chapter dependencies are split so you can use a separate Python environment per chapter, cutting down on version conflicts.
Limitations of this tutorial
- △Main content is written in Chinese; only the top-level README has an English version, so non-Chinese readers lean on translation for the actual chapters.
- △The Extra Chapter blog section is explicitly marked in progress, so parts of the curriculum are still being filled in.
- △The content license (CC BY-NC-SA 4.0) blocks commercial use of the material, which matters if you plan to reuse chapters in a paid course.
- △Chapter 8's Agentic RL coverage is introductory — the README points readers to a separate, more frequently updated repo for deeper work on those algorithms.
Alternatives to Happy-LLM
Frequently asked questions
Happy-LLM is completely free — the README describes it as fully open for study, and its PDF export is explicitly free, with a watermark added only to stop resellers from selling it to beginners. The content itself is licensed CC BY-NC-SA 4.0, which forbids commercial use of the material.
self-llm is Datawhale's earlier project on deploying and using existing open-source LLMs. Happy-LLM was created afterward for readers who finished self-llm and wanted to understand how those models are actually built and trained, so it goes deeper into architecture, pretraining, and fine-tuning from scratch rather than model usage.
The README recommends some programming experience, particularly in Python, before starting Happy-LLM. Background knowledge of deep learning plus familiarity with NLP concepts and terminology also helps, since the early chapters build on those ideas quickly rather than re-teaching them.
Yes, Happy-LLM includes hands-on code exercises. Chapter 5 has you implement and train a LLaMA2-style model and its tokenizer in PyTorch, and Chapter 6 walks through pretraining, supervised fine-tuning, and LoRA/QLoRA using the Hugging Face Transformers framework.
Happy-LLM's content is licensed under Creative Commons BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike), as stated in the repository's license section. That permits sharing and adapting the material with attribution, but blocks commercial use and requires derivatives to carry the same license.
Yes, Happy-LLM covers both. Chapter 6 walks through supervised fine-tuning plus LoRA and QLoRA for efficient fine-tuning, and Chapter 7 covers retrieval-augmented generation (RAG) alongside model evaluation and Agent workflows.
Related repositories
Want a second opinion on happy-llm?
Ask an AI that can read this page — one click and you get its take on happy-llm.