LLMs-from-scratch: Build LLM PyTorch Step by Step
/LLMs-from-scratch
Build a Large Language Model (From Scratch) is Sebastian Raschka's code companion to his Manning book of the same name, walking through pretraining and finetuning a GPT-style model chapter by chapter in PyTorch. Reach for it if you want to write every layer of a transformer yourself and understand why it works, not just call an API. Skip it if you want a production LLM stack, since the repository is built for learning, not deployment.
What is it?
Build a Large Language Model (From Scratch) is the official GitHub code repository for Sebastian Raschka's Manning book of the same title, ISBN 9781633437166. It contains Jupyter notebooks and Python scripts that build a GPT-like model from raw text handling through pretraining and two kinds of finetuning, organized into seven numbered chapters plus five appendices.
Who Benefits from This Learning Resource
Developers who already know Python well and want to understand transformer internals instead of importing a library get the most out of this repo, since the README lists a strong Python foundation as the main prerequisite and treats prior deep-learning experience as a bonus, not a requirement. Appendix A gives a from-scratch PyTorch primer for readers new to the framework, so you don't need PyTorch experience going in, just patience for reading tensor code line by line. Skip it if you want a ready-made model API instead of a chapter-by-chapter build.
Comprehensive LLM Implementation Topics
- โChapter 2 covers text data handling: tokenization, byte pair encoding, and building data loaders for training.
- โChapter 3 codes attention mechanisms from scratch, including a multi-head attention implementation.
- โChapter 4 implements a full GPT-style model architecture in PyTorch, with a standalone gpt.py script.
- โChapter 5 pretrains the model on unlabeled text and includes bonus material on loading pretrained weights and converting the GPT model to a Llama architecture.
- โChapter 6 finetunes the model for text classification; Chapter 7 finetunes it to follow instructions, including a DPO (Direct Preference Optimization) notebook for alignment.
- โAppendix E covers LoRA, a parameter-efficient finetuning method, and Appendix D adds extras like learning rate schedulers to the training loop.
- โBonus notebooks outside the numbered chapters rebuild newer architectures from scratch too, including Qwen3, Gemma 3, Olmo 3, and Llama 3.2 variants, plus mechanisms like KV caching and grouped-query attention.
Strengths
- โChapter-by-chapter structure ties directly to a published Manning book, so the code has a fixed reference and a companion video course rather than being a loose pile of scripts.
- โBuilding attention, the GPT architecture, pretraining, and two finetuning approaches from scratch forces an understanding library shortcuts usually hide.
- โRuns on ordinary laptops for the main chapters and automatically uses a GPU if one's available, so you don't need a cluster to follow along.
- โBonus material stays current with newer open architectures: Qwen3, Gemma 3, Olmo 3, and Llama 3.2 rebuilds sit alongside the core seven chapters.
- โEvery chapter includes exercises with solutions summarized in Appendix C inside the repo, plus an optional 170-page self-test PDF you can download from Manning for extra practice.
Scope and Contribution Guidelines
- โณSebastian Raschka states plainly that the code exists to teach LLM mechanics, producing a compact model meant for educational purposes, not one sized or optimized for production use.
- โณThe maintainer won't accept pull requests that extend or change the main chapter code, since deviating from the printed book would break the one-to-one mapping readers rely on.
- โณThe license isn't clearly documented here. Confirm the exact terms on the GitHub repository page before reusing any of the code.
- โณSome of the study material sits outside the repository itself: the 170-page self-test PDF and parts of the book text are distributed through Manning, not bundled with the code.
Related Learning Resources
Frequently Asked Questions
Build a Large Language Model (From Scratch) treats solid Python skills as its core requirement before you start. The README notes that prior exposure to deep neural networks helps but isn't required, and PyTorch familiarity is useful rather than mandatory since Appendix A introduces PyTorch basics.
The main chapters of Build a Large Language Model (From Scratch) run comfortably on an everyday laptop in a reasonable amount of time, with no specialized hardware needed. The code automatically switches to a GPU when one is available, per the README.
Build a Large Language Model (From Scratch) is built to teach the mechanics behind a GPT-like model, not to hand you a production system. The README frames the resulting model as compact and meant for educational purposes, so treat this as a study project rather than a deployment-ready LLM stack.
The rasbt/LLMs-from-scratch repository is the official code companion for Sebastian Raschka's Manning book Build a Large Language Model (From Scratch), ISBN 9781633437166. Every notebook and script maps to a chapter or appendix in the printed book, and a separate 17-hour, 15-minute video course mirrors the same structure.
Pull requests that extend or change the main chapter code aren't accepted for Build a Large Language Model (From Scratch), since deviating from the printed book would break the mapping between the two. The README does welcome general feedback and questions through GitHub Discussions or the Manning Forum instead.
The license isn't clearly documented in the facts available for Build a Large Language Model (From Scratch), so confirm the exact terms on the GitHub repository page before reusing the code elsewhere.
Best use cases
- โขWorking through Build a Large Language Model (From Scratch) chapter by chapter alongside the book or the companion video course, running each notebook as you read.
- โขUsing it as a hands-on PyTorch refresher: Appendix A's from-scratch tensor and training-loop code doubles as a standalone PyTorch primer.
- โขExtending the bonus notebooks to experiment with newer architectures the README lists as drop-in comparisons, such as Qwen3, Gemma 3, and Llama 3.2 rebuilt from scratch.
- โขTesting your understanding with the roughly 30 quiz questions per chapter in the free 170-page self-test PDF Manning distributes alongside the book.
Related repositories
Want a second opinion on LLMs-from-scratch?
Ask an AI that can read this page โ one click and you get its take on LLMs-from-scratch.