makemore character-level language model guide
/makemore
makemore is a small Python command-line program for learning how autoregressive character models generate new text from examples. It is worth using for education and experimentation because one script exposes bigram, neural-network, and Transformer approaches without packaging overhead. Skip it when you need a maintained library, pretrained models, or production-serving features.
A minimal character-level language model
makemore is a Python command-line program that learns an autoregressive character-level language model from a plain text file. Each line becomes a training item, and the learned model generates name-like or word-like strings. It is a deliberately compact teaching codebase, not a production text service.
Supported model architectures
- โThe model choices span bigram, MLP, RNN, LSTM, GRU, and Transformer architectures.
- โCharacter-level modeling generates each next token one character at a time rather than starting from whole words.
- โA CNN implementation is included but remains marked as in progress in the README.
- โPyTorch is the only stated software requirement.
- โThe default configuration uses a small 200K-parameter Transformer.
- โTraining progress and generated samples are printed while the model is learning.
- โSaved logs and model files are written to the output directory.
- โThe included names.txt file provides a ready-made example corpus.
Training and sampling names
Train with one item per line in the input file: bash python makemore.py -i names.txt -o names Training progress, logs, and the model are written to the `names` directory. Samples appear during training. To load the best saved model and request more output, run this in another terminal: bash python makemore.py -i names.txt -o names --sample-only Then sample again.
Strengths
- โThe one-file design keeps the complete training and sampling workflow close enough to read in one sitting.
- โPyTorch is the only stated dependency, and a GPU is optional rather than mandatory.
- โArchitecture choices make it possible to compare a simple bigram with MLP, recurrent, convolutional, and Transformer approaches.
- โSamples are visible during training, while the saved best model can generate more output later.
- โMIT licensing gives users broad permission to study, modify, and reuse the code.
- โThe included name dataset makes the basic command usable without preparing an external corpus first.
Designed for learning, not production
- โณThe project is explicitly designed for education, so it omits the packaging, controls, and production behavior expected from a maintained library.
- โณCharacter-level generation works from line-separated examples; it is not intended to model rich document structure or long prose formatting.
- โณThe CNN model is still described as being implemented in progress.
- โณGeneration quality depends on the training corpus and hyperparameters, so useful results may require tuning the available settings.
- โณThere is no documented API, service layer, or deployment workflow.
Choosing a model architecture
Frequently asked questions
makemore is a command-line program, not a library meant for import into another application. The README describes it as one hackable file, with PyTorch as its only stated requirement.
makemore includes bigram, MLP, CNN, RNN, LSTM, GRU, and Transformer architectures. The README marks the CNN implementation as still in progress, while the other listed models describe the intended range from count tables to GPT-style modeling.
Provide a text file containing one name per line, then run `python makemore.py -i names.txt -o names`. Training prints generated candidates and writes the model, logs, and progress into the `names` output directory.
Run `python makemore.py -i names.txt -o names --sample-only` from the project directory. The command loads the best model saved so far and prints additional samples on demand.
No. makemore can train on ordinary hardware without special acceleration, although the README notes that a GPU will make training faster.
Yes. makemore is distributed under the MIT license, which permits commercial use subject to the license terms. The code is compact and educational, so production suitability still depends on your own testing and deployment needs.
Best use cases
- โขTurn a collection of names into new name candidates while learning how autoregressive generation works at character level.
- โขExplore the differences between bigram, MLP, RNN, LSTM, GRU, and Transformer training approaches through source code that users can modify.
- โขGenerate company-name ideas from a line-based list of existing company names.
- โขCreate English-like babble from a list of valid Scrabble words.
- โขTeach neural-network language modeling with examples that progress from character counts to a GPT-style Transformer.
How to install / try
Install from the repository source. The README does not document a package-manager installation command; it states that PyTorch is the only requirement. Clone the repository, ensure PyTorch is available in the Python environment, then run `python makemore.py` from the project directory.
Who should try it โ and who should skip
Try makemore if you are learning autoregressive modeling, want a readable PyTorch implementation across several architectures, or need a quick experiment with name and word generation. Skip it if you need pretrained models, stable APIs, scalable text generation, or production support.
Is makemore worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of makemore.