ML-From-Scratch: Learn Machine Learning From Scratch
/ML-From-Scratch
Machine Learning From Scratch is worth studying if you want to see the actual math behind an algorithm instead of a black-box call to .fit(). Reading regression.py next to gradient_boosting.py next to the GAN implementation shows how the same autograd-free tensor operations repeat across supervised, unsupervised, and deep learning. It won't teach you production ML engineering — for that, pair it with a framework tutorial.
What is Machine Learning From Scratch?
Machine Learning From Scratch is a Python collection of machine learning algorithms implemented in pure NumPy, from linear regression through convolutional and generative adversarial networks. Each model lives in its own file inside the mlfromscratch package, split into supervised, unsupervised, reinforcement, and deep learning folders. The code favors readability over speed, not an optimized backend.
Why learn ML by building from scratch
Implementing an algorithm from scratch forces you to answer the questions a framework call hides: what shape is the gradient, why batch normalization needs a running mean, how a convolution turns into matrix multiplication. There's no way around the math. ML-From-Scratch structures that exercise algorithm by algorithm instead of leaving it as a one-off blog post, and it covers unsupervised, reinforcement, and evolutionary methods that most from-scratch tutorials skip in favor of just regression and MLPs.
Algorithms covered in ML-From-Scratch
- ✓Supervised learning implementations covering Linear, Ridge, Lasso, Elastic Net, Polynomial and Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, XGBoost, Adaboost, Naive Bayes, K Nearest Neighbors, Support Vector Machine, Perceptron, Multilayer Perceptron, and Linear Discriminant Analysis.
- ✓Unsupervised learning code for K-Means, DBSCAN, Gaussian Mixture Model, PCA, Autoencoder, Restricted Boltzmann Machine, Apriori, and FP-Growth.
- ✓A deep learning module with hand-written layers — Convolutional, Dense, Dropout, Batch Normalization, RNN, Max/Average Pooling, Flatten, Reshape, Zero/Constant Padding — assembled into CNN, MLP, and RNN examples.
- ✓A Generative Adversarial Network example that trains a generator and discriminator to produce handwritten digits.
- ✓A Deep Q-Network reinforcement learning example solving the CartPole-v1 environment from OpenAI gym.
- ✓Evolutionary methods: a Genetic Algorithm example and a Neuroevolution example that evolves a neural network's weights instead of training them with gradient descent.
What you can build and learn
- •Step through the CNN example's printed layer table (Conv2D, BatchNorm, Dense, Softmax) to see exactly how parameter counts and output shapes change layer by layer.
- •Read the GAN example to see a generator and discriminator defined as two ordinary NumPy models trained against each other, instead of relying on a framework's adversarial-training wrapper.
- •Trace the Genetic Algorithm example, which evolves a population of candidate strings toward a target string using fitness-proportional selection and single-point crossover.
- •Compare Ridge, Lasso, and Elastic Net side by side since all three live in the same regression.py file, showing exactly where the regularization term differs in the code.
- •Follow the Deep Q-Network example against the CartPole-v1 environment in OpenAI gym to see a minimal reinforcement learning loop end to end.
What makes this approach effective
- ✓No framework dependency to fight — the code is NumPy, so every operation is either a matrix multiply, a broadcast, or a loop you can step through.
- ✓The examples print concrete output (parameter counts, layer shapes, training accuracy) so you can check your understanding of a layer's math against a real run, not just its formula.
- ✓Related algorithms sit in the same file — Linear, Ridge, Lasso, Elastic Net, and Polynomial Regression are all variants inside regression.py — so the difference between them is a diff, not a search across separate libraries.
- ✓MIT license, so the code can be copied into your own study notes or a course without asking permission.
When to use production libraries instead
- △The README says explicitly the goal is transparency, not computational efficiency — these implementations are not optimized and shouldn't be benchmarked against scikit-learn or PyTorch.
- △Installation is python setup.py install, an old-style packaging approach with no listed Python version requirement or dependency pinning in the README.
- △There's no test suite or CI badge mentioned in the README, so correctness of any given algorithm rests on reading the code yourself, not on a maintainer's guarantee.
- △No mention of ongoing maintenance activity in the README — treat it as a fixed teaching snapshot rather than a project tracking new techniques.
Other ways to learn ML fundamentals
Is this repo right for you
ML-From-Scratch fits a specific reader: someone who already knows Python and enough linear algebra to follow a matrix multiply, and who learns by reading code rather than watching video lectures. It's a poor fit if you're brand new to Python — the code assumes you can parse NumPy broadcasting and class inheritance without hand-holding — or if your goal is shipping a model quickly, in which case a framework tutorial gets you there faster.
Frequently asked questions
Machine Learning From Scratch is not built for production. The README states its purpose is to present each algorithm's inner workings transparently and accessibly rather than to make them as computationally efficient as possible, so the implementations are meant for reading and study, not for deploying in a live pipeline.
Reading Machine Learning From Scratch comfortably requires working Python and NumPy skills — array broadcasting, indexing, and class-based object-oriented code — plus enough linear algebra and calculus to follow matrix multiplication and gradients, since the repo doesn't teach those basics itself.
Machine Learning From Scratch does cover both: the deep_learning module's Neural Network and Layers files implement forward and backward passes — the mechanism behind backpropagation — and the training loops in the examples run gradient descent to update Dense, Convolutional, and Batch Normalization layer weights.
Machine Learning From Scratch and TensorFlow or PyTorch solve different problems: TensorFlow and PyTorch give you an optimized, GPU-accelerated backend for building real models fast, while Machine Learning From Scratch spells out the NumPy math those frameworks hide behind autograd and compiled kernels — useful before you use a framework, not instead of one.
The supervised_learning folder in Machine Learning From Scratch includes Linear, Ridge, Lasso, Elastic Net, Polynomial and Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, XGBoost, Adaboost, Naive Bayes, K Nearest Neighbors, Support Vector Machine, Perceptron, Multilayer Perceptron, and Linear Discriminant Analysis.
Machine Learning From Scratch includes exactly that: a Convolutional Neural Network example that classifies digits with a printed layer-by-layer parameter table, and a Generative Adversarial Network example whose generator and discriminator are both plain NumPy models trained against each other to produce handwritten digits.
Related repositories
Want a second opinion on ML-From-Scratch?
Ask an AI that can read this page — one click and you get its take on ML-From-Scratch.