Pro Pro lesson
Overfitting
Why a model that aces its homework can still fail the exam.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 9 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Overfitting”
What you’ll learn
- A pattern, seen through noise
- Curves that can bend
- The test set tells the truth
- Too simple, too flexible, just right
- More data calms the wiggles
- Regularisation: a penalty for wiggling
Key terms
- Noise
- Random variation in measurements that is not part of the underlying pattern, like the day-to-day weather on top of the seasons.
- Polynomial
- A curve built from powers of x, such as a + bx + cx². Its degree is the highest power used.
- Training error
- How far a model’s predictions are from the examples it was fitted to.
- Test error
- How far a model’s predictions are from held-out examples it never saw during fitting.
- Underfitting
- When a model is too simple to capture the pattern, so it does poorly even on its training data.
- Overfitting
- When a model fits the noise in its training data, doing well there but badly on new data.
- Regularisation
- A penalty added during training that discourages overly complex models, such as large coefficients.
Sources
- Hadley Centre Central England Temperature (HadCET) dataset (Met Office) — Source of the daily temperatures: the longest instrumental temperature record in the world; the daily mean series begins in 1772 (daily series by Parker et al., 1992). Open Government Licence.
- An Update to the Central England Temperature Series — HadCET v2.1 (Legg, Packman, Caton Harrison & McCarthy, 2025), Geoscience Data Journal — Describes the current version of the series the temperatures come from; the Met Office asks users to cite it alongside Parker et al. (1992).
- Deep Learning, Chapter 5: Machine Learning Basics (Goodfellow, Bengio & Courville, 2016) — Capacity, underfitting and overfitting, illustrated with polynomials of rising degree; generalisation error has a U-shaped curve as capacity grows; weight decay adds λ times the squared weights to the MSE.
- Deep Learning, Chapter 7: Regularization for Deep Learning (Goodfellow, Bengio & Courville, 2016) — Regularisation: the L2 parameter-norm penalty (weight decay, also called ridge regression), and the point that training on more data is the best way to make a model generalise better.
- Overfitting (Google Machine Learning Crash Course) — Defines overfitting (matching the training set so closely the model fails on new data) and underfitting, using the gap between training and test performance.
- Overfitting: L2 regularization (Google Machine Learning Crash Course) — L2 regularisation penalises large weights; the regularisation rate λ sets its strength — higher gives simpler models, lower raises the risk of overfitting.