Gradient descent

How a model finds the bottom of a valley it cannot see, one small step at a time.

This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 10 minutes.

See Pro plans Sign in

Already Pro? Sign in and you’ll go straight in.

Inside “Gradient descent”

What you’ll learn

Lost in the fog
Step downhill, then look again
The learning rate
Two dials, one landscape
Bumpy landscapes

Key terms

Loss landscape: The loss for every possible setting of a model’s parameters, pictured as hills and valleys.
Gradient: The steepness of the loss along every parameter at once, gathered into one arrow; it points uphill.
Gradient descent: Repeatedly nudging the parameters a little way against the gradient, so the loss goes down.
Learning rate: A hyperparameter that sets how big each gradient descent step is.
Diverging: When steps overshoot by more each time and the loss grows instead of shrinking.
Local minimum: A point lower than everything around it, but not necessarily the lowest point overall.
Global minimum: The lowest loss anywhere in the landscape.
Loss curve: The loss plotted against one parameter, such as the slope of a line, with everything else held fixed or chosen at its best.

Sources

Deep Learning, Chapter 4: Numerical Computation (Goodfellow, Bengio & Courville, 2016) — Gradient descent: step against the derivative, x′ = x − ε∇f(x), with learning rate ε. Defines local and global minima, and shows that long, narrow (poorly conditioned) valleys force small steps to avoid overshooting.
CS229 Lecture Notes (Ma & Ng, Stanford University) — The gradient-descent update θ := θ − α ∂J/∂θ, with α the learning rate, applied to fitting a least-squares line.
Linear regression: Gradient descent (Google Machine Learning Crash Course) — Gradient descent repeatedly computes the loss and nudges a line’s weight and bias in the direction that reduces it.
Linear regression: Hyperparameters (Google Machine Learning Crash Course) — The learning rate is a hyperparameter: too low and the model takes a long time to converge; too high and it bounces around the minimum instead of converging.
Deep Learning, Chapter 8: Optimization for Training Deep Models (Goodfellow, Bengio & Courville, 2016) — Local minima in neural networks: for large enough networks most local minima have low cost, so finding the true global minimum matters little. Also describes momentum (Polyak, 1964).
Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project (De Cock, 2011) — Source of the 24 Ames house sales whose mean squared error forms the one-dial valley and the two-dial landscape.
Station 9414290, San Francisco, CA (NOAA Tides & Currents) — Source of the 72 verified hourly sea-level readings (1–3 June 2023) behind the bumpy loss curve. Public domain US Government data.
Types and Causes of Tidal Cycles (NOAA National Ocean Service) — Mixed semidiurnal tides — two high and two low tides a day, of unequal height — are typical of the US West Coast, which is why a twice-a-day wave fits San Francisco best.
Frequency of Tides: The Lunar Day (NOAA National Ocean Service) — A lunar day lasts 24 hours and 50 minutes, and coasts get two high and two low tides in that time — which is why the best-fitting wave makes slightly fewer than two cycles per 24-hour day.