The chain rule

Follow a nudge through a chain of calculations, and see how networks find every gradient at once.

This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 10 minutes.

Already Pro? Sign in and you’ll go straight in.

Inside “The chain rule”

Composition: Feeding the output of one function into another, written f(g(x)).
Chain rule: The rate of change of a composition is the product of the local rates: dy/dx = dy/du · du/dx.
Computational graph: A diagram of a calculation broken into simple steps, with arrows showing which values feed into which.
Forward pass: Computing the values of a graph from its inputs through to the output or loss.
Local derivative: How much one node’s output changes with one of its own inputs, ignoring the rest of the graph.
Backward pass: Working from the loss back to the inputs, multiplying local derivatives to get the gradient of the loss at every node. (When a value feeds several nodes, the gradients coming back along each path are added.)
Backpropagation: Using the backward pass to find how the loss depends on every weight in a neural network.
Finite difference: Estimating a derivative by nudging an input by a small amount and dividing the change in output by the nudge.

CS231n: Backpropagation, Intuitions (Stanford) — Chain rule as multiplication of local gradients; forward pass computes values, backward pass applies the chain rule from the end.
Goodfellow, Bengio & Courville, Deep Learning, ch. 6.5: Back-Propagation — Computational graphs; back-propagation’s cost is of the same order as the forward computation.
CS231n: Neural Networks Part 3, Gradient Checks (Stanford) — Numerical gradient checks; the centred difference has error O(h²) versus O(h) for the one-sided formula.
Brown et al. (2020), Language Models are Few-Shot Learners — GPT-3 has 175 billion parameters.