Pro Pro lesson
Training a network
Stack neurons into layers and watch them learn to draw real borders on a map.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 12 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Training a network”
What you’ll learn
- Why one neuron is not enough
- The forward pass
- Loss and backpropagation
- Untangling a patchwork
- Learning rate and capacity
- What the hidden neurons learn
Key terms
- Hidden layer
- A layer of neurons between the inputs and the output. Nobody tells it what its outputs should be — the training data only gives the right answer for the final output.
- Forward pass
- Running an input through the network, layer by layer, to get a prediction.
- Loss
- A single number measuring how wrong the network’s predictions are. Training tries to make it smaller.
- Backpropagation
- The method for working out how much each weight contributed to the loss, by passing errors backwards through the network.
- Epoch
- One full pass over all the training data.
- Learning rate
- How big a step each weight takes on every update. Too small is slow; too large is unstable.
- Capacity
- How complex a pattern a model can represent, roughly set by how many neurons and layers it has.
- Learned feature
- A pattern a hidden neuron has learned to respond to, such as an edge or a corner. Unlike the input features in the first lesson, nobody chooses these.
Sources
- GeoNames geographical database — Source of the place positions and populations used for the Swiss, Iberian and euro maps. CC BY 4.0.
- GeoNames download readme (export/dump) — Defines cities15000, the town list the maps were drawn from: all cities with a population over 15,000, or capitals.
- Countries using the euro (European Union) — 21 of 27 EU countries use the euro, including Bulgaria, Croatia, Lithuania and Slovakia; Czechia, Hungary, Poland, Romania and Sweden have not adopted it, and Denmark has an opt-out.
- Learning representations by back-propagating errors (Rumelhart, Hinton & Williams, 1986) — Popularised back-propagation for training networks of neurone-like units: repeatedly adjust the weights to shrink the difference between the network’s output and the desired output.
- Deep Learning, Chapter 6: Deep Feedforward Networks (Goodfellow, Bengio & Courville, 2016) — Hidden layers (so called because the training data does not show their desired output), forward propagation, back-propagation, and why depth and width add capacity.
- CS231n notes: Backpropagation, Intuitions (Stanford University) — Backpropagation computes gradients by recursive application of the chain rule, passing a gradient signal backwards through each part of the network.
- Machine Learning Glossary: epoch (Google for Developers) — Defines an epoch: a full training pass over the entire training set, so that each example has been processed once.
- Linear regression: Hyperparameters (Google Machine Learning Crash Course) — The learning rate sets the step size: too low and training is slow to converge, too high and it bounces around instead of converging.