Pro Pro lesson
The artificial neuron
Build the tiny decision-maker that every neural network is made of.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 9 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “The artificial neuron”
What you’ll learn
- Weigh up the evidence
- Bias and a decision
- Different ways to squash
- A neuron draws a line
- Let it find the weights
- What one neuron cannot do
Key terms
- Neuron
- The basic unit of a neural network: it weights its inputs, adds a bias and applies an activation function.
- Weight
- A number that sets how strongly, and in which direction, an input influences a neuron.
- Bias
- A number added to a neuron’s weighted sum that shifts how easily it fires.
- Weighted sum
- Each input multiplied by its weight, all added together. With the bias added on, it is often written z.
- Activation function
- The function a neuron applies to its weighted sum, such as sigmoid, tanh or ReLU.
- ReLU
- Rectified linear unit: outputs z when z is positive and 0 otherwise.
- Linearly separable
- Data that a single straight line (or flat plane) can split perfectly into its classes, like Adelie and Gentoo penguins by bill depth and flipper length.
- XOR
- “Exclusive or”: true when exactly one of two inputs is true. A classic pattern no single line can separate.
Sources
- CS231n notes: Neural Networks Part 1 (Stanford University) — The artificial neuron: a weighted sum of inputs plus a bias passed through an activation function; a very coarse model of a biological neuron; sigmoid (0 to 1), tanh (−1 to 1) and ReLU; a single neuron is a linear classifier.
- Deep Learning, Chapter 6: Deep Feedforward Networks (Goodfellow, Bengio & Courville, 2016) — Section 6.1, “Learning XOR”: XOR is 1 when exactly one input is 1, a linear model cannot represent it, and a hidden layer solves it. Also introduces rectified linear units.
- Deep Sparse Rectifier Neural Networks (Glorot, Bordes & Bengio, 2011) — Showed that rectifying (ReLU) neurons train deep networks as well as or better than tanh neurons, despite the hard non-linearity at zero.
- palmerpenguins: Palmer Archipelago (Antarctica) penguin data (Horst, Hill & Gorman, 2020) — Source of the penguin measurements: 344 Adelie, Chinstrap and Gentoo penguins from the Palmer Archipelago, Antarctica, collected by Dr Kristen Gorman and Palmer Station LTER. CC0.
- Ecological Sexual Dimorphism and Environmental Variability within a Community of Antarctic Penguins (Genus Pygoscelis) (Gorman, Williams & Fraser, 2014) — The original study behind the data: bill (culmen) and flipper measurements of Adélie, chinstrap and gentoo penguins near Anvers Island, over the 2007/08–2009/10 seasons.