What you’ll cover

Open a step to read its text without the interactive.

1. Rules versus examples

   A traditional program follows rules a person wrote: “if the email mentions a prize, mark it as spam.” Spammers adapt, and the rules never keep up.

   Machine learning flips this around. Instead of rules, we hand the computer examples that are already labelled, our training data, and let it work out the pattern on its own.

   These dots are real emails, collected at Hewlett-Packard in 1999 and sorted into spam or not by HP staff and their postmaster — 140 of the 4,601 in a public dataset. Each email is placed by two numbers that describe it, called features: across, how much of its text is exclamation marks; up, how many capital letters it contains. Teal emails are fine. Rose emails are spam.

   Both axes are stretched (a log-like scale) so that emails with a little and emails with a lot both fit. Notice how spam drifts up and to the right — more capitals, more “!” — and how the colours still overlap, as real data always does. Nearly half of these emails have no “!” at all, so they line up along the left edge.

2. Your first model: copy your neighbour

   A new email arrives — the ?. The simplest possible idea: find the training email that looks most similar, and give the new one the same label.

   “Most similar” just means closest on this map. This is called the nearest neighbour classifier, and despite being almost embarrassingly simple, it works surprisingly often.

   Drag the ? around. The dashed line shows which email it is copying, and the readout turns its position back into real numbers.

3. The whole map at once

   If we ask the same question for every spot on the map, the plane splits into regions. The line where teal meets rose is the decision boundary: the model’s entire understanding of spam, drawn in one picture.

   The model didn’t receive this boundary. It emerged from the examples — change the examples and you change what the model believes.

   Click to add emails of your own. Choose “Add spam”, then click to drop a few rose points deep in teal territory and watch an island appear. “Reset points” puts back just the real ones.

4. Ask more neighbours

   Copying a single neighbour makes the model jumpy. Real emails are messy: one shouty newsletter that isn’t spam, or one quiet spam message, carves out its own little island.

   Instead, let the k closest emails vote. With more voters, odd emails get outvoted and the boundary becomes smoother. k is a hyperparameter — a setting we choose, not something the model learns.

   Slide k from 1 up to 25 and watch the islands dissolve.

5. The real test: emails it has never seen

   Getting the training emails right is easy — with k = 1 the model simply memorises them, so training accuracy is a perfect 100% (here, where no two emails sit on exactly the same spot).

   What matters is how it handles new emails. So we kept 70 more real emails hidden away as a test set. Doing well on unseen data is called generalisation, and it is the whole point of machine learning.

   In a real project you would choose k using a separate validation set — a slice of the training emails held back for tuning — and touch the test set only once, at the very end. Otherwise you risk picking the k that happens to suit these particular 70 emails. Here we peek at the test score for every k only to show the gap.

   With only 70 test emails, each one is worth about 1.4%, so don’t read too much into a difference of a point or two. This sample is also a little lucky: a fairer check across all 4,601 emails puts k = 9 at about 83%.

   Compare training and test accuracy at k = 1 and at k = 9. Where is the gap between them biggest?

Key terms

Training data

Labelled examples a model learns from.

Feature

A measurable property that describes each example, such as the share of an email’s characters that are exclamation marks.

Label

The known answer attached to a training example, such as spam or not spam.

Classifier

A model that sorts inputs into categories, like spam or not spam.

Nearest neighbours (k-NN)

Classify a new point by letting the k most similar training examples vote.

Decision boundary

The border in input space where the model switches from predicting one class to another.

Hyperparameter

A setting chosen by a person before training, such as k in k-NN (later lessons meet another: the learning rate).

Test set

Labelled examples kept hidden during training, used only to measure how well a model generalises.

Validation set

Labelled examples held back from training and used to choose settings such as k, so the test set can stay untouched until the very end.

Generalisation

How well a model performs on new data it has never seen.