What you’ll cover

Open a step to read its text without the interactive.

1. The text marks its own homework

   Before a model can chat, it goes through pre-training: it reads a vast amount of text and, at every position, tries to predict the next token. You watched GPT-2 make these predictions in Predicting the next word; here is how a model learns to make them.

   Nobody has to label anything. The text itself shows what really came next, so every token of every document is a free practice question. This is called self-supervised learning.

   These are real predictions from Qwen2.5-0.5B, a small model that has only been pre-trained. Before it sees “Paris” it already gives it 49%, and once it has read “The Eiff” it gives “el” over 99%.

   Step through the Eiffel Tower sentence with Next token. Where is the model confident, and where is it caught out?

2. Loss: a score for surprise

   Each prediction gets a score called the loss: minus the natural logarithm of the probability the model gave the real next token. A confident, correct guess scores close to 0. A token the model thought almost impossible scores high.

   Pre-training adjusts the model to make the average loss over its training text as small as possible. Some text stays hard however much the model reads: nothing in it says that this particular cat is called Biscuit. The model gave the name’s first token, “B”, just 0.6% (its 20th choice). Overall the sentence averages a loss of 3.72, against 1.91 for the Eiffel Tower sentence, though the very worst surprises are “neighbour” and the start of “sardines”, which nothing before them predicts either.

   Choose “A personal detail” and find the loss on “B”. Which tokens score even higher, and why? Then compare the average loss of all three sentences.

3. Parameters: the numbers that get adjusted

   What exactly gets adjusted? A language model is a very large network of artificial neurons like the one in The neuron, and every weight and bias in it is a number called a parameter. Training nudges all of them, a tiny step at a time, in whatever direction lowers the loss.

   Parameter counts have grown fast. The largest GPT-2 had 1.5 billion in 2019 (the small GPT-2 used in other lessons here has 124 million); GPT-3 had 175 billion the next year; Meta’s largest Llama 3.1 model has 405 billion. The model making the predictions in this lesson has 0.49 billion, small enough that we ran it on an ordinary computer’s processor to prepare these examples.

   Tap each model to read the exact figure from its paper or announcement. Roughly how many times bigger is GPT-3 than the largest GPT-2?

4. Training tokens: how much it reads

   The other number that matters is how much text a model learns from, counted in training tokens. GPT-3 was trained on 300 billion. In 2022 DeepMind’s Chinchilla study found that, for a fixed amount of computing, model size and training tokens should grow together, at roughly 20 tokens per parameter. Chinchilla itself had 70 billion parameters and 1.4 trillion tokens.

   Models have since been trained far past that ratio. Meta says Llama 3 was trained on over 15 trillion tokens, and that its 8-billion-parameter model kept improving long after the roughly 200 billion tokens the Chinchilla rule suggests. A smaller model trained for longer is cheaper to run once it is finished.

   Switch to Tokens per parameter. Which model sits near the dashed Chinchilla line, which falls short of it, and which is far past it?

5. Ask a base model a question

   A model that has only been pre-trained is called a base model. It has one skill: continuing text the way text tends to continue. Nobody has taught it that a question deserves an answer.

   Here is Qwen2.5-0.5B again, given each prompt as plain text and always taking its most likely next token. Asked how to boil an egg, it carries on as if the question were one item in a quiz about whether two questions mean the same. Asked to write an email, it gets stuck repeating a sentence. Asked for Australia’s capital, it writes something like an encyclopaedia entry: it names Canberra correctly, then wrongly places it in New South Wales.

   Play “A how-to question” and “A practical task”. Does the base model ever do what the prompt asks?

6. What a base model can and can’t do

   A base model is still useful, because it follows patterns. Write the start of the document you want and it continues in kind. Show it a question with its answer, then a new question, and it supplies the next answer. The GPT-3 paper calls this one-shot prompting (with several examples, few-shot): the examples sit in the prompt, and the model’s parameters do not change.

   But it often doesn’t know when to stop, it states mistakes as fluently as facts, and it has no idea it is talking to anyone. Turning it into a helpful assistant takes more training, which is the subject of Instruction tuning and RLHF.

   Play “A pattern to continue”. Where does the answer you wanted end, and what does the model do after it?

Key terms

Pre-training

The first, largest stage of training, in which a model learns to predict the next token across a huge amount of text.

Self-supervised learning

Learning in which the training targets come from the data itself, such as the next token of a text, rather than from human labels.

Loss

A score for how wrong a prediction was; for next-token prediction, minus the logarithm of the probability given to the real next token.

Parameter

One of the numbers inside a model, such as a connection weight, that training adjusts.

Training tokens

The amount of text a model learned from, counted in tokens.

Base model

A model that has been pre-trained but not yet tuned to follow instructions; it continues text.

Few-shot prompting

Putting a few worked examples of a task in the prompt so the model continues the pattern, without changing its parameters.