Pro Pro lesson
Predicting the next word
Watch a real language model, GPT-2, read text as tokens and write it one token at a time.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 12 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Predicting the next word”
What you’ll learn
- Text becomes tokens
- A probability for every token
- Rolling weighted dice
- One token at a time
- How far back can it see?
- The loop behind every chatbot
Key terms
- Token
- A chunk of text — a word, part of a word, a character or even a single byte — that a language model reads and writes.
- Tokeniser
- The step that splits text into tokens and turns each one into an ID number.
- Byte-pair encoding
- A way to build a tokeniser’s vocabulary: start from single bytes and repeatedly merge the pairs that appear most often in the training text.
- Language model
- A model that gives a probability for every possible next token, given the text so far.
- GPT-2
- A language model released by OpenAI in 2019. The small version has 124 million parameters and a vocabulary of 50,257 tokens, and was trained on about 40 GB of web text.
- Sampling
- Choosing the next token at random, weighted by the model’s probabilities.
- Temperature
- A setting that sharpens (below 1) or flattens (above 1) the probabilities before sampling.
- Greedy decoding
- Always choosing the single most likely next token.
- Top-k sampling
- Sampling only from the k most likely tokens, ignoring the rest.
- Context window
- How much of the earlier text a model can see when predicting the next token. GPT-2’s is 1,024 tokens.
Sources
- Language Models are Unsupervised Multitask Learners (Radford et al., 2019) — WebText: slightly over 8 million documents, 40 GB of text. GPT-2 uses byte-level byte-pair encoding with a vocabulary of 50,257 and a context of 1,024 tokens.
- GPT-2 model card, openai-community/gpt2 (Hugging Face) — The exact model used in every widget here: the smallest GPT-2, 124M parameters, byte-level BPE with 50,257 tokens, inputs of 1,024 consecutive tokens.
- Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow & Birch, 2016) — Byte-pair encoding for subword tokenisation: adapts a compression method that repeatedly replaces the most frequent pair of symbols with a new one.
- Speech and Language Processing (3rd ed. draft), Chapter 3: N-gram Language Models (Jurafsky & Martin) — Older language models: bigram and trigram models predict the next word from just the previous one or two words; 4-gram and 5-gram models look a little further.
- Speech and Language Processing (3rd ed. draft), Chapter 7: Transformers and Pretraining (Jurafsky & Martin) — Generating text one token at a time: greedy decoding, and temperature sampling, which divides the logits by τ before the softmax, as well as top-k sampling.
- Hierarchical Neural Story Generation (Fan, Lewis & Dauphin, 2018) — Early use of top-k random sampling for text generation, widely cited for it: at each step, sample only from the k most likely words (they used k = 10).
- The Curious Case of Neural Text Degeneration (Holtzman et al., 2020) — Maximisation-based decoding such as greedy and beam search produces bland text that gets stuck in repetitive loops; discusses temperature and top-k sampling.
- How to generate text: using different decoding methods for language generation with Transformers (von Platen, Hugging Face blog) — Uses GPT-2 and the prompt “I enjoy walking with my cute dog” to show greedy search quickly repeating itself, then explains temperature and top-k sampling.
- Training language models to follow instructions with human feedback (Ouyang et al., 2022) — Assistants are pre-trained language models fine-tuned further, with human demonstrations and feedback, to follow instructions.
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (Gemini Team, 2024) — Today’s long context windows: near-perfect retrieval up to at least 10M tokens, compared with Claude 3.0 (200k; the first version of the paper said Claude 2.1) and GPT-4 Turbo (128k).