What you’ll cover

Open a step to read its text without the interactive.

1. Models count tokens, not words

   In Predicting the next word you met tokens: the chunks a language model reads and writes, each with an ID number. Here are two real tokenisers, the programs that do the chopping. GPT-2’s has a vocabulary of 50,257 tokens; Qwen2.5’s, from 2024, has 151,665 (counting its special tokens).

   For everyday English a token is often a whole word with its leading space, so ten words come out as eleven tokens: the question mark gets one of its own. Numbers, rare words and other writing systems are a different story.

   Pick “One long word”, then “Numbers and dates”, then “Emoji”. Watch the words-to-tokens count drift apart.

2. Same text, different tokens

   A tokeniser’s vocabulary is built from the text its makers chose, using their own rules, so two tokenisers can split the same sentence very differently.

   Qwen2.5 always gives each digit its own token, so the “2024” in this date costs four tokens, where GPT-2 split it into two (“20” and “24”; with a space in front, “ 2024” is a single GPT-2 token). On the other hand Qwen2.5 needs fewer tokens than GPT-2 for the French and Japanese sentences, and for the indentation in the code. Neither is “right”: the split simply decides how much of the model’s budget each text uses.

   Step through the numbers, code, French and Japanese samples. Which tokeniser is cheaper for each?

3. How many tokens is a word?

   You will often see rules of thumb like “a token is about three-quarters of a word”. Rather than take that on trust, we counted: four whole public-domain novels and five Wikipedia articles, about 371,000 words, run through both tokenisers.

   Across all of them GPT-2 used 1.37 tokens per word (0.73 words per token) and Qwen2.5 used 1.25 (0.80 words per token). Individual texts ranged from about 1.2 to 1.6. So for English, 100 words is roughly 125–140 tokens, depending on the tokeniser and the text. Other languages, code and numbers can cost far more.

   Find the text with the most tokens per word for GPT-2, and the one with the fewest. Is the gap the same for Qwen2.5?

4. The context window is a fixed budget

   A model can only take in a limited number of tokens at once: its context window. Everything counts against it: the hidden system message the app adds, every message so far, and the reply being written.

   Below is a real chat with a small assistant model, Qwen2.5-0.5B-Instruct. The questions were written for this lesson; every reply is the model’s own, cut off at 160 tokens. Each message is tokenised for real, so you can see what it costs. Published windows range from GPT-2’s 1,024 tokens to a million or more: Google offered 2 million for Gemini 1.5 Pro in 2024, and Meta announced 10 million for Llama 4 Scout in 2025. This little chat would fit in any of them, so to see the limit bite, shrink the window yourself. The widget keeps 100 tokens free for the model’s answer to the last question, which we capped at 100 tokens (earlier replies were allowed 160).

   Drag the window down slowly from the top. Watch which messages grey out first, and notice the system message stays in view until not even the newest question fits.

5. When the chat outgrows the window

   When a conversation no longer fits, something has to give. Anthropic’s documentation, for example, says its API rejects a request whose input is too long, while chat apps can drop the oldest turns on a “first in, first out” basis, or summarise the earlier part of the conversation.

   Here the app drops the oldest exchanges first, a kind of truncation. With the whole chat in view, the model answers that Sam is allergic to peanuts. Once your first message falls outside the window, it says it has no access to that information, even though a later reply still mentions a peanut-free dinner. The model has no memory of the conversation beyond what is inside its window.

   These replies come from a very small model, and they are not good ones: for a peanut-free dinner it offered a “peanut butter and jelly salad”.

   Drag the window down until your first message turns grey, and read the new answer. Then keep going until not even the last question fits.

6. Why longer context costs more

   Why not give every model an enormous window? Part of the answer is attention. In a model that writes text, every token is scored against itself and every earlier token, so n tokens means n × (n + 1) ÷ 2 scores, for every attention head in every layer. The original transformer paper lists self-attention’s cost per layer as growing with n².

   Double the context and those comparisons roughly quadruple. Engineers have found many ways to make this cheaper, but the basic pressure remains, and Anthropic’s documentation also warns that “more context isn’t automatically better”: accuracy and recall can degrade as the token count grows.

   Slide the toy example from 4 to 8 tokens and count the shaded cells. Then pick Llama 4 Scout’s 10M-token row and compare its pairs with GPT-2’s.

Key terms

Token

A chunk of text that a language model reads and writes one at a time: often a word or part of a word, sometimes a single character or a few bytes.

Tokeniser

The program that splits text into tokens and turns each into an ID number, using a fixed vocabulary.

Vocabulary

The full list of tokens a tokeniser can produce; GPT-2’s has 50,257 entries.

Context window

The maximum number of tokens a model can take in at once, including the reply it is writing.

System message

Instructions an app places at the start of the conversation, usually hidden from the user, which also use up tokens.

Truncation

Cutting text, often the oldest messages, so that a conversation fits inside the context window.

Self-attention

The transformer step in which each token is scored against other tokens to decide what to draw information from.