How Models Learn

Context window

How much of the earlier text a model can see when predicting the next token. GPT-2’s is 1,024 tokens.

GPT-2 can look back up to 1,024 tokens: its context window.
From Predicting the next word, step “How far back can it see?”

See it in action

Try context window hands-on in the lesson Predicting the next word, part of the AI fundamentals course.

Open the lesson