Pre-training
The first, largest stage of training, in which a model learns to predict the next token across a huge amount of text.
Before a model can chat, it goes through pre-training: it reads a vast amount of text and, at every position, tries to predict the next token.
See it in action
Try pre-training hands-on in the lesson Pre-training, part of the How AI assistants work course.