How Models Learn

Perplexity

exp of a language model’s average cross-entropy (in nats) per token; lower means less surprised by the text.

To report how good a model is, people often use perplexity: exp of the average cross-entropy (with ln).
From Softmax and cross-entropy, step “What language models minimise”

See it in action

Try perplexity hands-on in the lesson Softmax and cross-entropy, part of the The maths behind AI course.

Open the lesson