Pro Pro lesson
Softmax and cross-entropy
How a model turns raw scores into probabilities, and the loss that tells it how wrong it was.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 11 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Softmax and cross-entropy”
What you’ll learn
- From scores to probabilities
- Why exponentials?
- Temperature
- Logarithms measure surprise
- Cross-entropy: the loss for classifiers
- What language models minimise
Key terms
- Logit
- A raw score a model gives one class before softmax; it can be any number, positive or negative.
- Softmax
- Turns a list of scores into positive weights that add up to 1 (probabilities): take exp of each score and divide by the total. A score one point higher than another gets about 2.7 times its weight, so large gaps in score become very large gaps in weight.
- Temperature
- A number T that every score is divided by before softmax; below 1 sharpens the probabilities, above 1 flattens them.
- Natural logarithm (ln)
- The logarithm with base e ≈ 2.718: ln y is the power you raise e to in order to get y.
- Surprise
- How unexpected an outcome was, −ln p for an outcome given probability p; also called self-information.
- Nat
- The unit of surprise or cross-entropy when natural logs are used; logs base 2 give bits instead.
- Cross-entropy loss
- For one example, −ln of the probability the model gave the correct answer; also called log-loss.
- Perplexity
- exp of a language model’s average cross-entropy (in nats) per token; lower means less surprised by the text.
Sources
- CS231n: Linear classification, Softmax classifier (Stanford) — Softmax definition; cross-entropy loss −log of the correct class probability; shifting scores by the max for numeric stability.
- Goodfellow, Bengio & Courville, Deep Learning, ch. 4.1: Overflow and Underflow — Softmax is unchanged by adding a constant to every input; subtracting the max prevents overflow.
- Goodfellow, Bengio & Courville, Deep Learning, ch. 3.13: Information Theory — Self-information I(x) = −log P(x) with the natural log, measured in nats; base 2 gives bits.
- Jurafsky & Martin, Speech and Language Processing (3rd ed. draft, Aug 2026), ch. 7: Transformers and Pretraining — Language models are trained on the cross-entropy loss −log p(next token); perplexity is the exponent of the mean cross-entropy (natural log); temperature divides the logits; perplexity depends on tokenisation.
- Hinton, Vinyals & Dean (2015), Distilling the Knowledge in a Neural Network — Softmax with a temperature T; a higher T gives a softer probability distribution.
- GPT-2 small (openai-community/gpt2), Hugging Face, pinned revision — The model behind this lesson’s real numbers, computed offline: raw next-token logits (data-pipeline/latent_data/gpt2_logits.py) and per-token probabilities for the perplexity step (gpt2_perplexity.py: perplexity 9.25, 9.96 and 2620.88).
- Radford et al. (2019), Language Models are Unsupervised Multitask Learners (GPT-2) — GPT-2: language models trained on WebText to predict the next token; byte-level BPE input (§2.2) and a vocabulary of 50,257 tokens (§2.3).
- Wikipedia: Colorless green ideas sleep furiously — Chomsky’s 1957 example (Syntactic Structures) of a grammatically correct but meaningless sentence.