Why models hallucinate

See a real assistant answer confidently about things that don’t exist, and learn how to catch it.

This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 10 minutes.

See Pro plans Sign in

Already Pro? Sign in and you’ll go straight in.

Inside “Why models hallucinate”

What you’ll learn

Fluent is not the same as true
A predictor, not a database
Confident answers about things that don’t exist
Why training rewards a good guess
Ask again: disagreement is a clue
Spotting and reducing hallucinations

Key terms

Hallucination: When a model generates false or made-up content and presents it as if it were true.
Token: A chunk of text, often a word or part of a word, that a language model reads and writes one at a time.
Next-token probability: The chance a model gives each possible next token; it writes text by repeatedly choosing among them.
Sampling: Choosing each next token at random, weighted by its probability (often reshaped by settings such as temperature), so the same prompt can give different answers.
Confidence target: An instruction that wrong answers cost points, so a model should answer only when it is confident enough.
Knowledge cut-off: The date up to which a model’s training text was collected; it knows nothing later unless told.
Calibration: How well a model’s stated or implied confidence matches how often it is actually right.

Sources

Survey of Hallucination in Natural Language Generation (Ji et al., ACM Computing Surveys) — Defines hallucination as generated content that is nonsensical or unfaithful to the provided source content.
Why Language Models Hallucinate (Kalai, Nachum, Vempala & Zhang, 2025) — OpenAI/Georgia Tech. Errors arise from statistical pressures in pre-training (facts seen once); binary right/wrong grading rewards guessing; proposes explicit confidence targets with penalty t/(1−t). Source of the Kalai dissertation question and true title.
Language Models (Mostly) Know What They Know (Kadavath et al., 2022) — Larger models are well calibrated on multiple-choice and true/false questions in the right format, but calibration of P(IK) on new tasks is weaker.
Large Language Models Struggle to Learn Long-Tail Knowledge (Kandpal et al., 2023) — A model’s ability to answer a fact-based question relates to how many relevant documents it saw in pre-training.
When Not to Trust Language Models (Mallen et al., ACL 2023) — Models struggle with less popular factual knowledge; scaling mainly improves popular knowledge; retrieval helps in the long tail.
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection (Manakul, Liusie & Gales, 2023) — If a model knows a fact, sampled responses tend to agree; for hallucinated facts they tend to diverge and contradict one another.
Fabrication and errors in the bibliographic citations generated by ChatGPT (Walters & Wilder, Scientific Reports, 2023) — In 84 short literature reviews, 55% of GPT-3.5 citations and 18% of GPT-4 citations were fabricated.
Reduce hallucinations (Anthropic documentation) — Basic strategies: allow the model to say “I don’t know”, ground answers in direct quotes, verify with citations; these reduce but don’t eliminate hallucinations.
ChatGPT and large language models: what’s the risk? (UK National Cyber Security Centre) — LLMs can ‘hallucinate’ incorrect facts; a model’s data is typically static after training (knowledge cut-off).
Qwen2.5-0.5B-Instruct model card (Qwen team) — The model behind every answer here: 0.49B parameters, instruction-tuned, Apache-2.0; recommended generation settings sample (temperature 0.7).
The Nobel Prize in Literature 1987 (NobelPrize.org) — Awarded to Joseph Brodsky (Mo Yan won in 2012).
Moons of Mars (NASA Science) — Mars has two moons, Phobos and Deimos.
Svalbard Global Seed Vault (Wikipedia) — Opened on 26 February 2008.
Alexander Fleming, Biographical (NobelPrize.org) — Fleming worked at St Mary’s in London, where in 1928 he noticed the mould that led to penicillin.
Fleming (1929), On the antibacterial action of cultures of a penicillium, British Journal of Experimental Pathology — Fleming’s paper describing penicillin, published in 1929.
Gold (Wikipedia) — Gold’s chemical symbol Au comes from the Latin aurum.