Pro Pro lesson
Retrieval-augmented generation
Give a model the right pages before it answers, and watch what changes (and what doesn’t).
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 10 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Retrieval-augmented generation”
What you’ll learn
- An open-book exam
- Search by meaning, not keywords
- How many passages go in? Top-k
- From memory versus from the sources
- When RAG goes wrong
Key terms
- Retrieval-augmented generation (RAG)
- Searching for relevant text and adding it to the prompt so a model answers from those sources.
- Embedding
- A list of numbers that represents a piece of text so that texts with similar meanings get similar lists.
- Cosine similarity
- A score from −1 to 1 for how closely two embeddings point in the same direction; higher means more similar.
- Top-k retrieval
- Keeping only the k highest-scoring passages from a search to put in the prompt.
- Chunking
- Splitting long documents into short passages so each can be embedded and retrieved separately.
- Grounding
- Tying a model’s answer to specific source text that a reader can check.
- Citation
- A pointer in an answer, like [1], to the source a fact came from.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020) — Introduced the term (Facebook AI Research, UCL, NYU): a generator combined with a dense vector index of Wikipedia accessed by a neural retriever, fine-tuned together; more specific, factual answers with provenance. Our demo uses the simpler prompt-only form.
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019) — Sentence embeddings that can be compared with cosine similarity, making semantic search fast; the approach behind the sentence-transformers models.
- all-MiniLM-L6-v2 model card (sentence-transformers) — The embedding model used here: maps sentences and paragraphs to a 384-dimensional vector space for semantic search; mean pooling and L2 normalisation; Apache-2.0.
- Qwen2.5-0.5B-Instruct model card (Qwen team) — The instruct model that wrote both sets of answers: 0.49B parameters, Apache-2.0.
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — Performance is often highest when relevant information is at the start or end of the context and drops when it is in the middle: models don’t robustly use long contexts.
- When Not to Trust Language Models (Mallen et al., ACL 2023) — Models struggle with less popular facts, and retrieval augmentation helps significantly in those cases.
- Prompt engineering guide (OpenAI documentation) — Recommends including relevant context information in the prompt, sometimes called retrieval-augmented generation, to give the model data outside its training set.
- Prompting best practices: long context prompting (Anthropic documentation) — Advises placing long documents above the query; our prompt puts the sources first and the question last.
- Wikipedia: Reusing Wikipedia content — Wikipedia text is licensed CC BY-SA 4.0; the 48 passages credit their articles and link to the exact revisions used.