Instruction tuning and RLHF

How a text predictor is trained into an assistant, with demonstrations, preferences and rewards.

This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 12 minutes.

Already Pro? Sign in and you’ll go straight in.

Inside “Instruction tuning and RLHF”

Post-training: Training that comes after pre-training, to turn a base model into one that follows instructions and behaves as intended.
Supervised fine-tuning (SFT): Further training of a pre-trained model on example prompts paired with the replies people want.
Demonstration: An example reply, written or chosen by a person, used to show a model the behaviour wanted.
Chat template: The fixed text format, with special tokens, that lays out a conversation for an assistant model.
Preference data: Records of which of two or more model responses people judged better.
Reward model: A model trained on preference data to give a response a single score predicting how much people would like it.
RLHF: Reinforcement learning from human feedback: fine-tuning a model to earn higher scores from a reward model trained on human preferences.
DPO: Direct preference optimisation: training a model directly on preference pairs with a classification-style loss, without a separate reward model.

Ouyang et al. (2022), Training language models to follow instructions with human feedback (InstructGPT) — The three steps; ~13k SFT, 33k RM, 31k PPO prompts; 4–9 responses ranked, giving K(K−1)/2 comparisons per prompt; 6B reward model and its loss; KL penalty; 1.3B preferred to 175B GPT-3; 85 ± 3%; still makes up facts.
Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model — DPO solves the RLHF problem with a simple classification loss, without fitting a reward model or sampling from the model during fine-tuning.
Qwen Team (2024), Qwen2.5 Technical Report — Qwen2.5 post-training: over 1 million SFT examples, about 150,000 training pairs for offline RL (DPO), then online RL (GRPO).
Qwen2.5-0.5B-Instruct model card (Hugging Face) — The assistant model in the widgets: same architecture as Qwen2.5-0.5B, pre-training and post-training, Apache-2.0.
Qwen2.5-0.5B model card (Hugging Face) — The base model in the comparisons: pre-training only.
NASA Space Place, Why Is the Sky Blue? — Sunlight is scattered by the gases and particles in the air, and blue light is scattered more than other colours.