DPO
Direct preference optimisation: training a model directly on preference pairs with a classification-style loss, without a separate reward model.
Direct preference optimisation (DPO, Rafailov et al., 2023) trains the model directly on preference pairs with a simple classification-style loss, with no separate reward model and no sampling during fine-tuning.
See it in action
Try dpo hands-on in the lesson Instruction tuning and RLHF, part of the How AI assistants work course.