Glossary · Phase 6 · Modern LLM engineering
Implicit reward
In DPO, the score you can read off from how much more likely the trained model makes an answer than the reference model does.
Try it now
DPO skips the reward model and learns straight from the pairs.
Learn it properly
DPO and direct preference methods · Preference training without a separate reward model. About 40 minutes.
Other terms from the same lesson
- DPO: Direct preference optimization: training straight from preference pairs, without a separate reward model.
- preference pair: One prompt with two answers, one marked as preferred (chosen) and one not (rejected).
- reference model: A frozen copy of the model before preference training. Training is penalized for drifting too far from it.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.