Glossary · Phase 6 · Modern LLM engineering
Reward model
A model that scores answers by how much people would prefer them.
Try it now
The leash matters. Without it, the model finds ways to game the reward model.
Learn it properly
RLHF · Reward models and reinforcement learning from human feedback. About 40 minutes.
Other terms from the same lesson
- best-of-n sampling: Generating several answers and keeping the one a reward model scores highest. Simple and strong, but it costs n generations per question.
- policy: In reinforcement learning, the model being trained: the thing that decides what to do (here, which token to write next).
- PPO: Proximal policy optimization: the reinforcement-learning method classic RLHF uses, which nudges the model toward higher-scoring answers in small, careful steps.
- reward hacking: When a model finds ways to score highly with the reward model that don't make its answers better, like padding or flattery.
- RLHF: Reinforcement learning from human feedback: training a model toward answers people prefer, using a learned scorer.
- value model: In RLHF with PPO, a helper model that predicts the score an answer is likely to get from the part written so far.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.