Glossary · Phase 6 · Modern LLM engineering
PPO
Proximal policy optimization: the reinforcement-learning method classic RLHF uses, which nudges the model toward higher-scoring answers in small, careful steps.
Try it now
The leash matters. Without it, the model finds ways to game the reward model.
Learn it properly
RLHF · Reward models and reinforcement learning from human feedback. About 40 minutes.
Other terms from the same lesson
- best-of-n sampling: Generating several answers and keeping the one a reward model scores highest. Simple and strong, but it costs n generations per question.
- policy: In reinforcement learning, the model being trained: the thing that decides what to do (here, which token to write next).
- reward hacking: When a model finds ways to score highly with the reward model that don't make its answers better, like padding or flattery.
- reward model: A model that scores answers by how much people would prefer them.
- RLHF: Reinforcement learning from human feedback: training a model toward answers people prefer, using a learned scorer.
- value model: In RLHF with PPO, a helper model that predicts the score an answer is likely to get from the part written so far.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.