Glossary · Phase 6 · Modern LLM engineering

Value model

In RLHF with PPO, a helper model that predicts the score an answer is likely to get from the part written so far.

Try it now

The leash matters. Without it, the model finds ways to game the reward model.

RLHF · Reward models and reinforcement learning from human feedback. About 40 minutes.

best-of-n sampling: Generating several answers and keeping the one a reward model scores highest. Simple and strong, but it costs n generations per question.
policy: In reinforcement learning, the model being trained: the thing that decides what to do (here, which token to write next).
PPO: Proximal policy optimization: the reinforcement-learning method classic RLHF uses, which nudges the model toward higher-scoring answers in small, careful steps.
reward hacking: When a model finds ways to score highly with the reward model that don't make its answers better, like padding or flattery.
reward model: A model that scores answers by how much people would prefer them.
RLHF: Reinforcement learning from human feedback: training a model toward answers people prefer, using a learned scorer.

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.