Reward hacking

Also written: over-optimization, overoptimization

When a model finds ways to score highly with the reward model that don't make its answers better, like padding or flattery.

Other terms from the same lesson

• best-of-n sampling: Generating several answers and keeping the one a reward model scores highest. Simple and strong, but it costs n generations per question.
• policy: In reinforcement learning, the model being trained: the thing that decides what to do (here, which token to write next).
• PPO: Proximal policy optimization: the reinforcement-learning method classic RLHF uses, which nudges the model toward higher-scoring answers in small, careful steps.
• reward model: A model that scores answers by how much people would prefer them.
• RLHF: Reinforcement learning from human feedback: training a model toward answers people prefer, using a learned scorer.
• value model: In RLHF with PPO, a helper model that predicts the score an answer is likely to get from the part written so far.