Glossary · Phase 4 · Deep learning
Momentum
An optimizer trick: keep a running average of recent steps so training keeps rolling in a steady direction instead of zigzagging.
Try it now
On a narrow valley, a step big enough for the gentle direction overshoots in the steep one. Momentum and Adam exist to fix exactly this.
Learn it properly
Optimizers · Momentum and Adam. About 35 minutes.
Other terms from the same lesson
- AdamW: Adam with weight decay applied separately from the gradient step. It's the default optimizer for training most language models.
- optimizer: The rule that decides how to change the weights each step, given the gradients. Adam is the usual choice.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.