Optimizer
The rule that decides how to change the weights each step, given the gradients. Adam is the usual choice.
Think of the volume-leveling setting in a music app. Loud songs get turned down and quiet songs get turned up, so everything plays at about the same volume. Adam does this for gradients. It tracks how big each weight's gradients usually are, and divides by that. Big gradients get turned down, and small ones get turned up. Adam also uses momentum. Its name is short for "adaptive moment estimation".
Try it now
On a narrow valley, a step big enough for the gentle direction overshoots in the steep one. Momentum and Adam exist to fix exactly this.
Learn it properly
Optimizers · Momentum and Adam. About 35 minutes.
Other terms from the same lesson
- AdamW: Adam with weight decay applied separately from the gradient step. It's the default optimizer for training most language models.
- momentum: An optimizer trick: keep a running average of recent steps so training keeps rolling in a steady direction instead of zigzagging.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.