Glossary · Phase 3 · Machine learning
Learning-rate schedule
A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.
Try it now
Run 10 steps at 0.02 (too small). Then reset and try 0.4, 0.95 and 1.05. Watch the status line.
Learn it properly
Learning rate · Step size, convergence and divergence. About 25 minutes.
Other terms from the same lesson
- cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.
- diverge: When training blows up: the loss grows instead of settling, often because the learning rate is too big.
- learning rate: How big a step training takes each time. Too big overshoots; too small crawls.
- learning-rate sweep: Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.
- warmup: Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.