Glossary · Phase 3 · Machine learning

Warmup

Also written: warm-up

Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.

Try it now

Run 10 steps at 0.02 (too small). Then reset and try 0.4, 0.95 and 1.05. Watch the status line.

Learning rate · Step size, convergence and divergence. About 25 minutes.

cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.
diverge: When training blows up: the loss grows instead of settling, often because the learning rate is too big.
learning rate: How big a step training takes each time. Too big overshoots; too small crawls.
learning-rate schedule: A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.
learning-rate sweep: Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.