Glossary · Phase 3 · Machine learning

Learning rate

How big a step training takes each time. Too big overshoots; too small crawls.

Try it now

Run 10 steps at 0.02 (too small). Then reset and try 0.4, 0.95 and 1.05. Watch the status line.

Learning rate · Step size, convergence and divergence. About 25 minutes.

cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.
diverge: When training blows up: the loss grows instead of settling, often because the learning rate is too big.
learning-rate schedule: A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.
learning-rate sweep: Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.
warmup: Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.