Learning-rate sweep
Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.
What it means
The learning rate is the setting that matters most in training, and you can't work out the best value ahead of time. A learning rate sweep finds it by testing: train briefly with several rates, each about three times the last, and compare the loss after the same number of steps.
A sweep in Python
def final_loss(lr, steps=30): w = 10.0 # the best value is 3, where the loss is 0 for _ in range(steps): w -= lr * 2 * (w - 3) # one step of gradient descent on (w - 3) squared if abs(w) > 1e6: return float("inf") return (w - 3) ** 2 rates = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0, 1.3] results = {lr: final_loss(lr) for lr in rates} for lr, loss in results.items(): print(f"lr {lr:<6} loss after 30 steps: {loss:.4f}") print("best:", min(results, key=results.get))
lr 0.001 loss after 30 steps: 43.4539 lr 0.003 loss after 30 steps: 34.1491 lr 0.01 loss after 30 steps: 14.5801 lr 0.03 loss after 30 steps: 1.1964 lr 0.1 loss after 30 steps: 0.0001 lr 0.3 loss after 30 steps: 0.0000 lr 1.0 loss after 30 steps: 49.0000 lr 1.3 loss after 30 steps: inf best: 0.3
The small rates barely move in 30 steps. 0.1 and 0.3 get there. At 1.0 the weight bounces between the same two values forever, and at 1.3 training diverges. The usable range is wide, but the failure at the top end is sudden, so it pays to see where that edge is.
In practice
Space the rates evenly on a log scale, since the differences that matter are ratios. A few hundred steps is usually enough to rule most of them out. Then refine around the best one, or settle a little below it for safety. A learning rate range test, from Leslie Smith's work on cyclical learning rates, gets the same answer by raising the rate gradually during a single short run.
You can't calculate the best rate ahead of time for a real model. So you test. Train briefly with several rates and compare the loss after a fixed number of steps. This is a learning-rate sweep.
Try it now
Run 10 steps at 0.02 (too small). Then reset and try 0.4, 0.95 and 1.05. Watch the status line.
Learn it properly
Learning rate · Step size, convergence and divergence. About 25 minutes.
Other terms from the same lesson
- cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.
- diverge: When training blows up: the loss grows instead of settling, often because the learning rate is too big.
- learning rate: How big a step training takes each time. Too big overshoots; too small crawls.
- learning-rate schedule: A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.
- warmup: Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.