Learning-rate schedules in PyTorch
Warmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.
What it covers6
Why it matters
A training run does not use one learning rate from the first step to the last. Big steps cover ground early. Small steps settle the weights at the end. And the first few steps are the shakiest, so most real runs start the rate low and raise it. Every training script you will read has a scheduler for this. It is a small object, and most of the bugs it causes come from one place: counting the wrong thing.
What you'll be able to do
- Work out a warmup-then-cosine rate at any step by hand
- Write the schedule as a function of the step, and size it in steps, not epochs
- Drive a PyTorch optimizer with LambdaLR, and know where scheduler.step() goes
- Keep a schedule right when you accumulate gradients
The idea: the rate is a function of the step
This lesson rests on Learning rate, which showed what one fixed step size does, and on Optimizers, where an Adam step came out close to the rate itself. Here the rate stops being one number. It becomes a number for each step.
Show the answer
About 0.03
Each step moves about 0.01, so three steps move about 3 × 0.01 = 0.03. A schedule changes that 0.01 from step to step, so the total depends on the whole list of rates.
Early in training the weights are far from good, and a big step covers ground. Late in training a big step bounces across the bottom of the valley, and a small step settles into it. So the rate should fall.
The start is different. At step 0 the network's gradients are rough, and Adam's running averages have seen only one of them. A full-size step on that is a gamble. Warmup starts the rate low and raises it over the first steps. After warmup the rate follows half a cosine wave down toward zero over the remaining steps. Call the peak rate , the step , and the total number of steps .
In plain words: during warmup the rate is a growing share of the peak: 1 of , then 2 of , up to all of it. After warmup, says how far through the fall you are, from 0 to 1. The rate is the peak times a number that slides from 1 down to 0.
Use small numbers: a peak of 0.1, warmup and steps in all. Step through the picture.
Warmup up, then half a cosine wave down. The picture's numbers are the ones the code below prints.
Write a sentence or two, then compare it with a good answer in the lesson.
From scratch: a schedule is a function
The schedule is a plain function from a step number to a rate. The stepper runs it for step 6, where .
The same function, in the setup above this cell, gives all ten rates. They are the numbers in the picture.
Setup (runs before every cell) · 7 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math def lr_at(step, peak, warmup, total): if step < warmup: return peak * (step + 1) / warmup p = min(1.0, (step - warmup) / (total - warmup)) return peak * (1 + math.cos(math.pi * p)) / 2
rates = [lr_at(t, 0.1, 4, 10) for t in range(10)] print([round(r, 4) for r in rates])
import math print(round(0.1 * (1 + math.cos(math.pi * 1.0)) / 2, 3))
Show the answer
0.0
cos(π) is −1, so 1 + (−1) = 0 and the rate is 0.1 × 0 ÷ 2 = 0. The cosine falls all the way to zero, unless you give it a floor.
Three schedules that people use: step decay (cut the rate by a factor every so many steps), cosine (no warmup), and warmup then cosine. Plot them over 100 steps with the same peak.
Uses the setup above, 7 lines.
import matplotlib.pyplot as plt steps = range(100) step_decay = [0.1 * 0.1 ** (t // 40) for t in steps] # divide by 10 every 40 steps cosine = [lr_at(t, 0.1, 0, 100) for t in steps] warm_cosine = [lr_at(t, 0.1, 10, 100) for t in steps] plt.plot(steps, step_decay, label="step decay") plt.plot(steps, cosine, label="cosine") plt.plot(steps, warm_cosine, label="warmup, then cosine") plt.xlabel("step") plt.ylabel("learning rate") plt.legend() print(round(warm_cosine[0], 4), round(cosine[50], 4), round(step_decay[50], 4))
The first printed number is the rate at step 0 with warmup, 0.01: a tenth of the peak. The second is the cosine at step 50, the middle, half the peak. The third is step decay after its first cut. The curves differ in shape, and they all fall by about the same amount over the run.
The mistake that wastes a run
Here is the slip the sixth ladder problem planted. A schedule is sized in some unit: steps, or epochs. The scheduler counts calls to step(). If the two do not match, the schedule runs at the wrong speed. The example below has 10 epochs of 100 steps and a schedule sized for 1,000 steps. It calls step() once per step, then once per epoch.
Worked example: sized in steps, stepped by epoch
Run it and compare the last rate of the two runs. Then change per_epoch to 50 and run it again.
Uses the setup above, 7 lines.
import matplotlib.pyplot as plt epochs, per_epoch = 10, 100 total = epochs * per_epoch # sized in steps: 1,000 right = [lr_at(t, 0.1, 0, total) for t in range(total)] # one call per step wrong = [lr_at(e, 0.1, 0, total) for e in range(epochs)] # one call per epoch print(f"stepped every step, last rate: {right[-1]:.2e}") print(f"stepped every epoch, last rate: {wrong[-1]:.5f}") plt.plot(range(total), right, label="stepped every step") plt.plot([e * per_epoch for e in range(epochs)], wrong, marker="o", label="stepped every epoch") plt.xlabel("training step") plt.ylabel("learning rate") plt.legend()
Stepped every step, the rate ends near zero, as designed: 2.47e-07. Stepped every epoch, it ends at 0.09998, so the run never left the peak. No error appears and the loss still falls, so the run looks fine. It never settles. The fix is to size the schedule in the unit you step it in, and to count the steps from the data. With per_epoch = 50 the total is 500 and the wrong run ends at 0.09992.
peak, floor = 0.1, 0.01 print(round(floor + (peak - floor) * (1 + 0) / 2, 3))
Show the answer
0.055
The distance from peak to floor is 0.09. Halfway down is 0.045 below the peak, so the rate sits 0.045 above the floor: 0.01 + 0.045 = 0.055.
The warmup rate
Write warmup_rate(step, peak, warmup): the rate for step step (counting from 0) when the rate climbs linearly over warmup steps and then stays at peak. Step 0 must not be 0: it uses 1 / warmup of the peak.
Uses the setup above, 7 lines.
def warmup_rate(step, peak, warmup): return peak
Size the schedule in steps
Write total_steps(epochs, n_examples, batch_size): how many optimizer steps a run takes, counting the short last batch of every epoch. 1,437 examples in batches of 32 make 45 steps an epoch.
Uses the setup above, 7 lines.
def total_steps(epochs, n_examples, batch_size): return epochs * n_examples
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the scheduler object
Purpose. Hand the schedule to the optimizer through LambdaLR, check it against the hand numbers, put scheduler.step() in the right place, and see what warmup does on a real run. This page runs a teaching subset of PyTorch: the same code, CPU only, and seeded numbers that differ from a real install.
LambdaLR(optimizer, lr_lambda) sets the rate to the optimizer's starting rate times lr_lambda(step). So the optimizer holds the peak, and the lambda returns a number from 0 to 1. Here the lambda is the function from the setup with a peak of 1. The cell trains nothing. It only watches the rate the optimizer would use at each step.
Uses the setup above, 7 lines.
import torch from torch.optim.lr_scheduler import LambdaLR model = torch.nn.Linear(2, 1) optimizer = torch.optim.SGD(model.parameters(), lr=0.1) # the peak rate scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, 4, 10)) seen = [] for step in range(10): seen.append(optimizer.param_groups[0]["lr"]) # the rate this step uses optimizer.step() scheduler.step() # the rate for the next step by_hand = [lr_at(t, 0.1, 4, 10) for t in range(10)] print([round(r, 4) for r in seen]) same = all(abs(a - b) < 1e-9 for a, b in zip(seen, by_hand)) print("matches the hand numbers:", same) print("get_last_lr():", scheduler.get_last_lr())
It matches. Three facts about the object:
- The scheduler sets the first rate when you build it, and a new rate on every
scheduler.step(). optimizer.param_groups[0]["lr"]is the rate in use.scheduler.get_last_lr()gives the same number as a list, one for each parameter group. Log it every step.scheduler.step()goes afteroptimizer.step(), once for each optimizer step. Calling it first skips the first rate, and PyTorch warns.
Show the answer
16
The scheduler counts optimizer steps. There is one every 4 micro-batches, so 64 ÷ 4 = 16. Size the schedule in those 16 steps too.
Show the answer
The first rate in the schedule is skipped, and PyTorch warns
The scheduler sets the rate the next step will use. Called first, it has already moved on to rate 2 before step 1 has used rate 1. With warmup, that skips the gentlest rate.
With accumulation (the idea from Mini-batches and SGD), the scheduler goes inside the if that takes the optimizer step. The cell counts how many optimizer steps get a non-zero rate when the scheduler is placed inside the if, and when it is placed in the loop body by mistake. The schedule has 16 steps and a peak of 0.1.
Uses the setup above, 7 lines.
import warnings import torch from torch.optim.lr_scheduler import LambdaLR warnings.simplefilter("ignore") # real PyTorch warns about the wrong order def rates_used(step_inside_if, accum=4, micro_batches=64): model = torch.nn.Linear(2, 1) optimizer = torch.optim.SGD(model.parameters(), lr=0.1) scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, 0, 16)) used = [] for i in range(micro_batches): if (i + 1) % accum == 0: used.append(optimizer.param_groups[0]["lr"]) optimizer.step() if step_inside_if: scheduler.step() if not step_inside_if: scheduler.step() return used right, wrong = rates_used(True), rates_used(False) print(len(right), sum(r > 1e-9 for r in right), "steps at a rate above zero") print(len(wrong), sum(r > 1e-9 for r in wrong), "steps at a rate above zero")
Placed inside the if, all 16 optimizer steps run at a rate above zero. Placed in the loop body, the scheduler ticks 4 times as fast, so the schedule ends after the first 4 optimizer steps and the other 12 steps use a rate of 0. The weights stop moving for three quarters of the run.
Now the reason for warmup, on a real run. The cell trains the digits network twice with Adam and a high peak rate of 0.3, both with a cosine fall. One has 30 steps of warmup. The other has none. It prints the highest loss seen and the accuracy on 397 held-back digits.
Uses the setup above, 7 lines.
import math import torch import torch.nn as nn import matplotlib.pyplot as plt from torch.optim.lr_scheduler import LambdaLR from sklearn.datasets import load_digits digits = load_digits() X = torch.tensor(digits.data / 16.0, dtype=torch.float32) y = torch.tensor(digits.target) X_train, y_train, X_val, y_val = X[:1400], y[:1400], X[1400:], y[1400:] steps = 300 def train(warmup): torch.manual_seed(0) model = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10)) optimizer = torch.optim.Adam(model.parameters(), lr=0.3) scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, warmup, steps)) losses = [] for _ in range(steps): idx = torch.randint(0, len(X_train), (64,)) optimizer.zero_grad() loss = nn.functional.cross_entropy(model(X_train[idx]), y_train[idx]) loss.backward() optimizer.step() scheduler.step() losses.append(loss.item()) with torch.no_grad(): acc = (model(X_val).argmax(1) == y_val).float().mean().item() return losses, acc for warmup in [0, 30]: losses, acc = train(warmup) print(f"warmup {warmup:>2}: highest loss {max(losses):5.1f}, " f"validation accuracy {acc:.3f}") smooth = [sum(losses[i:i + 10]) / 10 for i in range(0, steps, 10)] plt.plot(range(0, steps, 10), smooth, label=f"warmup {warmup}") plt.yscale("log") plt.xlabel("step") plt.ylabel("loss, mean of 10 steps") plt.legend()
Without warmup, the first steps at the full rate throw the weights far out: the loss climbs above 30 and the network ends up wrong more often than right. With 30 steps of warmup the loss never rises far past the 2.3 it started at, and the network is right about nine times in ten. The only difference between the runs is how the rate started.
Show the answer
The run without warmup took its first steps at the full rate, before Adam's averages had settled
Both reach the peak 0.3. One gets there in 30 gentle steps, and the other starts there. Both use the same seed, so they start from the same weights and see the same batches.
On the job
Four slips show up in a real week. A schedule is sized in steps and stepped by epoch, so the rate never falls. An accumulating loop calls scheduler.step() on every micro-batch. A resumed run builds a fresh scheduler, and the rate starts again from step 0 (the scheduler has its own state_dict(), like the optimizer). And a big peak rate with Adam has no warmup, so the first steps wreck the weights. Log get_last_lr()[0] next to the loss at every step. A rate curve that does not look like the one you drew is the quickest bug report there is.
Goes deeper in
- Optimizers: what the step does with the rate.
- Training at scale: scaling the rate with the batch, and what accumulation costs.
- Learning rate: why a fixed rate converges or diverges.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
What is warmup for?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 4 · about 24 hours of study
Deep learning
Neural networks, backprop and training them well
31 lessons 1 free
- The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
- Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
- Multi-layer networksStack layers to learn complex functions.35 min
- The forward passFollow numbers through a network to a prediction.36 min
- Computation graphsEvery calculation as a graph of small operations.40 min
- BackpropagationThe chain rule applied backwards to get every gradient.42 min
- Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
- PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
- PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
- The training loopForward, loss, backward, step, repeat.40 min
- Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
- OptimizersMomentum and Adam.41 min
- Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
- Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
- Weight initializationStart weights at the right scale.33 min
- Normalization layersBatch and layer norm keep training stable.40 min
- Residual connectionsSkip connections that let gradients flow.40 min
- DropoutRandomly switch off neurons to prevent overfitting.35 min
- Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
- Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
- Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
- Convolutional networksPattern detectors for images.44 min
- Embedding layersLearned lookup tables that turn tokens into vectors.38 min
- Recurrent networksProcess sequences step by step, and their limits.43 min
- LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
- Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
- AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
- GPUs and computeWhy deep learning runs on graphics cards.40 min
- Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
- A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
- Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write