Index Ai Academy
Phase 4 · Training

Learning-rate schedules in PyTorch

Warmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.

About 25 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A training run does not use one learning rate from the first step to the last. Big steps cover ground early. Small steps settle the weights at the end. And the first few steps are the shakiest, so most real runs start the rate low and raise it. Every training script you will read has a scheduler for this. It is a small object, and most of the bugs it causes come from one place: counting the wrong thing.

What you'll be able to do

  • Work out a warmup-then-cosine rate at any step by hand
  • Write the schedule as a function of the step, and size it in steps, not epochs
  • Drive a PyTorch optimizer with LambdaLR, and know where scheduler.step() goes
  • Keep a schedule right when you accumulate gradients

The idea: the rate is a function of the step

This lesson rests on Learning rate, which showed what one fixed step size does, and on Optimizers, where an Adam step came out close to the rate itself. Here the rate stops being one number. It becomes a number for each step.

Quick check
From the optimizers lesson: when gradients are steady, an Adam step is about as big as the learning rate. A weight gets steady gradients for 3 steps at a rate of 0.01. About how far does it move in all?
  1. About 0.03
  2. About 0.01
  3. About 0.003
  4. About 0.3
Show the answer

About 0.03

Each step moves about 0.01, so three steps move about 3 × 0.01 = 0.03. A schedule changes that 0.01 from step to step, so the total depends on the whole list of rates.

Early in training the weights are far from good, and a big step covers ground. Late in training a big step bounces across the bottom of the valley, and a small step settles into it. So the rate should fall.

The start is different. At step 0 the network's gradients are rough, and Adam's running averages have seen only one of them. A full-size step on that is a gamble. Warmup starts the rate low and raises it over the first WW steps. After warmup the rate follows half a cosine wave down toward zero over the remaining steps. Call the peak rate η\eta, the step tt, and the total number of steps TT.

warmup, t<W:ηt=η⋅t+1W\text{warmup, } t < W:\quad \eta_t = \eta \cdot \frac{t + 1}{W}
cosine, t≥W:ηt=η⋅1+cos⁡(πp)2,p=t−WT−W\text{cosine, } t \ge W:\quad \eta_t = \eta \cdot \frac{1 + \cos(\pi p)}{2}, \qquad p = \frac{t - W}{T - W}

In plain words: during warmup the rate is a growing share of the peak: 1 of WW, then 2 of WW, up to all of it. After warmup, pp says how far through the fall you are, from 0 to 1. The rate is the peak times a number that slides from 1 down to 0.

Use small numbers: a peak of 0.1, warmup W=4W = 4 and T=10T = 10 steps in all. Step through the picture.

Try it now

Warmup up, then half a cosine wave down. The picture's numbers are the ones the code below prints.

In your own words
In your own words: why does the rate start low, and why does it end low?

Write a sentence or two, then compare it with a good answer in the lesson.

From scratch: a schedule is a function

The schedule is a plain function from a step number to a rate. The stepper runs it for step 6, where p=1/3p = 1/3.

The same function, in the setup above this cell, gives all ten rates. They are the numbers in the picture.

Setup (runs before every cell) · 7 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math

def lr_at(step, peak, warmup, total):
    if step < warmup:
        return peak * (step + 1) / warmup
    p = min(1.0, (step - warmup) / (total - warmup))
    return peak * (1 + math.cos(math.pi * p)) / 2
rates = [lr_at(t, 0.1, 4, 10) for t in range(10)]
print([round(r, 4) for r in rates])
Quick check
The rate at the very end of the cosine part, where p = 1, is what this prints?
import math
print(round(0.1 * (1 + math.cos(math.pi * 1.0)) / 2, 3))
  1. -0.1
  2. 0.1
  3. 0.0
  4. 0.05
Show the answer

0.0

cos(π) is −1, so 1 + (−1) = 0 and the rate is 0.1 × 0 ÷ 2 = 0. The cosine falls all the way to zero, unless you give it a floor.

Three schedules that people use: step decay (cut the rate by a factor every so many steps), cosine (no warmup), and warmup then cosine. Plot them over 100 steps with the same peak.

Uses the setup above, 7 lines.

import matplotlib.pyplot as plt

steps = range(100)
step_decay = [0.1 * 0.1 ** (t // 40) for t in steps]      # divide by 10 every 40 steps
cosine = [lr_at(t, 0.1, 0, 100) for t in steps]
warm_cosine = [lr_at(t, 0.1, 10, 100) for t in steps]
plt.plot(steps, step_decay, label="step decay")
plt.plot(steps, cosine, label="cosine")
plt.plot(steps, warm_cosine, label="warmup, then cosine")
plt.xlabel("step")
plt.ylabel("learning rate")
plt.legend()
print(round(warm_cosine[0], 4), round(cosine[50], 4), round(step_decay[50], 4))

The first printed number is the rate at step 0 with warmup, 0.01: a tenth of the peak. The second is the cosine at step 50, the middle, half the peak. The third is step decay after its first cut. The curves differ in shape, and they all fall by about the same amount over the run.

The mistake that wastes a run

Here is the slip the sixth ladder problem planted. A schedule is sized in some unit: steps, or epochs. The scheduler counts calls to step(). If the two do not match, the schedule runs at the wrong speed. The example below has 10 epochs of 100 steps and a schedule sized for 1,000 steps. It calls step() once per step, then once per epoch.

Worked example

Worked example: sized in steps, stepped by epoch

Run it and compare the last rate of the two runs. Then change per_epoch to 50 and run it again.

Uses the setup above, 7 lines.

import matplotlib.pyplot as plt

epochs, per_epoch = 10, 100
total = epochs * per_epoch                                # sized in steps: 1,000
right = [lr_at(t, 0.1, 0, total) for t in range(total)]   # one call per step
wrong = [lr_at(e, 0.1, 0, total) for e in range(epochs)]  # one call per epoch
print(f"stepped every step, last rate:  {right[-1]:.2e}")
print(f"stepped every epoch, last rate: {wrong[-1]:.5f}")
plt.plot(range(total), right, label="stepped every step")
plt.plot([e * per_epoch for e in range(epochs)], wrong, marker="o",
         label="stepped every epoch")
plt.xlabel("training step")
plt.ylabel("learning rate")
plt.legend()
In the lesson you can change this and run it.

Stepped every step, the rate ends near zero, as designed: 2.47e-07. Stepped every epoch, it ends at 0.09998, so the run never left the peak. No error appears and the loss still falls, so the run looks fine. It never settles. The fix is to size the schedule in the unit you step it in, and to count the steps from the data. With per_epoch = 50 the total is 500 and the wrong run ends at 0.09992.

Quick check
With a floor, the cosine part runs from the peak down to the floor, not to zero. At the middle (p = 0.5, cosine term 0), what does this print?
peak, floor = 0.1, 0.01
print(round(floor + (peak - floor) * (1 + 0) / 2, 3))
  1. 0.1
  2. 0.01
  3. 0.055
  4. 0.05
Show the answer

0.055

The distance from peak to floor is 0.09. Halfway down is 0.045 below the peak, so the rate sits 0.045 above the floor: 0.01 + 0.045 = 0.055.

Drill

The warmup rate

Write warmup_rate(step, peak, warmup): the rate for step step (counting from 0) when the rate climbs linearly over warmup steps and then stays at peak. Step 0 must not be 0: it uses 1 / warmup of the peak.

Uses the setup above, 7 lines.

def warmup_rate(step, peak, warmup):
    return peak
In the lesson, your code is checked as you write it.
Drill

Size the schedule in steps

Write total_steps(epochs, n_examples, batch_size): how many optimizer steps a run takes, counting the short last batch of every epoch. 1,437 examples in batches of 32 make 45 steps an epoch.

Uses the setup above, 7 lines.

def total_steps(epochs, n_examples, batch_size):
    return epochs * n_examples
In the lesson, your code is checked as you write it.
In your own words
In your own words: a schedule is sized for 1,000 steps but your script steps it once per epoch. What happens to the run, and how would you notice?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the scheduler object

Purpose. Hand the schedule to the optimizer through LambdaLR, check it against the hand numbers, put scheduler.step() in the right place, and see what warmup does on a real run. This page runs a teaching subset of PyTorch: the same code, CPU only, and seeded numbers that differ from a real install.

LambdaLR(optimizer, lr_lambda) sets the rate to the optimizer's starting rate times lr_lambda(step). So the optimizer holds the peak, and the lambda returns a number from 0 to 1. Here the lambda is the function from the setup with a peak of 1. The cell trains nothing. It only watches the rate the optimizer would use at each step.

Uses the setup above, 7 lines.

import torch
from torch.optim.lr_scheduler import LambdaLR

model = torch.nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)      # the peak rate
scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, 4, 10))
seen = []
for step in range(10):
    seen.append(optimizer.param_groups[0]["lr"])             # the rate this step uses
    optimizer.step()
    scheduler.step()                                     # the rate for the next step
by_hand = [lr_at(t, 0.1, 4, 10) for t in range(10)]
print([round(r, 4) for r in seen])
same = all(abs(a - b) < 1e-9 for a, b in zip(seen, by_hand))
print("matches the hand numbers:", same)
print("get_last_lr():", scheduler.get_last_lr())

It matches. Three facts about the object:

  • The scheduler sets the first rate when you build it, and a new rate on every scheduler.step().
  • optimizer.param_groups[0]["lr"] is the rate in use. scheduler.get_last_lr() gives the same number as a list, one for each parameter group. Log it every step.
  • scheduler.step() goes after optimizer.step(), once for each optimizer step. Calling it first skips the first rate, and PyTorch warns.
Quick check
A script does 64 micro-batches in one epoch and accumulates gradients over 4 of them before each optimizer.step(). How many times should scheduler.step() be called in the epoch?
  1. 1
  2. 16
  3. 64
  4. 4
Show the answer

16

The scheduler counts optimizer steps. There is one every 4 micro-batches, so 64 ÷ 4 = 16. Size the schedule in those 16 steps too.

Quick check
A loop calls scheduler.step() before optimizer.step() on every iteration. What goes wrong?
  1. The schedule runs backward
  2. The weights are never updated at all after that
  3. The first rate in the schedule is skipped, and PyTorch warns
  4. Nothing: the order does not matter
Show the answer

The first rate in the schedule is skipped, and PyTorch warns

The scheduler sets the rate the next step will use. Called first, it has already moved on to rate 2 before step 1 has used rate 1. With warmup, that skips the gentlest rate.

With accumulation (the idea from Mini-batches and SGD), the scheduler goes inside the if that takes the optimizer step. The cell counts how many optimizer steps get a non-zero rate when the scheduler is placed inside the if, and when it is placed in the loop body by mistake. The schedule has 16 steps and a peak of 0.1.

Uses the setup above, 7 lines.

import warnings
import torch
from torch.optim.lr_scheduler import LambdaLR

warnings.simplefilter("ignore")     # real PyTorch warns about the wrong order

def rates_used(step_inside_if, accum=4, micro_batches=64):
    model = torch.nn.Linear(2, 1)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
    scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, 0, 16))
    used = []
    for i in range(micro_batches):
        if (i + 1) % accum == 0:
            used.append(optimizer.param_groups[0]["lr"])
            optimizer.step()
            if step_inside_if:
                scheduler.step()
        if not step_inside_if:
            scheduler.step()
    return used

right, wrong = rates_used(True), rates_used(False)
print(len(right), sum(r > 1e-9 for r in right), "steps at a rate above zero")
print(len(wrong), sum(r > 1e-9 for r in wrong), "steps at a rate above zero")

Placed inside the if, all 16 optimizer steps run at a rate above zero. Placed in the loop body, the scheduler ticks 4 times as fast, so the schedule ends after the first 4 optimizer steps and the other 12 steps use a rate of 0. The weights stop moving for three quarters of the run.

Now the reason for warmup, on a real run. The cell trains the digits network twice with Adam and a high peak rate of 0.3, both with a cosine fall. One has 30 steps of warmup. The other has none. It prints the highest loss seen and the accuracy on 397 held-back digits.

Uses the setup above, 7 lines.

import math
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
from torch.optim.lr_scheduler import LambdaLR
from sklearn.datasets import load_digits

digits = load_digits()
X = torch.tensor(digits.data / 16.0, dtype=torch.float32)
y = torch.tensor(digits.target)
X_train, y_train, X_val, y_val = X[:1400], y[:1400], X[1400:], y[1400:]
steps = 300

def train(warmup):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
    optimizer = torch.optim.Adam(model.parameters(), lr=0.3)
    scheduler = LambdaLR(optimizer, lambda s: lr_at(s, 1.0, warmup, steps))
    losses = []
    for _ in range(steps):
        idx = torch.randint(0, len(X_train), (64,))
        optimizer.zero_grad()
        loss = nn.functional.cross_entropy(model(X_train[idx]), y_train[idx])
        loss.backward()
        optimizer.step()
        scheduler.step()
        losses.append(loss.item())
    with torch.no_grad():
        acc = (model(X_val).argmax(1) == y_val).float().mean().item()
    return losses, acc

for warmup in [0, 30]:
    losses, acc = train(warmup)
    print(f"warmup {warmup:>2}: highest loss {max(losses):5.1f}, "
          f"validation accuracy {acc:.3f}")
    smooth = [sum(losses[i:i + 10]) / 10 for i in range(0, steps, 10)]
    plt.plot(range(0, steps, 10), smooth, label=f"warmup {warmup}")
plt.yscale("log")
plt.xlabel("step")
plt.ylabel("loss, mean of 10 steps")
plt.legend()

Without warmup, the first steps at the full rate throw the weights far out: the loss climbs above 30 and the network ends up wrong more often than right. With 30 steps of warmup the loss never rises far past the 2.3 it started at, and the network is right about nine times in ten. The only difference between the runs is how the rate started.

Quick check
In that run, why does the highest loss differ so much between the two runs, when the peak rate is the same?
  1. The run with warmup used a smaller peak rate for all of its steps, so its loss stayed low
  2. The run without warmup took its first steps at the full rate, before Adam's averages had settled
  3. Warmup changes the loss function
  4. The run with warmup saw different data
Show the answer

The run without warmup took its first steps at the full rate, before Adam's averages had settled

Both reach the peak 0.3. One gets there in 30 gentle steps, and the other starts there. Both use the same seed, so they start from the same weights and see the same batches.

On the job

Four slips show up in a real week. A schedule is sized in steps and stepped by epoch, so the rate never falls. An accumulating loop calls scheduler.step() on every micro-batch. A resumed run builds a fresh scheduler, and the rate starts again from step 0 (the scheduler has its own state_dict(), like the optimizer). And a big peak rate with Adam has no warmup, so the first steps wreck the weights. Log get_last_lr()[0] next to the loss at every step. A rate curve that does not look like the one you drew is the quickest bug report there is.

Goes deeper in

In your own words
In your own words: where does scheduler.step() go in a loop with gradient accumulation, and what does the schedule have to be sized in?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What is warmup for?

  1. It keeps the first, shaky steps small, before the optimizer's averages have settled
  2. It makes each batch bigger at the start of training
  3. It changes the loss function to an easier one for the first steps
  4. It makes the cosine fall more slowly later in the run

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 4 · about 24 hours of study

Deep learning

Neural networks, backprop and training them well 31 lessons 1 free
  1. The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
  2. Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
  3. Multi-layer networksStack layers to learn complex functions.35 min
  4. The forward passFollow numbers through a network to a prediction.36 min
  5. Computation graphsEvery calculation as a graph of small operations.40 min
  6. BackpropagationThe chain rule applied backwards to get every gradient.42 min
  7. Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
  8. PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
  9. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
  10. The training loopForward, loss, backward, step, repeat.40 min
  11. Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
  12. OptimizersMomentum and Adam.41 min
  13. Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
  14. Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
  15. Weight initializationStart weights at the right scale.33 min
  16. Normalization layersBatch and layer norm keep training stable.40 min
  17. Residual connectionsSkip connections that let gradients flow.40 min
  18. DropoutRandomly switch off neurons to prevent overfitting.35 min
  19. Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
  20. Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
  21. Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
  22. Convolutional networksPattern detectors for images.44 min
  23. Embedding layersLearned lookup tables that turn tokens into vectors.38 min
  24. Recurrent networksProcess sequences step by step, and their limits.43 min
  25. LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
  26. Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
  27. AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
  28. GPUs and computeWhy deep learning runs on graphics cards.40 min
  29. Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
  30. A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
  31. Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write