Index Ai Academy
Phase 4 · Compute and scale

Training at scale: precision, accumulation, data and many GPUs

Gradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A training loop that works on a laptop does not simply run faster on bigger hardware. The batch you want doesn't fit in memory. Half the GPU's time goes on waiting for data. Eight GPUs turn out to be five times as fast as one, at eight times the price. The four techniques in this lesson are what engineers reach for in every real training run, from a fine-tune on one rented card to a model trained on thousands. Each rests on a small piece of arithmetic that you can check here in the page.

What you'll be able to do

  • Say what gradient accumulation costs and what memory it does and does not save
  • Scale the learning rate with the batch size, and find where the rule stops working
  • Explain mixed precision: what 16-bit numbers lose, and how loss scaling repairs the gradients
  • Diagnose a data loader that starves the GPU, and size its workers
  • Explain how several GPUs share one batch, and why the speedup falls short of their number
Quick check
A guess before you start. Your model trains best with batches of 64 examples, and your GPU's memory holds only 16 at a time. Without buying a bigger GPU, what could you do?
  1. Add up the gradients from 4 batches of 16, then update once
  2. Train on batches of 16, at 4 times the learning rate
  3. Run each batch of 16 four times before updating
  4. Skip three examples in every four, to save memory
Show the answer

Add up the gradients from 4 batches of 16, then update once

A batch's gradient is an average over its examples, and an average can be built in pieces. Run 16 examples, keep the gradient, run the next 16, and so on. After four, the sum is the gradient of all 64, and a single update uses it. Memory only ever held 16. This is gradient accumulation.

The idea: four shortages, four fixes

This lesson rests on GPUs and compute: training takes about 16 bytes a parameter, and a GPU's speed is a share of its peak. It also rests on Mini-batches and SGD, where gradient accumulation is built. Here the question is what each fix costs. A run in the real world runs short of four things, one at a time.

  • Memory for the batch you want. Fix: accumulate gradients.
  • Updates when the batch grows. Fix: scale the learning rate.
  • Room and speed in each number. Fix: mixed precision.
  • Data reaching the GPU, and GPUs to share the work. Fix: loader workers and data-parallel training.

Start with the memory, by hand. Four batches of 16 make an effective batch of 64. An epoch of 4,096 examples still runs 4,096 / 16 = 256 small batches, so it takes as long as before. But the weights change once for every four: 64 updates. What shrinks is the activations, the in-between results kept for the backward pass, because they grow with the batch in memory: 16 examples' worth instead of 64. What does not shrink is the 16 bytes a parameter for weights, gradients, the master copy and Adam's averages. They are there whatever the batch.

Quick check
From the lesson on GPUs and compute: training with mixed precision and Adam takes about 16 bytes of memory for every weight, before activations. Four of those bytes are a 32-bit "master copy" of the weight. What is it for?
  1. So that tiny updates aren't rounded away
  2. To make the forward pass more accurate
  3. So that Adam can skip its running averages
  4. As a backup in case the GPU crashes
Show the answer

So that tiny updates aren't rounded away

The 16 bytes are 2 for the 16-bit weight used in the arithmetic, 2 for its gradient, 4 for the 32-bit master copy, and 8 for Adam's two running averages. Updates are added to the master copy, where a change of one part in ten thousand survives.

Quick check
A GPU fits batches of 16, and gradients are added up 4 at a time. The dataset has 4,096 examples. What does this print: the effective batch, the batches run in one epoch, and the updates made?
print(16 * 4, 4096 // 16, 4096 // 16 // 4)
  1. 64 256 64
  2. 64 256 256
  3. 16 256 64
  4. 64 64 64
Show the answer

64 256 64

Four batches of 16 make an effective batch of 64. An epoch still runs all 256 small batches. But the weights change only once for every four: 64 updates. Accumulation changes how many examples stand behind each update. It doesn't reduce the work.

Small numbers lose to the grid

A 16-bit float has about three decimal digits. Near 1, the next number above 1 is 2−102^{-10} away, which is 0.000977. Anything closer rounds to a neighbor. A typical weight update is 0.0001, less than half that gap, so 1 + 0.0001 rounds back to exactly 1: the update vanishes. In plain words: a number only has so many digits, and a tiny change to a big number is lost. Step through it.

Try it now

The gap near 1 is 2 to the power of minus 10, about 0.001.

The same trouble hits gradients: a gradient of 0.00000002 is below the smallest 16-bit number and becomes exactly 0. Loss scaling multiplies the loss by a large number, 65,536 (which is 2162^{16}), before the backward pass, so every gradient comes out that much larger. Then it divides by the same number in 32 bits before the update.

Waiting for data, and for each other

A GPU can only train on batches it has been given. Suppose it needs 20 ms for a batch, and one loader worker needs 80 ms to prepare one. Four workers deliver a batch every 80 / 4 = 20 ms: just enough.

In your own words
A colleague says accumulation "lets us train faster with a bigger batch". Which part of that is right?

Write a sentence or two, then compare it with a good answer in the lesson.

From scratch: measure each fix

The network below is small enough for its gradient to be written out by hand in NumPy. Its code is in this lesson's setup, folded above the first cell. The first cell compares one batch of 64 with the same rows cut into four batches of 16, then trains three ways.

Setup (runs before every cell) · 44 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

def examples(n, seed):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    return X, np.sin(X[:, 0]) + 0.5 * X[:, 1] * X[:, 2] + 0.1 * rng.normal(0, 1, n)

def new_weights(hidden=32):
    rng = np.random.default_rng(0)
    return {"W1": rng.normal(0, 0.3, (8, hidden)), "b1": np.zeros(hidden),
            "W2": rng.normal(0, 0.3, hidden), "b2": 0.0}

def loss_and_gradient(w, X, y):
    """A network with one hidden layer, and the gradient of its mean squared error,
    worked out by hand."""
    h = np.tanh(X @ w["W1"] + w["b1"])
    error = h @ w["W2"] + w["b2"] - y
    d_out = 2 * error / len(y)
    d_h = np.outer(d_out, w["W2"]) * (1 - h ** 2)
    return (float((error ** 2).mean()),
            {"W1": X.T @ d_h, "b1": d_h.sum(axis=0), "W2": h.T @ d_out,
             "b2": float(d_out.sum())})

X, y = examples(4096, 0)
X_new, y_new = examples(2000, 1)

def train(batch, rate, epochs, accumulate=1):
    """Plain gradient descent. With accumulate > 1, gradients from several batches are
    added up before each update."""
    w, rng, updates = new_weights(), np.random.default_rng(0), 0
    for epoch in range(epochs):
        order = rng.permutation(len(y))
        total, held = None, 0
        for start in range(0, len(y) - batch + 1, batch):
            rows = order[start:start + batch]
            loss, g = loss_and_gradient(w, X[rows], y[rows])
            total = g if total is None else {k: total[k] + g[k] for k in g}
            held += 1
            if held == accumulate:
                for k in w:
                    # the average of the gradients held
                    w[k] = w[k] - rate * total[k] / accumulate
                total, held, updates = None, 0, updates + 1
    return loss_and_gradient(w, X_new, y_new)[0], updates, w
w = new_weights()
# one batch of 64
loss, big = loss_and_gradient(w, X[:64], y[:64])
# the same rows, as four batches of 16
parts = [loss_and_gradient(w, X[i:i + 16], y[i:i + 16])[1] for i in range(0, 64, 16)]
average = {k: sum(part[k] for part in parts) / 4 for k in big}
difference = max(float(np.abs(big[k] - average[k]).max()) for k in big)
print("the batch-of-64 gradient and the average of the four agree to "
      "12 decimal places:", difference < 1e-12)
print("largest difference from the first batch of 16 alone:",
      round(max(float(np.abs(big[k] - parts[0][k]).max()) for k in big), 3))

print()
print("training for 5 epochs at a learning rate of 0.05       | updates | "
      "error on new data")
a, b, c = train(64, 0.05, 5), train(16, 0.05, 5, accumulate=4), train(16, 0.05, 5)
print("batches of 64                                          |", str(a[1]).rjust(7),
      "|", round(a[0], 4))
print("batches of 16, gradients added up 4 at a time          |", str(b[1]).rjust(7),
      "|", round(b[0], 4))
print("batches of 16, an update after every one               |", str(c[1]).rjust(7),
      "|", round(c[0], 4))
print("the first two end with the same weights, to 12 decimal places:",
      max(float(np.abs(a[2][k] - b[2][k]).max()) for k in a[2]) < 1e-12)

The gradient of the batch of 64 and the average of the four small gradients agree to 12 decimal places. They are the same sum, added up in a different order. One small batch alone is a different matter: its gradient differs by as much as 0.768.

Batches of 64, and batches of 16 with gradients added up 4 at a time, make the same 320 updates, reach the same error of 0.1984, and end with the same weights. The third row is the surprise. Batches of 16 with an update after each one reach an error of 0.0456, far better. It made 1,280 updates from the same data, four times as many.

So accumulation reproduces a big batch, with a big batch's smoother gradient and its smaller number of updates. It saves activation memory. It saves no time, and it does not make training better than small batches would.

Bigger batches need bigger steps

Why want a big batch, if small ones made more updates and did better? On many GPUs a big batch is the whole point: it is the work that gets shared out. The price is fewer updates for each pass through the data. The usual compensation is to multiply the learning rate by the same factor as the batch. A gradient averaged over more examples is less noisy, so it can be trusted further. Every run below sees the same data eight times.

Uses the setup above, 44 lines.

import matplotlib.pyplot as plt

batches, unscaled, scaled = (16, 64, 256, 1024), [], []
print("batch | updates | error at a rate of 0.02 | rate scaled with the batch | "
      "error at that rate")
for batch in batches:
    same = train(batch, 0.02, 8)
    rate = 0.02 * batch / 16
    with np.errstate(all="ignore"):
        big = train(batch, rate, 8)
    unscaled.append(same[0])
    scaled.append(min(big[0], 1.0))
    result = f"{big[0]:.4f}" if big[0] < 1000 else "blew up"
    print(str(batch).rjust(5), "|", str(same[1]).rjust(7), "|", f"{same[0]:23.4f}", "|",
          f"{rate:26.2f}", "|", result)

plt.plot(batches, unscaled, marker="o", label="rate fixed at 0.02")
plt.plot(batches, scaled, marker="s", label="rate scaled (capped at 1.0 for the plot)")
plt.xscale("log", base=2)
plt.xlabel("batch size")
plt.ylabel("error on new data")
plt.legend()
plt.show()

With the rate left at 0.02, the error climbs as the batch grows: 0.0557 at a batch of 16, then 0.2424, 0.2950 and 0.3115. Nothing is wrong with those runs except that they took too few steps of an unchanged size: 32 updates at a batch of 1,024, against 2,048.

At a batch of 64 with the rate scaled to 0.08, the error is 0.0580, almost all the way back. At 256 with a rate of 0.32 it is 0.1269: better than unscaled, and no longer a full recovery. At 1,024 the rule asks for a rate of 1.28, and training blows up. The scaling rule holds over a moderate range, and then runs into the largest step the loss surface will tolerate. Large runs push the limit out a little by raising the rate gradually over the first steps, called warmup. On a problem this small it makes no useful difference.

Quick check
A run that worked with a batch of 32 and a learning rate of 0.001 is moved to 8 GPUs, with a total batch of 256. Its loss now falls much more slowly per epoch. What is the first thing to try?
  1. Add more GPUs, to make the batch larger still
  2. Raise the learning rate, toward 8 times its old value
  3. Lower the learning rate, since there is more data in each step
  4. Reduce the model's size, to match the new batch
Show the answer

Raise the learning rate, toward 8 times its old value

With a batch eight times larger, each epoch makes an eighth as many updates. Steps of the old size no longer cover the same ground. Scaling the learning rate with the batch restores most of the progress, up to the largest rate the model can take, and a gradual warmup helps reach it safely.

Half the bits: mixed precision

A 16-bit number takes half the memory, and modern GPUs multiply 16-bit matrices considerably faster. Mixed precision runs most of the forward and backward pass in 16 bits. The word "mixed" is there for a reason, and the cell shows it.

Uses the setup above, 44 lines.

half = np.finfo(np.float16)
print("16-bit floats: largest", float(half.max), "| smallest normal number",
      float(half.tiny), "| gap between 1 and the next number", float(half.eps))
with np.errstate(over="ignore"):
    print("300 x 300 in 16 bits:", np.float16(300) * np.float16(300))
print("1 + 0.0001 in 16 bits:", np.float16(1.0) + np.float16(0.0001), "| in 32 bits:",
      np.float32(1.0) + np.float32(0.0001))
print("a gradient of 0.00000002 in 16 bits:", np.float16(2e-8))
# multiply the loss by 65,536 first, so the gradient is that much bigger
scaled = np.float16(2e-8 * 65536)
print("the same gradient with the loss scaled up:", scaled,
      "| divided back down in 32 bits:", np.float32(scaled) / np.float32(65536))

loss, g = loss_and_gradient(new_weights(), X[:64], y[:64])
print()
print("typical size of gradient | entries that become exactly 0 in 16 bits | "
      "with the loss scaled by 65,536")
# deeper networks and larger batches give smaller gradients
for shrink in (1e-3, 1e-4, 1e-5, 1e-6):
    small = g["W1"] * shrink
    print(f"{np.abs(small).mean():24.1e}", "|",
          f"{100 * (small.astype(np.float16) == 0).mean():38.1f}%", "|",
          f"{100 * ((small * 65536).astype(np.float16) == 0).mean():.1f}%")

weights = np.zeros(1_000_000, dtype=np.float32)
print()
print("a million numbers: 32-bit", weights.nbytes, "bytes | 16-bit",
      weights.astype(np.float16).nbytes, "bytes")

A 16-bit float runs out of room at both ends, and in the middle.

  • At the top. The largest value is 65,504. 300 × 300 is already inf.
  • In the middle. Near 1, the gap to the next number is about 0.001. Adding 0.0001 to a weight of 1 changes nothing. So the weights themselves stay in 32 bits, and only a 16-bit copy is used for the arithmetic.
  • At the bottom. A gradient of 0.00000002 becomes exactly 0, and a weight with a zero gradient doesn't learn.

When gradient entries are typically around 7e-07 in size, 11.7% of them vanish in 16 bits. At 7e-08, 42.2% do. With loss scaling, the table's last column is 0.0% throughout, and the gradient of 0.00000002 comes back as 1.9994332e-08.

So mixed precision is three things together: 16-bit arithmetic for speed and memory, 32-bit weights for small updates, and loss scaling for small gradients. A newer 16-bit format, bfloat16, keeps the range of 32-bit numbers and gives up precision, so it needs no loss scaling.

Quick check
What does this print: whether adding 0.0001 to 1 leaves it unchanged, in 16-bit and then in 32-bit floats?
import numpy as np

print(np.float16(1.0) + np.float16(0.0001) == np.float16(1.0),
      np.float32(1.0) + np.float32(0.0001) == np.float32(1.0))
  1. False True
  2. False False
  3. True False
  4. True True
Show the answer

True False

Near 1, 16-bit floats are about 0.001 apart, so 1.0001 rounds back to 1: the addition does nothing. In 32 bits the spacing near 1 is about a ten-millionth, and 1.0001 is stored without trouble.

Feeding the GPU

The cell simulates a loader. The GPU needs 20 ms to train on a batch. One worker needs 80 ms to prepare one.

Uses the setup above, 44 lines.

def run(step_ms, load_ms, workers, batches=400):
    """Each worker takes load_ms to prepare a batch. The GPU takes step_ms to train on
    one, and can't start until it is ready."""
    gpu_free, waiting = 0.0, 0.0
    for k in range(batches):
        # with several workers, that many batches are ready at once
        ready = (k // workers + 1) * load_ms
        start = max(ready, gpu_free)
        waiting += start - gpu_free
        gpu_free = start + step_ms
    return 100 * (1 - waiting / gpu_free), 1000 * batches / gpu_free

print("the GPU needs 20 ms for a batch. "
      "One worker needs 80 ms to load and prepare one.")
print("workers | GPU busy | batches a second")
for workers in (1, 2, 4, 8):
    busy, rate = run(20, 80, workers)
    print(str(workers).rjust(7), "|", f"{busy:7.0f}%", "|", round(rate, 1))

print()
print("a faster GPU, 10 ms for a batch, with the same 4 workers:")
busy, rate = run(10, 80, 4)
print("   GPU busy", f"{busy:.0f}%", "| batches a second", round(rate, 1))
It runs right here in your browser, with nothing to install.

With one worker the GPU is busy 25% of the time, and trains 12.5 batches a second. Two workers double the speed. Four bring the GPU to 99% busy and 49.5 batches a second. Eight add nothing: the GPU is now the limit, at one batch every 20 ms.

The last lines are the expensive mistake. Swap in a GPU twice as fast and keep the four workers. It is busy 50% of the time, and the speed is 49.8 batches a second: the same as before. The faster GPU bought nothing, since the data arrives no sooner.

So before paying for more compute, look at GPU utilization as monitoring tools report it: the share of time the GPU has any work to do. (The lesson on GPUs used the word for the share of peak speed.) If the GPU is often idle, the problem is upstream. The cures, cheapest first: more workers until the CPU cores run out, doing slow preparation once ahead of time, smaller files, and a few prepared batches waiting in a queue.

Drill

How many workers keep the GPU busy?

Write workers_needed(step_ms, load_ms): the smallest whole number of workers for which batches arrive at least as fast as the GPU uses them. The GPU takes step_ms for a batch, and one worker takes load_ms to prepare one, so n workers deliver a batch every load_ms / n. Always return at least 1.

Uses the setup above, 44 lines.

def workers_needed(step_ms, load_ms):
    return 1


print(workers_needed(20, 80), workers_needed(10, 80), workers_needed(100, 80))
In the lesson, your code is checked as you write it.

Many GPUs, one batch

Four batches of 16 can be run one after another on one GPU, with the gradients added up. Or they can be run at the same moment on four GPUs, each holding a copy of the model, with the gradients averaged. The arithmetic is identical, and so is the result: one update from a batch of 64. This is data-parallel training. Every GPU must apply the same update, so at each step the gradients are exchanged and averaged. That takes time, and some work can't be shared.

Uses the setup above, 44 lines.

def step_time(compute_ms, fixed_ms, sync_ms, gpus):
    """Time for one update on a batch of fixed total size. The compute is shared out,
    and sharing has a cost."""
    return compute_ms / gpus + fixed_ms + (sync_ms if gpus > 1 else 0)

# ms: work that splits across GPUs, work that doesn't, and exchanging gradients
compute, fixed, sync = 120, 4, 6
print("a simple model of one update: 120 ms of work that splits, 4 ms that doesn't, "
      "6 ms to exchange gradients")
print("GPUs | ms for each update | times faster than one GPU | share of the ideal")
one = step_time(compute, fixed, sync, 1)
counts, speedups = (1, 2, 4, 8, 16), []
for gpus in counts:
    t = step_time(compute, fixed, sync, gpus)
    speedups.append(one / t)
    print(str(gpus).rjust(4), "|", f"{t:18.1f}", "|", f"{one / t:25.2f}", "|",
          f"{100 * one / t / gpus:.0f}%")
print("the limit, however many GPUs:", round(one / (fixed + sync), 1), "times faster")

import matplotlib.pyplot as plt
plt.plot(counts, counts, linestyle="--", color="gray", label="ideal")
plt.plot(counts, speedups, marker="o", label="this model")
plt.xlabel("GPUs")
plt.ylabel("times faster than one GPU")
plt.legend()
plt.show()

This is a model with made-up but plausible costs, and its shape is the point. Two GPUs are 1.77 times as fast as one, and four are 3.10 times as fast. Eight give 4.96, and sixteen give 7.09: 44% of what sixteen GPUs ideally would. The 4 ms that doesn't split and the 6 ms of exchange stay as the shared work shrinks. With sixteen GPUs the fixed costs are 10 ms, more than half of every step. However many GPUs are added, this job can never be more than 12.4 times faster. The principle is known as Amdahl's law.

Two consequences. To use many GPUs well, each must be given enough work that the fixed costs stay small, which means a bigger total batch, and that leads back to the learning rate. And layers that compute statistics over a batch, such as batch normalization, see only their own GPU's share unless they are told to combine across GPUs.

Drill

Combine gradients from uneven batches

Write combine(gradients, sizes). Each entry of gradients is a list of numbers: the mean gradient over one batch. sizes gives the number of examples in each batch. Return the mean gradient over all the examples together, as a list. A larger batch must count for more: weight each gradient by its size. Return [] when there are no batches.

Uses the setup above, 44 lines.

def combine(gradients, sizes):
    return []


print(combine([[1.0, 2.0], [3.0, 6.0]], [16, 16]), combine([[0.0], [10.0]], [30, 10]))
In the lesson, your code is checked as you write it.
Worked example

Worked example: planning a run

The four ideas combine into a plan. The job: a model with 300 million weights, a target batch of 512 that earlier experiments say it needs, and GPUs with 24 GB of memory. Each example's activations take 0.55 GB in 32-bit. How should the batch be made up?

Uses the setup above, 44 lines.

import math

weights = 300e6
gpu_gb, target = 24, 512
# 16 bytes for each weight: weights, gradients and Adam's two averages
fixed_gb = weights * 16 / 1e9
# activations kept for the backward pass, for one example, in 32-bit
per_example_gb = 0.55

def fits(per_example):
    return math.floor((gpu_gb - fixed_gb) / per_example)

print("weights, gradients and optimizer:", fixed_gb, "GB, whatever the batch")
print("largest batch on one GPU, 32-bit:", fits(per_example_gb))
print("largest batch on one GPU, mixed precision (activations in 16-bit):",
      fits(per_example_gb / 2))

per_gpu = 64                                      # a round number under the limit
print()
print("GPUs | batch on each | batches added up before an update | effective batch | "
      "updates in an epoch of 1,000,000 examples")
for gpus in (1, 2, 4, 8):
    accumulate = math.ceil(target / (per_gpu * gpus))
    effective = per_gpu * gpus * accumulate
    print(str(gpus).rjust(4), "|", str(per_gpu).rjust(13), "|",
          str(accumulate).rjust(33), "|", str(effective).rjust(15), "|",
          format(1_000_000 // effective, ","))
In the lesson you can change this and run it.

The weights, gradients and optimizer take 4.8 GB whatever happens: 16 bytes for each of 300 million weights. That leaves 19.2 GB for activations. In 32-bit, 34 examples fit. With mixed precision the activations take half the space and 69 fit, so a batch of 64 on each GPU is a comfortable round number. From there the table reaches 512 four ways. One GPU adds up 8 batches before each update. Two GPUs add up 4, four add up 2, and eight need no accumulation at all. Every row makes the same 1,953 updates in an epoch of a million examples, and to the model they are the same training run. What differs is the wall-clock time and the bill: one GPU takes roughly eight times as long as eight, less whatever the eight lose to exchanging gradients. Notice that accumulation never touched the 4.8 GB: it only lets the activations stay small. Two cautions belong with any such plan. The learning rate has to suit a batch of 512, whichever row is chosen. And every row needs enough data workers for the GPUs it uses.

Quick check
A GPU has 24 GB. The weights, gradients and optimizer take 4.8 GB, and each example needs 0.55 GB. Batches of 64 run on each of 2 GPUs, and the target batch is 512. What does this print: the largest batch that fits in 32-bit, and the batches to add up before each update?
import math

print(math.floor((24 - 4.8) / 0.55), math.ceil(512 / (64 * 2)))
  1. 43 4
  2. 35 2
  3. 34 8
  4. 34 4
Show the answer

34 4

After the fixed 4.8 GB, 19.2 GB is left, and 19.2 / 0.55 is about 34.9, so 34 examples fit: memory rounds down. Two GPUs with 64 each make 128 examples a step, and 512 / 128 = 4 steps must be added up for each update.

In your own words
Accumulating gradients over 4 batches on one GPU, and averaging gradients from 4 GPUs: what is the same about them, and what is different?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the loop that holds all four

Each technique is a few lines in a real training loop. The card below runs on real PyTorch. On a machine with a GPU it uses float16 and the GradScaler. On a CPU it falls back to bfloat16, which needs no scaling, so the same code runs everywhere. It runs 8 batches, adds up 4 at a time, and counts the updates, as the from-scratch cell did.

On the job

  • Many GPUs use DistributedDataParallel: one process for each GPU, each given a different share of the data by a DistributedSampler, with gradients averaged at every step. It needs several GPUs to run, so this page has no card for it. On a cluster it is started with torchrun.
  • Measure first. Read GPU utilization before you buy anything. A job at 30 percent is waiting for data.
  • Order of attack. Memory first, since a job that crashes makes no progress: batch, mixed precision, accumulation. Then data loading. Only then more GPUs, with the learning rate scaled to the total batch.
  • What breaks. Gradients that are not divided by the number of accumulation steps give an update that is 4 times too large. A loss scale that is too big makes gradients inf, and the scaler skips that step and lowers the scale.
  • Two kinds of batch. Batch norm sees only its own GPU's share. Tell it to combine across GPUs, or use layer norm.
In your own words
You inherit a training job that is slow and sometimes runs out of memory. In what order would you investigate, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

A run adds up gradients over 8 batches of 32 before each update. What has changed, compared with updating after every batch of 32?

  1. Each update is based on 256 examples
  2. Each epoch takes an eighth of the time
  3. The GPU needs eight times the memory
  4. The model sees each example eight times

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 4 · about 24 hours of study

Deep learning

Neural networks, backprop and training them well 31 lessons 1 free
  1. The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
  2. Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
  3. Multi-layer networksStack layers to learn complex functions.35 min
  4. The forward passFollow numbers through a network to a prediction.36 min
  5. Computation graphsEvery calculation as a graph of small operations.40 min
  6. BackpropagationThe chain rule applied backwards to get every gradient.42 min
  7. Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
  8. PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
  9. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
  10. The training loopForward, loss, backward, step, repeat.40 min
  11. Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
  12. OptimizersMomentum and Adam.41 min
  13. Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
  14. Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
  15. Weight initializationStart weights at the right scale.33 min
  16. Normalization layersBatch and layer norm keep training stable.40 min
  17. Residual connectionsSkip connections that let gradients flow.40 min
  18. DropoutRandomly switch off neurons to prevent overfitting.35 min
  19. Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
  20. Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
  21. Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
  22. Convolutional networksPattern detectors for images.44 min
  23. Embedding layersLearned lookup tables that turn tokens into vectors.38 min
  24. Recurrent networksProcess sequences step by step, and their limits.43 min
  25. LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
  26. Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
  27. AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
  28. GPUs and computeWhy deep learning runs on graphics cards.40 min
  29. Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
  30. A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
  31. Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write