Free Python developer course
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Where a training step spends its time, Amdahl's rule by hand, and cProfile on a loop with a slow batch function.
A training run that takes ten hours is not ten hours of one thing. It is thousands of steps, and each step is a little data loading, a forward pass, a backward pass and an update. Making the run faster means finding which of those is big, and changing only that part. Profiling is how you find it. Most engineers who guess at the slow part are wrong, and an afternoon spent speeding up the wrong stage saves nothing. This lesson gives you the arithmetic that says how much a fix can save, then the tools that show where the time really goes.
Measure the stages first, then speed up the biggest
A speedup helps only in proportion to how much of the step the stage takes. Without a measurement you are guessing which stage that is. This lesson is the measuring.
This rests on Performance and profiling: measure first, then fix the biggest stage. A training run is that idea repeated thousands of times.
One training step takes four stages, in order:
The step takes the sum of the four. A stage that is a tenth of the step can save at most a tenth, however fast you make it. That is Amdahl's rule: if a fraction f of the time is made s times faster, the new time is (1 − f) + f / s of the old. Make a stage infinitely fast and you still pay for everything else.
65%
The other 30% is unchanged, and the 70% is halved to 35%. 30% + 35% = 65% of the original time. Speeding up the biggest part by two saves much less than half.
times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
total = sum(times.values())
print(total, times["backward"] / total)160 0.5
The four stages add up to 160 ms, and the backward pass is 80 of them, a share of 0.5.
The picture. One step, drawn along a time axis in milliseconds. The numbers are made up, to practice with: they are not a measurement of any real model. Move the cursor along the step and read how much of it each stage takes.
Step through. The widest span is where a fix pays most.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Measure rather than guess, first for one piece of numeric code and then for a whole training loop.
From scratch, then the library. Adding up squares in a Python loop is the from-scratch version. NumPy does the same sum in compiled code. The two give the same number, and a plot at six sizes shows what the difference costs as the data grows:
from academy import measure import numpy as np def make(n): return list(range(n)), np.arange(n, dtype=np.float64) check = make(1000) print(sum(v * v for v in check[0]), float(check[1] @ check[1])) measure( {"python loop": lambda d: sum(v * v for v in d[0]), "NumPy": lambda d: float(d[1] @ d[1])}, [1_000, 2_000, 4_000, 8_000, 16_000, 32_000], setup=make, repeat=2, xlabel="numbers", )
The first line proves they agree. The chart shows both lines rising about in step with n, a slope near 1, since each must visit every number. The gap between the lines is a constant factor, and it is large: that gap is what moving the loop into NumPy buys. Never quote a time from it; times change between machines.
The constant factor is much smaller, so every step is cheaper
Big O hides the constant: both lines do work proportional to n, but each Python step goes through the interpreter, and a NumPy step is a tight compiled loop. A constant factor of 50 is the difference between a run that ends tonight and one that ends next week.
Code flow. The arithmetic of the ladder, as code. Four stage times, their total, the biggest share, and the new total after a speedup to one stage:
128
16 rows in each of 8 steps is 16 × 8 = 128 calls. A count like that, a multiple of the batch size, points at code that runs once per row.
A whole training loop. The cell below trains a tiny model for 8 steps with a deliberately wasteful batch function: it builds each batch row by row in Python, and the loop calls .item() on the loss at every step. cProfile counts every call. Times change from run to run, but call counts do not, and they point straight at the suspects:
import cProfile import pstats import torch import torch.nn as nn torch.manual_seed(0) X = torch.randn(64, 4) y = (X[:, 0] > 0).float().reshape(64, 1) def slow_batch(idx): rows = [X[i].tolist() for i in idx] return torch.tensor(rows), y[idx] def train(steps): model = nn.Linear(4, 1) opt = torch.optim.SGD(model.parameters(), lr=0.1) losses = [] for step in range(steps): idx = list(range(step % 4 * 16, step % 4 * 16 + 16)) xb, yb = slow_batch(idx) loss = nn.functional.mse_loss(model(xb), yb) opt.zero_grad() loss.backward() opt.step() losses.append(loss.item()) return losses profiler = cProfile.Profile() profiler.runcall(train, 8) calls = {} stats = pstats.Stats(profiler).stats for func, (primitive, total, own, cumulative, callers) in stats.items(): name = func[2] # "tolist" here, "{method 'tolist' ...}" in CPython for word in ("tolist", "item", "backward"): if name == word or f"'{word}'" in name: calls[word] = calls.get(word, 0) + total print("tolist:", calls["tolist"], "| item:", calls["item"]) print("backward:", calls["backward"])
(This page runs a teaching subset of PyTorch: same calls, CPU only.) Eight steps made 128 calls to tolist, 16 rows for each step: the batch function does a Python-level loop, one row at a time. And .item() ran once per step. On a GPU that call waits for all the work queued so far to finish, so calling it every step stops the GPU from running ahead of the Python.
tolist was called 128 times for 8 steps of 16 rows. What is the likely fix?Index the tensor once, X[idx], instead of converting row by row
One call to X[idx] gathers all 16 rows in one compiled operation, where the loop makes 16 Python-level trips. The counts tell you where the loop is, and the fix removes it.
import numpy as np x = np.arange(4) print(sum(int(v) * int(v) for v in x), int(x @ x))
14 14
0 + 1 + 4 + 9 = 14, by the loop and by the dot product. The two are the same sum, and the profile is about how fast each one gets there.
Write shares(times): given a dict of stage times, return a dict of each stage's share of the total, rounded to 3 decimals. Then write bottleneck(times): the name of the biggest stage.
def shares(times): ... def bottleneck(times): ... times = {"data": 30, "forward": 40, "backward": 80, "update": 10} print(shares(times), bottleneck(times))
Write new_time(times, stage, speedup): the step time after the named stage is made speedup times faster. Everything else stays the same.
def new_time(times, stage, speedup): ... times = {"data": 30, "forward": 40, "backward": 80, "update": 10} print(new_time(times, "forward", 2))
Three candidate fixes for the step above: halve the data stage, halve the forward pass, or halve the backward pass. Which saves the most, and how much of the step does each save?
times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
total = sum(times.values())
for stage in ["data", "forward", "backward"]:
saved = times[stage] / 2
print(stage, "saves", saved, "ms,", round(saved / total * 100, 1), "% of the step")Halving a stage saves half of its time, so the biggest stage saves the most: backward saves 40 ms, a quarter of the step, while halving data saves 15 ms, under a tenth. The rule is the same as in the ladder. The order of the fixes is the order of the stages by size, and a profile tells you the sizes.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. On a real model the same four stages are there, and the tool is a profiler that also sees the GPU.
A GPU is only useful while it has work. If the data loader cannot deliver the next batch in time, the GPU sits idle between steps, and no amount of work on the model will help. The signs are a profile where the data stage is large, or a GPU whose usage in a monitor keeps dropping to zero between steps. The usual fixes, cheapest first: load batches in background workers so they overlap with compute, keep the data in memory, index a whole batch at once, and avoid .item() or .tolist() on a GPU tensor inside the step.
Here is the batch gathering done two ways on a NumPy array, one row at a time and all at once, plotted as the batch grows:
from academy import measure import numpy as np def make(n): rng = np.random.default_rng(0) return rng.standard_normal((10_000, 8)), list(range(n)) data, idx = make(100) rows = np.array([data[i] for i in idx]) print(np.array_equal(rows, data[idx])) measure( {"row by row": lambda d: np.array([d[0][i] for i in d[1]]), "index at once": lambda d: d[0][d[1]]}, [100, 200, 400, 800, 1600, 3200], setup=make, repeat=2, xlabel="rows in the batch", )
The first line proves both give the same batch. Both lines rise with the batch size, but the row-by-row line sits far above the other: each trip through the Python loop is expensive and the gather is one compiled step.
On the job: profile before you change anything, and after every change, so you know the fix worked and by how much. Keep the numbers: the profile before and after is what you put in the pull request. For a running service,
py-spy topshows where a live Python process spends its time without stopping it. And remember the order: measure, find the biggest stage, fix that one, measure again.
Write a sentence or two, then compare it with a good answer in the lesson.
A stage is 10% of a step. What is the best total saving from speeding it up, however much?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write