Index Ai Academy
Phase 7 · How computers run code

Profiling numeric code and a training loop

Where a training step spends its time, Amdahl's rule by hand, and cProfile on a loop with a slow batch function.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A training run that takes ten hours is not ten hours of one thing. It is thousands of steps, and each step is a little data loading, a forward pass, a backward pass and an update. Making the run faster means finding which of those is big, and changing only that part. Profiling is how you find it. Most engineers who guess at the slow part are wrong, and an afternoon spent speeding up the wrong stage saves nothing. This lesson gives you the arithmetic that says how much a fix can save, then the tools that show where the time really goes.

What you'll be able to do

  • Split a training step into data, forward, backward and update, and say what share each takes
  • Use Amdahl's rule by hand to work out how much a speedup to one part can save
  • Time a Python loop against NumPy at six sizes, plot it, and read the slope
  • Run cProfile on a small training loop and find the two classic slow spots: a per-row Python loop and an .item() inside the step
Quick check
A training run is too slow. You can speed up exactly one of four stages. Which choice gives you the best chance of a real gain?
  1. Measure the stages first, then speed up the biggest
  2. Speed up the one that sounds hardest
  3. Speed up the one you know best
  4. Rewrite the update step in a faster language
Show the answer

Measure the stages first, then speed up the biggest

A speedup helps only in proportion to how much of the step the stage takes. Without a measurement you are guessing which stage that is. This lesson is the measuring.

The idea: a step is four stages, and only the big one matters

This rests on Performance and profiling: measure first, then fix the biggest stage. A training run is that idea repeated thousands of times.

One training step takes four stages, in order:

  1. data: get the next batch, ready to use;
  2. forward: run the model on the batch and compute the loss;
  3. backward: work out the slopes of the loss for every weight;
  4. update: let the optimizer move every weight a small step.

The step takes the sum of the four. A stage that is a tenth of the step can save at most a tenth, however fast you make it. That is Amdahl's rule: if a fraction f of the time is made s times faster, the new time is (1 − f) + f / s of the old. Make a stage infinitely fast and you still pay for everything else.

Quick check
From the performance lesson: a profile shows one function takes 70% of the run. If you make that function twice as fast, what share of the original time does the whole run now take?
  1. 50%
  2. 35%
  3. 85%
  4. 65%
Show the answer

65%

The other 30% is unchanged, and the 70% is halved to 35%. 30% + 35% = 65% of the original time. Speeding up the biggest part by two saves much less than half.

Quick check
What does this print?
times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
total = sum(times.values())
print(total, times["backward"] / total)
  1. 80 0.5
  2. 160 0.25
  3. 160 0.5
  4. 160 80
Show the answer

160 0.5

The four stages add up to 160 ms, and the backward pass is 80 of them, a share of 0.5.

The picture. One step, drawn along a time axis in milliseconds. The numbers are made up, to practice with: they are not a measurement of any real model. Move the cursor along the step and read how much of it each stage takes.

Try it now

Step through. The widest span is where a fix pays most.

In your own words
In your own words: why can making one stage of a training step much faster still save little time?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: timing, plotting and cProfile

Purpose. Measure rather than guess, first for one piece of numeric code and then for a whole training loop.

From scratch, then the library. Adding up squares in a Python loop is the from-scratch version. NumPy does the same sum in compiled code. The two give the same number, and a plot at six sizes shows what the difference costs as the data grows:

from academy import measure
import numpy as np

def make(n):
    return list(range(n)), np.arange(n, dtype=np.float64)

check = make(1000)
print(sum(v * v for v in check[0]), float(check[1] @ check[1]))

measure(
    {"python loop": lambda d: sum(v * v for v in d[0]),
     "NumPy": lambda d: float(d[1] @ d[1])},
    [1_000, 2_000, 4_000, 8_000, 16_000, 32_000],
    setup=make,
    repeat=2,
    xlabel="numbers",
)
It runs right here in your browser, with nothing to install.

The first line proves they agree. The chart shows both lines rising about in step with n, a slope near 1, since each must visit every number. The gap between the lines is a constant factor, and it is large: that gap is what moving the loop into NumPy buys. Never quote a time from it; times change between machines.

Quick check
On the chart both lines rise with n, with a slope near 1. If both are O(n), why does moving the loop into NumPy still matter?
  1. NumPy changes the slope of the line down to zero
  2. The constant factor is much smaller, so every step is cheaper
  3. NumPy skips most of the numbers
  4. The Python loop is O(n²)
Show the answer

The constant factor is much smaller, so every step is cheaper

Big O hides the constant: both lines do work proportional to n, but each Python step goes through the interpreter, and a NumPy step is a tight compiled loop. A constant factor of 50 is the difference between a run that ends tonight and one that ends next week.

Code flow. The arithmetic of the ladder, as code. Four stage times, their total, the biggest share, and the new total after a speedup to one stage:

Quick check
A function is called once for each of the 16 rows in a batch, and the loop runs 8 steps. How many calls does a profile show?
  1. 16
  2. 24
  3. 8
  4. 128
Show the answer

128

16 rows in each of 8 steps is 16 × 8 = 128 calls. A count like that, a multiple of the batch size, points at code that runs once per row.

A whole training loop. The cell below trains a tiny model for 8 steps with a deliberately wasteful batch function: it builds each batch row by row in Python, and the loop calls .item() on the loss at every step. cProfile counts every call. Times change from run to run, but call counts do not, and they point straight at the suspects:

import cProfile
import pstats
import torch
import torch.nn as nn

torch.manual_seed(0)
X = torch.randn(64, 4)
y = (X[:, 0] > 0).float().reshape(64, 1)

def slow_batch(idx):
    rows = [X[i].tolist() for i in idx]
    return torch.tensor(rows), y[idx]

def train(steps):
    model = nn.Linear(4, 1)
    opt = torch.optim.SGD(model.parameters(), lr=0.1)
    losses = []
    for step in range(steps):
        idx = list(range(step % 4 * 16, step % 4 * 16 + 16))
        xb, yb = slow_batch(idx)
        loss = nn.functional.mse_loss(model(xb), yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return losses

profiler = cProfile.Profile()
profiler.runcall(train, 8)
calls = {}
stats = pstats.Stats(profiler).stats
for func, (primitive, total, own, cumulative, callers) in stats.items():
    name = func[2]          # "tolist" here, "{method 'tolist' ...}" in CPython
    for word in ("tolist", "item", "backward"):
        if name == word or f"'{word}'" in name:
            calls[word] = calls.get(word, 0) + total
print("tolist:", calls["tolist"], "| item:", calls["item"])
print("backward:", calls["backward"])

(This page runs a teaching subset of PyTorch: same calls, CPU only.) Eight steps made 128 calls to tolist, 16 rows for each step: the batch function does a Python-level loop, one row at a time. And .item() ran once per step. On a GPU that call waits for all the work queued so far to finish, so calling it every step stops the GPU from running ahead of the Python.

Quick check
In the profile, tolist was called 128 times for 8 steps of 16 rows. What is the likely fix?
  1. Move the loop to a different file
  2. Index the tensor once, X[idx], instead of converting row by row
  3. Call tolist fewer times by using 8 rows per batch
  4. Use a larger learning rate
Show the answer

Index the tensor once, X[idx], instead of converting row by row

One call to X[idx] gathers all 16 rows in one compiled operation, where the loop makes 16 Python-level trips. The counts tell you where the loop is, and the fix removes it.

Quick check
The from-scratch loop and the NumPy line must agree. What does this print?
import numpy as np

x = np.arange(4)
print(sum(int(v) * int(v) for v in x), int(x @ x))
  1. 14 6
  2. 6 14
  3. 30 30
  4. 14 14
Show the answer

14 14

0 + 1 + 4 + 9 = 14, by the loop and by the dot product. The two are the same sum, and the profile is about how fast each one gets there.

Drill

What share of the step is each stage?

Write shares(times): given a dict of stage times, return a dict of each stage's share of the total, rounded to 3 decimals. Then write bottleneck(times): the name of the biggest stage.

def shares(times):
    ...


def bottleneck(times):
    ...


times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
print(shares(times), bottleneck(times))
In the lesson, your code is checked as you write it.
Drill

Amdahl's rule

Write new_time(times, stage, speedup): the step time after the named stage is made speedup times faster. Everything else stays the same.

def new_time(times, stage, speedup):
    ...


times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
print(new_time(times, "forward", 2))
In the lesson, your code is checked as you write it.
Worked example

Worked example: which fix first?

Three candidate fixes for the step above: halve the data stage, halve the forward pass, or halve the backward pass. Which saves the most, and how much of the step does each save?

times = {"data": 30, "forward": 40, "backward": 80, "update": 10}
total = sum(times.values())
for stage in ["data", "forward", "backward"]:
    saved = times[stage] / 2
    print(stage, "saves", saved, "ms,", round(saved / total * 100, 1), "% of the step")
In the lesson you can change this and run it.

Halving a stage saves half of its time, so the biggest stage saves the most: backward saves 40 ms, a quarter of the step, while halving data saves 15 ms, under a tenth. The rule is the same as in the ladder. The order of the fixes is the order of the stages by size, and a profile tells you the sizes.

In your own words
A profile shows a training step spends 10% of its time in the batch function. A teammate wants to rewrite it in C. What do you tell them?

Write a sentence or two, then compare it with a good answer in the lesson.

In a real codebase: reading a profile of a training loop

Purpose. On a real model the same four stages are there, and the tool is a profiler that also sees the GPU.

A GPU is only useful while it has work. If the data loader cannot deliver the next batch in time, the GPU sits idle between steps, and no amount of work on the model will help. The signs are a profile where the data stage is large, or a GPU whose usage in a monitor keeps dropping to zero between steps. The usual fixes, cheapest first: load batches in background workers so they overlap with compute, keep the data in memory, index a whole batch at once, and avoid .item() or .tolist() on a GPU tensor inside the step.

Here is the batch gathering done two ways on a NumPy array, one row at a time and all at once, plotted as the batch grows:

from academy import measure
import numpy as np

def make(n):
    rng = np.random.default_rng(0)
    return rng.standard_normal((10_000, 8)), list(range(n))

data, idx = make(100)
rows = np.array([data[i] for i in idx])
print(np.array_equal(rows, data[idx]))

measure(
    {"row by row": lambda d: np.array([d[0][i] for i in d[1]]),
     "index at once": lambda d: d[0][d[1]]},
    [100, 200, 400, 800, 1600, 3200],
    setup=make,
    repeat=2,
    xlabel="rows in the batch",
)

The first line proves both give the same batch. Both lines rise with the batch size, but the row-by-row line sits far above the other: each trip through the Python loop is expensive and the gather is one compiled step.

On the job: profile before you change anything, and after every change, so you know the fix worked and by how much. Keep the numbers: the profile before and after is what you put in the pull request. For a running service, py-spy top shows where a live Python process spends its time without stopping it. And remember the order: measure, find the biggest stage, fix that one, measure again.

In your own words
A GPU training run shows the GPU at 40% usage, dropping to zero between steps. Where do you look first, and what might you change?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A stage is 10% of a step. What is the best total saving from speeding it up, however much?

  1. 10% of the step
  2. 50% of the step
  3. 90% of the step
  4. Nothing: speedups never help

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 7 · about 48 hours of study

Software engineering

How software is built, from bits to tested, well-designed programs 59 lessons 1 free
  1. Iterators and generators FreeThe iteration protocol, yield, lazy pipelines and itertools, in small memory.45 min
  2. Context managersSetup and cleanup that always run: with, contextlib, transactions and atomic writes.35 min
  3. Closures and decoratorsFunctions that remember, wrap and extend other functions.40 min
  4. Exceptions in depthError hierarchies, chaining, notes, exception groups and warnings.40 min
  5. Pattern matchingmatch and case: literal, sequence, mapping and class patterns with guards.35 min
  6. Text, bytes and encodingsstr and bytes, UTF-8, decoding errors, normalization and base64.35 min
  7. Binary filesBytes as numbers, endianness, struct records, file signatures and checking uploads.45 min
  8. Testing your codeAutomated tests that prove code works and keep it working.45 min
  9. Testing with pytestFixtures, parametrize, mocks, temporary files and coverage.45 min
  10. Testing numeric and random codeClose enough for floats, seeds and what they fix, properties that hold for any input, a gradient check and overfit-one-batch.40 min
  11. Debugging systematicallyFind any bug with a method instead of guesswork.35 min
  12. Readable code and refactoringWrite code people can understand, and improve it safely.35 min
  13. Code style and toolsPEP 8, formatters, linters and type checkers, and how they fit together.35 min
  14. Types and data modelingType hints, dataclasses and checking data at the edges of a program.35 min
  15. Typing in depthGenerics, protocols, TypedDict, Literal and narrowing, checked by mypy.45 min
  16. Classes in depthProperties, alternative constructors, abstract base classes and tuned dataclasses.40 min
  17. Make your classes feel built inSpecial methods for printing, equality, ordering, arithmetic and containers.45 min
  18. Software designModules, interfaces and dependencies that keep big programs changeable.45 min
  19. LoggingLoggers, levels, handlers and configuration, without leaking secrets.40 min
  20. How computers run codeBits, memory, the processor, and what really happens when Python runs.40 min
  21. Arrays in memoryBuffers, dtypes, shapes and strides: why a transpose is free, and which operations copy.40 min
  22. Algorithm complexityCount the work code does, and see why some programs stop scaling.40 min
  23. Concurrency and asyncDo many things at once: processes, threads and the event loop.45 min
  24. Performance and profilingMeasure before you optimize, then fix the part that matters.45 min
  25. Profiling numeric code and a training loopWhere a training step spends its time, Amdahl's rule by hand, and cProfile on a loop with a slow batch function.40 min
  26. Reading C, C++, CUDA and RustPointers and loops in C, references and templates in C++, one-thread-per-element CUDA kernels, and Rust's Option, Result and ownership.30 min
  27. Core data structuresArrays, linked lists, stacks, queues and the hash table behind every dict.45 min
  28. Heaps and priority queuesA list read as a tree: the best k of a million without sorting, and where top-k hides in AI work.35 min
  29. Recursion, trees and graphsFunctions that call themselves, and the shapes most real data takes.45 min
  30. Searching, sorting and dynamic programmingThe classic algorithms, and the ideas behind them.45 min
  31. Dynamic programming in depthA method in five questions: state, choices, recurrence, base case and order. Coins where greedy fails, rebuilding the solution, packing a token budget, the likeliest path behind noisy readings, and what it costs.45 min
  32. Binary search trees and traversalsInsert, search and delete in a sorted tree, walk it four ways, and see why balance and B-trees matter.45 min
  33. Patterns for coding problemsTwo pointers, counting with dictionaries, prefix sums and sliding windows: O(n) answers to classic problems.45 min
  34. Project: a spell checker, three waysA sorted list, a hash set and a trie, measured for speed and memory, with suggestions.45 min
  35. Searching for solutionsBacktracking with pruning, Dijkstra's algorithm and A*, ending with an optimal sliding-puzzle solver.45 min
  36. Project: an unbeatable game opponentGame state and rules, minimax with a cache, and an exhaustive test that proves it never loses.45 min
  37. Databases and SQLStore, query and join data with the language every database speaks.40 min
  38. SQL for analysisSubqueries, CASE, common table expressions, outer joins and window functions on real order data.45 min
  39. Designing a databaseFrom one big table to related tables: keys, constraints, many-to-many and a loader that is safe to rerun.40 min
  40. Databases with an ORMSQLAlchemy models, sessions, relationships, transactions and the N+1 problem.45 min
  41. Changing a database safelyNumbered migrations in transactions, required columns and backfills, and expand-and-contract changes.40 min
  42. Secure codingInjection, escaping, secrets and passwords: the mistakes attackers look for.40 min
  43. Projects, packaging and CILay out a real project, pin its dependencies and test it on every push.35 min
  44. Config files: YAML and TOMLWrite YAML and TOML that parse as you meant, dodge the type traps, and validate what you load.25 min
  45. Command-line programsargparse subcommands, stdin, exit codes and testable main functions.40 min
  46. Packaging your codepyproject.toml, the src layout, versions, entry points and wheels.45 min
  47. Pipes and text tools in the shellgrep, wc, sort, uniq, head and cut, chained with pipes to answer questions about logs and data.25 min
  48. JSON on the command line: curl and jqCall APIs with curl, fail on HTTP errors, and pull values out of JSON with jq paths and select.25 min
  49. Shell scripts that fail safelyTurn commands into scripts with variables, set -euo pipefail and clear exit codes, and know when to switch to Python.25 min
  50. Reading and writing XMLElementTree, attributes and text, namespaces, building XML safely, and untrusted input.35 min
  51. Project: a word-guessing gamePlan a game with a flowchart, then build it from tested functions and one input loop.40 min
  52. Project: a card game of 21Turn written rules into functions, and make randomness testable.40 min
  53. Project: a drinks kioskA program with state and commands, built with functions, then refactored into classes.45 min
  54. Project: a flashcard trainerCards from a file, review boxes, and progress saved between runs.45 min
  55. Project: a price-watch alert botAn API client, alert rules, digest emails and a state file, so reruns never repeat.45 min
  56. Project: Pack Train, an arcade gameA grid game on the turtle canvas: game state as classes, a timer loop, key handlers and a saved best score.45 min
  57. Project: a private password-breach checkHash locally, send five characters, match locally: k-anonymity, safe logging and failure handling.40 min
  58. Capstone: a command-line appDesign, build and test a complete program with a database.45 min
  59. Capstone: ship a small model libraryA typed, tested and profiled package with a train and an eval command, a YAML config, logging and a wheel.45 min
Part of a free course 127 lessons

Free Python developer course

Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write