Lesson 81 found every slope of a small network by hand: a forward pass that kept each value, then a backward pass that multiplied along each path and added across paths. PyTorch does that bookkeeping for you. It's called automatic differentiation, or autograd, and the idea behind it is a tape: as your code runs, each step is written down with what its slope will need, and backward() reads the tape from the end. This lesson builds that tape in plain Python, then matches it with PyTorch, slope for slope. Knowing what gets recorded explains zero_grad(), no_grad() and the commonest autograd error messages.
What you'll be able to do
Record a calculation as a tape: one entry per step, each keeping its inputs and its local slopes
Walk the tape backward to get every slope, multiplying along a path and adding across paths
Use requires_grad, backward() and .grad, and match PyTorch's slopes to the ones found by hand
Explain why slopes add up unless they're cleared, what no_grad() turns off, and what detach() is for
Quick check
Have a guess before you start. You walk into the woods taking five turns, and write each turn in a notebook as you go. To walk back out, how do you use the notebook?
From the first turn to the last
Only the last turn
From the last turn back to the first
Throw it away and draw a new map
Show the answer
From the last turn back to the first
The last turn you took is the first one to undo, so you read the record backward. A backward pass reads its record the same way: last step first.
The math: record every step, then read the record backward
Purpose. Every slope of a calculation, from a record of how it ran.
A tape is a list of the steps a calculation ran, in the order they ran. Each step adds one entry, and an entry keeps three things: the value the step made, the inputs it used, and its local slope for each input, worked out right then, while the input values are at hand. Take the graph from lesson 81, x = t², y = c·t and z = x·y, at t = 2 and c = 3. Running it writes three entries:
entry
made
from
local slopes
#1
x = 2² = 4
t
dx/dt = 2t = 4
#2
y = 3 × 2 = 6
c, t
∂y/∂c = t = 2, ∂y/∂t = c = 3
#3
z = 4 × 6 = 24
x, y
∂z/∂x = y = 6, ∂z/∂y = x = 4
In plain words: the forward pass fills the tape, so every local slope is ready before the backward pass starts.
Quick check
From lesson 81: a value reaches the loss along two paths. The local slopes are 2 and 3 along the first, and 4 and 0.5 along the second. What is the value's slope?
8
9.5
12
6
Show the answer
8
Multiply along each path, then add across them: 2 × 3 + 4 × 0.5 = 6 + 2 = 8.
The picture. Here is that tape being written, then read. Forward, each step's circle gets its value and its place on the tape: x #1, y #2, z #3. Backward, the walk starts at the last entry. Each entry sends its slope back along its arrows to its inputs, times the local slope it wrote down, and then leaves the tape. The slopes are written under the circles (∂) and the local slopes on the arrows (×). Watch t: two entries send it a piece, and the pieces add. The rule at the top of the flow is the one the walk applies at every arrow, with v the entry passing a slope back and u the input it reaches.
Try it now
The tape ends empty, as PyTorch's does after backward(). To walk it again, you run the forward pass again, which records a fresh tape.
Symbols and formula. The rule at the top of the flow reads more easily with one new symbol. Write a bar for "the slope of the final value with respect to": v̄ = ∂z/∂v, starting with z̄ = 1. Then the walk reads each entry v, last to first, and for each input u of it:
uˉ←uˉ+vˉ⋅∂u∂v
In plain words: each entry passes its slope to each of its inputs, times the local slope it wrote down, and each input adds what it gets to what it already has.
Both of lesson 81's clauses are in that one line: the × is "multiply along a path" and the + is "add across paths". In the flow, t got 4 × 3 = 12 from entry #2 and 6 × 4 = 24 from entry #1: 36.
Why from the end? An entry can pass its slope back only once that slope is complete, and every step that used its value was recorded after it. Read backward, those later entries have all been read already. PyTorch then throws the tape away to free its memory, which is why the picture ends with an empty tape.
Quick check
An entry recorded u = v · w at v = 5 and w = −2. Its slope arrives: ∂L/∂u = 3. What does the entry add to v's slope?
−2
1
−6
15
Show the answer
−6
The local slope of v · w with respect to v is the other input, w = −2. So the entry passes 3 × (−2) = −6 to v.
Quick check
Why does each entry work out its local slopes during the forward pass?
It makes the forward pass faster
They need the input values, which are at hand right then
Local slopes never change, so any time would do
The backward walk can only add, not multiply
Show the answer
They need the input values, which are at hand right then
The local slope of t² is 2t, so it needs t's value. The forward pass has every value in hand as it goes, so each entry stores what its slope will need.
In your own words
In your own words: why must the tape be read from its last entry to its first?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a tape you can read
Purpose. Build the tape as a list of small lists, and walk it backward.
Each value gets a place in the list vals, and t, c, x, y and z hold those places, the way a PyTorch tensor is a handle on numbers kept elsewhere. grads holds one stored slope per value. leaf adds an input, a value no step made, and returns its place. record makes one step: it stores the value the same way and appends [its place, its inputs' places, its local slopes] to tape. backward is the math part's formula. reversed(tape) reads the entries from the last, i, ins, local = entry unpacks one entry into its three parts, and [0.0] * len(vals) is a list of zeros, one per value, for this walk's slopes.
vals = []
grads = []
tape = []
defleaf(v):
vals.append(v)
grads.append(0.0)
returnlen(vals) - 1defrecord(v, ins, local):
i = leaf(v)
tape.append([i, ins, local])
return i
defbackward(out):
g = [0.0] * len(vals)
g[out] = 1.0for entry inreversed(tape):
i, ins, local = entry
for k inrange(len(ins)):
g[ins[k]] = g[ins[k]] + g[i] * local[k]
for k inrange(len(g)):
grads[k] = grads[k] + g[k]
t = leaf(2.0)
c = leaf(3.0)
x = record(vals[t] ** 2, [t], [2 * vals[t]])
y = record(vals[c] * vals[t], [c, t], [vals[t], vals[c]])
z = record(vals[x] * vals[y], [x, y], [vals[y], vals[x]])
print(vals)
print(tape)
backward(z)
print(grads[t], grads[c])
backward(z)
print(grads[t], grads[c])
The tape is the math part's table: entry #3, [4, [2, 3], [6.0, 4.0]], says z sits at place 4, was made from places 2 and 3 (x and y), and has local slopes 6 and 4. One backward(z) gives the slopes found by hand in lesson 81: 36 for t and 8 for c. The last two lines are a slip, on purpose: more on that after the code flow.
Code flow. Here is the backward walk on its own, with the printed tape typed in and g starting with z's slope, 1, at place 4. Step through it and watch the boxes of g: each entry passes its slope back to its inputs' places, last entry first.
A named mistake: two walks, no zeroing.backward adds its slopes into grads instead of replacing them, exactly as PyTorch adds into .grad. That's on purpose: it lets you add up the slopes of several batches before one update. But a training loop runs a walk every step, and if the stored slopes aren't set back to 0 in between, each step's slopes land on top of the last ones. The cell's second walk gave 72 and 16: double the truth. (Our tape stays put after a walk, so the cell could walk it twice. PyTorch frees its tape once backward() has used it, so there the doubling comes from a loop: each step records a fresh tape and walks it.) The fix is to clear the stored slopes before each walk:
for k inrange(len(grads)):
grads[k] = 0.0
That's what PyTorch's zero_grad() is for.
Worked example
Worked example: a branch, on the tape
Lesson 81 worked L = (u − 3)² + u with u = w · x, at w = 2 and x = 1. Record it step by step: u feeds two entries, so its slope arrives in two pieces.
vals = []
grads = []
tape = []
defleaf(v):
vals.append(v)
grads.append(0.0)
returnlen(vals) - 1defrecord(v, ins, local):
i = leaf(v)
tape.append([i, ins, local])
return i
defbackward(out):
g = [0.0] * len(vals)
g[out] = 1.0for entry inreversed(tape):
i, ins, local = entry
for k inrange(len(ins)):
g[ins[k]] = g[ins[k]] + g[i] * local[k]
for k inrange(len(g)):
grads[k] = grads[k] + g[k]
w = leaf(2.0)
x = leaf(1.0)
u = record(vals[w] * vals[x], [w, x], [vals[x], vals[w]])
d = record(vals[u] - 3, [u], [1.0])
s = record(vals[d] ** 2, [d], [2 * vals[d]])
L = record(vals[s] + vals[u], [s, u], [1.0, 1.0])
backward(L)
print(vals)
print(grads[u], grads[w], grads[x])
In the lesson you can change this and run it.
Four entries: u = 2, d = u − 3 = −1, s = d² = 1 and L = s + u = 3. The walk reads L's entry first, which passes 1 to s and 1 to u. s's entry passes 1 × 2d = −2 to d, and d's entry passes −2 × 1 on to u, so u ends with 1 + (−2) = −1, the value found by hand. Then u's entry passes −1 × x = −1 to w and −1 × w = −2 to x. The 3 is a plain number, not a value on the tape, so nothing passes a slope to it.
Quick check
What does this print?
tape = [[1, [0], [3.0]], [2, [1, 0], [2.0, 5.0]]]
g = [0.0, 0.0, 1.0]
for entry inreversed(tape):
i, ins, local = entry
for k inrange(len(ins)):
g[ins[k]] = g[ins[k]] + g[i] * local[k]
print(g)
[5.0, 2.0, 1.0]
[11.0, 2.0, 1.0]
[30.0, 2.0, 1.0]
[6.0, 2.0, 1.0]
Show the answer
[11.0, 2.0, 1.0]
The last entry is read first: it passes 1 × 2.0 = 2.0 to place 1 and 1 × 5.0 = 5.0 to place 0. Then the first entry passes place 1's 2.0 × 3.0 = 6.0 to place 0, which adds it: 5.0 + 6.0 = 11.0.
Drill
Walk the tape the right way
The starter's backward has two slips, and zero_grad does nothing yet. Fix backward so one walk gives t the slope 36 and c the slope 8, and write zero_grad so it sets every stored slope in grads back to 0.
vals = [2.0, 3.0, 4.0, 6.0, 24.0]
grads = [0.0, 0.0, 0.0, 0.0, 0.0]
tape = [[2, [0], [4.0]], [3, [1, 0], [2.0, 3.0]], [4, [2, 3], [6.0, 4.0]]]
defbackward(out):
g = [0.0] * len(vals)
g[out] = 1.0for entry in tape:
i, ins, local = entry
for k inrange(len(ins)):
g[ins[k]] = g[i] * local[k]
for k inrange(len(g)):
grads[k] = grads[k] + g[k]
defzero_grad():
pass
backward(4)
print(grads)
zero_grad()
print(grads)
In the lesson, your code is checked as you write it.
In your own words
Why does backward add into grads instead of replacing what's there, and what does that ask of a training loop?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: what PyTorch records for you
Purpose. See the tape behind requires_grad, backward() and .grad, and the three habits that keep slopes right.
PyTorch's tape. A tensor made with requires_grad=True is a leaf that PyTorch watches. Every step that uses it, or uses something made from it, adds an entry to a tape, just as record did. One detail differs: a PyTorch entry keeps what its local slopes need, such as both inputs of x · y, and works them out on the way back. Those kept values are the memory backward() frees. L.backward() walks that tape from L by the math part's rule, and adds each leaf's slope into its .grad. Here is the 2-2-1 network from lesson 81 again, as tensors. The cell checks PyTorch's slopes against that lesson's hand slopes, then takes one step downhill:
import torch
x = torch.tensor([1.0, 2.0])
y = 1.0
W1 = torch.tensor([[0.5, -0.2], [0.3, 0.4]], requires_grad=True)
b1 = torch.tensor([0.1, -0.1], requires_grad=True)
w2 = torch.tensor([0.6, -0.5], requires_grad=True)
b2 = torch.tensor(0.2, requires_grad=True)
L = (w2 @ torch.relu(W1 @ x + b1) + b2 - y) ** 2
L.backward()
print(round(L.item(), 4), w2.grad, b2.grad)
print(torch.allclose(W1.grad, torch.tensor([[-1.416, -2.832], [1.18, 2.36]])))
print(torch.allclose(b1.grad, torch.tensor([-1.416, 1.18])))
with torch.no_grad():
for p in [W1, b1, w2, b2]:
p -= 0.1 * p.grad
p.grad = None
L = (w2 @ torch.relu(W1 @ x + b1) + b2 - y) ** 2print(round(L.item(), 4), W1.grad, L.requires_grad)
The loss is 1.3924, as in that lesson, and w2.grad and b2.grad hold its hand values. torch.allclose compares two tensors number by number, allowing for float rounding: True twice, so the six first-layer slopes match too. Nine slopes from one call. The update runs inside with torch.no_grad():, and p.grad = None clears each slope once it's used. The next forward pass records a fresh tape (the last True is L.requires_grad: the new loss is on it). The loss is 0.0015, the same fall as by hand, and W1.grad is None until the next walk.
Quick check
After L.backward() in that cell, which tensors hold a slope in .grad?
None of them until an optimizer step runs
W1, b1, w2 and b2, made with requires_grad=True
Every variable in the cell, x and y too
Only L, the loss
Show the answer
W1, b1, w2 and b2, made with requires_grad=True
PyTorch keeps slopes for the leaves it watches. x was made without requires_grad and y is a plain number, so nothing is stored for them, and middle values drop theirs unless you call retain_grad().
Habit 1: clear the slopes every step. Take the four days of tent data from lesson 79, fit by a line ŷ = w·x + b. This cell forgets to clear: it runs the forward pass and backward() twice, and prints the stored slopes after each walk.
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
y = torch.tensor([3.0, 5.0, 6.0, 9.0])
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for run inrange(2):
L = ((w * x + b - y) ** 2).mean()
L.backward()
print(w.grad.item(), b.grad.item())
The first walk gives the gradient from lesson 78, (−33.5, −11.5). The second adds the same again, the Python part's named mistake: (−67, −23), so an update would step twice as far as it should. The fix is w.grad = None after each update. That's the job of opt.zero_grad() in lesson 79's loop: it does exactly that for every weight the optimizer holds.
Habit 2: update under no_grad().Lesson 50 used it to set weights by hand, and an update sets weights too. The update w -= 0.05 * w.grad is a calculation. If PyTorch recorded it, the next loss would trace back through every old update. So for a leaf that requires grad it refuses, with the error "a leaf Variable that requires grad is being used in an in-place operation" (the drill below starts with it). with torch.no_grad(): turns the recording off for the lines under it: the update changes the numbers and leaves the tape alone.
Habit 3: detach() to take numbers out. A value you keep for later, such as each step's loss for a plot, should come off the tape: a tensor still on it keeps the tape alive with it. .detach() gives the same numbers with requires_grad False. NumPy can't follow a tape at all, so PyTorch refuses .numpy() on a tensor that's on one. Here is the fitted line's prediction for the four days, w = 1.9 and b = 1.0:
import torch
w = torch.tensor(1.9, requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
pred = w * x + 1.0print(pred.detach().numpy())
print(pred.numpy())
The first print works. The second stops with an error that names its own fix. .item(), used above, does the same job for a single number.
Quick check
What does this print?
import torch
w = torch.tensor(2.0, requires_grad=True)
a = w * 3with torch.no_grad():
b = w * 3
c = a.detach()
print(a.requires_grad, b.requires_grad, c.requires_grad, c.item())
True False True 6.0
True False False 6.0
False False False 6.0
True True True 6.0
Show the answer
True False False 6.0
a is made from w with recording on, so it's on the tape. b is made under no_grad(), so it isn't. c is a's numbers taken off the tape. All three hold 6.0: no_grad() and detach() never change a value, only whether it's recorded.
Drill
Train the tent line with autograd
Train the tent line for 1,000 steps at η = 0.05, the loop lesson 79 ran in NumPy. The starter breaks two habits: its update is recorded, and its slopes are never cleared. Fix both, so first holds the first step's slopes and w and b end where the NumPy loop did, 1.9 and 1.0.
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
y = torch.tensor([3.0, 5.0, 6.0, 9.0])
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
first = []
for step inrange(1000):
L = ((w * x + b - y) ** 2).mean()
L.backward()
if step == 0:
first = [w.grad.item(), b.grad.item()]
w -= 0.05 * w.grad
b -= 0.05 * b.grad
print(first, round(w.item(), 4), round(b.item(), 4), round(L.item(), 4))
In the lesson, your code is checked as you write it.
In your own words
A friend's training loss jumps around and grows instead of falling, and their loop has no zero_grad(). Explain what is going wrong.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Build an autograd engine: write a tape and its backward walk yourself, for a whole small library.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write