Every function so far had one input. Real models have many: a line has a slope and an intercept, a network has millions of weights, and the loss depends on all of them at once. So "how does the loss change?" needs a sharper question: how does it change when I nudge this one weight and leave every other one alone? The answer is a partial derivative, written with a curly ∂. It's the ordinary derivative of lessons 72 to 74, applied to one input at a time. By the end of the lesson you'll take partials by hand and by nudging in Python, compute the two slopes that train a straight-line model, and watch PyTorch find the same two with one call.
What you'll be able to do
Picture a function of two inputs as a table, a contour map and a surface
Take a partial derivative: change one input, hold the others still, written ∂f/∂x
Estimate partials in Python by nudging one input at a time, and check them against the hand rule
Find ∂L/∂w and ∂L/∂b for a line's loss, one slope per weight, and get both from one PyTorch backward()
Quick check
Have a guess before you start. A tent's price depends on its size and its weight. To see how size alone affects price, what do you compare?
Any two tents
Tents of different sizes but the same weight
Tents of different weights but the same size
Tents of the same size and weight
Show the answer
Tents of different sizes but the same weight
Vary the one thing you're asking about and hold the others fixed. That's a partial derivative's recipe.
The math: one input at a time
Purpose. Change when there are several inputs.
A function of two inputs, like f(x, y) = x² + 3y², gives one number for each pair (x, y). You can picture it as a table (rows for x, columns for y), as a contour map (lines joining points of equal height, like a hiking map), or as a surface over a floor.
To ask how f changes with x, hold y still and differentiate as usual, treating y as a constant:
∂x∂f=2x∂y∂f=6y
In plain words: ∂f/∂x ("partial f, partial x") is the slope in the x direction with y frozen; ∂f/∂y is the slope in the y direction with x frozen.
Quick check
From lesson 73: what is the derivative of 3y² with respect to y?
6y
6y²
3y
2y
Show the answer
6y
Power rule with a constant multiple: 3 × 2y = 6y. The partial ∂f/∂y uses exactly this rule.
At the point (1, 2): ∂f/∂x = 2 × 1 = 2 and ∂f/∂y = 6 × 2 = 12, so the surface is six times steeper in the y direction there. The flow measures both by nudging each input a tiny bit each way while holding the other still.
The picture. The drawing is the bowl seen from above, as a contour map. Each oval joins the points where f has the same value, like the contour lines on a hiking map, and the ovals crowd together where the bowl gets steep. Step through and watch the point's own oval light up, then the dashed line each nudge moves along, then one arrow per slope. The arrows are drawn shorter than their true size but in proportion, so the arrow along y is six times as long as the arrow along x. The code's PyTorch tab finds both slopes with a single call, f.backward().
Try it now
Then set x to 0 and press Use these numbers: the x-slope is 0 there, since the bowl's lowest line runs along x = 0, and the picture has no arrow along x.
Symbols and formula. Holding y still makes it a constant, so every rule from lesson 73 applies unchanged:
∂x∂f=h→0limhf(x+h,y)−f(x,y)
In plain words: the same shrinking secant as lesson 72, moving only in x.
Some practice with the trick of freezing: for f = x²y + y, ∂f/∂x = 2xy (y is just a number in front) and ∂f/∂y = x² + 1 (x² is just a number in front of y). For f = xy, ∂f/∂x = y and ∂f/∂y = x. A term with no x in it, like 5y, has ∂/∂x = 0: with y frozen, it's a constant.
Quick check
For f(x, y) = x²y + y, what is ∂f/∂x at (2, 3)?
4
15
5
12
Show the answer
12
∂f/∂x = 2xy (the + y is constant in x), and 2 × 2 × 3 = 12.
In your own words
In your own words: why can you use the ordinary derivative rules for a partial derivative?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: nudge one input at a time
Purpose. Measure each partial, and check it against the hand rule.
It runs right here in your browser, with nothing to install.
Each function nudges just one input, both ways, and divides by 2h: lesson 73's central difference, one direction at a time. The measured slopes match the hand rules, 2 and 12.
Code flow. Here are the same two nudges written out a line at a time, at (1, 2). The nudge h is 0.5 this time, so every value stays short enough to read. For a bowl built from squares like this one, the two-way nudge happens to give the exact slope for any h; the tiny h in the cell above is what works for every function. Step through and watch up and down: first only x moves, then only y.
A named mistake: moving both inputs. Nudging x and y together, f(x + h, y + h), measures the slope along a diagonal, a mix of both partials, not either one. For x² + 3y² at (1, 2) it gives 14, the sum of the two partials, which matches neither. Each partial must move its own input only.
Cost. With n inputs, nudging needs two function calls per input: 2n calls for all the partials. For a handful of inputs that's nothing; for a network with millions of weights it's hopeless, which is why lesson 81's backpropagation matters.
Worked example
Worked example: both partials of three functions
Measure both partials of three functions at a point, and compare with the hand rules.
h = 0.000001defpx(f, x, y):
return (f(x + h, y) - f(x - h, y)) / (2 * h)
defpy(f, x, y):
return (f(x, y + h) - f(x, y - h)) / (2 * h)
defmix(x, y):
return x ** 2 * y + y
defprod(x, y):
return x * y
defsaddle(x, y):
return x ** 2 - y ** 2print(round(px(mix, 2, 3), 4), round(py(mix, 2, 3), 4), 2 * 2 * 3, 2 ** 2 + 1)
print(round(px(prod, 3, 4), 4), round(py(prod, 3, 4), 4))
print(round(px(saddle, 1, 1), 4), round(py(saddle, 1, 1), 4))
In the lesson you can change this and run it.
For x²y + y at (2, 3) the measured partials are 12 and 5, matching 2xy and x² + 1. For xy at (3, 4) each partial is the other input: 4 and 3. The saddle x² − y² at (1, 1) has partials 2 and −2: rising in x, falling in y, which is what makes it a saddle, a shape lesson 83 returns to.
Quick check
What does this print?
deff(x, y):
return4 * x + x * y
h = 0.000001print(round((f(2 + h, 5) - f(2 - h, 5)) / (2 * h), 4), round((f(2, 5 + h) - f(2, 5 - h)) / (2 * h), 4))
9.0 2.0
4.0 2.0
2.0 9.0
9.0 5.0
Show the answer
9.0 2.0
∂f/∂x = 4 + y = 9 at y = 5. ∂f/∂y = x = 2.
Reading a contour map. On a contour map, lines join points of equal height, like a hiker's map. Where the lines crowd together the surface is steep; where they spread out it's gentle. A partial derivative reads the map along one axis: ∂f/∂x is how fast you cross contour lines walking due east, ∂f/∂y walking due north. For x² + 3y² the lines are squeezed in the y direction, as the ovals in the math part's picture are, which is why ∂f/∂y is the bigger of the two.
Drill
Nudge one input only
Fix partial_y so it nudges only y. Then set grad to the pair (∂f/∂x, ∂f/∂y) for f = x² + xy at (3, 1), each rounded to 4 places.
h = 0.000001defpartial_x(f, x, y):
return (f(x + h, y) - f(x - h, y)) / (2 * h)
defpartial_y(f, x, y):
return (f(x + h, y + h) - f(x - h, y - h)) / (2 * h)
deff(x, y):
return x ** 2 + x * y
grad = (0, 0)
print(grad)
In the lesson, your code is checked as you write it.
In your own words
Why does nudging x and y together give neither partial?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: one slope per weight
Purpose. A model has many weights, and each gets its own slope.
A line ŷ = w·x + b has two weights, so its mean squared error on some data is a function of both: L(w, b). Each partial says how L changes when that one weight moves:
∂w∂L=n2i∑(y^i−yi)xi∂b∂L=n2i∑(y^i−yi)
In plain words: both use the errors ŷ − y; the w-slope weights each error by its input, the b-slope doesn't.
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([3, 5, 6, 9])
defloss(w, b):
return np.mean((w * x + b - y) ** 2)
h = 0.000001
w, b = 1, 0print(loss(w, b))
print(round((loss(w + h, b) - loss(w - h, b)) / (2 * h), 4), round((loss(w, b + h) - loss(w, b - h)) / (2 * h), 4))
e = w * x + b - y
print(2 / 4 * np.sum(e * x), 2 / 4 * np.sum(e))
Both partials are negative, −18.5 and −6.5: raising either weight lowers the loss. The nudged slopes and the formulas agree.
Quick check
At some (w, b), ∂L/∂w = 3 and ∂L/∂b = −1. How should each weight move to lower the loss?
Raise both
Lower both
Raise w, lower b
Lower w, raise b
Show the answer
Lower w, raise b
Move each weight against its own slope: w's is positive, so lower it; b's is negative, so raise it.
Why the errors appear in both. By the chain rule (lesson 74), each squared error (ŷ − y)² has slope 2(ŷ − y) with respect to ŷ, and ŷ = wx + b changes by x per unit of w and by 1 per unit of b. Averaging over the examples gives the two formulas. A network works the same way, just with millions of partials, one per weight, each built from the errors flowing back through the chain.
In PyTorch. PyTorch runs that chain for you. Make w and b with requires_grad=True and PyTorch records every step that uses them. loss.backward() then runs the chain rule back through those steps and leaves each weight's partial derivative in its .grad:
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
y = torch.tensor([3.0, 5.0, 6.0, 9.0])
w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
loss = ((w * x + b - y) ** 2).mean()
loss.backward()
print(loss.item())
print(w.grad, b.grad)
The same loss, 11.75, and the same two slopes the NumPy cell found by nudging and by formula, −18.5 and −6.5, printed PyTorch's way with four decimals. One call filled both.
Quick check
Nudging took two loss evaluations per weight. How many backward() calls did PyTorch need for both slopes?
Four, two per weight
Two, one per weight
Four, one per data point
One for both weights
Show the answer
One for both weights
A single loss.backward() fills the .grad of every tensor made with requires_grad=True, however many weights there are. That's why training a network with millions of weights is possible at all: there's no extra call per weight.
Two slopes, one step. Training uses both partials together: each weight moves against its own slope in the same step. At w = 1, b = 0 both slopes are negative, so both weights rise; later, when the line fits well, both slopes shrink toward 0 and the weights settle.
Quick check
At w = 2, b = 1, the line nearly fits. What does this print?
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([3, 5, 6, 9])
e = 2 * x + 1 - y
print(2 / 4 * np.sum(e * x), 2 / 4 * np.sum(e))
1.5 0.5
0.5 1.5
3.0 1.0
-1.5 -0.5
Show the answer
1.5 0.5
The errors are 0, 0, 1 and 0. ∂L/∂w = (2/4)(1 × 3) = 1.5 and ∂L/∂b = (2/4)(1) = 0.5: lower both a little.
Drill
The two slopes of a line's loss
For the data below at w = 2, b = 1, set e to the errors ŷ − y as a list, and dw and db to the two partials from the formulas.
import numpy as np
x = np.array([1, 2, 3])
y = np.array([4, 4, 8])
w, b = 2, 1
e = []
dw = 0
db = 0print(e, dw, db)
In the lesson, your code is checked as you write it.
In your own words
Why does ∂L/∂w weight each error by its input x, while ∂L/∂b doesn't?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write