Lesson 77 gave one slope per input. Put them side by side and they make a vector, the gradient, and that vector has a meaning of its own: it points in the direction the function climbs fastest, and its length says how steep that climb is. Turn it around and it points straight downhill. That's the direction gradient descent (lesson 79) moves a model's weights, every step of every training run. This lesson builds the gradient from partials, measures it numerically for any number of inputs, and computes it for a line's loss, where it turns out to be a dot product you met in lesson 50. Then PyTorch's loss.backward() finds the same gradient for you.
What you'll be able to do
Collect all the partial derivatives into one vector, the gradient ∇f
Read the gradient as the direction of steepest climb, its length as the steepness, and its opposite as steepest descent
Write numerical_gradient(f, point) for any number of inputs, and check it against the hand formula
Compute the gradient of a line's mean squared error, with ∂L/∂w as a dot product of errors and inputs, and get the same gradient from PyTorch's loss.backward()
Quick check
Have a guess before you start. On a hillside, the ground rises 2 m per step east and 6 m per step north. Which way is the climb steepest?
Mostly north, a little east
Due east, along the gentler slope
Due south, downhill
Due north, the steeper slope
Show the answer
Mostly north, a little east
The steepest direction leans toward the steeper slope, north, while still taking some of the east slope: the arrow (2, 6).
The math: all the slopes as one arrow
Purpose. All the slopes at once, as a direction.
The gradient of f is the vector of its partial derivatives, written ∇f ("del f" or "grad f"):
∇f=(∂x∂f,∂y∂f)
In plain words: one slope per input, stacked into an arrow.
For f = x² + 3y² at (1, 2), the partials are 2 and 12, so ∇f = (2, 12). Three facts make it useful:
it points in the direction where f rises fastest;
its length is that fastest rate: √(2² + 12²) ≈ 12.17;
its opposite, −∇f = (−2, −12), points where f falls fastest.
Quick check
From lesson 49: what is the length of the vector (6, 8)?
10
48
100
14
Show the answer
10
√(36 + 64) = √100 = 10. A gradient of (6, 8) means the fastest climb rises 10 per unit step.
The flow measures both partials of x² + 3y² at (1, 2), which together are the gradient.
The picture. The drawing is the contour map from lesson 77: ovals where f has one value, around the bottom of the bowl. This time the arrow for ∂f/∂y starts at the tip of the arrow for ∂f/∂x, so the two add up, tip to tail, into one arrow: the gradient. Watch where it points at the last step: mostly north, a little east, like the hillside in the opening question, and straight across the oval through the point, at a right angle to it. The arrows are drawn shorter than their true size, in proportion. The code's PyTorch tab gets the whole gradient from one f.backward().
Try it now
Then set x and y to 0 and press Use these numbers: at the bottom of the bowl the gradient is (0, 0), with no arrow and no direction uphill at all.
Symbols and formula. With any number of inputs, the gradient has one entry per input:
∇f(x)=(∂x1∂f,…,∂xn∂f)Δf≈∇f⋅Δx
In plain words: a small move Δx changes f by about the gradient dotted with the move (lesson 50's dot product).
That second formula is lesson 72's Δy ≈ f′·Δx in many dimensions. It also shows why the gradient is the steepest direction: the dot product of ∇f with a move of fixed length is biggest when the move points along ∇f (lesson 51). At the bottom of a bowl every partial is 0, so ∇f = 0: no direction is uphill.
The gradient is perpendicular to the contour lines. Walking along a contour line keeps f the same, so the gradient dotted with that direction is 0: the gradient meets every contour line at a right angle, pointing straight across them toward higher ground, as the flow's last picture shows. On a map, that's the line a stream would take, downhill, against the gradient.
Many inputs, same picture. With three inputs the contour lines become contour surfaces, and with a million inputs nothing can be drawn at all. But every fact above still holds: the gradient still has one entry per input, still points uphill most steeply, and its opposite still points downhill. That's why the same few lines of gradient descent can train a model of any size.
Quick check
∇f = (3, −4) at a point. Moving by (0.1, 0.1), about how much does f change?
In your own words: why does the gradient point in the direction of steepest climb?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a numerical gradient
Purpose. Measure the gradient of any function of any number of inputs.
Store the point as a NumPy array, and nudge one coordinate at a time:
import numpy as np
defnumerical_gradient(f, point):
h = 0.000001
g = np.zeros(len(point))
for i inrange(len(point)):
up = point.copy()
down = point.copy()
up[i] = up[i] + h
down[i] = down[i] - h
g[i] = (f(up) - f(down)) / (2 * h)
return g
deff(p):
return p[0] ** 2 + 3 * p[1] ** 2print(np.round(numerical_gradient(f, np.array([1.0, 2.0])), 4).tolist())
It runs right here in your browser, with nothing to install.
point.copy() makes separate copies to nudge, so the original point is never changed. The answer, [2, 12], matches (2x, 6y).
Code flow. Step through the same loop at (1, 2). As in lesson 77, h is 0.5 here to keep the values short, and for this bowl the two-way nudge is exact at any h. Watch up and down as rows of boxes: each pass starts from a fresh copy of point and moves one entry only, and g fills in one entry per pass.
A named mistake: nudging the original. Writing up = point doesn't copy the array; both names point at the same numbers. Nudging up then nudges point too, so the point drifts as the loop runs, and later partials are measured from the wrong place. Write down = point as well and it gets worse: up and down are then one array, the two nudges cancel, and every entry of the gradient comes out 0. .copy() prevents both.
Any number of inputs. The same function works for three inputs, or three thousand: the loop runs once per coordinate, two evaluations each. That's fine for checking, and far too slow for training big models, which is why lesson 81 computes gradients a different way.
Worked example
Worked example: gradients of two functions at three points
Measure the gradient of two functions at three points each, and compare with the hand formulas.
import numpy as np
defnumerical_gradient(f, point):
h = 0.000001
g = np.zeros(len(point))
for i inrange(len(point)):
up = point.copy()
down = point.copy()
up[i] = up[i] + h
down[i] = down[i] - h
g[i] = (f(up) - f(down)) / (2 * h)
return g
defbowl(p):
return p[0] ** 2 + p[1] ** 2defthree(p):
return p[0] * p[1] + p[2] ** 2for pt in [[3.0, 4.0], [0.0, 1.0], [-1.0, 2.0]]:
print(pt, np.round(numerical_gradient(bowl, np.array(pt)), 4).tolist())
print(np.round(numerical_gradient(three, np.array([1.0, 2.0, 3.0])), 4).tolist())
In the lesson you can change this and run it.
For x² + y² the gradient is (2x, 2y): (6, 8), (0, 2) and (−2, 4) at the three points, each pointing straight away from the bottom at the origin. For xy + z² it's (y, x, 2z) = (2, 1, 6): three inputs, three entries, from the same function.
∂f/∂x = 2 and ∂f/∂y = 2y = 6 at y = 3. Adding (h, 0) nudges only the first input.
Checking a hand-worked gradient. Whenever you work out a gradient by hand, compare it with numerical_gradient at a point or two. The two are computed in completely different ways, so agreement is strong evidence both are right. Real libraries test their own gradient code exactly like this, on small random inputs, before trusting it on a model with millions of weights.
Drill
A numerical gradient that copies
Fix numerical_gradient so it nudges copies of the point, not the point itself. Then set g to the gradient of x² + xy + y² at (1, 2), rounded to 4 places, as a list.
import numpy as np
defnumerical_gradient(f, point):
h = 0.000001
g = np.zeros(len(point))
for i inrange(len(point)):
up = point
down = point
up[i] = up[i] + h
down[i] = down[i] - h
g[i] = (f(up) - f(down)) / (2 * h)
return g
deff(p):
return p[0] ** 2 + p[0] * p[1] + p[1] ** 2
g = []
print(g)
In the lesson, your code is checked as you write it.
In your own words
In the drill's starter, up = point and down = point. Why does that make every entry of the gradient 0?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the gradient of a line's loss
Purpose. The direction to move both weights at once.
For a line ŷ = w·x + b with mean squared error L, lesson 77's two partials make the gradient. The w-entry is a sum of errors times inputs, which is a dot product:
∇L=(n2e⋅x,n2i∑ei),e=y^−y
In plain words: the error vector dotted with the inputs, and the errors summed, each times 2/n.
Watch it computed point by point on four days of tent data, starting from w = 0, b = 0. The picture draws each error as the gap between a day's point and the line, which starts flat along the bottom, ŷ = 0.
The same gradient in code. The flow's lines, with @ for the dot product, on the same four days:
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([3, 5, 6, 9])
w, b = 0, 0
e = w * x + b - y
grad = np.array([2 / 4 * (e @ x), 2 / 4 * np.sum(e)])
print(e.tolist(), grad.tolist())
print(round(float(np.linalg.norm(grad)), 2))
A line stuck at 0 predicts too low on every day, so every error is negative, and so are both entries of the gradient: raise w and b. The second line is the gradient's length (lesson 49).
In PyTorch. Now the same loss in PyTorch, with loss.backward() finding the gradient:
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
y = torch.tensor([3.0, 5.0, 6.0, 9.0])
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
loss = ((w * x + b - y) ** 2).mean()
loss.backward()
print(w.grad, b.grad)
The two .grad entries are the formula's gradient, −33.5 and −11.5, the same numbers the NumPy cell printed. PyTorch never saw the formula. It ran the chain rule back through the loss, which works the same way for a line's two weights or a network's millions.
Quick check
The NumPy cell printed the gradient's length, 35.42. What does that number tell you?
How many steps training will take to finish
How fast the loss climbs, at its steepest
How far w and b are from the best fit
The size of the loss at w = 0, b = 0
Show the answer
How fast the loss climbs, at its steepest
A gradient's length is the steepest rate of change: moving (w, b) a small distance along the gradient raises the loss about 35.42 times that distance. Against the gradient, it falls just as fast.
Quick check
The gradient of a loss is (0, 0) at some (w, b). What does that mean?
A flat spot: the bottom of a convex loss
The loss is 0
The weights are 0
Training should take a bigger step
Show the answer
A flat spot: the bottom of a convex loss
A zero gradient means every partial is 0, the condition for a minimum; for the line's convex loss, it's the best fit.
Why the gradient shrinks as the line improves. Both entries are built from the errors. As training pulls the line closer to the points, the errors shrink, and so does the gradient: at w = 2, b = 0.5 it's only (−1, −0.5), a fraction of the starting (−33.5, −11.5). That's what makes training slow down near the end, and what lets it stop: a gradient of zero means no errors left to fix, or at least none a straight line can fix.
Quick check
At w = 2, b = 0, what does this print?
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([3, 5, 6, 9])
e = 2 * x + 0 - y
print(2 / 4 * (e @ x), 2 / 4 * np.sum(e))
3.5 1.5
-7.0 -3.0
-3.5 -1.5
-1.5 -3.5
Show the answer
-3.5 -1.5
The errors are −1, −1, 0 and −1. e·x = −1 − 2 + 0 − 4 = −7, times 2/4 is −3.5; Σe = −3, times 2/4 is −1.5.
Drill
The loss gradient, as a dot product
For the tent data at w = 2, b = 0.5, set e to the error vector, grad to the gradient [∂L/∂w, ∂L/∂b] (using @ for the dot product), and better to whether one step of 0.05 against the gradient lowers the loss.
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([3, 5, 6, 9])
defloss(w, b):
return np.mean((w * x + b - y) ** 2)
w, b = 2, 0.5
e = np.zeros(4)
grad = [0, 0]
better = Falseprint(e, np.round(grad, 4).tolist(), better)
In the lesson, your code is checked as you write it.
In your own words
Why is ∂L/∂w a dot product of the errors with the inputs?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Gradients: gradients for functions of many inputs.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write