A rate across an interval, from lesson 70, depends on the interval you choose. The derivative removes that choice: shrink the interval toward zero and, for a smooth function, the average rate settles on one number, the slope at that exact point. That number answers the most useful question in all of machine learning. If I nudge this weight up a little, does the loss go up or down, and by how much? Training a model asks it millions of times, once for every weight at every step. This lesson builds the derivative from shrinking secants, by hand and in a few lines of Python, then asks it of a real loss.
What you'll be able to do
Find the slope at a single point by shrinking a secant until its slope settles
Read the derivative as a new function, f′(x) or dy/dx, and use Δy ≈ f′(x)·Δx
Estimate derivatives in Python with slope(f, x, h) and a small h
Ask a loss the question training asks: if this weight goes up a little, does the loss go up or down?
Quick check
Have a guess before you start. A car's speedometer reads 50 km/h. Is that an average over the whole trip, or the speed right now?
The top speed so far
The average over the whole trip
The speed right now, at this instant
The average over the last hour
Show the answer
The speed right now, at this instant
A speedometer shows the rate of change of distance at one instant. That's a derivative.
The math: shrink the gap
Purpose. The rate of change at one instant.
Take f(x) = x² and ask for its slope at x = 3. Pick a small step h and compute the secant's slope, (f(3 + h) − f(3)) ÷ h. As h shrinks, the slopes are 7, 6.1, 6.01, 6.001: they settle on 6. That settled value is the derivative of f at 3, the slope of the tangent, the line that just touches the curve there.
f′(x)=h→0limhf(x+h)−f(x)
In plain words: the slope of a secant, as its two points slide together. Lesson 71's limit is what lets h head to 0 without ever being 0.
Quick check
From lesson 70: what is the average rate of x² from x = 3 to x = 3.1?
6.1
6
0.61
9.61
Show the answer
6.1
(9.61 − 9) ÷ 0.1 = 6.1. Shrink the step further and these averages close in on 6.
Watch the secant slopes settle as h shrinks from 1 to 0.001.
The picture. The curve is y = x², and the big dot is (3, 9), the point whose slope we want. Each step puts a second point h further along the curve, the hollow dot, and draws the secant through the two. Its slope is written in the top corner, the same number the step writes in its cell. As h shrinks, the hollow dot slides in and the secant swings toward the curve. On the last step it has become the tangent.
Try it now
Then set x to 1, and then to 2, and press Use these numbers: the slopes settle on 2 and 4. Each time the derivative is twice x.
Symbols and formula. The derivative is itself a function: put in any x and it gives the slope there. For x² the slopes at 1, 2 and 3 are 2, 4 and 6, so f′(x) = 2x. Two notations mean the same thing:
f′(x)=dxdyΔy≈f′(x)Δx
In plain words: dy/dx reads "the rate of y with respect to x". And near a point, a small change in x changes y by about the slope times that change.
At x = 3, nudging x by 0.1 changes x² by about 6 × 0.1 = 0.6. The true change, 9.61 − 9 = 0.61, is close: the smaller the nudge, the closer the estimate.
Quick check
f′(5) = −2. If x goes from 5 to 5.1, about how does f change?
It falls by about 0.2
It rises by about 0.2
It falls by about 2
It doesn't change
Show the answer
It falls by about 0.2
Δy ≈ f′(x)·Δx = −2 × 0.1 = −0.2: a small drop.
Reading the sign. A positive derivative means the function rises as x grows; negative means it falls; zero means it's flat right there, which happens at the bottom of a valley or the top of a hill (lesson 75). The size says how steeply. For x², the slope at −2 is −4 (falling steeply), at 0 it's 0 (flat), and at 2 it's 4 (rising steeply).
In your own words
In your own words: why does the slope of a secant settle on one number as h shrinks, for a smooth curve?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a slope function
Purpose. Estimate the derivative of any function at any point.
defslope(f, x, h):
return (f(x + h) - f(x)) / h
deff(x):
return x ** 2for h in [1, 0.1, 0.01, 0.001]:
print(h, round(slope(f, 3, h), 6))
It runs right here in your browser, with nothing to install.
slope takes the function itself as its first input (lesson 71 passed functions the same way) and returns the secant slope for a step h. With h = 0.00001 the estimate is good to several decimals.
Code flow. Step through the same estimator, this time keeping each estimate in a list. Watch the jump into slope on every pass: f is called twice, at x + h and at x, and the list grows one box at a time to the numbers the flow's cells filled in.
The derivative of x², found numerically. Estimate the slope at several points and the pattern appears:
defslope(f, x, h):
return (f(x + h) - f(x)) / h
deff(x):
return x ** 2for x in [1, 2, 3]:
print(x, round(slope(f, x, 0.00001), 4))
The slopes are 2, 4 and 6: twice x each time. A named mistake: h too big, or too small. A big h gives a long secant, not a tangent: h = 1 gives 7 instead of 6. A tiny h, like 1e-15, runs into float noise and gives garbage. Steps around 1e-5 or 1e-6 are a good middle for most functions.
Worked example
Worked example: numerical derivatives of three functions
Estimate the slope of x³, eˣ and √x at a few points, and compare with what you'd expect.
x³ has slope 3 at x = 1 and 12 at x = 2: it steepens fast, as 3x² would predict (lesson 73). eˣ has slope 1 at 0, the same as its value there, a special property of e. √x has slope 0.25 at 4, gentle, since square roots grow ever more slowly. math.exp and math.sqrt are passed to slope just like a function you wrote.
A straight line has the same slope everywhere, 3, and every secant gives exactly that.
Drill
A slope estimator
Fix slope(f, x, h) so it returns the secant slope. Then set s1 to the slope of x² at 5 with h = 0.0001, rounded to 2 places, and change to the estimated change in x² when x goes from 5 to 5.2, using Δy ≈ f′(x)·Δx.
In the lesson, your code is checked as you write it.
In your own words
Why not just use h = 0 in slope(f, x, h)?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: which way should a weight move?
Purpose. The question training asks millions of times.
A model ŷ = w·x predicts tent rentals from temperature. Its loss, the mean squared error over three days, is a function of the weight w alone. The derivative of the loss says what a small increase in w would do:
dwdL<0⇒raise wdwdL>0⇒lower w
In plain words: if the loss falls as w rises, raise w; if it rises, lower w. The size says how urgently.
import numpy as np
x = np.array([1, 2, 3])
y = np.array([2, 4, 7])
defloss(w):
return np.mean((w * x - y) ** 2)
defslope(f, w, h):
return (f(w + h) - f(w)) / h
for w in [1, 2, 3]:
print(w, round(loss(w), 4), round(slope(loss, w, 0.000001), 3))
At w = 1 the slope is −11.333: raising w helps a lot. At w = 2 it's −2: raise it a little. At w = 3 it's +7.333: w has gone too far, so lower it. The best w is between 2 and 3.
The picture. Here is that loss drawn as a curve of w, with the tangent at w = 1 and at w = 3, and an arrow for the move: against the slope, and longer when the slope is steeper.
Quick check
At some w, the loss's slope is 0. What does that suggest?
w is at a flat spot, likely the bottom
The loss is at its highest point
The model is broken
w should be raised a lot
Show the answer
w is at a flat spot, likely the bottom
A slope of 0 means small changes in w don't change the loss: for a bowl-shaped loss, that's the bottom.
Many weights, one question each. A real network has millions of weights, and training asks this question for each one: the slope of the loss with respect to that weight, holding the others still. Lesson 77 calls these partial derivatives, and lesson 78 gathers them into the gradient. Computing each one by nudging, as here, would take one extra loss evaluation per weight, far too slow; lesson 81's backpropagation gets them all at once.
In PyTorch. Here is a first taste of PyTorch's autograd, the machine that does that, which lesson 82 opens up. Make w a tensor with requires_grad=True and PyTorch records every calculation w goes into. Compute the loss L from w, call L.backward(), and PyTorch works the slope of L with respect to w back through that record into w.grad. No nudge, no h:
import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([2.0, 4.0, 7.0])
for v in [1, 2, 3]:
w = torch.tensor(float(v), requires_grad=True)
L = torch.mean((w * x - y) ** 2)
L.backward()
print(v, round(L.item(), 4), round(w.grad.item(), 3))
The same three rows as the NumPy cell: losses 7.0, 0.3333 and 3.0, and slopes −11.333, −2.0 and 7.333. The nudge only estimated each slope; w.grad is the exact one. Each pass makes a fresh w, because backward() adds to whatever w.grad already holds.
Overshooting the best value. The slope also says when a weight has gone too far. Past the bottom of the loss, the slope turns positive, so the same rule, "move against the slope", now says to bring w back down. The rule never needs to know where the bottom is; it only reads the slope where it stands, then takes another reading after every step.
Quick check
At w = 2.5 the tent loss's slope is measured. What does this print, and which way should w move?
import numpy as np
x = np.array([1, 2, 3])
y = np.array([2, 4, 7])
defloss(w):
return np.mean((w * x - y) ** 2)
h = 0.000001print(round((loss(2.5 + h) - loss(2.5 - h)) / (2 * h), 3))
0.0
-2.667
2.667
2.5
Show the answer
2.667
The slope at 2.5 is positive, 2.667: w is past the best value of about 2.21, so lower it.
Drill
Which way to move w
For the loss below, set slopes to its slope at w = 0, 1 and 2 (h = 0.000001, each rounded to 2 places), and moves to "up" or "down" for each: the way w should move to lower the loss.
import numpy as np
x = np.array([1, 2])
y = np.array([3, 5])
defloss(w):
return np.mean((w * x - y) ** 2)
defslope(f, w, h):
return (f(w + h) - f(w)) / h
slopes = []
moves = []
print(slopes, moves)
In the lesson, your code is checked as you write it.
In your own words
Why does the size of the loss's slope matter, not just its sign?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write