Lesson 72 found slopes by shrinking secants, one point at a time. That works, but it's slow, and it only gives numbers, never a formula. A handful of rules give the derivative of most everyday functions as a formula, in one line: the derivative of x³ is 3x², of 5 is 0, of eˣ is eˣ. With them you can write down the slope of a loss as an expression and see what drives it. For the squared error of a one-weight model, that slope turns out to contain the error itself, which is why big mistakes push the weights hard and small ones barely move them. This lesson learns the rules, checks each one against a numerical slope, and applies them to that loss.
What you'll be able to do
Differentiate constants, powers, constant multiples and sums with four short rules
Use the two special derivatives: eˣ is its own derivative, and ln x gives 1/x
Check every rule against a numerical slope, and write d_poly for any polynomial
Find the slope of a squared-error loss, 2x(wx − y), and see the error inside it
Quick check
Have a guess before you start. f(x) = 7 for every x. What is its slope?
It has no slope
0: it never changes
1
7
Show the answer
0: it never changes
A constant function is a flat line: its rate of change is 0 everywhere.
The math: four rules and two special cases
Purpose. Slopes without the limit every time.
Constant: the derivative of a number is 0.
Power rule: bring the power down, then lower it by one.
Constant multiple: a number in front stays in front: (5x³)′ = 5 × 3x² = 15x².
Sum: differentiate each term and add: (x² + x)′ = 2x + 1.
dxdxn=nxn−1
In plain words: x⁴ becomes 4x³, x² becomes 2x, and x becomes 1 (since x⁰ = 1).
Lesson 72 found the slope of x² to be 2x by estimating; the power rule gives it at once.
Quick check
From lesson 72: the slope of x² at x = 3 is 6. What does the power rule give for (x²)′ at x = 3?
3² = 9
3 − 1 = 2
2 × 3 = 6
2 × 3² = 18
Show the answer
2 × 3 = 6
(x²)′ = 2x, and 2 × 3 = 6, matching the shrinking secants.
Step through 1 − 4x + 2x³ term by term: each term follows its own rule, and the results add up.
The picture. The thin curve, f(x), is 1 − 4x + 2x³. Each step adds one term's slope to the thick curve, f′(x), the slope of f so far, and writes the rule it used in the corner. On the last step f′ is whole. Then, at x = 1, look at both: the tangent to f there climbs 2 for each step across, and f′(1) = −4 + 6 × 1² = 2 is the height of the slope curve at the same x.
Try it now
Then set the numbers on 1, x, x², … to 0, 0, 0, 0, 1, which is x⁴, and press Use these numbers: the derivative is 4x³.
Symbols and formula. The rules together, plus the two functions with special derivatives:
(c)′=0(cf)′=cf′(f+g)′=f′+g′(ex)′=ex(lnx)′=x1
In plain words: eˣ grows exactly as fast as its own value, and ln x grows by 1/x, ever more slowly.
Why eˣ is special. Every exponential bˣ has a slope proportional to itself; e ≈ 2.718 is the one base where the factor is exactly 1. That's why e appears in so many ML formulas: the sigmoid, softmax and the normal distribution all use it, and their slopes come out tidy because (eˣ)′ = eˣ.
Quick check
What is the derivative of 3x⁴ − 2x + 9?
3x³ − 2
12x³ − 2
12x³ − 2 + 9
12x⁴ − 2
Show the answer
12x³ − 2
3x⁴ → 3 × 4x³ = 12x³; −2x → −2; 9 → 0.
In your own words
In your own words: why does the derivative of a sum split into the derivatives of its terms?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: check each rule numerically
Purpose. Trust a rule because you've watched it agree with a measured slope.
A central difference, (f(x + h) − f(x − h)) ÷ 2h, looks both ways and is far more accurate than lesson 72's one-sided secant for the same h:
It runs right here in your browser, with nothing to install.
Each line prints the measured slope next to the rule's answer: 32 and 32 for x⁴, 2.7183 twice for eˣ, and 0.5 twice for ln x.
Code flow. Now put the power rule against the measured slope at three points in a row. Step through: each pass calls slope, which calls f twice, once just right of x and once just left, and then the rule, 4x³, is worked out beside it.
A polynomial's derivative from its coefficients. Store 1 − 4x + 2x³ as its coefficients [1, −4, 0, 2], one per power. The power rule then works on the list: the coefficient of xⁿ, times n, becomes the coefficient of xⁿ⁻¹.
coeffs = [1, -4, 0, 2]
d = []
for n inrange(1, len(coeffs)):
d.append(n * coeffs[n])
print(d)
[−4, 0, 6] is −4 + 0x + 6x². A named mistake: forgetting to lower the power. Writing 2x³ → 6x³ keeps the power; the derivative must lose one, giving 6x². Checking against a numerical slope catches it at once.
Worked example
Worked example: d_poly, verified at three points
Write d_poly(coeffs) for any polynomial and confirm it against the central difference at x = 0, 1 and 2.
defd_poly(coeffs):
d = []
for n inrange(1, len(coeffs)):
d.append(n * coeffs[n])
return d
defvalue(coeffs, x):
total = 0for n inrange(len(coeffs)):
total = total + coeffs[n] * x ** n
return total
c = [1, -4, 0, 2]
dc = d_poly(c)
h = 0.000001for x in [0, 1, 2]:
measured = (value(c, x + h) - value(c, x - h)) / (2 * h)
print(x, value(dc, x), round(measured, 4))
In the lesson you can change this and run it.
value evaluates a polynomial from its coefficients, so the same function evaluates both f and its derivative. At x = 0, 1 and 2 the rule gives −4, 2 and 20, and the measured slopes agree. Two independent methods matching is the best evidence that the rule, and the code, are right.
Quick check
What does this print?
coeffs = [3, 0, 5, 1]
d = []
for n inrange(1, len(coeffs)):
d.append(n * coeffs[n])
print(d)
[0, 5, 1]
[10, 3]
[3, 0, 10, 3]
[0, 10, 3]
Show the answer
[0, 10, 3]
3 + 5x² + x³ has derivative 0 + 10x + 3x²: the list [0, 10, 3].
Quick check
What does this print?
import math
h = 0.000001print(round((math.exp(2 + h) - math.exp(2 - h)) / (2 * h), 3), round(math.exp(2), 3))
2.0 7.389
7.389 7.389
14.778 7.389
7.389 2.718
Show the answer
7.389 7.389
The slope of eˣ at 2 equals its value there, e² ≈ 7.389: eˣ is its own derivative.
The power rule for any power. The rule works for negative and fractional powers too. 1/x is x⁻¹, so its derivative is −1·x⁻² = −1/x². √x is x^½, so its derivative is ½·x^(−½) = 1/(2√x): at x = 4 that's 0.25, the slope lesson 72 measured. Rewriting a function as powers of x first often turns a hard-looking derivative into a one-line one.
Drill
Differentiate any polynomial
Fix d_poly(coeffs) so it returns the derivative's coefficients. Then set at2 to the derivative of 5 + 2x − x² + x⁴ at x = 2, using value.
defd_poly(coeffs):
d = []
for n inrange(1, len(coeffs)):
d.append(coeffs[n])
return d
defvalue(coeffs, x):
total = 0for n inrange(len(coeffs)):
total = total + coeffs[n] * x ** n
return total
c = [5, 2, -1, 0, 1]
at2 = 0print(d_poly(c), at2)
In the lesson, your code is checked as you write it.
In your own words
Why check a rule against a numerical slope if the rule is already known to be true?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the slope of a squared error
Purpose. See what drives a weight's update.
A one-weight model predicts ŷ = w·x, and its squared error on one example is L = (w·x − y)². Expand it and apply the rules (lesson 74 shows a faster way):
L=w2x2−2wxy+y2⇒dwdL=2wx2−2xy=2x(wx−y)
In plain words: the slope is 2 times the input times the error, (ŷ − y). No error, no slope.
w = 1.5for x, y in [(2, 5), (1, 1), (3, 4)]:
error = w * x - y
print(x, y, error, 2 * x * error)
The first example's prediction, 3, misses 5 by −2, giving a slope of −8: raise w. The others overshoot, giving positive slopes: lower w. Bigger errors and bigger inputs give bigger slopes.
Quick check
For L = (w·x − y)², the prediction is exactly right on one example. What is dL/dw for it?
0: the error is 0
y, the target
1, the slope of a line
2x, the input doubled
Show the answer
0: the error is 0
dL/dw = 2x(wx − y), and a perfect prediction makes wx − y = 0.
What the formula says about training. Because the error sits inside the slope, a weight is pushed hardest by the examples it gets most wrong, and not at all by the ones it already gets right. The input x scales the push too: an example with a big input has more leverage over w. Averaging these slopes over all the examples gives the slope of the mean squared error, the number lesson 79's gradient descent follows downhill.
In PyTorch. PyTorch never writes out 2x(wx − y). It records how L was computed from w and, when you call L.backward(), applies these same rules step by step to put dL/dw into w.grad (lesson 72 showed it first). First the rules on their own: the power, constant multiple, sum and constant rules in one polynomial, then eˣ and ln x:
import math
import torch
x = torch.tensor(2.0, requires_grad=True)
(3 * x ** 4 - 5 * x + 7).backward()
print(x.grad.item(), 3 * 4 * 2 ** 3 - 5)
x = torch.tensor(1.0, requires_grad=True)
torch.exp(x).backward()
print(round(x.grad.item(), 4), round(math.exp(1), 4))
x = torch.tensor(2.0, requires_grad=True)
torch.log(x).backward()
print(x.grad.item(), 1 / 2)
Each .grad matches its rule: 91.0 and 91 for 12x³ − 5 at x = 2, 2.7183 twice for eˣ at 1, and 0.5 twice for 1/x at 2.
Now the slope of the squared error itself, for the three examples at the start of this part, with w = 1.5. A fresh w for each example keeps their slopes apart, because backward() adds to whatever w.grad already holds:
import torch
for x, y in [(2, 5), (1, 1), (3, 4)]:
w = torch.tensor(1.5, requires_grad=True)
L = (w * x - y) ** 2
L.backward()
print(x, y, w.grad.item(), 2 * x * (1.5 * x - y))
w.grad is −8.0, 1.0 and 3.0, the same slopes 2x(wx − y) gave at the start of this part, printed beside them again.
Summing over examples. A real loss averages the squared error over many examples, and the sum rule says its slope is the average of their slopes. So each example adds its own push, 2x(wx − y), and the weight moves by the combined push. Examples the model gets badly wrong, with big inputs, dominate that sum; examples it gets right contribute nothing at all. That's also why noisy, mislabeled examples can do harm: the model sees a big error and pushes hard to fix it, even though the label was the mistake. Cleaning labels often helps more than any change to the model.
Drill
Slopes of a squared error
Write dL_dw(w, x, y) returning 2x(wx − y). Then set slopes to its values for the pairs below at w = 2, and avg to their mean.
defdL_dw(w, x, y):
return w * x - y
pairs = [(1, 3), (2, 3), (4, 7)]
slopes = []
avg = 0print(slopes, avg)
In the lesson, your code is checked as you write it.
In your own words
Why does an example the model already predicts perfectly not change the weight at all?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write