Calculus slips are quiet. A gradient step with a plus where a minus belongs still runs; it just climbs instead of descending. A chain rule missing its inner factor gives a slope that's off by a constant. A rate never divided by its step looks like a rate. Each gives a confident number, and the loss curve, or a careful check, is the only warning. This review gathers the module's commonest slips, each paired with the check that catches it, mixes problems from all of Module 5, and collects every rule on one sheet, ready for Module 6, where probability meets the losses these slopes are taken of.
What you'll be able to do
Spot the common slips of Module 5: a step with the wrong sign, a chain missing its inner factor, a rate not divided by its step, a slope overwritten instead of added, and a learning rate that diverges; and two from optimizers and autograd
Find and fix slips in calculus code that runs but gives a wrong answer
Mix problems from every lesson of the module, and run a short calculus program in your head, line by line
Use the module's formula sheet: each rule, what it says, and its Python line; and see that Module 5 is a model's backward row
Quick check
Have a guess before you start. A training loop's loss goes 5, 7, 11, 19, 35, doubling its rise each step. Which slip is most likely?
The loss uses the wrong formula entirely
The model is too accurate
The data is too small
The step adds the gradient, so it climbs
Show the answer
The step adds the gradient, so it climbs
A loss that grows steadily from the first step usually means θ + η∇L instead of θ − η∇L, or a learning rate far past 2/L″.
Find the slip
Purpose. Five slips cover most of Module 5's mistakes, and each has a quick check:
The wrong sign. θ + η∇L climbs. Check: the loss must fall on the first small step.
A missing inner factor. ((3x + 1)²)′ is 2u × 3, not 2u. Check: compare with a central difference.
A rate not divided by its step.np.diff(v) alone is a rate only when the step is 1. Check: the units.
A slope overwritten. A value feeding two steps needs both contributions added. Check: a gradient check.
A diverging learning rate. Past 2 ÷ L″, steps overshoot and grow. Check: the loss must not rise.
Below are twelve worked lines, with a comment saying what each should give. Seven have a slip: the five above, and one each from lesson 80 and lesson 82.
Try it now
How the checks catch them. Each slip fails a one-line test. One small step of w + lr * dw raises the loss instead of lowering it. A central difference of (3x + 1)² at x = 1 gives 24, not 8. Printing the units, km per half hour or loss per 100 steps, exposes the missing division. A gradient check exposes the overwritten slope. And 2 ÷ L″ = 1 rules out a learning rate of 1.5 before anything runs. The last two fail a small case worked by hand. One step from v = 0 with g = −2 leaves 0.001 × (−2) = −0.002 in v, a negative average of squares, which can't be. And with no zero_grad(), two batches whose backward() each finds a slope of 3 leave 3 + 3 = 6 in w.grad. Predict the next one before you run it.
Quick check
What does this print?
x = 0.0for step inrange(3):
x = x + 0.25 * 2 * (x - 2)
print(x)
1.75
0.0
-4.75
2.0
Show the answer
-4.75
The + makes each step move away from 2: x goes −1, −2.5, −4.75. The fix is x − 0.25 × slope, which gives 1.75.
Slips that hide in code. Like the earlier modules', these slips run without an error. A slope function that forgets to divide by h returns tiny numbers that look like slopes. A chain rule missing its inner factor is off by the same constant everywhere, so its shape looks right on a plot. The two drills below are slips of exactly this kind: each runs, each gives a confident wrong answer, and one numerical check exposes each.
Quick check
A slope function returns 0.0006 for x² at x = 3 with h = 0.0001. What's the most likely slip?
It used a central difference
It forgot to divide by h
h is too small
x² has no slope at 3
Show the answer
It forgot to divide by h
f(3.0001) − f(3) ≈ 0.0006 is the change; divided by h = 0.0001 it's the slope, 6.
Drill
A slope off by a constant
This slope_of is meant to return the derivative of (2x − 1)³, but at x = 1 it gives 3 instead of 6. Find the slip and fix it.
defslope_of(x):
u = 2 * x - 1return3 * u ** 2print(slope_of(1), slope_of(2))
In the lesson, your code is checked as you write it.
In your own words
Which numerical check would you run on any slope formula you write by hand, and why that one?
Write a sentence or two, then compare it with a good answer in the lesson.
Mixed practice
Purpose. Problems from across Module 5, mixed. Name each one's lesson before you start: rates, limits, derivatives, rules, chains, minima, approximation, partials, gradients, descent, momentum, backpropagation, autograd, curvature or areas. If one stops you, the steps show the way, and the formula sheet in the last part has every rule in one place.
Play computer. The program below runs the module in small: a rate from readings half an hour apart, a chain's slope with its inner factor, one step of descent, a slope collected from two paths, and a trapezoid area. Each line is written the right way, at a place where one of the first part's slips could hide. Work out every value in your head before you type it, the way Python would print it: 2.0 when a decimal is involved, 2 when only whole numbers are.
Worked example
Worked example: a training loop with its checks built in
Train a line, and check after the first step that the loss fell, and at the end that the gradient is near zero.
import numpy as np
x = np.array([1.0, 2.0, 3.0])
y = np.array([3.0, 5.0, 7.0])
defloss(w, b):
return np.mean((w * x + b - y) ** 2)
w, b = 0.0, 0.0
start = loss(w, b)
for step inrange(500):
e = w * x + b - y
dw = 2 / 3 * (e @ x)
db = 2 / 3 * np.sum(e)
w = w - 0.05 * dw
b = b - 0.05 * db
if step == 0:
print("fell on step 1:", loss(w, b) < start)
e = w * x + b - y
print(round(w, 3), round(b, 3), round(abs(2 / 3 * (e @ x)), 4))
It runs right here in your browser, with nothing to install.
The first check would catch a wrong sign or a diverging rate at once: the loss must fall on step one. The last line prints the size of ∂L/∂w at the end: close to 0 means training reached the bottom, where w ≈ 2 and b ≈ 1 fit y = 2x + 1. Two cheap checks, at the start and the end, catch most training bugs.
Quick check
What does this print?
import numpy as np
d = np.array([0.0, 1.0, 3.0, 6.0])
print((np.diff(d) / 2).tolist(), np.cumsum(np.diff(d)).tolist())
[0.5, 1.0, 1.5] [1.0, 3.0, 6.0]
[1.0, 2.0, 3.0] [1.0, 3.0, 6.0]
[1.0, 3.0, 6.0] [0.5, 1.0, 1.5]
[0.5, 1.0, 1.5] [1.0, 2.0, 3.0]
Show the answer
[0.5, 1.0, 1.5] [1.0, 3.0, 6.0]
The differences are 1, 2 and 3; divided by the step 2 they're 0.5, 1 and 1.5. Their running total rebuilds how far d has come.
Quick check
Which learning rate is safe for a bowl with curvature L″ = 4?
4.0
0.6
0.3
1.0
Show the answer
0.3
The limit is 2 ÷ 4 = 0.5, so 0.3 settles; the others overshoot and grow.
Drill
A loss that climbs
This loop should fit w to the data, but its loss grows every step. Find the one slip and fix it.
import numpy as np
x = np.array([1.0, 2.0, 3.0])
y = np.array([2.0, 4.0, 6.0])
w = 0.0for step inrange(100):
e = w * x - y
dw = 2 / 3 * (e @ x)
w = w + 0.05 * dw
print(round(w, 4))
In the lesson, your code is checked as you write it.
In your own words
Which kind of problem in the mixed ladder did you have to stop and think about most? What told you which method to use?
Write a sentence or two, then compare it with a good answer in the lesson.
The formula sheet
Purpose. Every rule of Module 5 in one place, with what it says and its Python line.
lesson
rule
in plain words
in Python
70
Δy/Δx = (f(b) − f(a))/(b − a)
change over change
np.diff(v) / h
72
f′(x) = lim (f(x + h) − f(x))/h
the slope at a point
(f(x + h) - f(x - h)) / (2 * h)
73
(xⁿ)′ = nxⁿ⁻¹; (eˣ)′ = eˣ; (ln x)′ = 1/x
power down, power lowered
n * c[n] per term
74
dy/dx = dy/du · du/dx
multiply the links
dy_du * du_dx
75
f′ = 0; f″ > 0 bowl, f″ < 0 dome
flat, then which way it bends
xs[np.argmin(f(xs))]
76
f(x) ≈ f(a) + f′(a)(x − a) + ½f″(a)(x − a)²
the curve near a point
fa + dfa * d + 0.5 * d2fa * d ** 2
77–78
∇f = (∂f/∂x, ∂f/∂y)
all the slopes, steepest climb
nudge copies, one input at a time
79
θ ← θ − η∇L, η < 2/L″
step downhill, not too far
w = w - lr * dw
80
momentum: v ← βv + g, w ← w − ηv; Adam: m ← β₁m + (1 − β₁)g, v ← β₂v + (1 − β₂)g², m̂ = m ÷ (1 − β₁ᵗ), v̂ = v ÷ (1 − β₂ᵗ), w ← w − η·m̂ ÷ (√v̂ + ε)
momentum piles up slopes that keep pointing one way; Adam divides the average slope by its usual size, so every weight moves about η
v = 0.9 * v + g; w = w - lr * mh / (np.sqrt(vh) + eps)
Quick check
Which row of the sheet so far catches a chain missing its inner factor?
Lesson 79: the descent step
Lesson 75: f′ = 0
Lesson 74: dy/dx = dy/du · du/dx
Lesson 70: the rate of change
Show the answer
Lesson 74: dy/dx = dy/du · du/dx
The chain rule says the slope is the product of every link's slope, including the inner one.
The rest of the sheet:
lesson
rule
in plain words
in Python
81
∂L/∂v = Σₖ ∂L/∂uₖ · ∂uₖ/∂v
multiply along a path, add across paths
dL_dv = dL_dv + ...
82
ū ← ū + v̄ · ∂v/∂u for each entry v, last to first (ū = ∂L/∂u)
record each step forward, walk the tape back; .grad adds up until zeroed
loss.backward(), then w.grad; optimizer.zero_grad()
83
Jᵢⱼ = ∂fᵢ/∂xⱼ; Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ
all slopes; all curvatures
J[:, j] = ..., np.linalg.eigvalsh(H)
84
∫ f dx ≈ Σ f(xᵢ)Δx
thin strips, added
dx * (np.sum(y) - (y[0] + y[-1]) / 2)
Each row is also a check: if an answer breaks a row, the slip is in that step.
Where Module 5 lives in a model. Module 5 is the model's backward row. The loss is one number that depends on every weight. The chain rule finds its slope for each one, multiplying along paths and adding across them, and autograd does that bookkeeping on a tape; each layer passes the slopes back through its Jacobian. All the slopes together are the gradient, and the update steps every weight against it: plainly, or with momentum and Adam. The map below lights that row, from the loss back to the weights.
Coming up. First, lesson 86 practices rearranging this module's formulas: a slope set to zero, a step solved for its learning rate. Then Module 6's probability gives the losses these slopes are taken of. A model's output becomes a probability, its loss becomes how surprised it is by the right answer (lesson 105), and gradient descent then follows the slope of that surprise. Areas come back too: for continuous values, every probability is an area.
In your own words
Pick the rule on the sheet you're most likely to slip on. What check would you use to catch the slip?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write