Training a model means finding the weights that make the loss as small as possible. So the question "where is this function lowest?" is the goal of nearly everything in machine learning. Calculus gives a sharp answer for smooth functions: at a lowest point the slope is zero, because the curve has stopped falling and not yet started rising. The second derivative, the slope of the slope, then says whether that flat spot is the bottom of a bowl or the top of a dome. This lesson finds turning points by hand and in Python, and then meets the problem real models face: a bumpy loss with more than one valley, where the bottom you find depends on where you start.
What you'll be able to do
Find turning points by setting the slope to zero
Use the second derivative to tell a bowl (a lowest point) from a dome (a highest point)
Find a minimum in Python by solving f′ = 0, or by searching a table of values, and plot it to check the ends
Tell local from global minima, and see why a convex loss is easy and a bumpy one isn't
Read PyTorch's x.grad as the slope, 0 at a turning point, and watch optim.SGD end in different valleys from different starts
Quick check
Have a guess before you start. You're walking along a valley floor in fog. How can you tell you're at the very bottom?
It's level, and every step leads up
The ground slopes down steeply ahead
It's darker there
You've walked a long way
Show the answer
It's level, and every step leads up
At the bottom the slope is zero, and every step leads up. That's exactly the test calculus uses.
The math: where the slope is zero
Purpose. Find where a function is smallest, or largest.
At a lowest or highest point of a smooth curve, the tangent is flat: f′(x) = 0. These are turning points. For f(x) = x² − 4x + 5:
f′(x)=2x−4=0⇒x=2,f(2)=1
In plain words: solve "slope = 0" for x, then put that x back in to get the height.
To the left of 2 the slope is negative (falling); to the right, positive (rising). So x = 2 is a lowest point, a minimum, with value 1.
Quick check
From lesson 73: what is the derivative of x² − 4x + 5?
2x − 4
2x² − 4
2x − 4 + 5
x − 4
Show the answer
2x − 4
x² → 2x, −4x → −4, and 5 → 0.
The second derivative. Differentiate twice and you get f″(x), how the slope itself is changing. If f″ > 0 the slope is increasing, so the curve bends up like a bowl: a flat spot there is a minimum. If f″ < 0 it bends down like a dome: a maximum. Step through it for 2x² − 8x + 3.
The picture. The drawing shows the curve. The first step lays a short tangent one unit to each side of the turning point, labeled with its slope: falling on the left, rising on the right. The next lays the flat tangent at the bottom, then marks the point and its value, and the last names the shape. The PyTorch tab finds the flat spot with autograd: at x = 2, x.grad is 0, as the AI part prints.
Try it now
Then set a to −1, b to 6 and c to 0: −x² + 6x has a dome, with its highest point at x = 3.
Symbols and formula. The second-derivative test:
f′(a)=0 and f′′(a)>0⇒minimumf′(a)=0 and f′′(a)<0⇒maximum
In plain words: flat and bending up is a bottom; flat and bending down is a top.
For x³ − 3x, f′ = 3x² − 3 is 0 at x = 1 and x = −1, and f″ = 6x. At x = 1, f″ = 6 > 0: a minimum, f(1) = −2. At x = −1, f″ = −6 < 0: a maximum, f(−1) = 2. When f″ = 0 too, the test can't decide and you look at the slope on each side instead.
Quick check
At x = 4, f′(4) = 0 and f″(4) = −3. What is x = 4?
Can't tell without more information
Not a turning point
A maximum: flat and bending down
A minimum, the bottom of a bowl
Show the answer
A maximum: flat and bending down
A negative second derivative means a dome, so the flat spot is its top.
Local and global. A local minimum is lower than everything near it; a global minimum is lower than everything anywhere. A bumpy curve can have several local minima, each the bottom of its own valley, and only one of them is global. A convex function, shaped like a single bowl with no bumps, has only one minimum, so local and global are the same. That makes convex functions easy to minimize: any way downhill leads to the one bottom.
In your own words
In your own words: why can't f′ = 0 alone tell a minimum from a maximum?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: solve, or search a table
Purpose. Find a minimum both ways, and know when each works.
When the slope is a simple formula, solve f′ = 0 by hand and let Python evaluate. When it isn't, compute the function at many points and pick the smallest:
import numpy as np
deff(x):
return x ** 2 - 4 * x + 5
xs = np.linspace(0, 4, 9)
ys = f(xs)
print(xs.tolist())
print(ys.tolist())
print(xs[np.argmin(ys)], ys.min())
It runs right here in your browser, with nothing to install.
np.linspace(0, 4, 9) makes 9 evenly spaced points from 0 to 4. np.argmin gives the position of the smallest value and .min() the value itself: x = 2, f = 1, matching f′ = 0.
Code flow. Under np.argmin is a plain search: keep the best x so far, and replace it whenever a new x gives a lower value. Step through it on seven points from 0 to 3. Watch best walk down the valley and stop at the bottom.
Finding local minima in a table. A value lower than both its neighbors is a local minimum of the table:
L = [5.0, 3.2, 2.1, 2.6, 3.0, 1.4, 0.9, 1.3, 2.2]
for i inrange(1, len(L) - 1):
if L[i] < L[i - 1] and L[i] < L[i + 1]:
print(i, L[i])
A named mistake: a table that's too coarse. A table only sees the points it computes. With points far apart, a narrow valley can fall between them and be missed, or a minimum can be reported a whole step away from where it really is. Use a finer table near anything interesting, or solve f′ = 0 where you can.
Worked example
Worked example: minima of three functions, both ways
Find each function's minimum on [−3, 3] by a fine table, and compare with solving f′ = 0.
import numpy as np
xs = np.linspace(-3, 3, 601)
deff1(x):
return x ** 2 - 4 * x + 5deff2(x):
return2 * x ** 2 + 3 * x
deff3(x):
return x ** 3 - 3 * x
for f in [f1, f2, f3]:
ys = f(xs)
print(round(float(xs[np.argmin(ys)]), 3), round(float(ys.min()), 3))
print(4 / 2, -3 / 4)
In the lesson you can change this and run it.
The table's 601 points are 0.01 apart. f1's minimum is at 2 and f2's at −0.75, matching f′ = 0 (the last line). f3 is the warning: its table minimum is at x = −3, the edge of the range, with value −18, not the local minimum at x = 1. Cubics fall forever to the left, so on a range the lowest point can be at an end, where the slope isn't 0. Checking the ends is part of finding a minimum on a range.
Plot it. A graph shows the ends at a glance. This draws x³ − 3x on the same range:
Read the plot. Where on the range −3 to 3 is x³ − 3x lowest?
At x = 1, its local minimum
At x = −3, the left end
At x = −1, where it turns
At x = 0, the middle
Show the answer
At x = −3, the left end
The curve dips to −2 at x = 1, but at the left end it falls all the way to −18: the table's answer in the worked example. A valley's bottom is only the lowest point nearby.
In the lesson, your code is checked as you write it.
In your own words
When would you search a table instead of solving f′ = 0?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: one valley or many
Purpose. Training looks for a low point of the loss, and the loss's shape decides how hard that is.
For the straight-line model of lesson 72, the mean squared error is a bowl in w: convex, with one minimum, where its slope is 0. Any walk downhill finds it. A neural network's loss is different: a landscape with many valleys, and the walk finds whichever one it starts in.
convex: one minimumbumpy: many local minima, one global
In plain words: in a single bowl every path downhill ends at the same bottom; in a bumpy landscape, where you start decides where you end.
defslope(x):
return x ** 3 - 4 * x + 0.8for start in [-3.0, 3.0]:
x = start
for step inrange(60):
x = x - 0.05 * slope(x)
print(start, round(x, 3), round(0.25 * x ** 4 - 2 * x ** 2 + 0.8 * x + 5, 3))
Started on the left, the walk ends at x ≈ −2.09, value −0.64: the global minimum. Started on the right, it stops at x ≈ 1.89, value 2.56: a local minimum. Same rule, same rate, different answers.
The picture. Both shapes, drawn. Press Next to go from one valley to two. The rings are the starts and the dots are where the walks stop.
Quick check
Why is fitting a straight line with mean squared error easier than training a deep network?
Networks have no minima
The line's loss is convex: one bowl, one bottom
Lines have fewer numbers to print
Mean squared error can't be used for networks
Show the answer
The line's loss is convex: one bowl, one bottom
A convex loss has a single minimum, so downhill steps always find it. A network's loss is bumpy.
Why deep learning works anyway. It turns out that in very high dimensions most local minima of a large network's loss are nearly as good as the global one, and the real obstacles are flat regions and saddle points (lesson 83), where the slope is near 0 but the point isn't a bottom at all. Practical training adds tricks, like momentum and random restarts, to keep moving through them.
In PyTorch. The same two walks, with PyTorch finding the slopes. First, the math part's turning point: at x = 2, y.backward() fills x.grad with the slope there. Then torch.optim.SGD([x], lr=0.05) makes an optimizer, the update on the map: opt.step() moves x by −0.05 × x.grad, the same step as the loop above, and opt.zero_grad() clears the old slope first, because backward() adds to .grad instead of replacing it.
import torch
x = torch.tensor(2.0, requires_grad=True)
y = 2 * x ** 2 - 8 * x + 3
y.backward()
print(y.item(), x.grad.item())
for start in [-3.0, 3.0]:
x = torch.tensor(start, requires_grad=True)
opt = torch.optim.SGD([x], lr=0.05)
for step inrange(60):
loss = 0.25 * x ** 4 - 2 * x ** 2 + 0.8 * x + 5
opt.zero_grad()
loss.backward()
opt.step()
print(start, round(x.item(), 3), round(loss.item(), 3))
The first line is the flow's height at x = 2, −5.0, and a slope of 0.0 there: a flat tangent. Then the two walks print exactly what the plain loop printed: −2.093 and −0.638 from the left, 1.891 and 2.558 from the right (each loss is the last pass's, worked out just before its step). Lesson 79 writes this loop out for a whole model.
Quick check
A loss was recorded at six points. What does this print?
L = [3.0, 2.2, 2.5, 1.9, 1.2, 1.6]
found = []
for i inrange(1, len(L) - 1):
if L[i] < L[i - 1] and L[i] < L[i + 1]:
found.append(i)
print(found)
[4]
[1, 4]
[0, 5]
[1, 3, 4]
Show the answer
[1, 4]
2.2 (position 1) and 1.2 (position 4) are each lower than both neighbors. 1.9 isn't: the next value, 1.2, is lower.
Drill
Minima in a loss table
Set mins to the positions of every local minimum in the loss table (values lower than both neighbors), and best to the position of the global minimum. Then set left_finds to the position a walk from the left end reaches: keep moving right while the next value is lower.
L = [4.0, 2.5, 1.8, 2.2, 2.9, 1.1, 0.6, 0.8, 1.9]
mins = []
best = 0
left_finds = 0print(mins, best, left_finds)
In the lesson, your code is checked as you write it.
In your own words
A walk downhill from the left stopped at a value of 1.8, but the table's lowest value is 0.6. What happened, and what could help?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Curvature: second derivatives and the shape of a loss.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write