Up close, every smooth curve looks like a straight line: its tangent. That simple fact is how calculus turns hard problems into easy ones. Near a point, replace the curve by its tangent line and you can predict values with one multiplication; add a curvature term and the prediction gets better still. The catch is the word "near": the farther you go, the worse the stand-in. Gradient descent relies on exactly this. Each step trusts the tangent's promise that moving downhill lowers the loss, and that promise only holds for small steps. This lesson builds the approximation, measures its error, and works out how long a step can be before the promise breaks.
What you'll be able to do
Approximate a curve near a point by its tangent line: f(a) + f′(a)(x − a)
Add a curvature term, ½f″(a)(x − a)², for a closer fit
Measure how the error grows as you move away from the point
See why learning rates must be small: a step only lowers the loss while the tangent is still right
Quick check
Have a guess before you start. √4 = 2. Without a calculator, about what is √4.1?
About 2.025
About 4.1
About 2.1
About 2.05
Show the answer
About 2.025
The slope of √x at 4 is 1/(2√4) = 0.25, so a step of 0.1 raises it by about 0.025. The true value is 2.02485.
The math: the tangent as a stand-in
Purpose. Replace a hard function by an easy one, near a point.
Near x = a, a smooth curve stays close to its tangent line. So for x near a:
f(x)≈f(a)+f′(a)(x−a)
In plain words: start at the known value, and add the slope times how far you've moved.
This is lesson 72's Δy ≈ f′(x)·Δx, written as a function of x. For √x near a = 4: f(4) = 2 and f′(4) = 0.25, so √4.1 ≈ 2 + 0.25 × 0.1 = 2.025. The true value is 2.024846: off by about 0.00015.
Quick check
From lesson 72: f′(4) = 8 for x². About how much does x² change between 4 and 4.05?
About 8
About 16.4
About 0.05
About 0.4
Show the answer
About 0.4
Δy ≈ 8 × 0.05 = 0.4. The tangent-line approximation is the same idea: 4.05² ≈ 16 + 0.4 = 16.4.
Adding curvature. The tangent is straight; the curve bends. The second derivative measures the bend, and one more term accounts for it:
f(x)≈f(a)+f′(a)(x−a)+21f′′(a)(x−a)2
In plain words: a parabola that matches the curve's value, slope and bend at a. It hugs the curve much more tightly than the tangent does.
Step through eˣ near 0, stepping 0.5 away: the tangent, then the curvature term, against the true value.
The picture. The curve is eˣ, the big dot is (0, 1), and the dashed line at x = 0.5 is where we want the value. The second step draws the tangent: the hollow dot where it meets the dashed line is its guess, and the filled dot on the curve is the true value. The third step adds the curvature term, a dashed parabola that bends with the curve, and its guess lands much nearer.
Try it now
Then set the step away to 0.1, and then to 2, and press Use these numbers: near the point both are close; far away even the parabola falls behind.
How the error grows. Near a, the tangent's error grows like the square of the distance: twice as far, about four times the error. The parabola's error grows like the cube: twice as far, about eight times, but starting much smaller. For √x around 4: at a distance of 0.1 the tangent is off by 0.00015; at 1, by 0.014; at 2, by 0.05. The approximation is excellent nearby and drifts away from the curve as you go farther.
Quick check
A tangent-line approximation is off by 0.002 at a distance of 0.1. Roughly how far off is it at a distance of 0.2?
About 0.004
About 0.2
About 0.008
About 0.002
Show the answer
About 0.008
The tangent's error grows like the distance squared: twice as far, about 2² = 4 times the error.
In your own words
In your own words: why is the curvature-term version exact for x², but not for eˣ?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: the stand-ins as functions
Purpose. Build the approximations once, and tabulate their errors.
import math
deftangent(f, df, a, x):
return f(a) + df(a) * (x - a)
defquadratic(f, df, d2f, a, x):
return tangent(f, df, a, x) + 0.5 * d2f(a) * (x - a) ** 2for d in [0.1, 0.5, 1, 2]:
x = 0 + d
print(d, round(tangent(math.exp, math.exp, 0, x), 4), round(quadratic(math.exp, math.exp, math.exp, 0, x), 4), round(math.exp(x), 4))
It runs right here in your browser, with nothing to install.
For eˣ the slope and curvature functions are eˣ itself, so math.exp is passed three times. At 0.1 all three columns nearly agree; at 2 the tangent says 3, the parabola 5, and the truth is 7.389.
Code flow. How fast does the tangent's error grow? Step through the tangent for eˣ at 0, doubling the step each pass, and watch the errors list fill one box at a time.
A named mistake: forgetting the ½. The curvature term is ½f″(a)(x − a)², not f″(a)(x − a)². Without the ½, the parabola bends twice as much as the curve and overshoots: for x² near 3 it would give 9.62 instead of the exact 9.61. Comparing with a true value at one small step catches it.
Why bother approximating at all? Many functions are slow or impossible to compute exactly, but their values, slopes and curvatures at one point are known. Physics, engineering and optimization all lean on these stand-ins, and the tangent version is the one gradient descent uses at every step.
Worked example
Worked example: the error growing with the step
Tabulate √x near 4: the true value, the tangent's and the parabola's, and their errors, at four distances.
import math
deff(x):
return math.sqrt(x)
defdf(x):
return1 / (2 * math.sqrt(x))
defd2f(x):
return -1 / (4 * x ** 1.5)
a = 4for d in [0.1, 0.5, 1, 2]:
true = f(a + d)
first = f(a) + df(a) * d
second = first + 0.5 * d2f(a) * d ** 2print(d, round(true, 4), round(abs(first - true), 6), round(abs(second - true), 6))
In the lesson you can change this and run it.
The tangent's error grows from 0.00015 to 0.0505 as the step goes from 0.1 to 2, faster than the step itself. The parabola's error starts at 0.000002 and is always much smaller. √x bends down (f″ < 0), so here the tangent overshoots and the curvature term pulls the estimate back down. Both are best near 4 and worse far away.
Quick check
What does this print?
deff(x):
return x ** 2
a = 5
d = 0.2print(round(f(a) + 2 * a * d, 6), round(f(a + d), 6))
27.0 27.0
27.04 27.0
27.0 27.04
25.0 27.04
Show the answer
27.0 27.04
The tangent gives 25 + 10 × 0.2 = 27. The truth is 5.2² = 27.04: the curvature adds the last 0.04.
Drill
Tangent and parabola
Fix quadratic so it adds the curvature term with its ½. Then set errs to the absolute errors of the tangent and of the parabola for eˣ at a = 0, x = 1, each rounded to 4 places.
import math
deftangent(f, df, a, x):
return f(a) + df(a) * (x - a)
defquadratic(f, df, d2f, a, x):
return tangent(f, df, a, x) + d2f(a) * (x - a) ** 2
errs = []
print(quadratic(math.exp, math.exp, math.exp, 0, 1), errs)
In the lesson, your code is checked as you write it.
In your own words
Why do both approximations get worse the farther x is from a?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: why learning rates must be small
Purpose. A gradient step trusts the tangent; curvature decides how far that trust goes.
Gradient descent (lesson 79) moves a weight against the slope: w ← w − η·L′(w). The tangent promises that this lowers the loss by about η·L′(w)². That promise only holds while the tangent is still a good stand-in, and the curvature decides for how long. For a bowl with curvature L″, steps are safe while
η<L′′2
In plain words: the more sharply the loss bends, the shorter a step must be to stay downhill.
defloss(w):
return (w - 2) ** 2 + 1
w = 0
slope = 2 * (w - 2)
for lr in [0.25, 0.5, 0.75, 1.0, 1.1]:
new_w = w - lr * slope
print(lr, new_w, round(loss(new_w), 4))
The loss starts at 5 with slope −4 and curvature 2. Steps with η = 0.25, 0.5 and 0.75 all lower it. At η = 1 the step jumps clean over the bottom to a point just as high, 5; at 1.1 the loss gets worse. The limit is 2 ÷ 2 = 1, exactly as the formula says.
The picture. Four of those steps, drawn. The arrow is the step, at the starting height; the dashed line is the starting loss, 5. Where the step lands on the curve says whether it helped: below the dashed line is better, above it is worse.
Quick check
A loss bowl has curvature L″ = 8. What is the largest safe learning rate?
2
0.25
4
8
Show the answer
0.25
η < 2 ÷ L″ = 2 ÷ 8 = 0.25. A sharply bent loss needs short steps.
Why one learning rate is hard to choose. A real loss bends differently in different directions and at different places: steep in some weights, gentle in others. A learning rate small enough for the steepest direction is painfully slow in the gentle ones. That tension is why optimizers like Adam adapt the step for each weight, and why lesson 83's Hessian, the table of all the curvatures, matters.
The promise, in numbers. At w = 0 the loss is 5 and its slope −4, so the tangent promises that a step of η lowers the loss by about 16η. For η = 0.25 it promises 4 and delivers 3; for η = 1 it promises 16 and delivers 0. The promise is good for small steps and grows ever more wrong as the step lengthens.
In PyTorch. The same loss, with the step done by PyTorch: backward() finds the slope, −4, and the update moves w against it. torch.no_grad() (lesson 50) lets the update change w in place without being recorded, exactly what an optimizer's step() does inside. A safe learning rate, then one too large:
import torch
for lr in [0.25, 1.1]:
w = torch.tensor(0.0, requires_grad=True)
loss = (w - 2) ** 2 + 1
loss.backward()
with torch.no_grad():
w -= lr * w.grad
print(lr, round(w.item(), 4), round(((w - 2) ** 2 + 1).item(), 4))
The same two rows as the Python cell: η = 0.25 lands at w = 1.0 with the loss down to 2.0, and η = 1.1 at 4.4 with the loss up to 6.76. PyTorch follows the slope it's given; only a smaller learning rate stops the overshoot.
Quick check
One step from w = 0 on (w − 2)² + 1, at two learning rates. What does this print?
η = 0.4 is under the limit of 1 and lowers the loss from 5 to 1.16; η = 1.2 overshoots to 4.8, where the loss is 8.84.
Drill
The largest safe step
For the loss 4(w − 1)² starting at w = 0, set limit to 2 ÷ L″, and ok to a list of True/False for whether one step with each learning rate in rates lowers the loss.
defloss(w):
return4 * (w - 1) ** 2
w = 0
slope = 8 * (w - 1)
rates = [0.1, 0.2, 0.25, 0.3]
limit = 0
ok = []
print(limit, ok)
In the lesson, your code is checked as you write it.
In your own words
A training run's loss suddenly starts rising. What does this lesson suggest you try, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Curvature: second derivatives, curvature and step sizes.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write