Almost no real model is trained with plain gradient descent. The optimizer in nearly every training script is Adam, and its update is three short formulas you can already read: two moving averages from lesson 27 and a step from lesson 79. This lesson runs momentum and Adam by hand, writes both as loops, sends all three optimizers down the same narrow valley, and then gets the same numbers out of PyTorch.
What you'll be able to do
Run momentum by hand for three steps, and say why its steps grow on a steady slope and cancel on a zigzag
Run Adam by hand for three steps, with its bias correction, and say what β₁, β₂ and ε each do
Write plain descent, momentum and Adam as loops in Python, and compare their loss curves on one bowl
Compare the three down a narrow valley, and get the hand numbers again from PyTorch's optim.SGD and optim.Adam
Quick check
Have a guess before you start. Gradient descent always steps straight downhill. In a long, narrow valley with steep sides, what does that do?
It climbs out over the top of the valley
It stops at the first wall it meets
It runs straight along the floor to the bottom
It zigzags from wall to wall and creeps along the floor
Show the answer
It zigzags from wall to wall and creeps along the floor
Straight downhill points mostly at the nearest steep wall, not along the valley. So the steps bounce across it and make little progress along it. Momentum and Adam are two fixes.
The math: averages of slopes
Purpose. Step down faster and steadier than plain descent by averaging the slopes as you go.
Plain gradient descent, from lesson 79, steps against today's slope and forgets every slope before it. In a round bowl that works well. In a long, narrow valley it doesn't. Take the valley
L(w1,w2)=0.1w12+5w22
It is gentle along w₁ and steep across w₂. Its slopes are g = (0.2w₁, 10w₂), so at (−6, 1) the slope across is 10 and the slope along is only −1.2: straight downhill points almost straight at the far wall. The steep wall also caps the learning rate. Its curvature is 10, so past η = 2 ÷ 10 = 0.2 the steps across overshoot by more each time (lesson 76), and under that cap the steps along the floor are tiny.
Momentum keeps a velocity v, a running total of the slopes, and steps along it:
v←βv+g,w←w−ηv
In plain words: keep β of the old velocity, add today's slope, and step along the velocity. Slopes that keep pointing the same way pile up; slopes that flip back and forth cancel.
Quick check
Momentum with β = 0.9 starts at v = 0 and meets the same slope, g = 4, twice. What is v after the second step?
8
4
7.6
0.76
Show the answer
7.6
First v = 0.9 × 0 + 4 = 4. Then v = 0.9 × 4 + 4 = 7.6. A steady slope piles up: v heads toward 4 ÷ (1 − 0.9) = 40, ten times one slope.
Adam keeps two running averages from lesson 27, both started at 0: m of the slopes, and v of the squared slopes. (Adam's v is not momentum's velocity; the letter is reused.) Each is corrected for its start at 0, and then the step divides one by the other. t counts the steps, from 1:
m←β1m+(1−β1)g,v←β2v+(1−β2)g2
m^=1−β1tm,v^=1−β2tv,w←w−ηv^+ϵm^
In plain words: m̂ is where the slopes point on average, and √v̂ is how big they usually are. Their ratio is about ±1 for every weight, so each weight moves about η, steep or gentle.
The usual settings are β₁ = 0.9, β₂ = 0.999 and ε = 10⁻⁸ (epsilon, a tiny number).
The picture. Step through Adam's first three steps down the valley, seen from above. Each level curve is at half the loss of the one outside it. At each step's slopes an arrow shows −g, straight downhill; then the path grows by the move Adam really makes. The last step runs momentum, plain descent and Adam on from the same start, 18 steps each.
Try it now
Then set ε (epsilon) to 1 and press Use these numbers: the gentle weight's first move shrinks, because 1.2 ÷ (1.2 + 1) is only about half. That's why ε is kept tiny.
What each setting does.
β₁ = 0.9 sets how far back the slopes' average looks: about 1 ÷ (1 − 0.9) = 10 steps. It is Adam's momentum: slopes that keep their sign keep their size in m, and slopes that flip cancel.
β₂ = 0.999 does the same for the squared slopes, over about 1,000 steps, so √v̂ is a slow, steady gauge of how big each weight's slopes run.
ε = 10⁻⁸ keeps the division safe. A weight whose slopes are all near 0 would otherwise divide by almost nothing.
The bias correction. Both averages start at 0, lesson 27's named mistake: after one step m holds only 0.1 of g, and v only 0.001 of g². Dividing by 1 − β₁¹ = 0.1 and 1 − β₂¹ = 0.001 undoes that exactly, so on the first step m̂ is g and v̂ is g², as the flow showed. As t grows, βᵗ shrinks toward 0, the divisor grows toward 1, and the correction fades away.
Quick check
From an earlier lesson: an exponential moving average with β = 0.9 starts at m = 0 and is fed the value 5 three times. It reads 1.355. What does dividing it by 1 − 0.9³ give?
1.355
0.271
4.5
5
Show the answer
5
1 − 0.9³ = 1 − 0.729 = 0.271, and 1.355 ÷ 0.271 = 5: the value itself. Three 5s average to 5, and the correction removes the pull of the starting 0. That's Adam's m̂.
In your own words
In your own words: why does Adam move every weight by about the learning rate, however steep its slope?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: three optimizers as loops
Purpose. Write each optimizer as a short loop, then compare them on one bowl.
Back to lesson 79's bowl, L(x) = (x − 2)² + 1, from x = 0, where the slope is 2(x − 2). Here are plain descent and momentum, three steps each with η = 0.1:
defslope(x):
return2 * (x - 2)
x = 0.0for t inrange(3):
x = x - 0.1 * slope(x)
print("plain", t + 1, round(x, 4))
x, v = 0.0, 0.0for t inrange(3):
v = 0.9 * v + slope(x)
x = x - 0.1 * v
print("momentum", t + 1, round(v, 4), round(x, 4))
Momentum's first step matches plain descent's, 0.4, because v starts at 0. After that the slopes keep pointing the same way, so v piles up, −4, −6.8, −7.96, and momentum pulls ahead: 1.876 after three steps, against plain descent's 0.976.
Code flow. Now Adam: the math part's six lines inside a loop, three passes from x = 0 with η = 0.5. Step through it and watch m and v, their corrected copies mh and vh, and x. (vh ** 0.5 is the square root.) Float noise from lesson 12 shows here and there: 1 − 0.9 isn't exactly 0.1 in a float, so the first mh reads −4.000000000000001.
The same loop, printing m̂, √v̂ and x after each pass, rounded:
x, m, v = 0.0, 0.0, 0.0for t inrange(1, 4):
g = 2 * (x - 2)
m = 0.9 * m + 0.1 * g
v = 0.999 * v + 0.001 * g ** 2
mh = m / (1 - 0.9 ** t)
vh = v / (1 - 0.999 ** t)
x = x - 0.5 * mh / (vh ** 0.5 + 1e-8)
print(t, round(mh, 4), round(vh ** 0.5, 4), round(x, 4))
m̂ ÷ √v̂ is −1 on the first pass and close to −1 on the next two (−3.4737 ÷ 3.5353 and −2.9363 ÷ 3.1122), so each step is nearly η: x moves 0.5, then 0.4913, then 0.4717, to 1.463.
Quick check
What does this print?
v, x = 0.0, 0.0for g in [-4.0, -4.0]:
v = 0.9 * v + g
x = x - 0.1 * v
print(round(v, 2), round(x, 2))
-4.0 0.4
-7.6 1.16
-7.6 0.76
-8.0 1.2
Show the answer
-7.6 1.16
v = −4, so x = 0 − 0.1 × (−4) = 0.4. Then v = 0.9 × (−4) − 4 = −7.6, so x = 0.4 + 0.76 = 1.16. The second step is bigger than the first: the velocity piled up.
A named mistake: leaving out the bias correction. Drop mh and vh and step with m and v as they are:
x, m, v = 0.0, 0.0, 0.0for t inrange(1, 4):
g = 2 * (x - 2)
m = 0.9 * m + 0.1 * g
v = 0.999 * v + 0.001 * g ** 2
x = x - 0.5 * m / (v ** 0.5 + 1e-8)
print(t, round(x, 4))
print(round(0.001 ** 0.5, 4), round(0.1 / 0.001 ** 0.5, 4))
The first step lands at 1.5811, not 0.5, and the second throws x past the bottom at 2 to 3.2989. You might expect the opposite: m alone starts too small, as an average started at 0 does in lesson 27. But v starts far smaller still, holding only 0.001 of g², so its square root holds only √0.001 = 0.0316 of |g|. So m ÷ √v comes out 0.1 ÷ √0.001 = 3.1623 times too big, and 0.5 × 3.1623 = 1.5811. The two biases don't cancel. The correction removes both, so the first step is η.
Worked example
Worked example: three optimizers, twelve steps, one bowl
Run plain descent and momentum with η = 0.1, and Adam with η = 0.5, for 12 steps each from x = 0, and keep the loss after every step, rounded to 2 places.
defloss(x):
return (x - 2) ** 2 + 1
x, plain = 0.0, []
for t inrange(1, 13):
x = x - 0.1 * 2 * (x - 2)
plain.append(round(loss(x), 2))
x, v, mom = 0.0, 0.0, []
for t inrange(1, 13):
v = 0.9 * v + 2 * (x - 2)
x = x - 0.1 * v
mom.append(round(loss(x), 2))
x, m, v, adam = 0.0, 0.0, 0.0, []
for t inrange(1, 13):
g = 2 * (x - 2)
m = 0.9 * m + 0.1 * g
v = 0.999 * v + 0.001 * g ** 2
x = x - 0.5 * (m / (1 - 0.9 ** t)) / ((v / (1 - 0.999 ** t)) ** 0.5 + 1e-8)
adam.append(round(loss(x), 2))
print(plain)
print(mom)
print(adam)
In the lesson you can change this and run it.
Plain descent falls smoothly to 1.02, near the floor of 1. Momentum falls faster at first, to 1.02 by step 3, but its velocity carries x past the bottom: the loss climbs back to 3.01 at step 6 before it swings down again. Adam's steps stay near 0.5 long, so it too overshoots and swings, reaching 1.36 at step 12. The plot below draws all three lists.
The plot types in those three lists and draws them against the step number:
Read the plot. Which optimizer has the lowest loss at step 12, and why?
Adam: its steps are always exactly η
None: all three end level at 1
Plain descent: on a round bowl, straight down is right
Momentum: its velocity makes it the fastest
Show the answer
Plain descent: on a round bowl, straight down is right
Plain descent ends at 1.02, close to the floor. On a round bowl, downhill points at the bottom, so a good plain rate is hard to beat; momentum's velocity and Adam's steady steps overshoot. They earn their keep where slopes differ in scale or keep flipping, like the math part's valley, where Adam got the loss to 0.1132 in 18 steps and plain descent only to 1.0051.
Drill
Two optimizers as functions
Fix momentum(lr, steps) so it keeps PyTorch's velocity, v = 0.9v + g, and adam(lr, steps) so it corrects both averages before it steps. Both start at x = 0 on the bowl and return the final x. The starter has one slip in each.
defslope(x):
return2 * (x - 2)
defmomentum(lr, steps):
x, v = 0.0, 0.0for t inrange(steps):
v = 0.9 * v + 0.1 * slope(x)
x = x - lr * v
return x
defadam(lr, steps):
x, m, v = 0.0, 0.0, 0.0for t inrange(1, steps + 1):
g = slope(x)
m = 0.9 * m + 0.1 * g
v = 0.999 * v + 0.001 * g ** 2
x = x - lr * m / (v ** 0.5 + 1e-8)
return x
print(round(momentum(0.1, 3), 4), round(adam(0.5, 3), 4))
In the lesson, your code is checked as you write it.
In your own words
On the bowl, momentum was ahead after three steps and behind after twelve. Why?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the optimizer in every training loop
Purpose. Nearly every training script steps with Adam. Read its settings, and get your hand numbers out of PyTorch.
Steps the same size, whatever the units. The trail club predicts how long a walk takes, in hours, from its length in km and its climb in meters: ŷ = w₁ × length + w₂ × climb. A climb runs to hundreds of meters, so the slope for w₂ comes out far bigger than the slope for w₁. Here is the first step from w = (0, 0), both ways, with η = 0.001:
import numpy as np
km = np.array([4.0, 8.0, 10.0, 14.0])
climb = np.array([200.0, 600.0, 300.0, 900.0])
hours = np.array([1.5, 3.5, 3.0, 5.5])
e = 0.0 * km + 0.0 * climb - hours
g = np.array([2 / 4 * (e @ km), 2 / 4 * (e @ climb)])
print(g.tolist(), round(g[1] / g[0], 1))
lr = 0.001print("plain:", np.round(-lr * g, 4).tolist())
m = 0.1 * g
v = 0.001 * g ** 2print("Adam: ", np.round(-lr * (m / 0.1) / (np.sqrt(v / 0.001) + 1e-8), 4).tolist())
The slopes are −70.5 and −4,125: the climb weight's is 58.5 times bigger, only because climb is counted in meters. Plain descent's first move follows suit, 0.0705 for length and 4.125 for climb: a claim that every meter of climb adds four hours. Adam moves both weights 0.001, which is η. Its step is set by the learning rate, not by the units. That's why Adam's default rate, 0.001, works for so many models, while plain descent's rate has to be tuned to its steepest weight.
Quick check
One weight's slopes are always about 100 times another's. In plain descent and in Adam, how do their steps compare?
Plain: the same. Adam: 100 times bigger
Plain: 100 times bigger. Adam: the same, η
About the same size in both
100 times bigger in both
Show the answer
Plain: 100 times bigger. Adam: the same, η
Plain descent's step is η × g, so it grows with the slope. Adam's is η × m̂ ÷ √v̂, and that ratio is about ±1 whatever the slope's scale.
In PyTorch. Both optimizers are built in. Each is made from the weights it moves and its hyperparameters, the settings you choose: lr is η, momentum is β, betas is (β₁, β₂). Each training step then runs lesson 79's three calls: zero_grad() clears the old slopes, backward() finds the new ones, and step() makes the move. Three steps of each on the bowl, from x = 0:
import torch
x = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([x], lr=0.1, momentum=0.9)
for t inrange(3):
opt.zero_grad()
loss = (x - 2) ** 2 + 1
loss.backward()
opt.step()
print("momentum", round(x.item(), 4))
x = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([x], lr=0.5, betas=(0.9, 0.999))
for t inrange(3):
opt.zero_grad()
loss = (x - 2) ** 2 + 1
loss.backward()
opt.step()
print("Adam", round(x.item(), 4))
The same numbers as your loops: 0.4, 1.08 and 1.876 for momentum, and 0.5, 0.9913 and 1.463 for Adam. Read the two formulas side by side once more. PyTorch's momentum keeps v = 0.9v + g, with no (1 − β) on the slope: its v is a running total, which settles at 1 ÷ (1 − 0.9) = 10 times a steady slope. Adam's two averages do use (1 − β), as true moving averages do, and its bias correction then divides by 1 − βᵗ. Some libraries write momentum with (1 − β) too; their η must then be 10 times bigger to take the same steps.
Quick check
PyTorch's momentum meets the same slope three times. What does this print?
v = 0.0for g in [2.0, 2.0, 2.0]:
v = 0.9 * v + g
print(round(v, 2))
5.42
2.0
6.0
0.54
Show the answer
5.42
v goes 2, then 0.9 × 2 + 2 = 3.8, then 0.9 × 3.8 + 2 = 5.42, on its way to 2 ÷ 0.1 = 20. With (1 − β) on the slope, as Adam's m has, it would read 0.54 and head for 2.
Drill
One Adam step for many weights
Fix adam_step(w, g, m, v, t, lr) so it corrects m and v for their start at 0 before it steps, with β₁ = 0.9, β₂ = 0.999 and ε = 1e-8. It takes and returns NumPy arrays, one number per weight. As written, its first step goes too far: the math part's flow moved each weight 0.4.
import numpy as np
defadam_step(w, g, m, v, t, lr):
m = 0.9 * m + 0.1 * g
v = 0.999 * v + 0.001 * g ** 2
w = w - lr * m / (np.sqrt(v) + 1e-8)
return w, m, v
w, m, v = adam_step(np.array([-6.0, 1.0]), np.array([-1.2, 10.0]), np.zeros(2), np.zeros(2), 1, 0.4)
print(np.round(w, 4).tolist())
In the lesson, your code is checked as you write it.
In your own words
A model trains well with Adam at η = 0.001. Someone switches to plain descent with the same η, and the loss barely moves. What's going on?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Optimizers: momentum, Adam and their cousins in real training.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write