Index Ai Academy
Foundation · Foundation mathCalculus

Faster descent: momentum and Adam

About 40 minLooking ahead: some prerequisites aren't done yet

Why this matters

Almost no real model is trained with plain gradient descent. The optimizer in nearly every training script is Adam, and its update is three short formulas you can already read: two moving averages from lesson 27 and a step from lesson 79. This lesson runs momentum and Adam by hand, writes both as loops, sends all three optimizers down the same narrow valley, and then gets the same numbers out of PyTorch.

  • Run momentum by hand for three steps, and say why its steps grow on a steady slope and cancel on a zigzag
  • Run Adam by hand for three steps, with its bias correction, and say what β₁, β₂ and ε each do
  • Write plain descent, momentum and Adam as loops in Python, and compare their loss curves on one bowl
  • Compare the three down a narrow valley, and get the hand numbers again from PyTorch's optim.SGD and optim.Adam
Quick check
Have a guess before you start. Gradient descent always steps straight downhill. In a long, narrow valley with steep sides, what does that do?

The math: averages of slopes

Purpose. Step down faster and steadier than plain descent by averaging the slopes as you go.

Plain gradient descent, from lesson 79, steps against today's slope and forgets every slope before it. In a round bowl that works well. In a long, narrow valley it doesn't. Take the valley

L(w1,w2)=0.1 w12+5 w22L(w_1, w_2) = 0.1\,w_1^2 + 5\,w_2^2

It is gentle along w₁ and steep across w₂. Its slopes are g = (0.2w₁, 10w₂), so at (−6, 1) the slope across is 10 and the slope along is only −1.2: straight downhill points almost straight at the far wall. The steep wall also caps the learning rate. Its curvature is 10, so past η = 2 ÷ 10 = 0.2 the steps across overshoot by more each time (lesson 76), and under that cap the steps along the floor are tiny.

Momentum keeps a velocity v, a running total of the slopes, and steps along it:

v←β v+g,w←w−η vv \leftarrow \beta\, v + g, \qquad w \leftarrow w - \eta\, v

In plain words: keep β of the old velocity, add today's slope, and step along the velocity. Slopes that keep pointing the same way pile up; slopes that flip back and forth cancel.