Phase 2 · Math, when you need it · Calculus

Optimization by walking downhill

Repeatedly step against the gradient to find a minimum.

About 30 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

Every neural network you've heard of learned the same way. It made a guess, measured how wrong the guess was, worked out which way to nudge its numbers, and took a small step. Then it did it again. A large language model repeats that loop hundreds of thousands of times over trillions of tokens. Here you'll run the loop yourself on a small problem. You'll see how the step size, the starting point and the shape of the problem decide whether training works.

What you'll be able to do

Write the gradient descent update and explain each symbol
Choose a learning rate by recognizing slow, good and diverging runs
Fit a linear model to campaign data by gradient descent, from scratch
Explain local minima, stochastic (mini-batch) gradients, and why optimizers like Adam exist

Walking down a hill in fog

Picture yourself on a hillside in thick fog. You want to reach the bottom of the valley, but you can't see more than a step ahead. Still, you can feel the ground under your feet. If it slopes down to the left, you step left. Then you feel again and take another step. Sooner or later the ground goes flat. You've reached a low point.

Training a model works the same way. The "height" is the loss: a score for how wrong the model's guesses are. Lower is better. The model's position on the hill is set by its weights, the adjustable numbers inside it. In the last lesson you met the gradient: it points in the uphill direction. So to go down, you step the opposite way.

This method is called gradient descent. "Descent" means going down.

The update rule

Here is the whole method as one line. First, the symbols, one at a time:

$\mathbf{w}$ is the list of weights. It's bold because it's a vector.
$L(\mathbf{w})$ is the loss for those weights.
$\nabla L(\mathbf{w})$ , said "grad L", is the gradient: the uphill direction for the loss.
$\eta$ , said "eta", is the learning rate: how big a step to take.

Start with any weights. Then repeat this step:

\mathbf{w} \leftarrow \mathbf{w} - \eta\, \nabla L(\mathbf{w})

In plain words: your new weights are your old weights minus a small step uphill. The arrow means "replace with". The minus sign turns the uphill direction into a downhill one. The learning rate decides how far you go.

That's all of gradient descent. Each step uses only the slope right where you stand, like feeling the ground in the fog.

Worked example

Worked example: three steps by hand

Take a simple loss with one weight: $L(w) = (w - 2)^2 + 1$ . Its lowest point is at $w = 2$ , where the loss is 1. Its slope (gradient) is $2(w - 2)$ .

Start at $w = -1.5$ with a learning rate of 0.1.

Slope here: 2 × (−1.5 − 2) = 2 × (−3.5) = −7.
The slope is negative, so downhill is to the right.
Step: −1.5 − 0.1 × (−7) = −1.5 + 0.7 = −0.8.

The weight moved from −1.5 to −0.8, closer to 2. The code repeats this three times.

def gradient(w):
    return 2 * (w - 2)        # slope of the loss (w - 2)² + 1

w = -1.5                      # starting guess
learning_rate = 0.1
for step in range(1, 4):
    w = w - learning_rate * gradient(w)
    print("step", step, "w =", round(w, 3))

It runs right here in your browser, with nothing to install.

The weight goes −0.8, then −0.24, then 0.208. Each step is a little smaller than the last. That's because the slope gets gentler as you near the bottom. Nobody told the loop where the bottom was. It found its way by feeling the slope. The line inside the loop is the update rule, word for word.

Quick check

Same loss, slope 2 × (w − 2). You start at w = 0 and use a learning rate of 0.5. Where does one step take you?

w = 0.0
w = w - 0.5 * 2 * (w - 2)
print(w)

1.0
-2.0
0.5
2.0

Show the answer

2.0

The slope at 0 is 2 × (0 − 2) = −4. The step is 0 − 0.5 × (−4) = 2, and Python prints that as 2.0. On this bowl, a rate of 0.5 lands on the bottom in one step.

Try it now

This is the bowl from the worked example. Press Step a few times at 0.1. Then try 0.9: it bounces across the valley but still settles. Then try 1.05: each step overshoots further, and it flies off (diverges).

Choosing the learning rate

The learning rate is the one number you have to pick yourself. Think of it as stride length on that foggy hill. Tiny steps are safe but slow. Giant leaps can jump clean over the valley and land higher up the other side.

Learning rate	What you see	Loss curve
too small	tiny steps, very slow progress	falls, but slowly
about right	steady progress, settles at the bottom	falls quickly, then flattens
too big	overshoots, bounces, may blow up	jumps around or rises to infinity

A loss curve is a chart of the loss after each step. It's the main way you watch training.

In practice, people try a few rates that differ by a factor of ten each time (0.1, 0.01, 0.001). They watch the loss curve and keep the one that falls fastest without bouncing. Many also lower the rate as training goes on. That plan for changing the rate is a learning-rate schedule.

Go deeper Why 0.9 settles but 1.05 blows up

On the bowl $L(w) = (w - 2)^2 + 1$ , one step does this to your distance from the bottom:

(w_{\text{new}} - 2) = (1 - 2\eta)(w - 2)

Read it as: each step multiplies the distance from the bottom by $1 - 2\eta$ .

Rate 0.1: the distance is multiplied by 0.8. It shrinks a bit each step.
Rate 0.9: it's multiplied by −0.8. The sign flips, so you land on the other side, but the distance still shrinks. You bounce your way in.
Rate 1.05: it's multiplied by −1.1. You flip sides and get further away every step.

Any multiplier between −1 and 1 shrinks the distance. On this bowl, that means any rate between 0 and 1 settles. At exactly 1 you bounce between the same two points forever.

for rate in [0.1, 0.9, 1.05]:
    print(rate, "->", round(1 - 2 * rate, 2))

Quick check

Your training loss shoots up to infinity within a few steps. What's the most likely fix?

Train for more steps
Raise the learning rate
Lower the learning rate
Remove the bias

Show the answer

Lower the learning rate

A loss that explodes is the classic sign of steps that overshoot. Smaller steps stop the overshooting.

Local minima

A bowl has one bottom, so walking downhill always finds it. Real loss surfaces look more like hill country, with several dips. Walking downhill finds a low point: the bottom of whichever valley you started in. That's called a local minimum. The lowest point of all is the global minimum.

For huge neural networks, this matters less than you'd fear. They have millions of weights, so the "hill" has millions of directions. Most dips have a way out in some direction. And many low points turn out to be about equally good.

Two other shapes cause more trouble. A plateau is a wide flat area. A saddle point is like a horse's saddle: it goes down one way and up another. In both, the slope is tiny, so the steps are tiny too. These slow training down more often than bad local minima do.

Quick check

On the two-valley curve, gradient descent from 2.8 stops in the right-hand valley. What does that tell you?

The learning rate was too big
Gradient descent finds the bottom of the valley it starts in, not always the lowest point overall
The gradient was computed wrongly
The right-hand valley must be the deepest

Show the answer

Gradient descent finds the bottom of the valley it starts in, not always the lowest point overall

Each step only feels the local slope. It can't see that a deeper valley exists on the left.

Fitting a model by gradient descent

Now the real thing. The code below predicts revenue from campaign spend with a straight-line model. It has two weights:

a bias (a baseline amount, also called the intercept), and
a weight for spend (how much revenue each unit of spend adds).

Two set-up steps come first. Spend is standardized, as in the matrices lesson: rescaled so its average is 0 and its spread is 1. That keeps one learning rate sensible for both weights. A column of ones gives the model its bias.

Then the loop does four things each step:

Make predictions and find the errors (prediction minus true value).
Record the loss: the mean squared error from the gradients lesson.
Work out the gradient from the errors.
Take one step downhill.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(3)
spend = rng.uniform(500, 8000, 60)
revenue = 2.4 * spend + 900 + rng.normal(0, 1200, 60)

x = (spend - spend.mean()) / spend.std()
X = np.column_stack([np.ones_like(x), x])          # [bias, standardized spend]
y = revenue

w = np.zeros(2)
learning_rate = 0.1
history = []
for step in range(60):
    errors = X @ w - y
    history.append(np.mean(errors ** 2))
    gradient = 2 / len(y) * X.T @ errors
    w = w - learning_rate * gradient

print("weights:", w.round(1))
print("closed-form answer:", np.linalg.lstsq(X, y, rcond=None)[0].round(1))
plt.plot(history)
plt.yscale("log")
plt.xlabel("step")
plt.ylabel("mean squared error")

Sixty steps land on the same weights as the exact answer. That's the "closed-form answer": the one NumPy can calculate directly with a formula (np.linalg.lstsq, for "least squares").

The chart is the loss curve. Its vertical axis uses a log scale: each labeled mark is ten times the one below it. The loss drops fast at first, then goes flat. It can't reach zero, because the data has random noise that no straight line can explain.

For a straight-line model you'd use the exact formula. Neural networks have no such formula, so this loop is the only way to train them.

Go deeper The gradient line, as a formula

The line gradient = 2 / len(y) * X.T @ errors is the gradient of the mean squared error from the gradients lesson:

\nabla L(\mathbf{w}) = \frac{2}{n} X^\top (X\mathbf{w} - \mathbf{y})

Read it as: take the errors $X\mathbf{w} - \mathbf{y}$ . For each weight, add up "error times that weight's feature" over all $n$ examples. That's what $X^\top$ does. Then multiply by $2/n$ . A weight whose feature lines up with big errors gets a big gradient, so it moves the most.

Two upgrades used everywhere

Mini-batches (stochastic gradient descent). Real datasets can hold millions of examples. Working out the exact gradient over all of them for every single step is slow. Instead, you estimate it from a random handful, called a batch, of say 256 examples. Think of tasting a spoonful of soup instead of the whole pot. Each step is a bit noisier, but far cheaper. The noise can even help the walk jiggle its way off flat regions. "Stochastic" means "involving randomness".

Adaptive optimizers. Methods like Adam add two tricks to the plain update. First, momentum: they keep a running average of recent gradients, so the walk rolls through small bumps like a ball. Second, they scale each weight's step by how large its gradients have been. Adam, and its variant AdamW, is the default for training transformers.

Here is the same loop as it looks in PyTorch, a popular deep-learning library. You'll use it in Phase 4. You don't need to follow every line yet. Look for the two familiar parts: the gradient and the update.

# The same loop in PyTorch (static example; PyTorch isn't available in the browser):
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
for batch in loader:
    loss = loss_fn(model(batch.x), batch.y)
    optimizer.zero_grad()
    loss.backward()        # backpropagation: the gradient
    optimizer.step()       # the update: w ← w − η · (adjusted gradient)

Quick check

You train with mini-batches. The loss goes up a little on some steps, but falls over hundreds of steps. Is something broken?

No: each batch gives a noisy estimate of the gradient, so the trend matters, not each step
Yes: the batches are too big
Yes: the loss must fall on every step
No: the loss always goes up during training

Show the answer

No: each batch gives a noisy estimate of the gradient, so the trend matters, not each step

A small random batch only estimates the true gradient. Some steps point a little wrong, but on average they lead downhill.

Practice what you just read

4 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
4 review cards for spaced repetition

Create a free account

Try a quiz question

Why standardize features before gradient descent?

So one learning rate works for every weight; otherwise large-scale features force tiny steps
Gradient descent only accepts numbers between 0 and 1, so larger inputs stop it with an error
To make the model nonlinear, since a straight line can't fit features that are on different scales
To avoid computing the gradient, since on standardized data the slope is 1 for every single weight

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Gradients