Gradients

It points uphill

Two facts make the gradient useful.

1. Direction: the gradient points the way of steepest ascent. Of all the directions you could take a small step, this one raises $f$ the most. So the opposite arrow, $-\nabla f$, is the steepest way down.
2. Length: the arrow's length says how steep that steepest slope is. Near the bottom of a bowl the ground flattens out, so the gradient shrinks toward zero.

The length of a vector is written with double bars, as in $\|\nabla f\|$. You met the recipe in the vector lessons:

In plain words: square each slope, add them up, and take the square root. The answer is how steep the hill is in its steepest direction.

Worked at $(2, 1)$: the gradient is $[2, 8]$. Then 2² + 8² = 4 + 64 = 68. The square root of 68 is about 8.25. The code repeats this at points closer and closer to the bottom:

import math

def grad_f(x, y):
    return [x, 8 * y]

for point in [(2, 1), (1, 0.5), (0.2, 0.1)]:
    g = grad_f(*point)
    length = math.sqrt(g[0] ** 2 + g[1] ** 2)
    print(point, g, "length", round(length, 2))

The length falls from 8.25 to 4.12 to 0.82. The closer you get to $(0, 0)$, the flatter the ground.

Across the contours, not at the center

On a hiking map, contour lines join points at the same height. The gradient always crosses them at a right angle. Rain on a hillside does the same: it runs straight down across the contours, never along them.

So the downhill arrow often doesn't point at the bottom. At $(2, 1)$ the bottom of the bowl lies in the direction $[-2, -1]$. But $-\nabla f$ is $[-2, -8]$. It dives mostly in $y$, where the bowl is steepest.

The gradient of a loss

Now swap the hillside for a model. Say you predict each campaign's revenue from a few numbers about it, called its features, such as spend and clicks. A linear model multiplies each feature by a weight and adds up the results. Training means finding good weights.

Four symbols carry the story. Bold letters are vectors, as in the vector lessons.

• $X$ is the table of features: one row per campaign, one column per feature.
• $\mathbf{w}$ is the list of weights, one per feature.
• $\mathbf{y}$ holds the real revenue of each campaign.
• $n$ is the number of campaigns.

The predictions are $\hat{y} = X\mathbf{w}$, read "y-hat equals X times w". Each prediction is the dot product of one row with the weights.

A standard loss is the mean squared error (MSE):

In plain words: for each campaign, take the prediction minus the real value. Square it, so a miss in either direction counts as bad. Add these up and divide by $n$ to get the average.

Here $\mathbf{x}_i$ is row $i$ of the table: one campaign. The $\sum$ sign (sigma) means "add up for every $i$ from 1 to $n$", like a for loop. The right-hand version says the same thing with the squared length of the error vector (the double bars again).

Worked with small numbers: two campaigns, two features each. The table is [[1, 2], [2, 1]], the real values are [5, 4] and the weights are [1, 1]. The predictions are 1×1 + 2×1 = 3 and 2×1 + 1×1 = 3. The errors are 3 − 5 = −2 and 3 − 4 = −1. Their squares are 4 and 1, so the loss is (4 + 1) ÷ 2 = 2.5.

One slope per weight

Each weight gets its own slope: how fast the loss changes when only that weight moves. Those slopes, in a list, are the gradient of the loss. It has one entry per weight, so it has the same shape as $\mathbf{w}$. Worked out with the chain rule (the panel below shows how), it is:

The small $\mathbf{w}$ under the nabla says the slopes are taken with respect to the weights. $X^\top$ is "X transposed": the table flipped so its columns become rows.

In plain words: read it from the right. $X\mathbf{w} - \mathbf{y}$ is the list of errors, one per campaign. Multiplying by $X^\top$ gives, for each weight, "error times that weight's feature", added up over all campaigns. The $\frac{2}{n}$ scales the result: the 2 comes from the square, and dividing by $n$ takes the average.

A weight whose feature lines up with big errors gets a big gradient.

Back to the small example. The errors are [-2, -1]. Weight 1's feature column is [1, 2], so 1×(−2) + 2×(−1) = −4. Weight 2's column is [2, 1], so 2×(−2) + 1×(−1) = −5. Here $n = 2$, so $\frac{2}{n} = 1$, and the gradient is [−4, −5].

Both entries are negative, so raising either weight lowers the loss. That fits: both predictions were too low.

X = [[1, 2], [2, 1]]   # two campaigns, two features each
y = [5, 4]             # what really happened
w = [1, 1]             # the current weights
n = len(y)

preds = [row[0] * w[0] + row[1] * w[1] for row in X]
errors = [p - t for p, t in zip(preds, y)]
loss = sum(e ** 2 for e in errors) / n
grad = [2 / n * sum(errors[i] * X[i][j] for i in range(n)) for j in range(2)]
print("predictions:", preds, " errors:", errors)
print("loss:", loss, " gradient:", grad)

new_w = [w[j] - 0.1 * grad[j] for j in range(2)]
new_preds = [row[0] * new_w[0] + row[1] * new_w[1] for row in X]
new_loss = sum((p - t) ** 2 for p, t in zip(new_preds, y)) / n
print("new weights:", [round(v, 2) for v in new_w], " new loss:", round(new_loss, 3))

The same thing with NumPy

Real datasets have thousands of rows, and loops get slow. In NumPy, X @ w makes every prediction at once. Then X.T @ (X @ w - y) does every "error times feature" sum. One matrix multiplication computes the whole gradient.

The cell makes 50 made-up campaigns with 3 features each. Their revenue comes from the true weights [2, -1, 0.5], plus a little random noise. The model's weights start at zero.

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(50, 3))
true_w = np.array([2.0, -1.0, 0.5])
y = X @ true_w + rng.normal(0, 0.1, 50)

def loss(w):
    return np.mean((X @ w - y) ** 2)

def gradient(w):
    return 2 / len(y) * X.T @ (X @ w - y)

w = np.zeros(3)
print("loss at w = 0:", round(loss(w), 3))
print("gradient:", gradient(w).round(3))
print("loss after a small step against it:", round(loss(w - 0.1 * gradient(w)), 3))

One step against the gradient already lowers the loss. Keep stepping and you've invented gradient descent, the topic of the next lesson.

Always check gradients numerically

Gradients worked out by hand are easy to get subtly wrong. A 2 goes missing, or a table that should be transposed isn't. The standard test is to compare with central differences on a few weights: nudge one weight up and down and see how the loss changes. If the two methods agree to several decimal places, the formula is right. Here's the check for the small example:

X = [[1, 2], [2, 1]]
y = [5, 4]

def loss(w):
    preds = [row[0] * w[0] + row[1] * w[1] for row in X]
    return sum((p - t) ** 2 for p, t in zip(preds, y)) / len(y)

h = 1e-6
for j in range(2):
    up = [1.0, 1.0]
    down = [1.0, 1.0]
    up[j] += h
    down[j] -= h
    print("weight", j + 1, "numeric slope:", round((loss(up) - loss(down)) / (2 * h), 4))

The numeric slopes are −4.0 and −5.0, the same as the formula. Nudging costs two runs of the loss for every weight. That's fine for checking a small model, but far too slow for training a big one. In the exercise you'll build this check for any function.