Index Ai Academy
Phase 2 · Calculus at work

Backprop through a linear layer: gradients with shapes

The slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Every linear layer in a network, the attention projections and both halves of the feed-forward layer, computes Y = XW + b, and training needs the slope of the loss with respect to W, b and X. Autograd finds them, but an engineer who can't write them by hand can't read a shape error, can't tell when a gradient is off by a factor of the batch size, and can't follow a backward pass. Three formulas cover every linear layer at any batch size. This lesson derives them from one entry, finds them again from shapes alone, and checks them against PyTorch.

What you'll be able to do

  • Derive dW[i, j] = Σₙ X[n, i] dY[n, j] from one entry, and name the three rules dX = dY Wᵀ, dW = Xᵀ dY, db = column sums
  • Find the right transpose from shapes alone, and say why a gradient has the shape of its variable
  • Check hand gradients against autograd, and read nn.Linear's weight.grad = dYᵀ X for its (out, in) layout
  • Diagnose a gradient that is off by the batch size: the missing division of a mean loss
Quick check
Have a guess before you start. A layer computes Y = X W for X of shape (5, 3) and W of shape (3, 7). The gradient dW must have the same shape as W. Which product gives that shape?
  1. dY times W, shape (5, 7)(3, 7)
  2. Xᵀ times dY, shape (3, 5)(5, 7) = (3, 7)
  3. X times dY, which has no valid shape
  4. dY times Xᵀ, shape (5, 7)(3, 5)
Show the answer

Xᵀ times dY, shape (3, 5)(5, 7) = (3, 7)

Only Xᵀ dY multiplies and lands on (3, 7). Shapes alone often name the answer; the derivation below shows they're right.

The math: one entry, then three rules

Purpose. Find how the loss changes with each number of W, b and X, and write it as three matrix products.

Training a small network (Foundation lesson 111) used dW = Xᵀ dZ and fixed the transpose by shape. This lesson proves it. A linear layer takes a batch X with one row per example, multiplies by weights W, and adds a bias b to every row. Take X = [[1, 2], [0, 1], [3, −1]] (3 examples, 2 inputs), W = [[1, 0], [0.5, 1]] (2 inputs, 2 outputs) and b = (0, 0). Backprop hands the layer dY, the slope of the loss at each number of Y; take dY = [[1, 0], [0, 1], [1, 1]].

Quick check
From the chain rule: a number x is used in two places, once in a·x and once in b·x. How is the slope of the loss with respect to x built?
  1. Multiply them
  2. Add the slope along each use
  3. Take the larger
  4. Use only the first
Show the answer

Add the slope along each use

A value used twice collects the slope from both uses and adds them. A weight shared by every example in a batch collects one term per example.

One entry. Write one output number out: Y[n, j] = Σᵢ X[n, i] W[i, j] + b[j]. The weight W[i, j] appears in Y[n, j] for every example n, multiplied by X[n, i]. By the chain rule the slope of the loss with respect to W[i, j] adds one term per example:

dWij=∑nXni dYnj,dXni=∑jdYnj Wij,dbj=∑ndYnjdW_{ij} = \sum_{n} X_{ni}\, dY_{nj}, \qquad dX_{ni} = \sum_{j} dY_{nj}\, W_{ij}, \qquad db_{j} = \sum_{n} dY_{nj}

In plain words: each weight's slope is a sum over the examples of (the input it multiplied) times (the slope arriving at the output it fed). The bias has coefficient 1, so its slope is just the sum of the arriving slopes.

Read as matrix products, the three sums are dW = Xᵀ dY, dX = dY Wᵀ and db = the column sums of dY. The sum over n in dW is the row of Xᵀ meeting the column of dY, so the next picture is an ordinary matrix product.

Try it now

Then change the last row of dY to 0, 0: the third example stops contributing, and dW changes in every entry.

The other two follow the same way. dX = dY Wᵀ: row 3 of dY, (1, 1), times Wᵀ = [[1, 0.5], [0, 1]] gives (1, 1.5), so dX[2] = (1, 1.5). db = column sums of dY = (1 + 0 + 1, 0 + 1 + 1) = (2, 2). The bias is added to every row of Y, which stretched it down the rows (broadcasting): the slope of anything that was stretched is the sum over the stretched axis. Finally, a gradient always has the shape of its variable, which is how you recover the transposes: dW is (2, 2) like W, dX is (3, 2) like X, db is (2,) like b.

In your own words
In your own words: why does a weight's slope add one term per example?

Write a sentence or two, then compare it with a good answer in the lesson.

In NumPy and PyTorch: hand rules against autograd

Purpose. Write the three rules as code, then let PyTorch's autograd check them.

The code is three lines. The check is the part to keep: run the same layer through autograd and compare every entry, to four places. Shapes can tell you which product could be right; only autograd tells you it is.

Worked example

Worked example: the three rules, then autograd

A 3 × 2 input, a 2 × 2 weight and a chosen dY. Compute dX, dW and db by hand, then get them from backward().

import numpy as np
import torch

X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]])
W = np.array([[1.0, 0.0], [0.5, 1.0]])
dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
dX, dW, db = dY @ W.T, X.T @ dY, dY.sum(axis=0)
print(dX.tolist(), dW.tolist(), db.tolist())

Xt = torch.tensor(X, requires_grad=True)
Wt = torch.tensor(W, requires_grad=True)
bt = torch.zeros(2, requires_grad=True)
Y = Xt @ Wt + bt
(Y * torch.tensor(dY)).sum().backward()
pairs = [(Xt, dX), (Wt, dW), (bt, db)]
same = [np.allclose(g.grad.numpy(), h, atol=1e-4) for g, h in pairs]
print(same)
It runs right here in your browser, with nothing to install.

The first line prints dX = [[1, 0.5], [0, 1], [1, 1.5]], dW = [[4, 3], [1, 0]] and db = [2, 2]: the numbers of the by-hand part. The loss (Y * dY).sum() has slope exactly dY at Y, so backward() should return the same three gradients, and the list prints [True, True, True]. Each agreement is to four places, the habit for checking a hand formula against a library.

Quick check
What does this print?
import numpy as np

X = np.zeros((5, 3))
W = np.zeros((3, 7))
dY = np.ones((5, 7))
print((dY @ W.T).shape, (X.T @ dY).shape, dY.sum(axis=0).shape)
  1. (5, 3) (3, 7) (7,)
  2. (3, 5) (7, 3) (5,)
  3. (5, 3) (7, 3) (5,)
  4. (5, 7) (3, 7) (7,)
Show the answer

(5, 3) (3, 7) (7,)

dX matches X (5, 3), dW matches W (3, 7), db matches b (7,).

PyTorch stores the weight the other way. This course writes W as (in, out) and a layer as X @ W. nn.Linear(2, 2) stores its weight as (out, in) and computes X @ weight.T. Everything transposes: weight.grad = dYᵀ X, the transpose of our dW. The cell copies our W in (transposed), runs the layer, and compares. The page runs a teaching subset of PyTorch (the same lines run on a real install).

import torch
import torch.nn as nn

X = torch.tensor([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]])
dY = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
layer = nn.Linear(2, 2)
with torch.no_grad():
    layer.weight.copy_(torch.tensor([[1.0, 0.5], [0.0, 1.0]]))   # (out, in)
    layer.bias.zero_()
(layer(X) * dY).sum().backward()
print(layer.weight.grad.tolist(), torch.allclose(layer.weight.grad, dY.T @ X))
print(layer.bias.grad.tolist())

The weight's gradient is [[4, 1], [3, 0]]: our dW = [[4, 3], [1, 0]] transposed, and dY.T @ X matches it. The bias gradient is [2, 2].

Drill

The three gradients

Write linear_backward(X, W, dY) returning (dX, dW, db). The starter has the transposes in the wrong places.

import numpy as np


def linear_backward(X, W, dY):
    dX = dY @ W
    dW = X @ dY
    db = dY.sum()
    return dX, dW, db


X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]])
W = np.array([[1.0, 0.0], [0.5, 1.0]])
dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
print(linear_backward(X, W, dY)[1].tolist())
In the lesson, your code is checked as you write it.
In your own words
In your own words: how can shapes alone tell you where the transposes go, and what can't they tell you?

Write a sentence or two, then compare it with a good answer in the lesson.

In training code: checking a gradient, and the missing division

Purpose. See where this sits in a training step and the bug that matters most: a gradient off by the batch size.

Why this matters at scale. In a transformer the same three lines run inside every projection of every layer, with X holding thousands of token rows. The sum over n in dW is where the batch's information is combined into one update; a mistake of a factor of the batch size there makes a model train at the wrong speed without ever raising an error.

A check that costs one line. torch.autograd.gradcheck compares autograd's answer with a numerical slope (nudge each input, see how the loss moves) and returns True when they agree. It needs 64-bit numbers, and the page's PyTorch is a subset without it, so the card below is run on a real install. Writing a custom backward function without a gradcheck is how silent errors ship.

Quick check
A loss is the mean of the squared errors over 6 numbers (3 examples times 2 outputs). A teammate's hand-written dW comes out exactly 6 times what autograd gives. What did they forget?
  1. To transpose X
  2. To add the bias gradient to the weight gradient
  3. To use 64-bit numbers
  4. To divide dY by the 6 values the mean was taken over
Show the answer

To divide dY by the 6 values the mean was taken over

d(mean)/dY carries a factor 1/6 on every entry. Starting from 2 × (Y − target), without the division, makes every gradient 6 times too big: [[11, −8], [2, 12]] instead of [[1.83, −1.33], [0.33, 2]].

Quick check
What does this print?
import numpy as np

X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]])
W = np.array([[1.0, 0.0], [0.5, 1.0]])
t = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
dY = 2 * (X @ W - t) / t.size
print((X.T @ dY).round(2).tolist())
  1. [[11.0, -8.0], [2.0, 12.0]]
  2. [[4.0, 3.0], [1.0, 0.0]]
  3. [[1.83, 0.33], [-1.33, 2.0]]
  4. [[1.83, -1.33], [0.33, 2.0]]
Show the answer

[[1.83, -1.33], [0.33, 2.0]]

dY = 2 (Y − t) / 6 is the slope of the mean squared error at Y; Xᵀ dY then gives dW. It matches autograd on the same loss.

Quick check
What does this print?
import numpy as np

dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
W = np.array([[1.0, 0.0], [0.5, 1.0]])
print((dY @ W.T)[2].tolist())
  1. [2.0, 1.5]
  2. [1.0, 1.0]
  3. [1.5, 1.0]
  4. [1.0, 1.5]
Show the answer

[1.0, 1.5]

Row 2 of dY, (1, 1), times Wᵀ = [[1, 0.5], [0, 1]] gives 1 × 1 + 1 × 0 = 1 and 1 × 0.5 + 1 × 1 = 1.5.

Drill

The gradient of a mean squared error

Write mse_dW(X, W, t): dW for the loss mean((XW − t)²). Remember the mean divides by the number of values in t. The starter forgets the division.

import numpy as np


def mse_dW(X, W, t):
    dY = 2 * (X @ W - t)
    return X.T @ dY


X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]])
W = np.array([[1.0, 0.0], [0.5, 1.0]])
t = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
print(mse_dW(X, W, t).round(2).tolist())
In the lesson, your code is checked as you write it.
In your own words
In your own words: how do you check a hand-written gradient, and why does the batch size matter?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 2 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Y = XW + b with X (8, 4) and W (4, 6). What is the shape of dW?

  1. (4, 6)
  2. (8, 6)
  3. (6, 4)
  4. (8, 4)

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 2 · about 23 hours of study

Math for ML engineers

The Foundation's formulas at working depth: batches, shapes, autograd, stable arithmetic and error bars 30 lessons 1 free
  1. Reading a paper's equations, with shapes FreeΣ, subscripts, hats and bars, translated into Python you can run and test.35 min
  2. Floating point and log spaceGrowth, decay, and why logs turn products into sums.35 min
  3. Shapes at transformer scaleArrays with more axes: batches, sequences, heads.36 min
  4. The linear layer as algebraMany dot products at once: what a neural network layer computes.40 min
  5. Solving and least squaresThe matrix operations you'll see in every formula.36 min
  6. Projections, orthogonality and rank-1 updatesA matrix stretches, rotates and projects space.34 min
  7. Eigenvalues of symmetric matricesThe directions a transformation leaves pointing the same way.36 min
  8. SVD at workCompress data to its most important directions.40 min
  9. Strange geometry in high dimensionsNear-perpendicular random directions, bunched-up distances, and the curse of dimensionality.38 min
  10. Derivatives of the functions models usePower, sum and product rules, so you rarely compute from scratch.35 min
  11. The chain rule with vectorsDerivatives of functions inside functions: the heart of backprop.40 min
  12. Backprop through a linear layer: gradients with shapesThe slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.40 min
  13. Gradients in matrix form, and how to check themThe vector of all partial derivatives: which way is uphill.35 min
  14. Why training is fast or slowSecond derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method.40 min
  15. The chain rule of probabilityUpdate beliefs when new evidence arrives.40 min
  16. Distributions as objectsRandom variables and the shapes their outcomes take.35 min
  17. Expectation and variance of estimatorsExpected value, variance and standard deviation.35 min
  18. Gaussians at workThe bell curve, and why it shows up everywhere.35 min
  19. Covariance matrices and the multivariate normalCovariance and correlation, curves, outliers and common causes, rank correlation, skew and kurtosis.40 min
  20. Losses are negative log-likelihoodsChoose the parameters that make the data most probable.35 min
  21. Samples and confidence: error bars for an evalEstimate from a sample and say how sure you are.40 min
  22. Priors, MAP and regularizationPriors as pseudo-counts, Beta posteriors, shrinkage for small samples, and which of two rates is really higher.40 min
  23. Markov chainsStates, a table of what comes next, the chance of a path, and where a chain settles.38 min
  24. Softmax at working depthTurn any list of scores into probabilities.35 min
  25. Cross-entropy from logitsThe loss function language models are trained on.40 min
  26. KL divergence at workForward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.36 min
  27. A/B tests and significance, for prompts and modelsTell a real lift from noise.40 min
  28. Comparing two models on the same testPaired results, disagreements, a quick z and a bootstrap for the gap between two models.35 min
  29. Ranking from head-to-head resultsElo updates, Bradley-Terry strengths, and how model leaderboards turn votes into ratings.37 min
  30. Capstone: one problem through five tracksA three-class softmax classifier written from its formula: notation to NumPy, matrix products, cross-entropy, the gradient, and an honest report.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write