Backprop through a linear layer: gradients with shapes
The slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.
What it covers6
Why it matters
Every linear layer in a network, the attention projections and both halves of the feed-forward layer, computes Y = XW + b, and training needs the slope of the loss with respect to W, b and X. Autograd finds them, but an engineer who can't write them by hand can't read a shape error, can't tell when a gradient is off by a factor of the batch size, and can't follow a backward pass. Three formulas cover every linear layer at any batch size. This lesson derives them from one entry, finds them again from shapes alone, and checks them against PyTorch.
What you'll be able to do
- Derive dW[i, j] = Σₙ X[n, i] dY[n, j] from one entry, and name the three rules dX = dY Wᵀ, dW = Xᵀ dY, db = column sums
- Find the right transpose from shapes alone, and say why a gradient has the shape of its variable
- Check hand gradients against autograd, and read nn.Linear's weight.grad = dYᵀ X for its (out, in) layout
- Diagnose a gradient that is off by the batch size: the missing division of a mean loss
Show the answer
Xᵀ times dY, shape (3, 5)(5, 7) = (3, 7)
Only Xᵀ dY multiplies and lands on (3, 7). Shapes alone often name the answer; the derivation below shows they're right.
The math: one entry, then three rules
Purpose. Find how the loss changes with each number of W, b and X, and write it as three matrix products.
Training a small network (Foundation lesson 111) used dW = Xᵀ dZ and fixed the transpose by shape. This lesson proves it. A linear layer takes a batch X with one row per example, multiplies by weights W, and adds a bias b to every row. Take X = [[1, 2], [0, 1], [3, −1]] (3 examples, 2 inputs), W = [[1, 0], [0.5, 1]] (2 inputs, 2 outputs) and b = (0, 0). Backprop hands the layer dY, the slope of the loss at each number of Y; take dY = [[1, 0], [0, 1], [1, 1]].
Show the answer
Add the slope along each use
A value used twice collects the slope from both uses and adds them. A weight shared by every example in a batch collects one term per example.
One entry. Write one output number out: Y[n, j] = Σᵢ X[n, i] W[i, j] + b[j]. The weight W[i, j] appears in Y[n, j] for every example n, multiplied by X[n, i]. By the chain rule the slope of the loss with respect to W[i, j] adds one term per example:
In plain words: each weight's slope is a sum over the examples of (the input it multiplied) times (the slope arriving at the output it fed). The bias has coefficient 1, so its slope is just the sum of the arriving slopes.
Read as matrix products, the three sums are dW = Xᵀ dY, dX = dY Wᵀ and db = the column sums of dY. The sum over n in dW is the row of Xᵀ meeting the column of dY, so the next picture is an ordinary matrix product.
Then change the last row of dY to 0, 0: the third example stops contributing, and dW changes in every entry.
The other two follow the same way. dX = dY Wᵀ: row 3 of dY, (1, 1), times Wᵀ = [[1, 0.5], [0, 1]] gives (1, 1.5), so dX[2] = (1, 1.5). db = column sums of dY = (1 + 0 + 1, 0 + 1 + 1) = (2, 2). The bias is added to every row of Y, which stretched it down the rows (broadcasting): the slope of anything that was stretched is the sum over the stretched axis. Finally, a gradient always has the shape of its variable, which is how you recover the transposes: dW is (2, 2) like W, dX is (3, 2) like X, db is (2,) like b.
Write a sentence or two, then compare it with a good answer in the lesson.
In NumPy and PyTorch: hand rules against autograd
Purpose. Write the three rules as code, then let PyTorch's autograd check them.
The code is three lines. The check is the part to keep: run the same layer through autograd and compare every entry, to four places. Shapes can tell you which product could be right; only autograd tells you it is.
Worked example: the three rules, then autograd
A 3 × 2 input, a 2 × 2 weight and a chosen dY. Compute dX, dW and db by hand, then get them from backward().
import numpy as np import torch X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]]) W = np.array([[1.0, 0.0], [0.5, 1.0]]) dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) dX, dW, db = dY @ W.T, X.T @ dY, dY.sum(axis=0) print(dX.tolist(), dW.tolist(), db.tolist()) Xt = torch.tensor(X, requires_grad=True) Wt = torch.tensor(W, requires_grad=True) bt = torch.zeros(2, requires_grad=True) Y = Xt @ Wt + bt (Y * torch.tensor(dY)).sum().backward() pairs = [(Xt, dX), (Wt, dW), (bt, db)] same = [np.allclose(g.grad.numpy(), h, atol=1e-4) for g, h in pairs] print(same)
The first line prints dX = [[1, 0.5], [0, 1], [1, 1.5]], dW = [[4, 3], [1, 0]] and db = [2, 2]: the numbers of the by-hand part. The loss (Y * dY).sum() has slope exactly dY at Y, so backward() should return the same three gradients, and the list prints [True, True, True]. Each agreement is to four places, the habit for checking a hand formula against a library.
import numpy as np X = np.zeros((5, 3)) W = np.zeros((3, 7)) dY = np.ones((5, 7)) print((dY @ W.T).shape, (X.T @ dY).shape, dY.sum(axis=0).shape)
Show the answer
(5, 3) (3, 7) (7,)
dX matches X (5, 3), dW matches W (3, 7), db matches b (7,).
PyTorch stores the weight the other way. This course writes W as (in, out) and a layer as X @ W. nn.Linear(2, 2) stores its weight as (out, in) and computes X @ weight.T. Everything transposes: weight.grad = dYᵀ X, the transpose of our dW. The cell copies our W in (transposed), runs the layer, and compares. The page runs a teaching subset of PyTorch (the same lines run on a real install).
import torch import torch.nn as nn X = torch.tensor([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]]) dY = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) layer = nn.Linear(2, 2) with torch.no_grad(): layer.weight.copy_(torch.tensor([[1.0, 0.5], [0.0, 1.0]])) # (out, in) layer.bias.zero_() (layer(X) * dY).sum().backward() print(layer.weight.grad.tolist(), torch.allclose(layer.weight.grad, dY.T @ X)) print(layer.bias.grad.tolist())
The weight's gradient is [[4, 1], [3, 0]]: our dW = [[4, 3], [1, 0]] transposed, and dY.T @ X matches it. The bias gradient is [2, 2].
The three gradients
Write linear_backward(X, W, dY) returning (dX, dW, db). The starter has the transposes in the wrong places.
import numpy as np def linear_backward(X, W, dY): dX = dY @ W dW = X @ dY db = dY.sum() return dX, dW, db X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]]) W = np.array([[1.0, 0.0], [0.5, 1.0]]) dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) print(linear_backward(X, W, dY)[1].tolist())
Write a sentence or two, then compare it with a good answer in the lesson.
In training code: checking a gradient, and the missing division
Purpose. See where this sits in a training step and the bug that matters most: a gradient off by the batch size.
Why this matters at scale. In a transformer the same three lines run inside every projection of every layer, with X holding thousands of token rows. The sum over n in dW is where the batch's information is combined into one update; a mistake of a factor of the batch size there makes a model train at the wrong speed without ever raising an error.
A check that costs one line. torch.autograd.gradcheck compares autograd's answer with a numerical slope (nudge each input, see how the loss moves) and returns True when they agree. It needs 64-bit numbers, and the page's PyTorch is a subset without it, so the card below is run on a real install. Writing a custom backward function without a gradcheck is how silent errors ship.
Show the answer
To divide dY by the 6 values the mean was taken over
d(mean)/dY carries a factor 1/6 on every entry. Starting from 2 × (Y − target), without the division, makes every gradient 6 times too big: [[11, −8], [2, 12]] instead of [[1.83, −1.33], [0.33, 2]].
import numpy as np X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]]) W = np.array([[1.0, 0.0], [0.5, 1.0]]) t = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) dY = 2 * (X @ W - t) / t.size print((X.T @ dY).round(2).tolist())
Show the answer
[[1.83, -1.33], [0.33, 2.0]]
dY = 2 (Y − t) / 6 is the slope of the mean squared error at Y; Xᵀ dY then gives dW. It matches autograd on the same loss.
import numpy as np dY = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) W = np.array([[1.0, 0.0], [0.5, 1.0]]) print((dY @ W.T)[2].tolist())
Show the answer
[1.0, 1.5]
Row 2 of dY, (1, 1), times Wᵀ = [[1, 0.5], [0, 1]] gives 1 × 1 + 1 × 0 = 1 and 1 × 0.5 + 1 × 1 = 1.5.
The gradient of a mean squared error
Write mse_dW(X, W, t): dW for the loss mean((XW − t)²). Remember the mean divides by the number of values in t. The starter forgets the division.
import numpy as np def mse_dW(X, W, t): dY = 2 * (X @ W - t) return X.T @ dY X = np.array([[1.0, 2.0], [0.0, 1.0], [3.0, -1.0]]) W = np.array([[1.0, 0.0], [0.5, 1.0]]) t = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) print(mse_dW(X, W, t).round(2).tolist())
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Backpropagation: these gradients chained through a whole network.
- Autograd: how PyTorch records the graph that applies these rules for you.
- Attention's backward pass: the same three rules inside a transformer.
Practice what you just read
- 2 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
Y = XW + b with X (8, 4) and W (4, 6). What is the shape of dW?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 2 · about 23 hours of study
Math for ML engineers
The Foundation's formulas at working depth: batches, shapes, autograd, stable arithmetic and error bars
30 lessons 1 free
- Reading a paper's equations, with shapes FreeΣ, subscripts, hats and bars, translated into Python you can run and test.35 min
- Floating point and log spaceGrowth, decay, and why logs turn products into sums.35 min
- Shapes at transformer scaleArrays with more axes: batches, sequences, heads.36 min
- The linear layer as algebraMany dot products at once: what a neural network layer computes.40 min
- Solving and least squaresThe matrix operations you'll see in every formula.36 min
- Projections, orthogonality and rank-1 updatesA matrix stretches, rotates and projects space.34 min
- Eigenvalues of symmetric matricesThe directions a transformation leaves pointing the same way.36 min
- SVD at workCompress data to its most important directions.40 min
- Strange geometry in high dimensionsNear-perpendicular random directions, bunched-up distances, and the curse of dimensionality.38 min
- Derivatives of the functions models usePower, sum and product rules, so you rarely compute from scratch.35 min
- The chain rule with vectorsDerivatives of functions inside functions: the heart of backprop.40 min
- Backprop through a linear layer: gradients with shapesThe slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.40 min
- Gradients in matrix form, and how to check themThe vector of all partial derivatives: which way is uphill.35 min
- Why training is fast or slowSecond derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method.40 min
- The chain rule of probabilityUpdate beliefs when new evidence arrives.40 min
- Distributions as objectsRandom variables and the shapes their outcomes take.35 min
- Expectation and variance of estimatorsExpected value, variance and standard deviation.35 min
- Gaussians at workThe bell curve, and why it shows up everywhere.35 min
- Covariance matrices and the multivariate normalCovariance and correlation, curves, outliers and common causes, rank correlation, skew and kurtosis.40 min
- Losses are negative log-likelihoodsChoose the parameters that make the data most probable.35 min
- Samples and confidence: error bars for an evalEstimate from a sample and say how sure you are.40 min
- Priors, MAP and regularizationPriors as pseudo-counts, Beta posteriors, shrinkage for small samples, and which of two rates is really higher.40 min
- Markov chainsStates, a table of what comes next, the chance of a path, and where a chain settles.38 min
- Softmax at working depthTurn any list of scores into probabilities.35 min
- Cross-entropy from logitsThe loss function language models are trained on.40 min
- KL divergence at workForward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.36 min
- A/B tests and significance, for prompts and modelsTell a real lift from noise.40 min
- Comparing two models on the same testPaired results, disagreements, a quick z and a bootstrap for the gap between two models.35 min
- Ranking from head-to-head resultsElo updates, Bradley-Terry strengths, and how model leaderboards turn votes into ratings.37 min
- Capstone: one problem through five tracksA three-class softmax classifier written from its formula: notation to NumPy, matrix products, cross-entropy, the gradient, and an honest report.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write