A layer from lesson 56 gives every input its own weight for every output. For a picture, that's a lot of weights, and it ignores something true of pictures: an edge looks the same wherever it is. Image and audio models do something smarter. They slide one small pattern across the input and take a dot product at each stop. That's a convolution, and it's the dot product from lesson 50, repeated.
What you'll be able to do
Slide a 3-number pattern along a list and compute every output as a dot product
Predict how many outputs there are, (n + 2p − k) ÷ s + 1 rounded down, with stride s and padding p
Do the same in two dimensions: a 3 × 3 pattern over a 5 × 5 trail sign, with two loops
Say what a pattern detects from its numbers, and count the weights it saves by sharing them
Quick check
Have a guess before you start. A window 3 numbers wide slides along a list of 7 numbers, one place at a time, and never hangs off the end. How many places can it sit?
4
3
7
5
Show the answer
5
It can start at the 1st, 2nd, 3rd, 4th or 5th number. Starting at the 6th, it would need an 8th number that isn't there. 7 − 3 + 1 = 5.
The math: one pattern, many stops
Purpose. Slide a small pattern along a list, take a dot product at every stop, and count the stops.
Here's a row of 6 pixels from a photo, dark on the left and bright on the right: x = (2, 2, 2, 7, 7, 7). And here's a pattern, or kernel, of 3 numbers: w = (−1, 0, 1). Lay w over the first 3 pixels, the window, and take the dot product: 2 × (−1) + 2 × 0 + 2 × 1 = 0. Slide one place right and do it again. Each stop gives one output number. The list of outputs, y, is the convolution of x with w.
yj=w⋅(xj,xj+1,xj+2)
In plain words: output j is the pattern dotted with the window that starts at place j.
Quick check
From an earlier lesson: what is (1, 0, −1) · (4, 6, 9)?
−5
19
−13
5
Show the answer
−5
1 × 4 + 0 × 6 + (−1) × 9 = 4 + 0 − 9 = −5. Every stop of a convolution is one dot product like this.
The picture. Watch w slide along x. The top row of cells is the input, the middle row shows the pattern under the window it covers, and the bottom row collects the outputs, each dropped in under the middle of its window. The flow counts its stops from 1, as you would aloud; the Python lane's j counts the same stops from 0. Notice what (−1, 0, 1) computes: the window's right number minus its left one. Where would that be big?
Try it now
Then set stride s to 2 and press Use these numbers. The pattern jumps two places at a time, and only 2 stops fit.
Symbols and formula. For a list of n numbers and a pattern of k, two more choices shape the output. The stride s is how far the pattern moves between stops. The padding p is how many zeros are added at each end first. The number of outputs is
stops=⌊sn+2p−k⌋+1
In plain words: after padding, there's n + 2p − k room to move. Divide by the step size, round down (a stop needs a whole window), and add 1 for the first stop.
The flow had n = 6, k = 3, s = 1 and p = 0: (6 − 3) ÷ 1 + 1 = 4 stops. With s = 2, (6 − 3) ÷ 2 = 1.5 rounds down to 1, and + 1 makes 2. Padding with p = 1 gives (6 + 2 − 3) + 1 = 6 outputs, as many as inputs: that's why models pad.
Two dimensions. A picture is a grid, so the pattern is a small grid too. A 3 × 3 pattern sits over a 3 × 3 window of a 5 × 5 trail sign, its 9 numbers times the window's 9, all added. Then it slides across, and down. The formula works in each direction: (5 − 3) ÷ 1 + 1 = 3 stops across and 3 down, so the output is a 3 × 3 map.
Quick check
A list of 9 numbers, a pattern of 3, stride 2, no padding. How many outputs?
3
4
4.5
7
Show the answer
4
(9 − 3) ÷ 2 + 1 = 3 + 1 = 4. The windows start at places 0, 2, 4 and 6.
Quick check
A list of 8 numbers and a pattern of 3, stride 1. What padding p makes the output 8 long too?
p = 1
p = 3
p = 2
p = 0
Show the answer
p = 1
(8 + 2 × 1 − 3) ÷ 1 + 1 = 8. A pattern of 3 hangs one place over each end, so one zero on each side keeps the length.
In your own words
In your own words: why does the pattern (−1, 0, 1) give 0 on a flat stretch and a big number at an edge?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a loop of dot products
Purpose. Write the convolution as a loop over slices, then as two loops over a grid.
The colon slices from lesson 53 also take a start and an end: x[j:j + k] is the numbers from place j up to, not including, place j + k. That's k numbers, the window. So each stop is one line, a slice dotted with the pattern:
import numpy as np
x = np.array([2, 2, 2, 7, 7, 7])
w = np.array([-1, 0, 1])
k = len(w)
stops = len(x) - k + 1
y = np.zeros(stops)
for j inrange(stops):
y[j] = x[j:j + k] @ w
print(stops, y)
4 stops, and y = (0, 5, 5, 0): the flow's outputs.
Code flow. Step through the loop. Watch j pick each stop, and y fill one box at a time, each the window x[j:j + k] dotted with w.
Stride and padding in code.slide adds the stride: stop j's window starts at j * s, and the stop count is the formula, with // to round down. np.concatenate joins arrays end to end, here to pad x with a 0 at each end:
import numpy as np
defslide(x, w, s):
k = len(w)
stops = (len(x) - k) // s + 1
y = np.zeros(stops)
for j inrange(stops):
y[j] = x[j * s:j * s + k] @ w
return y
x = np.array([2, 2, 2, 7, 7, 7])
w = np.array([-1, 0, 1])
print(slide(x, w, 1), slide(x, w, 2))
xp = np.concatenate([[0], x, [0]])
print(xp, slide(xp, w, 1))
Stride 2 keeps every other stop, (0, 5). Padding brings back 6 outputs, but look at the ends: 2 and −7. The padded zeros look like edges of their own, a jump from 0 up to 2 and from 7 down to 0. Models pad anyway, to keep sizes steady, and learn to live with the borders.
Quick check
What does this print?
import numpy as np
x = np.array([1, 2, 3, 4, 5])
w = np.array([1, 1])
y = np.zeros(4)
for j inrange(4):
y[j] = x[j:j + 2] @ w
print(y.tolist())
[1.0, 2.0, 3.0, 4.0]
[2.0, 4.0, 6.0, 8.0]
[3.0, 5.0, 7.0, 9.0]
[3.0, 5.0, 7.0]
Show the answer
[3.0, 5.0, 7.0, 9.0]
Each stop adds two neighbors: 1 + 2, 2 + 3, 3 + 4, 4 + 5. Five numbers and a pattern of 2 give 5 − 2 + 1 = 4 outputs.
A named mistake: one stop too few, or too many. The stop count is n − k + 1. Leave off the + 1 and the loop quietly drops the last stop:
import numpy as np
x = np.array([2, 2, 2, 7, 7, 7])
w = np.array([-1, 0, 1])
y = []
for j inrange(len(x) - len(w)):
y.append(float(x[j:j + 3] @ w))
print(len(y), y)
Only 3 outputs: the last stop is gone, with no error to warn you. Loop to len(x) instead, one too many the other way, and a window near the end is too short for the pattern:
import numpy as np
x = np.array([2, 2, 2, 7, 7, 7])
w = np.array([-1, 0, 1])
for j inrange(len(x)):
print(j, x[j:j + 3] @ w)
j = 0 to 3 print, then x[4:7] holds only 2 numbers and @ stops with an error. Count the stops with the formula, and check len(y) against it.
Worked example
Worked example: two loops over a sign
Slide a 3 × 3 pattern over the 5 × 5 arrow sign. Its columns are −1, 2, −1: it looks for an ink column with blank on both sides, an upright stroke.
import numpy as np
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
P = np.array([[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]])
out = np.zeros((3, 3))
for i inrange(3):
for j inrange(3):
out[i, j] = np.sum(arrow[i:i + 3, j:j + 3] * P)
print(out)
print(arrow[0:3, 1:4] * P)
In the lesson you can change this and run it.
arrow[i:i + 3, j:j + 3] slices 3 rows and 3 columns: the window at stop (i, j). Multiplying it by P number by number and adding with np.sum is the dot product of two grids. The second print shows stop (0, 1)'s 9 products: the shaft's three 1s times 2, minus the two inked neighbors, 6 − 2 = 4. The map's middle column, 4, 4 and 6, runs down the arrow's shaft; it's biggest at the bottom, where the shaft stands alone.
A moving average is a convolution.Lesson 27 averaged each day of tent rentals with the two days before it. That's the pattern (1/3, 1/3, 1/3) slid along the week:
import numpy as np
rentals = np.array([12, 18, 9, 15, 21, 12, 18])
w = np.array([1/3, 1/3, 1/3])
smooth = np.zeros(5)
for j inrange(5):
smooth[j] = rentals[j:j + 3] @ w
print(smooth.round(2))
The same 13, 14, 15, 16 and 17. Here they are over the raw week, each output drawn at the last day of its window:
Read the plot. What does the smoothing pattern show that the raw rentals hide?
That rentals fall over the week
That day 5 was the busiest
A steady climb of one rental a day
That every day rents the same
Show the answer
A steady climb of one rental a day
The raw line zigzags between 9 and 21, but the smoothed line rises 13, 14, 15, 16, 17: one more rental each day. Every weight is 1/3 and they add to 1, so the pattern averages instead of finding edges.
Drill
A convolution with stride
Fix conv(x, w, s) so it returns one output per stop: the window starting at j * s dotted with w, for every stop that fits. The starter has two slips.
import numpy as np
defconv(x, w, s):
k = len(w)
stops = (len(x) - k) // s
y = np.zeros(stops)
for j inrange(stops):
y[j] = x[j:j + k] @ w
return y
x = np.array([2, 2, 2, 7, 7, 7])
print(conv(x, np.array([-1, 0, 1]), 1), conv(x, np.array([-1, 0, 1]), 2))
In the lesson, your code is checked as you write it.
In your own words
Why is it worth computing the number of stops with the formula, rather than trusting the loop?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: one small pattern for a whole picture
Purpose. See how a convolutional layer finds strokes in the trail signs with 9 weights.
The trail club's sign reader starts with patterns like the worked example's P: ink in the middle column, blank on either side. Watch it slide over the arrow sign. The picture lights the window at each stop, and the last step shows the output map, each number in the middle of its window.
Where it fires on each sign. The same pattern, the same 9 numbers, over the arrow, the tent and a water-drop sign. conv2d is the worked example's two loops as a function:
import numpy as np
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
tent = np.array([[0,0,1,0,0],[0,1,0,1,0],[0,1,0,1,0],[1,0,0,0,1],[1,1,1,1,1]])
drop = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,1,1,1,1],[1,1,1,1,1],[0,1,1,1,0]])
P = np.array([[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]])
defconv2d(x, w):
out = np.zeros((3, 3))
for i inrange(3):
for j inrange(3):
out[i, j] = np.sum(x[i:i + 3, j:j + 3] * w)
return out
for sign in [arrow, tent, drop]:
out = conv2d(sign, P)
print(out.tolist(), out.max())
The arrow's map peaks at 6, down its shaft. The tent fires at 3 on both sides, where its walls run nearly upright. The water drop barely fires at all, at most 2: it's a solid blob, and inside a solid patch every row of the window is 1, 1, 1, so −1 + 2 − 1 = 0. The pattern's numbers add to 0, so it ignores flat areas, ink or blank, and answers only to a stroke.
Quick check
A pattern's 9 numbers add up to 0. What does it give on a window that's all ink (all 1s)?
9
0
It depends on where the window is
1
Show the answer
0
On all 1s the dot product is just the pattern's numbers added up: 0. That's why the stroke pattern ignores the inside of the water drop.
Shared weights, counted. A plain layer from 25 pixels to the 9 numbers of the map would give every pixel a weight for every output. The convolution reuses one set of 9 at every stop:
import numpy as np
P = np.array([[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]])
print(P.size, 25 * 9)
side = 28 - 3 + 1print(28 * 28, side, side * side, 28 * 28 * side * side)
9 weights instead of 225. On a 28 × 28 photo, 784 pixels, the pattern stops 28 − 3 + 1 = 26 times each way, so the map is 26 × 26, 676 numbers. A plain layer from the 784 pixels to those 676 outputs needs a weight for every pair, 784 × 676 = 529,984, where the convolution still needs 9. Sharing also builds in the fact the opening line noted: a stroke is a stroke wherever it is, so the same pattern finds it at every stop. A real layer learns many patterns at once, each giving its own map, called a channel.
In PyTorch.F.conv1d and F.conv2d slide the patterns for you. They expect a batch of inputs with channels: a picture tensor has the shape (batch, channels, height, width), so the three signs stacked are (3, 1, 5, 5), and the pattern is (out channels, in channels, 3, 3), here (1, 1, 3, 3). view reshapes a tensor without changing its numbers:
import torch
import torch.nn.functional as F
x = torch.tensor([2., 2., 2., 7., 7., 7.])
w = torch.tensor([-1., 0., 1.])
print(F.conv1d(x.view(1, 1, 6), w.view(1, 1, 3)))
arrow = torch.tensor([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
tent = torch.tensor([[0,0,1,0,0],[0,1,0,1,0],[0,1,0,1,0],[1,0,0,0,1],[1,1,1,1,1]])
drop = torch.tensor([[0,0,1,0,0],[0,1,1,1,0],[1,1,1,1,1],[1,1,1,1,1],[0,1,1,1,0]])
P = torch.tensor([[-1., 2., -1.], [-1., 2., -1.], [-1., 2., -1.]])
signs = torch.stack([arrow, tent, drop]).float().view(3, 1, 5, 5)
out = F.conv2d(signs, P.view(1, 1, 3, 3))
print(signs.shape, out.shape)
print(out[0, 0])
print(out.view(3, 9).max(dim=1).values)
The math part's (0, 5, 5, 0), the arrow's map, and the three peaks 6, 3 and 2: the same numbers as the loops. torch.stack piles the signs into one tensor. In a model the pattern lives in a layer, nn.Conv2d(1, 1, 3), whose weight has shape (1, 1, 3, 3), 9 numbers, plus a bias, learned in training instead of written by hand. One more thing: in math and signal processing, a convolution flips the pattern before sliding it. Deep learning's doesn't (strictly, the unflipped version is called cross-correlation), and neither does this lesson.
Quick check
What does this print?
import torch
import torch.nn.functional as F
photos = torch.zeros(8, 1, 28, 28)
pattern = torch.zeros(1, 1, 3, 3)
print(F.conv2d(photos, pattern).shape)
torch.Size([8, 1, 28, 28])
torch.Size([8, 1, 26, 26])
torch.Size([8, 1, 25, 25])
torch.Size([1, 1, 26, 26])
Show the answer
torch.Size([8, 1, 26, 26])
The batch of 8 stays, one pattern gives one channel, and each side has (28 − 3) + 1 = 26 stops.
Drill
Which sign has a stroke?
Fix conv2d so it takes every stop of a 3 × 3 pattern over any grid (the output has len(x) - 2 rows and len(x[0]) - 2 columns). Then set peaks to each sign's biggest output, fires_most to the name of the sign with the biggest peak, and n_dense to how many weights a plain layer from 25 pixels to 9 outputs would need.
import numpy as np
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
tent = np.array([[0,0,1,0,0],[0,1,0,1,0],[0,1,0,1,0],[1,0,0,0,1],[1,1,1,1,1]])
drop = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,1,1,1,1],[1,1,1,1,1],[0,1,1,1,0]])
P = np.array([[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]])
defconv2d(x, w):
H = len(x) - 2
W = len(x[0]) - 2
out = np.zeros((H, W))
for i inrange(H):
for j inrange(W):
out[i, j] = np.sum(x[i:i + 2, j:j + 2] * w)
return out
names = ["tent", "drop", "arrow"]
peaks = []
for sign in [tent, drop, arrow]:
peaks.append(float(conv2d(sign, P).max()))
fires_most = names[0]
n_dense = 25 + 9print(peaks, fires_most, n_dense)
In the lesson, your code is checked as you write it.
In your own words
A convolutional layer has far fewer weights than a plain layer on the same picture. What does it give up, and what does it gain?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Convolutional networks: stacks of learned patterns, many channels, and pooling, trained on real images.
Build a vision transformer: cutting a picture into patches, a stride-sized convolution, before attention.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write