Sliding dot products: convolution
Why this matters
A layer from lesson 56 gives every input its own weight for every output. For a picture, that's a lot of weights, and it ignores something true of pictures: an edge looks the same wherever it is. Image and audio models do something smarter. They slide one small pattern across the input and take a dot product at each stop. That's a convolution, and it's the dot product from lesson 50, repeated.
- Slide a 3-number pattern along a list and compute every output as a dot product
- Predict how many outputs there are, (n + 2p − k) ÷ s + 1 rounded down, with stride s and padding p
- Do the same in two dimensions: a 3 × 3 pattern over a 5 × 5 trail sign, with two loops
- Say what a pattern detects from its numbers, and count the weights it saves by sharing them
The math: one pattern, many stops
Purpose. Slide a small pattern along a list, take a dot product at every stop, and count the stops.
Here's a row of 6 pixels from a photo, dark on the left and bright on the right: x = (2, 2, 2, 7, 7, 7). And here's a pattern, or kernel, of 3 numbers: w = (−1, 0, 1). Lay w over the first 3 pixels, the window, and take the dot product: 2 × (−1) + 2 × 0 + 2 × 1 = 0. Slide one place right and do it again. Each stop gives one output number. The list of outputs, y, is the convolution of x with w.
In plain words: output j is the pattern dotted with the window that starts at place j.