Convolutional networks

Why this beats a fully connected layer for images

Think of a "Where's Waldo?" puzzle. You don't learn a separate Waldo for each corner of the page. You learn one picture of Waldo and scan for it everywhere. CNNs work the same way, for three reasons.

• Shared weights. The same 9 weights are used at every spot. So a filter that finds an edge finds it anywhere in the picture. A 3 × 3 filter has 9 weights, whatever the picture size.
• Local connections. Each score looks only at a small neighborhood. That matches how pictures are built: a stroke or an edge is made of pixels that touch.
• Layers build up. Stack convolution layers and later filters see larger areas. The first layer finds edges. Later layers find textures and parts. The last layers find whole objects.

Compare a fully connected layer. Every pixel gets its own weight for every unit. A 224 × 224 photo has 50,176 pixels. Connect it to just 100 units and you need 50,176 × 100, about 5 million weights. And a pattern learned in one corner teaches the layer nothing about other corners.

How big is the output?

Four settings decide the size of a feature map. Here they are, one at a time:

• $n$ is the input size. An 8 × 8 picture has $n = 8$.
• $k$ is the kernel size. A 3 × 3 filter has $k = 3$.
• $s$ is the stride: how many pixels the filter moves each step. Stride 1 visits every spot. Stride 2 skips every other one.
• $p$ is the padding: how many rings of zeros you add around the border first. Padding lets the filter sit on edge pixels, so the output shrinks less, or not at all.

The output size is:

The $\lfloor\ \rfloor$ brackets mean "round down to a whole number".

In plain words: add the padding to both sides of the picture. Take away the filter size. That leaves the room the filter can travel. Divide by the step size, round down, and add 1 for the starting spot.

Try it with the digit above: $n = 8$, $k = 3$, $s = 1$, $p = 0$. That's 8 + 0 − 3 = 5. Then 5 ÷ 1 = 5, and 5 + 1 = 6. So the output is 6 × 6, as you saw.

Now add one ring of padding ($p = 1$): 8 + 2 − 3 = 7, and 7 + 1 = 8. The output stays 8 × 8.

The same calculation in Python, where // divides and rounds down:

def out_size(n, k, s=1, p=0):
    return (n + 2 * p - k) // s + 1

print(out_size(8, 3))          # the digit example
print(out_size(8, 3, p=1))     # one ring of padding
print(out_size(8, 2, s=2))     # 2 × 2 filter, stride 2

It prints 6, 8 and 4. The last line shows that stride 2 halves the size: (8 − 2) ÷ 2 + 1 = 4.

Channels and pooling

A layer usually has many filters, not one. Each filter makes its own feature map. The maps stack on top of each other like pages in a book. Each page is called a channel.

Pictures have channels too. A color photo starts with 3: one grid each for red, green and blue. A filter always reaches through every input channel. So on a color photo, one 3 × 3 filter has 3 × 3 × 3 = 27 weights, one 3 × 3 grid per color. It still gives one score per spot.

Pooling shrinks a feature map. The usual kind, max pooling, splits the map into 2 × 2 blocks and keeps only the biggest number in each block. It's like keeping each region's headline: "an edge showed up somewhere here". For example, the block below keeps 3.

1  3
2  0     →  3

Pooling halves the width and the height. That makes later layers cheaper. It also lets them see a wider area of the picture.

A small CNN, built from patches

Here's a useful fact. A convolution is a matrix multiply in disguise.

Cut out every 3 × 3 patch of the picture. Flatten each one into a row of 9 numbers. One filter is also 9 numbers. Its score on one patch is "multiply and add", which is a dot product. So put all the patch rows in one table and all the filters in another. One matrix multiply then applies every filter to every patch at once.

This trick is called im2col, short for "image to columns". (Some libraries lay the patches out as columns rather than rows. It's the same idea.) Many libraries run convolutions this way. It also lets this course's in-browser PyTorch, which has no built-in convolution layer (Conv2d), train a real CNN.

The code below trains two models on the 8 × 8 digits and compares them:

• TinyCNN. It has 8 learned 3 × 3 filters, then ReLU, then 2 × 2 max pooling. A final layer turns the pooled maps into 10 scores, one per digit.
• An MLP. It flattens the 64 pixels and uses one hidden layer of 64 units.

import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32)   # (1797, 8, 8)
y = torch.tensor(digits.target)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0)

# Row and column of every pixel in every 3 × 3 patch: 36 positions × 9 pixels
pos = [(r, c) for r in range(6) for c in range(6)]
ROWS = np.array([[r + i for i in range(3) for j in range(3)] for r, c in pos])
COLS = np.array([[c + j for i in range(3) for j in range(3)] for r, c in pos])

class TinyCNN(nn.Module):
    def __init__(self, filters=8):
        super().__init__()
        self.filters = filters
        self.kernel = nn.Linear(9, filters)               # one row of 9 weights per filter, plus a bias
        self.head = nn.Linear(filters * 3 * 3, 10)

    def forward(self, images):                            # images: (batch, 8, 8)
        patches = images[:, ROWS, COLS]                    # (batch, 36, 9)
        maps = torch.relu(self.kernel(patches))           # (batch, 36, filters): every filter at every position
        maps = maps.view(-1, 6, 6, self.filters)
        pooled = maps.view(-1, 3, 2, 3, 2, self.filters).max(dim=4).values.max(dim=2).values   # 2 × 2 max pooling
        return self.head(pooled.flatten(1))               # (batch, 10)

def train(model, epochs=12):
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    for epoch in range(epochs):
        for xb, yb in DataLoader(TensorDataset(X_train, y_train), batch_size=64, shuffle=True):
            opt.zero_grad()
            nn.functional.cross_entropy(model(xb), yb).backward()
            opt.step()
    with torch.no_grad():
        return (model(X_val).argmax(1) == y_val).float().mean().item()

torch.manual_seed(0)
cnn = TinyCNN()
mlp = nn.Sequential(nn.Flatten(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
for name, model in [("CNN", cnn), ("MLP", mlp)]:
    params = sum(p.numel() for p in model.parameters())
    print(f"{name}: {params:,} parameters, validation accuracy {train(model):.3f}")

The CNN has 810 parameters. The MLP has 4,810, about six times as many. Yet the CNN gets close to the MLP's accuracy on pictures it never trained on.

On tiny 8 × 8 digits, a small MLP still holds its own (it's slightly better here). Real photos are hundreds of pixels per side. There, fully connected layers grow far too large, and CNNs win clearly. That's why computer vision used them almost exclusively for a decade.

print(8 * 9 + 8)

The real thing in PyTorch

Full PyTorch has a ready-made convolution layer, nn.Conv2d, and a pooling layer, nn.MaxPool2d. Here's a typical small CNN for 28 × 28 grayscale pictures:

import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(in_channels=1, out_channels=16, kernel_size=3, padding=1),   # (B, 1, 28, 28) → (B, 16, 28, 28)
    nn.ReLU(),
    nn.MaxPool2d(2),                                                        # → (B, 16, 14, 14)
    nn.Conv2d(16, 32, kernel_size=3, padding=1),                           # → (B, 32, 14, 14)
    nn.ReLU(),
    nn.MaxPool2d(2),                                                        # → (B, 32, 7, 7)
    nn.Flatten(),
    nn.Linear(32 * 7 * 7, 10),
)

The comments show the shape after each layer. B is the batch size, the number of pictures handled together. Read the shapes as batch, channels, height, width.

• in_channels=1: grayscale input has one channel.
• out_channels=16: the layer learns 16 filters, so it makes 16 feature maps.
• padding=1 keeps the size at 28. Check it with the formula: 28 + 2 − 3 = 27, and 27 + 1 = 28.
• Each MaxPool2d(2) halves the size: 28 to 14, then 14 to 7.
• The final layer reads all 32 × 7 × 7 = 1,568 numbers and gives 10 class scores.

Where CNNs stand now

Newer models called vision transformers cut a picture into patches and use attention, which you'll meet in Phase 5. At large scale they now match or beat CNNs. Most multimodal LLMs use them to read images.

CNNs are still common where speed and efficiency matter: phones, cameras and real-time video. They also appear in hybrid models that mix both designs. And the idea of a small, shared, sliding filter lives on in audio models and some sequence models.