Index Ai Academy
Phase 5 · Beyond text

Segmentation: a label for every pixel

Masks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

A box says roughly where an object is. Some jobs need its exact outline: the area of a tumor on a scan, the edge of the road in front of a car, the part of a photo to keep when the background is removed. That is segmentation, and its answer is a label for every pixel. This lesson covers what a mask is, how to grade one honestly, the tension at the heart of every segmentation network, between seeing enough of the picture and keeping track of exactly where things are, and how to get from a mask back to countable objects.

What you'll be able to do

  • Describe a segmentation mask, and tell semantic segmentation from instance segmentation
  • Score a mask with IoU and Dice, and say why pixel accuracy misleads
  • Explain why each pixel's label needs context, and why too much context blurs small things
  • Describe the shrink-then-enlarge design of segmentation networks and what skip connections restore
  • Turn a mask into separate objects by finding connected regions, and say when that fails
Quick check
A guess before you start. A model marks the road in aerial photos, pixel by pixel, and 95% of its pixel labels are correct. Roads cover about 5% of each photo. What can you conclude?
  1. Very little: marking no road at all would also score 95%
  2. That its road outlines are accurate to within 5%
  3. That it finds 95% of the road pixels in each photo
  4. That 5% of its road pixels are wrongly marked
Show the answer

Very little: marking no road at all would also score 95%

When the thing you care about covers a small share of the image, a model that marks nothing is right almost everywhere. Pixel accuracy is dominated by the background. This lesson uses measures that look only at the object: the overlap between the predicted and the true region.

The idea: a label for every pixel

This rests on Object detection: boxes, overlap and suppression, whose IoU and average precision are reused here on regions of pixels, and on Convolutional networks.

A label for every pixel

A mask is an array the same height and width as the image, holding one class number per pixel. Where a detector says "a dog, somewhere in this rectangle", a segmentation model says which pixels are dog.

The task comes in two main kinds.

  • Semantic segmentation gives each pixel a class: road, car, sky. Two cars side by side become one region of "car".
  • Instance segmentation also separates the objects: this pixel belongs to car 1, that one to car 2.

The cell below builds a small mask by hand: a disk (class 1) and a tall bar (class 2) on a background (class 0). It prints every second row, then compares each object's mask with its bounding box.

Setup (runs before every cell) · 17 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

def scene(seed):
    g = np.random.default_rng(seed)
    y, x = np.mgrid[0:32, 0:32]
    mask = np.zeros((32, 32), dtype=bool)
    for _ in range(3):
        cy, cx, r = g.integers(6, 26), g.integers(6, 26), g.integers(3, 7)
        mask |= (y - cy) ** 2 + (x - cx) ** 2 <= r * r
    return np.where(mask, 0.65, 0.35) + g.normal(0, 0.35, size=(32, 32)), mask

def window_mean(img, k):
    padded, out = np.pad(img, k // 2, mode="edge"), np.zeros_like(img)
    for dy in range(k):
        for dx in range(k):
            out += padded[dy:dy + 32, dx:dx + 32]
    return out / (k * k)
import numpy as np

yy, xx = np.mgrid[0:24, 0:32]  # the row and column number of every pixel
mask = np.zeros((24, 32), dtype=int)
mask[(yy - 11) ** 2 + (xx - 9) ** 2 <= 36] = 1         # a disk of radius 6
mask[(yy >= 4) & (yy < 20) & (xx >= 22) & (xx < 25)] = 2    # a bar, 3 wide and 16 tall

for row in mask[::2]:
    print("".join(".o#"[v] for v in row))

for k, name in ((1, "disk"), (2, "bar")):
    ys, xs = np.where(mask == k)
    box_area = (xs.max() + 1 - xs.min()) * (ys.max() + 1 - ys.min())
    pixels = int((mask == k).sum())
    print(name, "| pixels in its mask:", pixels, "| pixels in its box:", int(box_area),
          "| share of the box that is object:", round(pixels / box_area, 2))
print("share of the image that is background:", round(float((mask == 0).mean()), 3))
It runs right here in your browser, with nothing to install.

The bar fills its box completely, so a box describes it perfectly. The disk fills 0.67 of its box: a third of the rectangle around it is something else. For a diagonal object, such as a pen lying across a desk, a box is nearly all background, and only a mask says where the object is.

Note the last line too. Even in this small picture, 0.79 of the pixels are background. In real images the share is often higher, and that shapes how masks have to be graded.

Quick check
What does this print: the number of object pixels, the area of the object's bounding box, and the share of the box the object fills?
mask = ["....#...",
        "...###..",
        "..#####.",
        "...###..",
        "....#..."]
pixels = sum(row.count("#") for row in mask)
columns = [x for row in mask for x, c in enumerate(row) if c == "#"]
rows = [y for y, row in enumerate(mask) if "#" in row]
box = (max(columns) - min(columns) + 1) * (max(rows) - min(rows) + 1)
print(pixels, box, pixels / box)
  1. 13 13 1.0
  2. 25 25 1.0
  3. 13 25 0.52
  4. 13 40 0.325
Show the answer

13 25 0.52

The diamond has 1 + 3 + 5 + 3 + 1 = 13 pixels. It spans columns 2 to 6 and rows 0 to 4, so its box is 5 by 5 = 25. The diamond fills about half of its own bounding box.

Scoring a mask

The obvious score is pixel accuracy: the share of pixels given the right class. The opening question showed its flaw, and the cell below measures it. The truth is the disk. One prediction is the right shape, 4 pixels too far right. The other predicts no object at all.

The measures that work are the ones from the detection lesson, applied to regions instead of boxes. IoU is the number of pixels in both regions divided by the number in either. Dice counts the shared pixels twice and divides by the two regions' sizes added together.

IoU=∣A∩B∣∣A∪B∣Dice=2 ∣A∩B∣∣A∣+∣B∣\text{IoU} = \frac{|A \cap B|}{|A \cup B|} \qquad \text{Dice} = \frac{2\,|A \cap B|}{|A| + |B|}

In plain words: ∣A∩B∣|A \cap B| is the number of pixels in both masks, ∣A∪B∣|A \cup B| the number in at least one, and ∣A∣|A| and ∣B∣|B| the sizes of the two masks. Both scores are 1 for a perfect match and 0 for no overlap.

Uses the setup above, 17 lines.

import numpy as np

yy, xx = np.mgrid[0:24, 0:32]
truth = (yy - 11) ** 2 + (xx - 9) ** 2 <= 36      # the disk, as True and False
shifted = (yy - 11) ** 2 + (xx - 13) ** 2 <= 36   # the same disk, 4 pixels to the right
nothing = np.zeros_like(truth)                    # no object predicted anywhere

def scores(pred, truth):
    both, either = (pred & truth).sum(), (pred | truth).sum()
    accuracy = (pred == truth).mean()
    return (round(float(accuracy), 3), round(float(both / either), 3),
            round(float(2 * both / (pred.sum() + truth.sum())), 3))

print("prediction                | pixel accuracy, IoU, Dice")
for name, pred in (("exact", truth), ("shifted 4 pixels", shifted),
                   ("no object at all", nothing)):
    print(name.ljust(25), "|", scores(pred, truth))

The prediction that finds nothing scores 0.853 on pixel accuracy, nearly the same as the shifted disk's 0.88. IoU tells them apart: 0.421 against 0.0.

Dice is always at least as large as IoU on the same masks (0.593 here), and the two always rank predictions in the same order. Medical imaging tends to report Dice, and general vision benchmarks report mean IoU: the IoU of each class, averaged over the classes, so that a small class counts as much as the background.

Quick check
From the lesson on object detection: two boxes of equal size overlap by exactly half of each. What is their IoU?
  1. One quarter
  2. One half
  3. Two thirds
  4. One third
Show the answer

One third

With each box's area as 1, the shared part is 0.5 and the union is 1 + 1 − 0.5 = 1.5, so the IoU is 1/3. Masks behave the same way: the union grows as the overlap shrinks, so IoU falls faster than the eye expects.

Quick check
A true mask and a predicted mask are compared. What does this print: the pixels in both, the pixels in either, the IoU and the Dice score?
truth = ["##..", "##..", "...."]
pred = [".##.", ".##.", "...."]
both = sum(t == "#" and p == "#" for tr, pr in zip(truth, pred) for t, p in zip(tr, pr))
either = sum(t == "#" or p == "#"
             for tr, pr in zip(truth, pred) for t, p in zip(tr, pr))
print(both, either, round(both / either, 3), 2 * both / (4 + 4))
  1. 2 8 0.25 0.5
  2. 4 6 0.667 1.0
  3. 2 6 0.333 0.333
  4. 2 6 0.333 0.5
Show the answer

2 6 0.333 0.5

The two squares share one column of 2 pixels. Together they cover 4 + 4 − 2 = 6 pixels. IoU is 2 / 6 = 0.333, and Dice is 2 × 2 / (4 + 4) = 0.5. Dice is the larger of the two, as always.

Try it now
Worked example

The numbers behind the picture

The same two masks as arrays: pixels in both, pixels in either, IoU, Dice and pixel accuracy.

Uses the setup above, 17 lines.

import numpy as np

truth = np.zeros((4, 5), bool)
truth[1:3, 1:4] = True
pred = np.zeros((4, 5), bool)
pred[1:3, 2:5] = True
both, either = (pred & truth).sum(), (pred | truth).sum()
print("in both:", int(both), " in either:", int(either))
print("IoU:", round(float(both / either), 2))
print("Dice:", round(float(2 * both / (pred.sum() + truth.sum())), 2))
print("pixel accuracy:", round(float((pred == truth).mean()), 2))
In the lesson you can change this and run it.

Four pixels are in both masks and eight in either: IoU 0.5. Dice counts the shared pixels twice, 8 over 12, so 0.67. Pixel accuracy is 0.8, because the 8 background pixels the two masks share also count as right: it looks good even though the masks overlap by half.

In your own words
What is the difference between semantic and instance segmentation? Give a case where it matters.

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: context, shrinking and regions

How much of the picture a pixel needs to see

A segmentation model is a classifier that runs at every pixel. The question is what it gets to look at when it decides. One pixel's brightness is rarely enough: real images are noisy, and "gray" could be road or roof.

The experiment below makes that concrete. Each image holds a few slightly brighter disks buried in noise. A logistic regression labels each pixel as object or background from one number: the average brightness of the square window around that pixel. Only the window's size changes. In a convolutional network this region is called the receptive field.

Uses the setup above, 17 lines.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression

train, test = [scene(s) for s in range(20)], [scene(s) for s in range(100, 120)]
shown = {}
for k in (1, 3, 5, 9, 15):
    model = LogisticRegression().fit(
        np.concatenate([window_mean(img, k).reshape(-1, 1) for img, m in train]),
        np.concatenate([m.reshape(-1) for img, m in train]))
    ious, accuracies = [], []
    for img, m in test:
        pred = model.predict(window_mean(img, k).reshape(-1, 1)).reshape(32, 32)
        ious.append((pred & m).sum() / (pred | m).sum())
        accuracies.append((pred == m).mean())
    shown[k] = model.predict(window_mean(test[0][0], k).reshape(-1, 1)).reshape(32, 32)
    print("window", str(k).rjust(2), "by", str(k).ljust(2), "| pixel accuracy",
          str(round(float(np.mean(accuracies)), 3)).ljust(5), "| IoU",
          round(float(np.mean(ious)), 3))

fig, axes = plt.subplots(1, 5, figsize=(12, 2.8))
panels = [("the image", test[0][0]), ("the truth", test[0][1]), ("window 1", shown[1]),
          ("window 5", shown[5]), ("window 15", shown[15])]
for ax, (title, picture) in zip(axes, panels):
    ax.imshow(picture, cmap="gray")
    ax.set_title(title)
    ax.axis("off")

A model that sees one pixel is nearly useless: an IoU of 0.082, with a pixel accuracy of 0.818 that would have hidden the failure. Averaging over a 5 by 5 window cancels much of the noise and lifts the IoU to 0.638.

Then it turns. At 15 by 15 the IoU is back down to 0.418. A window that large averages a small disk together with the background around it, and its edges blur into a smooth blob. The figure shows both failures: speckle on the left, a soft outline on the right.

This is the central tension of segmentation. Context says what a pixel is, and wants a large view. Location says exactly where the edge runs, and wants a small one.

Quick check
In the experiment, why did the largest window give a worse mask than the 5 by 5 window?
  1. It had more noise to deal with than the small windows
  2. A logistic regression can't use that many pixels
  3. It saw fewer training examples than the small windows
  4. It averaged small objects and their edges into the background
Show the answer

It averaged small objects and their edges into the background

Averaging over a large window removes noise, and it also removes detail. A disk 6 pixels across is a minority of a 15 by 15 window, so its average barely differs from background, and every edge becomes a slow ramp. A big view knows what is nearby and loses track of exactly where it ends.

The way out is to give the classifier both views. Below, each pixel is described by several window means at once, small and large, and the same model learns how to weigh them. The table also splits the mistakes by where they occur.

Uses the setup above, 17 lines.

import numpy as np
from sklearn.linear_model import LogisticRegression

def features(img, sizes):
    # one column per window size
    return np.stack([window_mean(img, k).reshape(-1) for k in sizes], axis=1)

train, test = [scene(s) for s in range(20)], [scene(s) for s in range(100, 120)]
print("windows used     | IoU   | wrong near an edge | wrong elsewhere")
for sizes in ((3,), (9,), (3, 9), (1, 3, 5, 9)):
    model = LogisticRegression().fit(
        np.concatenate([features(img, sizes) for img, m in train]),
        np.concatenate([m.reshape(-1) for img, m in train]))
    ious, at_edge, elsewhere = [], [], []
    for img, m in test:
        pred = model.predict(features(img, sizes)).reshape(32, 32)
        ious.append((pred & m).sum() / (pred | m).sum())
        blur = window_mean(m.astype(float), 3)
        # pixels within one step of the object's outline
        edge = (blur > 0.01) & (blur < 0.99)
        at_edge.append((pred != m)[edge].mean())
        elsewhere.append((pred != m)[~edge].mean())
    print(str(sizes).ljust(16), "|", str(round(float(np.mean(ious)), 3)).ljust(5), "|",
          str(round(float(np.mean(at_edge)), 3)).ljust(18), "|",
          round(float(np.mean(elsewhere)), 3))

A small window and a large one together reach 0.671, better than either alone (0.518 and 0.606), and four sizes reach 0.689. The large window says "there is an object around here", and the small one says "and this pixel is on the bright side of the edge".

The last two columns show where segmentation is hard. Away from any outline, about 2 pixels in 100 are wrong. Within a step of an outline, about 1 in 4 is. Nearly all the difficulty of segmentation is at the edges, which is also where the measurement matters.

Shrink to understand, enlarge to answer

Real segmentation networks are built around that tension. The encoder is an ordinary convolutional network: it applies filters and pools, so each stage sees a wider part of the image at a lower resolution. By the end it knows what is in the picture, on a grid perhaps 16 or 32 times coarser than the input.

A mask needs an answer for every pixel, so a decoder enlarges that coarse grid back to full size, a step called upsampling. The cell below shows what that round trip costs, with no network involved: take a perfect mask, shrink it by averaging blocks of pixels, and enlarge it again by repeating each value.

Uses the setup above, 17 lines.

import numpy as np

def shrink_and_enlarge(mask, f):
    h, w = mask.shape
    # a block is object if half of it is
    small = mask.reshape(h // f, f, w // f, f).mean(axis=(1, 3)) >= 0.5
    # each value becomes an f by f block
    return np.repeat(np.repeat(small, f, axis=0), f, axis=1)

yy, xx = np.mgrid[0:32, 0:32]
shapes = {
    "a disk of radius 10": (yy - 16) ** 2 + (xx - 16) ** 2 <= 100,
    "a disk of radius 3": (yy - 16) ** 2 + (xx - 16) ** 2 <= 9,
    "a line 2 pixels wide": (xx >= 15) & (xx < 17),
}
print("shape                | IoU after shrinking by 2, by 4, by 8")
for name, mask in shapes.items():
    row = []
    for f in (2, 4, 8):
        back = shrink_and_enlarge(mask, f)
        row.append(round(float((back & mask).sum() / (back | mask).sum()), 3))
    print(name.ljust(20), "|", row)

These are perfect masks, damaged only by a trip through a coarse grid. The large disk survives a shrinking by 8 with an IoU of 0.78: its body is intact and its outline has turned to staircase. The small disk is at 0.324 after shrinking by 4 and gone after 8. The thin line is gone after 4.

Thin and small things are what a coarse grid can't hold: a blood vessel, a lane marking, a cable. The fix is the one the last experiment pointed at. A skip connection carries the encoder's early, full-resolution features straight across to the decoder, so that when the decoder enlarges its coarse answer, it has the fine detail on hand to sharpen the edges. The best-known design built this way is the U-Net, named for the shape of its diagram: down one side, up the other, with skips across.

Go deeper Other parts you will meet
  • Transposed convolution and bilinear upsampling are two ways of enlarging a feature grid: the first learns how to fill in the new pixels, the second interpolates between neighbors.
  • Mask R-CNN does instance segmentation by detecting boxes first, then predicting a small mask inside each box.
  • Promptable models, such as Segment Anything, take a click or a box as a prompt and return the mask of whatever is there, without a fixed list of classes.
Quick check
A segmentation network's encoder ends at a grid 16 times coarser than the image. Without skip connections, what is most likely to be wrong with its masks?
  1. Thin and small objects are missing, and outlines are blocky
  2. The masks are sharp, and the classes are wrong
  3. Every pixel is given the background class
  4. Large objects are missing, and small ones are found
Show the answer

Thin and small objects are missing, and outlines are blocky

A grid that coarse can't represent anything much smaller than one of its cells, so thin lines and small blobs fall between them, and every outline comes back as a staircase when it is enlarged. The large shapes, which the coarse grid holds well, survive. Skip connections bring the fine detail back.

From a mask to separate objects

A semantic mask says which pixels are "cell". To count the cells, or measure each one, the mask has to be split into objects. The classic way is to find connected components: groups of object pixels that touch each other, up, down, left or right.

The method is a flood fill. Scan the mask. On reaching an object pixel that hasn't been visited, start a new component there and spread outward to every object pixel connected to it, marking each as visited.

Uses the setup above, 17 lines.

mask = ["..##....#.",
        "..##....#.",
        "........#.",
        ".#......#.",
        ".##..####.",
        "..#..##..."]

def components(rows):
    seen, sizes = set(), []
    for y in range(len(rows)):
        for x in range(len(rows[0])):
            if rows[y][x] != "#" or (y, x) in seen:
                continue
            stack, size = [(y, x)], 0                  # a new component starts here
            seen.add((y, x))
            while stack:
                cy, cx = stack.pop()
                size += 1
                for ny, nx in ((cy + 1, cx), (cy - 1, cx), (cy, cx + 1), (cy, cx - 1)):
                    inside = 0 <= ny < len(rows) and 0 <= nx < len(rows[0])
                    if inside and rows[ny][nx] == "#" and (ny, nx) not in seen:
                        seen.add((ny, nx))
                        stack.append((ny, nx))
            sizes.append(size)
    return sizes

print("components found:", len(components(mask)), "| their sizes:", components(mask))

A person would say this mask shows four things: a square at the top, a vertical bar on the right, a small shape at the lower left and a block at the bottom. The flood fill finds three. The bar's lowest pixel sits directly on the block's top row, and anything that touches is one component, here of 10 pixels.

That is the limit of the method. It separates objects only when background lies between them. Cells that press together, people in a crowd and stacked boxes all merge, which is why instance segmentation models predict the separation directly.

Worked example

Worked example: from a noisy image to a count

Put the pieces together the way a lab would to count cells: label each pixel using a small window, clean the mask by dropping specks, split it into regions and report the count and the sizes.

Uses the setup above, 17 lines.

import numpy as np

g = np.random.default_rng(3)
yy, xx = np.mgrid[0:40, 0:60]
truth = np.zeros((40, 60), dtype=bool)
# (row, column, radius)
cells = [(8, 10, 4), (10, 30, 5), (28, 14, 5), (30, 44, 6), (16, 50, 3)]
for cy, cx, r in cells:
    truth |= (yy - cy) ** 2 + (xx - cx) ** 2 <= r * r
image = np.where(truth, 0.7, 0.3) + g.normal(0, 0.25, size=truth.shape)

def window_mean(img, k):
    padded, out = np.pad(img, k // 2, mode="edge"), np.zeros_like(img)
    for dy in range(k):
        for dx in range(k):
            out += padded[dy:dy + img.shape[0], dx:dx + img.shape[1]]
    return out / (k * k)

def regions(mask):
    seen, found = np.zeros_like(mask), []
    for y, x in zip(*np.where(mask)):
        if seen[y, x]:
            continue
        stack, pixels = [(y, x)], []
        seen[y, x] = True
        while stack:
            cy, cx = stack.pop()
            pixels.append((cy, cx))
            for ny, nx in ((cy + 1, cx), (cy - 1, cx), (cy, cx + 1), (cy, cx - 1)):
                inside = 0 <= ny < mask.shape[0] and 0 <= nx < mask.shape[1]
                if inside and mask[ny, nx] and not seen[ny, nx]:
                    seen[ny, nx] = True
                    stack.append((ny, nx))
        found.append(pixels)
    return found

for name, mask in (("each pixel on its own", image > 0.5),
                   ("a 5 by 5 window", window_mean(image, 5) > 0.5)):
    found = regions(mask)
    # anything under 10 pixels is treated as a speck
    kept = [r for r in found if len(r) >= 10]
    iou = (mask & truth).sum() / (mask | truth).sum()
    print(name.ljust(21), "| IoU", round(float(iou), 3), "| regions found:",
          str(len(found)).rjust(3), "| of 10 pixels or more:", len(kept),
          "| their sizes:", sorted((len(r) for r in kept), reverse=True))
print("true cells:", len(cells), "| true sizes:", sorted(
    (int(((yy - cy) ** 2 + (xx - cx) ** 2 <= r * r).sum()) for cy, cx, r in cells),
    reverse=True))
In the lesson you can change this and run it.

Judged pixel by pixel, the mask is a mess: an IoU of 0.358 and 266 separate regions, nearly all of them specks of noise. Dropping the regions under 10 pixels leaves six, one more than there are cells, and the real ones are ragged and smaller than the truth. With a 5 by 5 window the same threshold gives an IoU of 0.796 and exactly five regions, with sizes of 110, 80, 78, 50 and 15 against true sizes of 113, 81, 81, 49 and 29. The count is right and four of the sizes are within a few pixels. The smallest cell, with a radius of 3, comes out at about half its true size: it is the one a 5 by 5 window blurs most, as the experiment on window sizes predicted. A count taken from a mask is only as good as the mask, and its errors land on the smallest objects first.

Drill

IoU and Dice for two masks

Write mask_scores(pred, truth). Each mask is a list of strings of the same size, where # marks an object pixel. Return a tuple (iou, dice), each rounded to 3 places. If both masks are empty, the prediction is perfectly right: return (1.0, 1.0).

Uses the setup above, 17 lines.

def mask_scores(pred, truth):
    return (0.0, 0.0)


print(mask_scores([".##.", ".##.", "...."], ["##..", "##..", "...."]))
In the lesson, your code is checked as you write it.
Worked example

Shrink and enlarge in PyTorch

The same round trip on a small disk, written as tensor reshapes: the mean over each f by f block, then each value copied back into a block.

Uses the setup above, 17 lines.

import torch

yy = torch.arange(32).view(32, 1)
xx = torch.arange(32).view(1, 32)
disk = ((yy - 16) ** 2 + (xx - 16) ** 2 <= 9).float()
for f in (2, 4, 8):
    n = 32 // f
    small = disk.view(n, f, n, f).mean(dim=(1, 3)) >= 0.5
    back = small.float().view(n, 1, n, 1).expand(n, f, n, f).reshape(32, 32)
    both = (back * disk).sum()
    either = ((back + disk) > 0).float().sum()
    print(f, tuple(small.shape), round(float(both / either), 3))
In the lesson you can change this and run it.

The coarse grid is (16, 16), then (8, 8), then (4, 4), and the IoU of the round trip is 0.743, 0.324 and 0. The small disk, radius 3, survives a coarse grid of 16 and is half gone at 8. It matches the NumPy cell above. In a network the enlarging is learned or interpolated, and the skip connections bring back the detail that this loss throws away.

Drill

Find the connected regions of a mask

Write blob_sizes(rows). rows is a list of equal-length strings, where # marks an object pixel. Two object pixels belong to the same region when one is directly above, below, left or right of the other, and regions chain: if A touches B and B touches C, all three are one region. Pixels that touch only at a corner are not connected. Return the sizes of the regions, largest first.

Uses the setup above, 17 lines.

def blob_sizes(rows):
    return []


print(blob_sizes(["##..#", "#...#", "..#.."]))
In the lesson, your code is checked as you write it.
In your own words
What do the encoder, the decoder and the skip connections of a U-Net each contribute?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: from a mask to a product

The ideas above are what you read in any segmentation report: the mask's size is the image's size, the score is IoU or Dice per class, and the last step is a count.

Where this shows up in an engineer's week

On the job: ask for the score per class, not the average over pixels, and look at the smallest class first. Check the resolution the network works at: a thin or small object can vanish in the coarse grid, as the cells above showed. For counts, look at how touching objects are split, since connected regions merge anything that touches.

Goes deeper in

In your own words
You are asked to count the cells in microscope images and report each one's area. Which outputs do you need from a model, and what can go wrong at each step?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 9 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

A model segments tumors, which cover about 2% of each scan. It reports a pixel accuracy of 98.5% and an IoU of 0.31 for the tumor class. How good is it?

  1. Weak: its tumor region mostly misses the true one
  2. Excellent: it labels 98.5% of the pixels correctly
  3. Unknown: the two numbers contradict each other
  4. Good: an IoU above 0.3 is high for a small object

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write