Segmentation: a label for every pixel
Masks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.
What it covers8
Why it matters
A box says roughly where an object is. Some jobs need its exact outline: the area of a tumor on a scan, the edge of the road in front of a car, the part of a photo to keep when the background is removed. That is segmentation, and its answer is a label for every pixel. This lesson covers what a mask is, how to grade one honestly, the tension at the heart of every segmentation network, between seeing enough of the picture and keeping track of exactly where things are, and how to get from a mask back to countable objects.
What you'll be able to do
- Describe a segmentation mask, and tell semantic segmentation from instance segmentation
- Score a mask with IoU and Dice, and say why pixel accuracy misleads
- Explain why each pixel's label needs context, and why too much context blurs small things
- Describe the shrink-then-enlarge design of segmentation networks and what skip connections restore
- Turn a mask into separate objects by finding connected regions, and say when that fails
Show the answer
Very little: marking no road at all would also score 95%
When the thing you care about covers a small share of the image, a model that marks nothing is right almost everywhere. Pixel accuracy is dominated by the background. This lesson uses measures that look only at the object: the overlap between the predicted and the true region.
The idea: a label for every pixel
This rests on Object detection: boxes, overlap and suppression, whose IoU and average precision are reused here on regions of pixels, and on Convolutional networks.
A label for every pixel
A mask is an array the same height and width as the image, holding one class number per pixel. Where a detector says "a dog, somewhere in this rectangle", a segmentation model says which pixels are dog.
The task comes in two main kinds.
- Semantic segmentation gives each pixel a class: road, car, sky. Two cars side by side become one region of "car".
- Instance segmentation also separates the objects: this pixel belongs to car 1, that one to car 2.
The cell below builds a small mask by hand: a disk (class 1) and a tall bar (class 2) on a background (class 0). It prints every second row, then compares each object's mask with its bounding box.
Setup (runs before every cell) · 17 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np def scene(seed): g = np.random.default_rng(seed) y, x = np.mgrid[0:32, 0:32] mask = np.zeros((32, 32), dtype=bool) for _ in range(3): cy, cx, r = g.integers(6, 26), g.integers(6, 26), g.integers(3, 7) mask |= (y - cy) ** 2 + (x - cx) ** 2 <= r * r return np.where(mask, 0.65, 0.35) + g.normal(0, 0.35, size=(32, 32)), mask def window_mean(img, k): padded, out = np.pad(img, k // 2, mode="edge"), np.zeros_like(img) for dy in range(k): for dx in range(k): out += padded[dy:dy + 32, dx:dx + 32] return out / (k * k)
import numpy as np yy, xx = np.mgrid[0:24, 0:32] # the row and column number of every pixel mask = np.zeros((24, 32), dtype=int) mask[(yy - 11) ** 2 + (xx - 9) ** 2 <= 36] = 1 # a disk of radius 6 mask[(yy >= 4) & (yy < 20) & (xx >= 22) & (xx < 25)] = 2 # a bar, 3 wide and 16 tall for row in mask[::2]: print("".join(".o#"[v] for v in row)) for k, name in ((1, "disk"), (2, "bar")): ys, xs = np.where(mask == k) box_area = (xs.max() + 1 - xs.min()) * (ys.max() + 1 - ys.min()) pixels = int((mask == k).sum()) print(name, "| pixels in its mask:", pixels, "| pixels in its box:", int(box_area), "| share of the box that is object:", round(pixels / box_area, 2)) print("share of the image that is background:", round(float((mask == 0).mean()), 3))
The bar fills its box completely, so a box describes it perfectly. The disk fills 0.67 of its box: a third of the rectangle around it is something else. For a diagonal object, such as a pen lying across a desk, a box is nearly all background, and only a mask says where the object is.
Note the last line too. Even in this small picture, 0.79 of the pixels are background. In real images the share is often higher, and that shapes how masks have to be graded.
mask = ["....#...", "...###..", "..#####.", "...###..", "....#..."] pixels = sum(row.count("#") for row in mask) columns = [x for row in mask for x, c in enumerate(row) if c == "#"] rows = [y for y, row in enumerate(mask) if "#" in row] box = (max(columns) - min(columns) + 1) * (max(rows) - min(rows) + 1) print(pixels, box, pixels / box)
Show the answer
13 25 0.52
The diamond has 1 + 3 + 5 + 3 + 1 = 13 pixels. It spans columns 2 to 6 and rows 0 to 4, so its box is 5 by 5 = 25. The diamond fills about half of its own bounding box.
Scoring a mask
The obvious score is pixel accuracy: the share of pixels given the right class. The opening question showed its flaw, and the cell below measures it. The truth is the disk. One prediction is the right shape, 4 pixels too far right. The other predicts no object at all.
The measures that work are the ones from the detection lesson, applied to regions instead of boxes. IoU is the number of pixels in both regions divided by the number in either. Dice counts the shared pixels twice and divides by the two regions' sizes added together.
In plain words: is the number of pixels in both masks, the number in at least one, and and the sizes of the two masks. Both scores are 1 for a perfect match and 0 for no overlap.
Uses the setup above, 17 lines.
import numpy as np yy, xx = np.mgrid[0:24, 0:32] truth = (yy - 11) ** 2 + (xx - 9) ** 2 <= 36 # the disk, as True and False shifted = (yy - 11) ** 2 + (xx - 13) ** 2 <= 36 # the same disk, 4 pixels to the right nothing = np.zeros_like(truth) # no object predicted anywhere def scores(pred, truth): both, either = (pred & truth).sum(), (pred | truth).sum() accuracy = (pred == truth).mean() return (round(float(accuracy), 3), round(float(both / either), 3), round(float(2 * both / (pred.sum() + truth.sum())), 3)) print("prediction | pixel accuracy, IoU, Dice") for name, pred in (("exact", truth), ("shifted 4 pixels", shifted), ("no object at all", nothing)): print(name.ljust(25), "|", scores(pred, truth))
The prediction that finds nothing scores 0.853 on pixel accuracy, nearly the same as the shifted disk's 0.88. IoU tells them apart: 0.421 against 0.0.
Dice is always at least as large as IoU on the same masks (0.593 here), and the two always rank predictions in the same order. Medical imaging tends to report Dice, and general vision benchmarks report mean IoU: the IoU of each class, averaged over the classes, so that a small class counts as much as the background.
Show the answer
One third
With each box's area as 1, the shared part is 0.5 and the union is 1 + 1 − 0.5 = 1.5, so the IoU is 1/3. Masks behave the same way: the union grows as the overlap shrinks, so IoU falls faster than the eye expects.
truth = ["##..", "##..", "...."] pred = [".##.", ".##.", "...."] both = sum(t == "#" and p == "#" for tr, pr in zip(truth, pred) for t, p in zip(tr, pr)) either = sum(t == "#" or p == "#" for tr, pr in zip(truth, pred) for t, p in zip(tr, pr)) print(both, either, round(both / either, 3), 2 * both / (4 + 4))
Show the answer
2 6 0.333 0.5
The two squares share one column of 2 pixels. Together they cover 4 + 4 − 2 = 6 pixels. IoU is 2 / 6 = 0.333, and Dice is 2 × 2 / (4 + 4) = 0.5. Dice is the larger of the two, as always.
The numbers behind the picture
The same two masks as arrays: pixels in both, pixels in either, IoU, Dice and pixel accuracy.
Uses the setup above, 17 lines.
import numpy as np truth = np.zeros((4, 5), bool) truth[1:3, 1:4] = True pred = np.zeros((4, 5), bool) pred[1:3, 2:5] = True both, either = (pred & truth).sum(), (pred | truth).sum() print("in both:", int(both), " in either:", int(either)) print("IoU:", round(float(both / either), 2)) print("Dice:", round(float(2 * both / (pred.sum() + truth.sum())), 2)) print("pixel accuracy:", round(float((pred == truth).mean()), 2))
Four pixels are in both masks and eight in either: IoU 0.5. Dice counts the shared pixels twice, 8 over 12, so 0.67. Pixel accuracy is 0.8, because the 8 background pixels the two masks share also count as right: it looks good even though the masks overlap by half.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: context, shrinking and regions
How much of the picture a pixel needs to see
A segmentation model is a classifier that runs at every pixel. The question is what it gets to look at when it decides. One pixel's brightness is rarely enough: real images are noisy, and "gray" could be road or roof.
The experiment below makes that concrete. Each image holds a few slightly brighter disks buried in noise. A logistic regression labels each pixel as object or background from one number: the average brightness of the square window around that pixel. Only the window's size changes. In a convolutional network this region is called the receptive field.
Uses the setup above, 17 lines.
import numpy as np import matplotlib.pyplot as plt from sklearn.linear_model import LogisticRegression train, test = [scene(s) for s in range(20)], [scene(s) for s in range(100, 120)] shown = {} for k in (1, 3, 5, 9, 15): model = LogisticRegression().fit( np.concatenate([window_mean(img, k).reshape(-1, 1) for img, m in train]), np.concatenate([m.reshape(-1) for img, m in train])) ious, accuracies = [], [] for img, m in test: pred = model.predict(window_mean(img, k).reshape(-1, 1)).reshape(32, 32) ious.append((pred & m).sum() / (pred | m).sum()) accuracies.append((pred == m).mean()) shown[k] = model.predict(window_mean(test[0][0], k).reshape(-1, 1)).reshape(32, 32) print("window", str(k).rjust(2), "by", str(k).ljust(2), "| pixel accuracy", str(round(float(np.mean(accuracies)), 3)).ljust(5), "| IoU", round(float(np.mean(ious)), 3)) fig, axes = plt.subplots(1, 5, figsize=(12, 2.8)) panels = [("the image", test[0][0]), ("the truth", test[0][1]), ("window 1", shown[1]), ("window 5", shown[5]), ("window 15", shown[15])] for ax, (title, picture) in zip(axes, panels): ax.imshow(picture, cmap="gray") ax.set_title(title) ax.axis("off")
A model that sees one pixel is nearly useless: an IoU of 0.082, with a pixel accuracy of 0.818 that would have hidden the failure. Averaging over a 5 by 5 window cancels much of the noise and lifts the IoU to 0.638.
Then it turns. At 15 by 15 the IoU is back down to 0.418. A window that large averages a small disk together with the background around it, and its edges blur into a smooth blob. The figure shows both failures: speckle on the left, a soft outline on the right.
This is the central tension of segmentation. Context says what a pixel is, and wants a large view. Location says exactly where the edge runs, and wants a small one.
Show the answer
It averaged small objects and their edges into the background
Averaging over a large window removes noise, and it also removes detail. A disk 6 pixels across is a minority of a 15 by 15 window, so its average barely differs from background, and every edge becomes a slow ramp. A big view knows what is nearby and loses track of exactly where it ends.
The way out is to give the classifier both views. Below, each pixel is described by several window means at once, small and large, and the same model learns how to weigh them. The table also splits the mistakes by where they occur.
Uses the setup above, 17 lines.
import numpy as np from sklearn.linear_model import LogisticRegression def features(img, sizes): # one column per window size return np.stack([window_mean(img, k).reshape(-1) for k in sizes], axis=1) train, test = [scene(s) for s in range(20)], [scene(s) for s in range(100, 120)] print("windows used | IoU | wrong near an edge | wrong elsewhere") for sizes in ((3,), (9,), (3, 9), (1, 3, 5, 9)): model = LogisticRegression().fit( np.concatenate([features(img, sizes) for img, m in train]), np.concatenate([m.reshape(-1) for img, m in train])) ious, at_edge, elsewhere = [], [], [] for img, m in test: pred = model.predict(features(img, sizes)).reshape(32, 32) ious.append((pred & m).sum() / (pred | m).sum()) blur = window_mean(m.astype(float), 3) # pixels within one step of the object's outline edge = (blur > 0.01) & (blur < 0.99) at_edge.append((pred != m)[edge].mean()) elsewhere.append((pred != m)[~edge].mean()) print(str(sizes).ljust(16), "|", str(round(float(np.mean(ious)), 3)).ljust(5), "|", str(round(float(np.mean(at_edge)), 3)).ljust(18), "|", round(float(np.mean(elsewhere)), 3))
A small window and a large one together reach 0.671, better than either alone (0.518 and 0.606), and four sizes reach 0.689. The large window says "there is an object around here", and the small one says "and this pixel is on the bright side of the edge".
The last two columns show where segmentation is hard. Away from any outline, about 2 pixels in 100 are wrong. Within a step of an outline, about 1 in 4 is. Nearly all the difficulty of segmentation is at the edges, which is also where the measurement matters.
Shrink to understand, enlarge to answer
Real segmentation networks are built around that tension. The encoder is an ordinary convolutional network: it applies filters and pools, so each stage sees a wider part of the image at a lower resolution. By the end it knows what is in the picture, on a grid perhaps 16 or 32 times coarser than the input.
A mask needs an answer for every pixel, so a decoder enlarges that coarse grid back to full size, a step called upsampling. The cell below shows what that round trip costs, with no network involved: take a perfect mask, shrink it by averaging blocks of pixels, and enlarge it again by repeating each value.
Uses the setup above, 17 lines.
import numpy as np def shrink_and_enlarge(mask, f): h, w = mask.shape # a block is object if half of it is small = mask.reshape(h // f, f, w // f, f).mean(axis=(1, 3)) >= 0.5 # each value becomes an f by f block return np.repeat(np.repeat(small, f, axis=0), f, axis=1) yy, xx = np.mgrid[0:32, 0:32] shapes = { "a disk of radius 10": (yy - 16) ** 2 + (xx - 16) ** 2 <= 100, "a disk of radius 3": (yy - 16) ** 2 + (xx - 16) ** 2 <= 9, "a line 2 pixels wide": (xx >= 15) & (xx < 17), } print("shape | IoU after shrinking by 2, by 4, by 8") for name, mask in shapes.items(): row = [] for f in (2, 4, 8): back = shrink_and_enlarge(mask, f) row.append(round(float((back & mask).sum() / (back | mask).sum()), 3)) print(name.ljust(20), "|", row)
These are perfect masks, damaged only by a trip through a coarse grid. The large disk survives a shrinking by 8 with an IoU of 0.78: its body is intact and its outline has turned to staircase. The small disk is at 0.324 after shrinking by 4 and gone after 8. The thin line is gone after 4.
Thin and small things are what a coarse grid can't hold: a blood vessel, a lane marking, a cable. The fix is the one the last experiment pointed at. A skip connection carries the encoder's early, full-resolution features straight across to the decoder, so that when the decoder enlarges its coarse answer, it has the fine detail on hand to sharpen the edges. The best-known design built this way is the U-Net, named for the shape of its diagram: down one side, up the other, with skips across.
Go deeper Other parts you will meet
- Transposed convolution and bilinear upsampling are two ways of enlarging a feature grid: the first learns how to fill in the new pixels, the second interpolates between neighbors.
- Mask R-CNN does instance segmentation by detecting boxes first, then predicting a small mask inside each box.
- Promptable models, such as Segment Anything, take a click or a box as a prompt and return the mask of whatever is there, without a fixed list of classes.
Show the answer
Thin and small objects are missing, and outlines are blocky
A grid that coarse can't represent anything much smaller than one of its cells, so thin lines and small blobs fall between them, and every outline comes back as a staircase when it is enlarged. The large shapes, which the coarse grid holds well, survive. Skip connections bring the fine detail back.
From a mask to separate objects
A semantic mask says which pixels are "cell". To count the cells, or measure each one, the mask has to be split into objects. The classic way is to find connected components: groups of object pixels that touch each other, up, down, left or right.
The method is a flood fill. Scan the mask. On reaching an object pixel that hasn't been visited, start a new component there and spread outward to every object pixel connected to it, marking each as visited.
Uses the setup above, 17 lines.
mask = ["..##....#.", "..##....#.", "........#.", ".#......#.", ".##..####.", "..#..##..."] def components(rows): seen, sizes = set(), [] for y in range(len(rows)): for x in range(len(rows[0])): if rows[y][x] != "#" or (y, x) in seen: continue stack, size = [(y, x)], 0 # a new component starts here seen.add((y, x)) while stack: cy, cx = stack.pop() size += 1 for ny, nx in ((cy + 1, cx), (cy - 1, cx), (cy, cx + 1), (cy, cx - 1)): inside = 0 <= ny < len(rows) and 0 <= nx < len(rows[0]) if inside and rows[ny][nx] == "#" and (ny, nx) not in seen: seen.add((ny, nx)) stack.append((ny, nx)) sizes.append(size) return sizes print("components found:", len(components(mask)), "| their sizes:", components(mask))
A person would say this mask shows four things: a square at the top, a vertical bar on the right, a small shape at the lower left and a block at the bottom. The flood fill finds three. The bar's lowest pixel sits directly on the block's top row, and anything that touches is one component, here of 10 pixels.
That is the limit of the method. It separates objects only when background lies between them. Cells that press together, people in a crowd and stacked boxes all merge, which is why instance segmentation models predict the separation directly.
Worked example: from a noisy image to a count
Put the pieces together the way a lab would to count cells: label each pixel using a small window, clean the mask by dropping specks, split it into regions and report the count and the sizes.
Uses the setup above, 17 lines.
import numpy as np g = np.random.default_rng(3) yy, xx = np.mgrid[0:40, 0:60] truth = np.zeros((40, 60), dtype=bool) # (row, column, radius) cells = [(8, 10, 4), (10, 30, 5), (28, 14, 5), (30, 44, 6), (16, 50, 3)] for cy, cx, r in cells: truth |= (yy - cy) ** 2 + (xx - cx) ** 2 <= r * r image = np.where(truth, 0.7, 0.3) + g.normal(0, 0.25, size=truth.shape) def window_mean(img, k): padded, out = np.pad(img, k // 2, mode="edge"), np.zeros_like(img) for dy in range(k): for dx in range(k): out += padded[dy:dy + img.shape[0], dx:dx + img.shape[1]] return out / (k * k) def regions(mask): seen, found = np.zeros_like(mask), [] for y, x in zip(*np.where(mask)): if seen[y, x]: continue stack, pixels = [(y, x)], [] seen[y, x] = True while stack: cy, cx = stack.pop() pixels.append((cy, cx)) for ny, nx in ((cy + 1, cx), (cy - 1, cx), (cy, cx + 1), (cy, cx - 1)): inside = 0 <= ny < mask.shape[0] and 0 <= nx < mask.shape[1] if inside and mask[ny, nx] and not seen[ny, nx]: seen[ny, nx] = True stack.append((ny, nx)) found.append(pixels) return found for name, mask in (("each pixel on its own", image > 0.5), ("a 5 by 5 window", window_mean(image, 5) > 0.5)): found = regions(mask) # anything under 10 pixels is treated as a speck kept = [r for r in found if len(r) >= 10] iou = (mask & truth).sum() / (mask | truth).sum() print(name.ljust(21), "| IoU", round(float(iou), 3), "| regions found:", str(len(found)).rjust(3), "| of 10 pixels or more:", len(kept), "| their sizes:", sorted((len(r) for r in kept), reverse=True)) print("true cells:", len(cells), "| true sizes:", sorted( (int(((yy - cy) ** 2 + (xx - cx) ** 2 <= r * r).sum()) for cy, cx, r in cells), reverse=True))
Judged pixel by pixel, the mask is a mess: an IoU of 0.358 and 266 separate regions, nearly all of them specks of noise. Dropping the regions under 10 pixels leaves six, one more than there are cells, and the real ones are ragged and smaller than the truth. With a 5 by 5 window the same threshold gives an IoU of 0.796 and exactly five regions, with sizes of 110, 80, 78, 50 and 15 against true sizes of 113, 81, 81, 49 and 29. The count is right and four of the sizes are within a few pixels. The smallest cell, with a radius of 3, comes out at about half its true size: it is the one a 5 by 5 window blurs most, as the experiment on window sizes predicted. A count taken from a mask is only as good as the mask, and its errors land on the smallest objects first.
IoU and Dice for two masks
Write mask_scores(pred, truth). Each mask is a list of strings of the same size, where # marks an object pixel. Return a tuple (iou, dice), each rounded to 3 places. If both masks are empty, the prediction is perfectly right: return (1.0, 1.0).
Uses the setup above, 17 lines.
def mask_scores(pred, truth): return (0.0, 0.0) print(mask_scores([".##.", ".##.", "...."], ["##..", "##..", "...."]))
Shrink and enlarge in PyTorch
The same round trip on a small disk, written as tensor reshapes: the mean over each f by f block, then each value copied back into a block.
Uses the setup above, 17 lines.
import torch yy = torch.arange(32).view(32, 1) xx = torch.arange(32).view(1, 32) disk = ((yy - 16) ** 2 + (xx - 16) ** 2 <= 9).float() for f in (2, 4, 8): n = 32 // f small = disk.view(n, f, n, f).mean(dim=(1, 3)) >= 0.5 back = small.float().view(n, 1, n, 1).expand(n, f, n, f).reshape(32, 32) both = (back * disk).sum() either = ((back + disk) > 0).float().sum() print(f, tuple(small.shape), round(float(both / either), 3))
The coarse grid is (16, 16), then (8, 8), then (4, 4), and the IoU of the round trip is 0.743, 0.324 and 0. The small disk, radius 3, survives a coarse grid of 16 and is half gone at 8. It matches the NumPy cell above. In a network the enlarging is learned or interpolated, and the skip connections bring back the detail that this loss throws away.
Find the connected regions of a mask
Write blob_sizes(rows). rows is a list of equal-length strings, where # marks an object pixel. Two object pixels belong to the same region when one is directly above, below, left or right of the other, and regions chain: if A touches B and B touches C, all three are one region. Pixels that touch only at a corner are not connected. Return the sizes of the regions, largest first.
Uses the setup above, 17 lines.
def blob_sizes(rows): return [] print(blob_sizes(["##..#", "#...#", "..#.."]))
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: from a mask to a product
The ideas above are what you read in any segmentation report: the mask's size is the image's size, the score is IoU or Dice per class, and the last step is a count.
Where this shows up in an engineer's week
On the job: ask for the score per class, not the average over pixels, and look at the smallest class first. Check the resolution the network works at: a thin or small object can vanish in the coarse grid, as the cells above showed. For counts, look at how touching objects are split, since connected regions merge anything that touches.
Goes deeper in
- Object detection: boxes, overlap and suppression: boxes instead of masks, and the matching rules for grading.
- Build a vision transformer: patches and attention in place of the shrink-then-enlarge design.
- Convolutional networks: the convolutions that the encoder and decoder are built from.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 9 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 11 review cards for spaced repetition
Try a quiz question
A model segments tumors, which cover about 2% of each scan. It reports a pixel accuracy of 98.5% and an IoU of 0.31 for the tumor class. How good is it?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write