Self-supervised learning: labels from the data itself
Pretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.
What it covers6
Why it matters
Labels are the expensive part of machine learning. Photos, audio and text exist by the billion, and almost none of them come with the answer attached. Self-supervised learning is the set of methods that train on the raw data anyway, by setting the model a task whose answer is already inside the data. It is how language models are pretrained, and it is how the vision encoders behind today's image models are made. This lesson builds one such method on images, measures what it buys when labels are scarce, and then breaks it in two instructive ways.
What you'll be able to do
- Explain how a model can be trained on images with no labels, by inventing a task from the images themselves
- Pretrain an encoder by matching two augmented views of each image, and test it with a linear probe
- Show that the augmentations decide what the encoder keeps and what it throws away
- Recognize collapse, and say why matching needs something to push against
- Compare contrastive pretraining with filling in masked parts, and say what each keeps
Show the answer
Pretrain on the million with no labels, then fit on the 40
The million photos can teach a model what photos are like: which differences matter and which don't. That takes no labels if the task is built from the photos themselves. The 40 labels then only have to name the groups the encoder has already learned to tell apart. This is pretraining, followed by a small supervised step.
The idea: labels from the data itself
This rests on Image-text models in depth: retrieval, negatives and limits, where a table of scores pulled matching pairs together, and on Logistic regression for the linear probe.
The label problem
Supervised learning needs an answer for every example. This lesson's task is to sort photos into 10 classes. Each photo is described by 48 numbers, and those numbers mix two things: what the photo is of, and how it was taken, the lighting, the framing and the color cast. The second part is large and has nothing to do with the class.
Start with the plain supervised approach: a classifier trained on raw pixels, with a growing number of labeled photos.
Setup (runs before every cell) · 101 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F from sklearn.linear_model import LogisticRegression g0 = np.random.default_rng(0) CLASSES = 10 centers = g0.normal(size=(CLASSES, 6)).astype(np.float32) # how the subject of a photo shows up in its pixels P = g0.normal(size=(6, 48)).astype(np.float32) # how lighting, framing and color cast show up Q = g0.normal(size=(6, 48)).astype(np.float32) def photos(n, seed): """n photos, described by what is in them (content), how they were taken (style), and a class label.""" g = np.random.default_rng(seed) label = g.integers(0, CLASSES, size=n) content = centers[label] + 0.4 * g.normal(size=(n, 6)).astype(np.float32) style = 2.0 * g.normal(size=(n, 6)).astype(np.float32) return content, style, label def render(content, style, g): return (content @ P + style @ Q + 0.3 * g.normal(size=(len(content), 48)).astype(np.float32)) def probe(features, n_labels, target="class"): """Fit a logistic regression on n_labels labeled photos, using the given features. Test on 2,000 others.""" g = np.random.default_rng(5) (c, s, y), (ct, st, yt) = photos(n_labels, 10), photos(2000, 11) X, X_test = render(c, s, g), render(ct, st, g) if target == "bright": # a different question: was the photo taken in bright light? y, yt = (s[:, 0] > 0).astype(int), (st[:, 0] > 0).astype(int) model = LogisticRegression(max_iter=2000).fit(features(X), y) return round(float(model.score(features(X_test), yt)), 3) def unit(v): return v / v.norm(dim=-1, keepdim=True) def new_encoder(): return nn.Sequential(nn.Linear(48, 64), nn.ReLU(), nn.Linear(64, 16)) def pretrain(loss_kind="contrastive", keep_brightness=False, steps=300, batch=128): """Train an encoder on 4,096 photos with no labels. Each step shows it two views of each photo in the batch.""" torch.manual_seed(0) encoder, g = new_encoder(), np.random.default_rng(0) opt = torch.optim.Adam(encoder.parameters(), lr=3e-3) content, style, _ = photos(4096, 1) losses = [] for step in range(steps): idx = g.permutation(4096)[:batch] # an augmentation: the same subject, under new lighting, framing and color style_a = 2.0 * g.normal(size=(batch, 6)).astype(np.float32) style_b = 2.0 * g.normal(size=(batch, 6)).astype(np.float32) if keep_brightness: # a gentler augmentation that leaves brightness as it was style_a[:, 0] = style_b[:, 0] = style[idx, 0] a = encoder(torch.tensor(render(content[idx], style_a, g))) b = encoder(torch.tensor(render(content[idx], style_b, g))) if loss_kind == "contrastive": scores = unit(a) @ unit(b).T / 0.1 # view a of photo i goes with view b of photo i labels = torch.arange(batch) loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2 else: # "agree": pull the two views together, and nothing else loss = ((a - b) ** 2).mean() opt.zero_grad() loss.backward() opt.step() losses.append(loss.item()) return encoder, losses def encoded_by(encoder): def features(X): with torch.no_grad(): return encoder(torch.tensor(X)).numpy() return features def pretrain_masked(steps=300, batch=128): """Hide half of each photo's pixels and train an encoder and decoder to fill them back in.""" torch.manual_seed(0) encoder, decoder, g = new_encoder(), nn.Linear(16, 48), np.random.default_rng(0) opt = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()), lr=3e-3) content, style, _ = photos(4096, 1) X = render(content, style, g) for step in range(steps): x = X[g.permutation(4096)[:batch]] hidden = g.random(size=x.shape) < 0.5 guess = decoder( encoder(torch.tensor(np.where(hidden, 0.0, x).astype(np.float32)))) loss = (((guess - torch.tensor(x)) ** 2) * torch.tensor(hidden.astype(np.float32))).sum() / float(hidden.sum()) opt.zero_grad() loss.backward() opt.step() return encoder
pixels = lambda X: X # use the raw pixel values as the features print("labeled photos | accuracy of a classifier trained on raw pixels") for n in (20, 40, 160, 640): print(str(n).rjust(14), "|", probe(pixels, n))
With 640 labels the classifier reaches 0.933. With 40 it gets 0.494, and with 20 only 0.318. The pixels vary so much with lighting and framing that a handful of examples can't show the classifier which variations to ignore.
Labels cost money and time, since a person has to look at every example. Unlabeled photos cost nothing. The question for the rest of the lesson is how to learn "which variations to ignore" from them.
classes = 10 print([labels // classes for labels in (20, 40, 640)])
Show the answer
[2, 4, 64]
Twenty labels over ten classes is two examples of each class, and forty is four. A classifier asked to learn what a class looks like from two or four noisy photos has very little to go on, which is why the first rows of the table are so low.
Two views of the same photo
Here is the invented task. Take a photo and make two altered copies of it: crop it differently, change the brightness, shift the colors. These alterations are called augmentations, and the two copies are two views. Whatever the photo shows, both views show it too.
Now train an encoder so that the two views of one photo get similar vectors, and views of different photos get different ones. No label is needed: the "right answer" for a view is its sibling, and the data supplies that for free.
This is the contrastive loss from the image-text lessons with both sides being images. A view's positive is the other view of the same photo. Its negatives are the views of every other photo in the batch. The method is known as SimCLR.
After pretraining comes the test. Freeze the encoder, run the labeled photos through it, and fit a plain logistic regression on its output vectors. This is a linear probe: if a simple classifier does well on the encoder's features, the encoder has already done the hard part.
Uses the setup above, 101 lines.
torch.manual_seed(0) untrained = new_encoder() encoder, losses = pretrain() print("contrastive loss: first step", round(losses[0], 2), "| last step", round(losses[-1], 2)) print("labeled photos | raw pixels | untrained encoder | pretrained encoder") for n in (20, 40, 160, 640): print(str(n).rjust(14), "|", str(probe(lambda X: X, n)).ljust(10), "|", str(probe(encoded_by(untrained), n)).ljust(17), "|", probe(encoded_by(encoder), n))
The encoder never saw a label. Trained on 4,096 unlabeled photos, it lifts the 40-label classifier from 0.494 to 0.674, and the 20-label one from 0.318 to 0.431.
Two things keep that honest. The untrained encoder does worse than raw pixels, so the gain comes from the pretraining and not from the network's shape. And by 160 labels the advantage is gone: 0.885 against 0.886. Pretraining buys label efficiency. It matters most when labels are few, which is the usual case.
Show the answer
The positive is the other view of the same photo; the negatives are views of the other photos
The pairing comes from how the batch was built, not from similarity or labels: two views were made from each photo, so each view knows its sibling. Every other view in the batch is treated as a wrong answer. No labels and no captions are involved.
n = 4 positives = [(i + n) % (2 * n) for i in range(2 * n)] print(positives)
Show the answer
[4, 5, 6, 7, 0, 1, 2, 3]
Row 0 is view a of photo 0, and view b of photo 0 sits 4 rows later, in row 4. Row 5 is view b of photo 1, whose sibling is row 1. Each row's positive is 4 rows away, wrapping around.
Positives and negatives in a batch of views
A batch is built from n photos by stacking view a of every photo, then view b of every photo, giving 2n rows. Write view_pairs(n). Return a tuple of two things: a list with one entry per row, holding the row number of that row's positive, and the number of negatives each row has (every other row except itself and its positive). For n of 0, return ([], 0).
Uses the setup above, 101 lines.
def view_pairs(n): return [], 0 print(view_pairs(4))
The classifier is the same each time. Only the features it reads changed.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: augmentations, collapse and filling in
The augmentations decide what is learned
The encoder was asked for one thing: give two views of a photo the same vector. So whatever differs between the views is, by instruction, something to ignore. The augmentations are where you tell the model what doesn't matter.
That cuts both ways. Below, the probe asks two questions of the same features: which class is the photo, and a different one, was it taken in bright light? Brightness is one of the things the augmentations changed. A second encoder is then pretrained with a gentler augmentation that leaves brightness alone.
Uses the setup above, 101 lines.
print("features " "| which class? (40 labels) | bright or dim? (640 labels)") print("raw pixels |", str(probe(lambda X: X, 40)).ljust(24), "|", probe(lambda X: X, 640, "bright")) encoder, _ = pretrain() print("pretrained, views differ in everything |", str(probe(encoded_by(encoder), 40)).ljust(24), "|", probe(encoded_by(encoder), 640, "bright")) encoder, _ = pretrain(keep_brightness=True) print("pretrained, views share brightness |", str(probe(encoded_by(encoder), 40)).ljust(24), "|", probe(encoded_by(encoder), 640, "bright"))
Raw pixels answer the brightness question almost perfectly, 0.987. The pretrained encoder scores 0.521 on it, which is a coin flip. Brightness was removed from the features, on purpose and completely, since it was one of the things that differed between views.
The second encoder, whose views kept the photo's brightness, answers the brightness question well, 0.932, and is worse at the class question with few labels, 0.553 against 0.674. It kept something the class doesn't need, and the probe has to learn to look past it.
So the choice of augmentations is a claim about your task. Color jitter says "color doesn't matter", which is right for telling a cat from a dog and wrong for telling a ripe banana from a green one. An encoder can't be asked later for something it was trained to ignore.
Show the answer
Pretraining taught the encoder to ignore color
Two views with different colors had to get the same vector, so the encoder learned to discard color. Ripeness is mostly a matter of color, so the information the probe needs was removed before the probe ever ran. The augmentations have to leave alone whatever the later task depends on.
Collapse: agreeing about nothing
Why does the loss need negatives at all? The goal is for two views to agree, so why not train for agreement alone, and make the two vectors as close as possible?
Because there is a way to agree perfectly that involves learning nothing: give every photo the same vector. Two views of a photo then match exactly, the loss is zero, and the encoder is worthless. This failure is called collapse. The cell below trains for agreement alone and watches how spread out the features are across 500 photos.
Uses the setup above, 101 lines.
test_photos = render(*photos(500, 3)[:2], np.random.default_rng(0)) print("training goal | loss: first, last " "| spread of the features | which class? (40 labels)") for loss_kind, name in (("contrastive", "match, among the batch"), ("agree", "agree, and nothing else")): encoder, losses = pretrain(loss_kind) spread = encoded_by(encoder)(test_photos).std(axis=0).mean() print(name.ljust(24), "|", (str(round(losses[0], 3)) + ", " + str(round(losses[-1], 4))).ljust(18), "|", str(round(float(spread), 3)).ljust(22), "|", probe(encoded_by(encoder), 40))
Training for agreement alone drives its loss from 1.79 to 0.0018, a far better number than the contrastive loss reaches. It got there by shrinking: the spread of its features is 0.126, a ninth of the contrastive encoder's 1.126, and still falling. A probe on those features scores 0.372 with 40 labels, worse than raw pixels.
A loss can only be trusted to mean what you hope when the easy, useless ways of lowering it are closed off. In contrastive training the negatives do that: an encoder that says the same thing about every photo fails every comparison. Other self-supervised methods drop the negatives and block collapse by different means, and every one of them needs some such device.
Show the answer
The encoder has collapsed to a single output vector
A loss near zero with features that don't vary means the encoder found the shortcut: map every image to the same point, so any two views agree. It satisfies the letter of the task and encodes nothing. Watching the spread of the features during pretraining is the standard check.
Detect collapse
Write spread(features) and collapsed(features, floor). features is a list of rows, one per image, each a list of numbers. spread returns the standard deviation of each column (divide by the number of rows, not by one less), averaged over the columns and rounded to 4 places, or 0.0 if there are no rows. collapsed returns True when the spread is below floor.
Uses the setup above, 101 lines.
def spread(features): return 0.0 def collapsed(features, floor): return False healthy = [[1.0, -2.0], [-1.0, 2.0], [1.0, 2.0], [-1.0, -2.0]] shrunk = [[0.51, 0.5], [0.49, 0.5], [0.5, 0.5], [0.5, 0.5]] print(spread(healthy), spread(shrunk), collapsed(healthy, 0.1), collapsed(shrunk, 0.1))
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: filling in what is hidden
Matching views is one way to invent a task. The other large family hides part of the input and asks the model to fill it in. Language models are pretrained this way, predicting hidden or upcoming words. For images the method is called masked image modeling: hide a large share of the patches and train a model to reconstruct them.
The cell below pretrains an encoder by hiding half of each photo's 48 numbers and predicting them from the rest, then probes it beside the contrastive encoder.
Uses the setup above, 101 lines.
contrastive, _ = pretrain() masked = pretrain_masked() print("features | which class? 40 labels | " "640 labels | bright or dim? (640 labels)") for name, features in (("raw pixels", lambda X: X), ("contrastive", encoded_by(contrastive)), ("masked, filled in", encoded_by(masked))): print(name.ljust(20), "|", str(probe(features, 40)).ljust(22), "|", str(probe(features, 640)).ljust(10), "|", probe(features, 640, "bright"))
The two encoders learned opposite things. The contrastive one is good at the class with few labels, 0.674, and blind to brightness, 0.521. The masked one is the reverse: 0.415 on the class with 40 labels, below raw pixels, and 0.971 on brightness.
That follows from the tasks. To fill in hidden pixels, a model has to keep whatever explains the pixels, and in these photos lighting and framing explain most of them. Nothing told it that they don't matter. The contrastive task has that knowledge built in through its augmentations.
Neither is better in general. Matching views gives features that a simple probe can use directly, at the price of deciding in advance what to throw away. Filling in keeps everything, needs no augmentations, and tends to pay off when the whole model is fine-tuned on the task afterward instead of probed.
Go deeper Names you will meet
- SimCLR and MoCo match augmented views against negatives, as in this lesson.
- BYOL, SimSiam and DINO match views with no negatives, and avoid collapse by other means, such as updating one of two copies of the network slowly.
- Masked autoencoders (MAE) hide most of an image's patches and reconstruct them with a vision transformer.
- CLIP is the same idea with a caption as the second view.
Show the answer
Filling in pixels requires keeping what explains them; matching views discards what differs between views
Each task rewards a different kind of memory. To reconstruct hidden pixels the model must hold on to lighting, since lighting changes many pixels at once. To match two views that were lit differently, the model must let go of lighting. The features show what each task asked for.
Worked example: how many labels is the pretraining worth?
A concrete way to price pretraining: find how many labeled photos a classifier on raw pixels needs before it matches the pretrained encoder with 40.
Uses the setup above, 101 lines.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F from sklearn.linear_model import LogisticRegression g0 = np.random.default_rng(0) CLASSES = 10 centers = g0.normal(size=(CLASSES, 6)).astype(np.float32) # how the subject of a photo shows up in its pixels P = g0.normal(size=(6, 48)).astype(np.float32) # how lighting, framing and color cast show up Q = g0.normal(size=(6, 48)).astype(np.float32) def photos(n, seed): """n photos, described by what is in them (content), how they were taken (style), and a class label.""" g = np.random.default_rng(seed) label = g.integers(0, CLASSES, size=n) content = centers[label] + 0.4 * g.normal(size=(n, 6)).astype(np.float32) style = 2.0 * g.normal(size=(n, 6)).astype(np.float32) return content, style, label def render(content, style, g): return (content @ P + style @ Q + 0.3 * g.normal(size=(len(content), 48)).astype(np.float32)) def probe(features, n_labels, target="class"): """Fit a logistic regression on n_labels labeled photos, using the given features. Test on 2,000 others.""" g = np.random.default_rng(5) (c, s, y), (ct, st, yt) = photos(n_labels, 10), photos(2000, 11) X, X_test = render(c, s, g), render(ct, st, g) if target == "bright": # a different question: was the photo taken in bright light? y, yt = (s[:, 0] > 0).astype(int), (st[:, 0] > 0).astype(int) model = LogisticRegression(max_iter=2000).fit(features(X), y) return round(float(model.score(features(X_test), yt)), 3) def unit(v): return v / v.norm(dim=-1, keepdim=True) def new_encoder(): return nn.Sequential(nn.Linear(48, 64), nn.ReLU(), nn.Linear(64, 16)) def pretrain(loss_kind="contrastive", keep_brightness=False, steps=300, batch=128): """Train an encoder on 4,096 photos with no labels. Each step shows it two views of each photo in the batch.""" torch.manual_seed(0) encoder, g = new_encoder(), np.random.default_rng(0) opt = torch.optim.Adam(encoder.parameters(), lr=3e-3) content, style, _ = photos(4096, 1) losses = [] for step in range(steps): idx = g.permutation(4096)[:batch] # an augmentation: the same subject, under new lighting, framing and color style_a = 2.0 * g.normal(size=(batch, 6)).astype(np.float32) style_b = 2.0 * g.normal(size=(batch, 6)).astype(np.float32) if keep_brightness: # a gentler augmentation that leaves brightness as it was style_a[:, 0] = style_b[:, 0] = style[idx, 0] a = encoder(torch.tensor(render(content[idx], style_a, g))) b = encoder(torch.tensor(render(content[idx], style_b, g))) if loss_kind == "contrastive": scores = unit(a) @ unit(b).T / 0.1 # view a of photo i goes with view b of photo i labels = torch.arange(batch) loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2 else: # "agree": pull the two views together, and nothing else loss = ((a - b) ** 2).mean() opt.zero_grad() loss.backward() opt.step() losses.append(loss.item()) return encoder, losses def encoded_by(encoder): def features(X): with torch.no_grad(): return encoder(torch.tensor(X)).numpy() return features encoder, _ = pretrain() target = probe(encoded_by(encoder), 40) print("pretrained encoder, 40 labels:", target) for n in (40, 60, 80, 100, 120, 160): score = probe(lambda X: X, n) print("raw pixels,", str(n).rjust(3), "labels:", str(score).ljust(5), "<- catches up" if score >= target else "")
The pretrained encoder with 40 labels scores 0.674. A classifier on raw pixels has 0.494 at 40 labels and passes that mark at 60, with 0.755. So in this small problem the pretraining was worth about 20 labels: half again as many as were on hand. That is a modest price, for an honest reason. These photos are simple enough that a linear classifier on raw pixels learns quickly once it has seen a few examples per class. On real photographs raw pixels are a far worse starting point, and the same comparison comes out much more lopsided. The method of the comparison carries over: fix a score, and count the labels each route needs to reach it.
On the job: the practical question is how many labels you can skip. Train the probe on 40, 160 and 640 labels with and without pretrained features, as in the table above, and read where the curves meet. If pretraining only helps below 100 labels and you have 5,000, a plain model may be enough. If it helps at every size, it is worth the extra step.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Image-text models in depth: retrieval, negatives and limits: the same pull-together, push-apart loss with captions in place of a second view.
- Build a vision transformer: the encoder most self-supervised image models use today.
- Tuning an embedding model for retrieval: adapting an encoder's features to your own task.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
What makes a learning method "self-supervised"?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write