Index Ai Academy
Phase 5 · Beyond text

Self-supervised learning: labels from the data itself

Pretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Labels are the expensive part of machine learning. Photos, audio and text exist by the billion, and almost none of them come with the answer attached. Self-supervised learning is the set of methods that train on the raw data anyway, by setting the model a task whose answer is already inside the data. It is how language models are pretrained, and it is how the vision encoders behind today's image models are made. This lesson builds one such method on images, measures what it buys when labels are scarce, and then breaks it in two instructive ways.

What you'll be able to do

  • Explain how a model can be trained on images with no labels, by inventing a task from the images themselves
  • Pretrain an encoder by matching two augmented views of each image, and test it with a linear probe
  • Show that the augmentations decide what the encoder keeps and what it throws away
  • Recognize collapse, and say why matching needs something to push against
  • Compare contrastive pretraining with filling in masked parts, and say what each keeps
Quick check
A guess before you start. You have a million photos with no labels and 40 photos with labels. Which plan makes the best use of both?
  1. Label the million by guessing, then train on all of them
  2. Train on the million with the labels of the 40 copied at random
  3. Train a classifier on the 40 and ignore the rest
  4. Pretrain on the million with no labels, then fit on the 40
Show the answer

Pretrain on the million with no labels, then fit on the 40

The million photos can teach a model what photos are like: which differences matter and which don't. That takes no labels if the task is built from the photos themselves. The 40 labels then only have to name the groups the encoder has already learned to tell apart. This is pretraining, followed by a small supervised step.

The idea: labels from the data itself

This rests on Image-text models in depth: retrieval, negatives and limits, where a table of scores pulled matching pairs together, and on Logistic regression for the linear probe.

The label problem

Supervised learning needs an answer for every example. This lesson's task is to sort photos into 10 classes. Each photo is described by 48 numbers, and those numbers mix two things: what the photo is of, and how it was taken, the lighting, the framing and the color cast. The second part is large and has nothing to do with the class.

Start with the plain supervised approach: a classifier trained on raw pixels, with a growing number of labeled photos.

Setup (runs before every cell) · 101 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.linear_model import LogisticRegression

g0 = np.random.default_rng(0)
CLASSES = 10
centers = g0.normal(size=(CLASSES, 6)).astype(np.float32)
# how the subject of a photo shows up in its pixels
P = g0.normal(size=(6, 48)).astype(np.float32)
# how lighting, framing and color cast show up
Q = g0.normal(size=(6, 48)).astype(np.float32)

def photos(n, seed):
    """n photos, described by what is in them (content), how they were taken
    (style), and a class label."""
    g = np.random.default_rng(seed)
    label = g.integers(0, CLASSES, size=n)
    content = centers[label] + 0.4 * g.normal(size=(n, 6)).astype(np.float32)
    style = 2.0 * g.normal(size=(n, 6)).astype(np.float32)
    return content, style, label

def render(content, style, g):
    return (content @ P + style @ Q
            + 0.3 * g.normal(size=(len(content), 48)).astype(np.float32))

def probe(features, n_labels, target="class"):
    """Fit a logistic regression on n_labels labeled photos, using the given
    features. Test on 2,000 others."""
    g = np.random.default_rng(5)
    (c, s, y), (ct, st, yt) = photos(n_labels, 10), photos(2000, 11)
    X, X_test = render(c, s, g), render(ct, st, g)
    if target == "bright":  # a different question: was the photo taken in bright light?
        y, yt = (s[:, 0] > 0).astype(int), (st[:, 0] > 0).astype(int)
    model = LogisticRegression(max_iter=2000).fit(features(X), y)
    return round(float(model.score(features(X_test), yt)), 3)

def unit(v):
    return v / v.norm(dim=-1, keepdim=True)

def new_encoder():
    return nn.Sequential(nn.Linear(48, 64), nn.ReLU(), nn.Linear(64, 16))

def pretrain(loss_kind="contrastive", keep_brightness=False, steps=300, batch=128):
    """Train an encoder on 4,096 photos with no labels. Each step shows it two views of
    each photo in the batch."""
    torch.manual_seed(0)
    encoder, g = new_encoder(), np.random.default_rng(0)
    opt = torch.optim.Adam(encoder.parameters(), lr=3e-3)
    content, style, _ = photos(4096, 1)
    losses = []
    for step in range(steps):
        idx = g.permutation(4096)[:batch]
        # an augmentation: the same subject, under new lighting, framing and color
        style_a = 2.0 * g.normal(size=(batch, 6)).astype(np.float32)
        style_b = 2.0 * g.normal(size=(batch, 6)).astype(np.float32)
        if keep_brightness:    # a gentler augmentation that leaves brightness as it was
            style_a[:, 0] = style_b[:, 0] = style[idx, 0]
        a = encoder(torch.tensor(render(content[idx], style_a, g)))
        b = encoder(torch.tensor(render(content[idx], style_b, g)))
        if loss_kind == "contrastive":
            scores = unit(a) @ unit(b).T / 0.1
            # view a of photo i goes with view b of photo i
            labels = torch.arange(batch)
            loss = (F.cross_entropy(scores, labels)
                    + F.cross_entropy(scores.T, labels)) / 2
        else:  # "agree": pull the two views together, and nothing else
            loss = ((a - b) ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return encoder, losses

def encoded_by(encoder):
    def features(X):
        with torch.no_grad():
            return encoder(torch.tensor(X)).numpy()
    return features

def pretrain_masked(steps=300, batch=128):
    """Hide half of each photo's pixels and train an encoder and
    decoder to fill them back in."""
    torch.manual_seed(0)
    encoder, decoder, g = new_encoder(), nn.Linear(16, 48), np.random.default_rng(0)
    opt = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()),
                           lr=3e-3)
    content, style, _ = photos(4096, 1)
    X = render(content, style, g)
    for step in range(steps):
        x = X[g.permutation(4096)[:batch]]
        hidden = g.random(size=x.shape) < 0.5
        guess = decoder(
            encoder(torch.tensor(np.where(hidden, 0.0, x).astype(np.float32))))
        loss = (((guess - torch.tensor(x)) ** 2)
                * torch.tensor(hidden.astype(np.float32))).sum() / float(hidden.sum())
        opt.zero_grad()
        loss.backward()
        opt.step()
    return encoder
pixels = lambda X: X                          # use the raw pixel values as the features
print("labeled photos | accuracy of a classifier trained on raw pixels")
for n in (20, 40, 160, 640):
    print(str(n).rjust(14), "|", probe(pixels, n))

With 640 labels the classifier reaches 0.933. With 40 it gets 0.494, and with 20 only 0.318. The pixels vary so much with lighting and framing that a handful of examples can't show the classifier which variations to ignore.

Labels cost money and time, since a person has to look at every example. Unlabeled photos cost nothing. The question for the rest of the lesson is how to learn "which variations to ignore" from them.

Quick check
The labeled photos are spread evenly over 10 classes. What does this print: the number of labeled examples of each class when there are 20, 40 and 640 labels?
classes = 10
print([labels // classes for labels in (20, 40, 640)])
  1. [10, 10, 10]
  2. [200, 400, 6400]
  3. [20, 40, 640]
  4. [2, 4, 64]
Show the answer

[2, 4, 64]

Twenty labels over ten classes is two examples of each class, and forty is four. A classifier asked to learn what a class looks like from two or four noisy photos has very little to go on, which is why the first rows of the table are so low.

Two views of the same photo

Here is the invented task. Take a photo and make two altered copies of it: crop it differently, change the brightness, shift the colors. These alterations are called augmentations, and the two copies are two views. Whatever the photo shows, both views show it too.

Now train an encoder so that the two views of one photo get similar vectors, and views of different photos get different ones. No label is needed: the "right answer" for a view is its sibling, and the data supplies that for free.

This is the contrastive loss from the image-text lessons with both sides being images. A view's positive is the other view of the same photo. Its negatives are the views of every other photo in the batch. The method is known as SimCLR.

After pretraining comes the test. Freeze the encoder, run the labeled photos through it, and fit a plain logistic regression on its output vectors. This is a linear probe: if a simple classifier does well on the encoder's features, the encoder has already done the hard part.

Uses the setup above, 101 lines.

torch.manual_seed(0)
untrained = new_encoder()
encoder, losses = pretrain()
print("contrastive loss: first step", round(losses[0], 2), "| last step",
      round(losses[-1], 2))
print("labeled photos | raw pixels | untrained encoder | pretrained encoder")
for n in (20, 40, 160, 640):
    print(str(n).rjust(14), "|", str(probe(lambda X: X, n)).ljust(10), "|",
          str(probe(encoded_by(untrained), n)).ljust(17), "|",
          probe(encoded_by(encoder), n))

The encoder never saw a label. Trained on 4,096 unlabeled photos, it lifts the 40-label classifier from 0.494 to 0.674, and the 20-label one from 0.318 to 0.431.

Two things keep that honest. The untrained encoder does worse than raw pixels, so the gain comes from the pretraining and not from the network's shape. And by 160 labels the advantage is gone: 0.885 against 0.886. Pretraining buys label efficiency. It matters most when labels are few, which is the usual case.

Quick check
From the lesson on image-text models: for one item in a contrastive batch, what are its positive and its negatives? Here the items are views of photos.
  1. The positive is the photo's caption; the negatives are other captions
  2. The positive is any view with the same class label; the negatives have other labels
  3. The positive is the other view of the same photo; the negatives are views of the other photos
  4. The positive is the most similar view in the batch; the negatives are the least similar
Show the answer

The positive is the other view of the same photo; the negatives are views of the other photos

The pairing comes from how the batch was built, not from similarity or labels: two views were made from each photo, so each view knows its sibling. Every other view in the batch is treated as a wrong answer. No labels and no captions are involved.

Quick check
A batch holds two views of each of 4 photos, laid out as view a of photos 0 to 3, then view b of photos 0 to 3. What does this print: for each of the 8 rows, the row that holds its positive?
n = 4
positives = [(i + n) % (2 * n) for i in range(2 * n)]
print(positives)
  1. [7, 6, 5, 4, 3, 2, 1, 0]
  2. [4, 5, 6, 7, 0, 1, 2, 3]
  3. [1, 0, 3, 2, 5, 4, 7, 6]
  4. [0, 1, 2, 3, 4, 5, 6, 7]
Show the answer

[4, 5, 6, 7, 0, 1, 2, 3]

Row 0 is view a of photo 0, and view b of photo 0 sits 4 rows later, in row 4. Row 5 is view b of photo 1, whose sibling is row 1. Each row's positive is 4 rows away, wrapping around.

Drill

Positives and negatives in a batch of views

A batch is built from n photos by stacking view a of every photo, then view b of every photo, giving 2n rows. Write view_pairs(n). Return a tuple of two things: a list with one entry per row, holding the row number of that row's positive, and the number of negatives each row has (every other row except itself and its positive). For n of 0, return ([], 0).

Uses the setup above, 101 lines.

def view_pairs(n):
    return [], 0


print(view_pairs(4))
In the lesson, your code is checked as you write it.
Try it now

The classifier is the same each time. Only the features it reads changed.

In your own words
What is a linear probe, and what does a good score on one tell you about an encoder?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: augmentations, collapse and filling in

The augmentations decide what is learned

The encoder was asked for one thing: give two views of a photo the same vector. So whatever differs between the views is, by instruction, something to ignore. The augmentations are where you tell the model what doesn't matter.

That cuts both ways. Below, the probe asks two questions of the same features: which class is the photo, and a different one, was it taken in bright light? Brightness is one of the things the augmentations changed. A second encoder is then pretrained with a gentler augmentation that leaves brightness alone.

Uses the setup above, 101 lines.

print("features                               "
      "| which class? (40 labels) | bright or dim? (640 labels)")
print("raw pixels                             |", str(probe(lambda X: X, 40)).ljust(24),
      "|", probe(lambda X: X, 640, "bright"))
encoder, _ = pretrain()
print("pretrained, views differ in everything |",
      str(probe(encoded_by(encoder), 40)).ljust(24), "|",
      probe(encoded_by(encoder), 640, "bright"))
encoder, _ = pretrain(keep_brightness=True)
print("pretrained, views share brightness     |",
      str(probe(encoded_by(encoder), 40)).ljust(24), "|",
      probe(encoded_by(encoder), 640, "bright"))

Raw pixels answer the brightness question almost perfectly, 0.987. The pretrained encoder scores 0.521 on it, which is a coin flip. Brightness was removed from the features, on purpose and completely, since it was one of the things that differed between views.

The second encoder, whose views kept the photo's brightness, answers the brightness question well, 0.932, and is worse at the class question with few labels, 0.553 against 0.674. It kept something the class doesn't need, and the probe has to learn to look past it.

So the choice of augmentations is a claim about your task. Color jitter says "color doesn't matter", which is right for telling a cat from a dog and wrong for telling a ripe banana from a green one. An encoder can't be asked later for something it was trained to ignore.

Quick check
A team pretrains an encoder on fruit photos with strong color changes as an augmentation, then uses it to sort bananas into ripe and unripe. The probe does badly. What is the most likely reason?
  1. A linear probe can never work on photos of fruit
  2. The encoder was trained on too many photos
  3. Pretraining taught the encoder to ignore color
  4. The ripe and unripe photos were not labeled during pretraining
Show the answer

Pretraining taught the encoder to ignore color

Two views with different colors had to get the same vector, so the encoder learned to discard color. Ripeness is mostly a matter of color, so the information the probe needs was removed before the probe ever ran. The augmentations have to leave alone whatever the later task depends on.

Collapse: agreeing about nothing

Why does the loss need negatives at all? The goal is for two views to agree, so why not train for agreement alone, and make the two vectors as close as possible?

Because there is a way to agree perfectly that involves learning nothing: give every photo the same vector. Two views of a photo then match exactly, the loss is zero, and the encoder is worthless. This failure is called collapse. The cell below trains for agreement alone and watches how spread out the features are across 500 photos.

Uses the setup above, 101 lines.

test_photos = render(*photos(500, 3)[:2], np.random.default_rng(0))
print("training goal            | loss: first, last  "
      "| spread of the features | which class? (40 labels)")
for loss_kind, name in (("contrastive", "match, among the batch"),
                        ("agree", "agree, and nothing else")):
    encoder, losses = pretrain(loss_kind)
    spread = encoded_by(encoder)(test_photos).std(axis=0).mean()
    print(name.ljust(24), "|",
          (str(round(losses[0], 3)) + ", " + str(round(losses[-1], 4))).ljust(18), "|",
          str(round(float(spread), 3)).ljust(22), "|", probe(encoded_by(encoder), 40))

Training for agreement alone drives its loss from 1.79 to 0.0018, a far better number than the contrastive loss reaches. It got there by shrinking: the spread of its features is 0.126, a ninth of the contrastive encoder's 1.126, and still falling. A probe on those features scores 0.372 with 40 labels, worse than raw pixels.

A loss can only be trusted to mean what you hope when the easy, useless ways of lowering it are closed off. In contrastive training the negatives do that: an encoder that says the same thing about every photo fails every comparison. Other self-supervised methods drop the negatives and block collapse by different means, and every one of them needs some such device.

Quick check
During self-supervised pretraining, the loss falls quickly to almost zero, and every feature has almost the same value for every image. What has happened?
  1. The encoder has collapsed to a single output vector
  2. The learning rate is too low for the encoder to change
  3. The batch contains too many negatives
  4. The encoder has finished learning and is ready to use
Show the answer

The encoder has collapsed to a single output vector

A loss near zero with features that don't vary means the encoder found the shortcut: map every image to the same point, so any two views agree. It satisfies the letter of the task and encodes nothing. Watching the spread of the features during pretraining is the standard check.

Drill

Detect collapse

Write spread(features) and collapsed(features, floor). features is a list of rows, one per image, each a list of numbers. spread returns the standard deviation of each column (divide by the number of rows, not by one less), averaged over the columns and rounded to 4 places, or 0.0 if there are no rows. collapsed returns True when the spread is below floor.

Uses the setup above, 101 lines.

def spread(features):
    return 0.0


def collapsed(features, floor):
    return False


healthy = [[1.0, -2.0], [-1.0, 2.0], [1.0, 2.0], [-1.0, -2.0]]
shrunk = [[0.51, 0.5], [0.49, 0.5], [0.5, 0.5], [0.5, 0.5]]
print(spread(healthy), spread(shrunk), collapsed(healthy, 0.1), collapsed(shrunk, 0.1))
In the lesson, your code is checked as you write it.
In your own words
What is collapse, and why isn't a loss near zero proof that pretraining worked?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: filling in what is hidden

Matching views is one way to invent a task. The other large family hides part of the input and asks the model to fill it in. Language models are pretrained this way, predicting hidden or upcoming words. For images the method is called masked image modeling: hide a large share of the patches and train a model to reconstruct them.

The cell below pretrains an encoder by hiding half of each photo's 48 numbers and predicting them from the rest, then probes it beside the contrastive encoder.

Uses the setup above, 101 lines.

contrastive, _ = pretrain()
masked = pretrain_masked()
print("features             | which class? 40 labels | "
      "640 labels | bright or dim? (640 labels)")
for name, features in (("raw pixels", lambda X: X),
                       ("contrastive", encoded_by(contrastive)),
                       ("masked, filled in", encoded_by(masked))):
    print(name.ljust(20), "|", str(probe(features, 40)).ljust(22), "|",
          str(probe(features, 640)).ljust(10), "|", probe(features, 640, "bright"))

The two encoders learned opposite things. The contrastive one is good at the class with few labels, 0.674, and blind to brightness, 0.521. The masked one is the reverse: 0.415 on the class with 40 labels, below raw pixels, and 0.971 on brightness.

That follows from the tasks. To fill in hidden pixels, a model has to keep whatever explains the pixels, and in these photos lighting and framing explain most of them. Nothing told it that they don't matter. The contrastive task has that knowledge built in through its augmentations.

Neither is better in general. Matching views gives features that a simple probe can use directly, at the price of deciding in advance what to throw away. Filling in keeps everything, needs no augmentations, and tends to pay off when the whole model is fine-tuned on the task afterward instead of probed.

Go deeper Names you will meet
  • SimCLR and MoCo match augmented views against negatives, as in this lesson.
  • BYOL, SimSiam and DINO match views with no negatives, and avoid collapse by other means, such as updating one of two copies of the network slowly.
  • Masked autoencoders (MAE) hide most of an image's patches and reconstruct them with a vision transformer.
  • CLIP is the same idea with a caption as the second view.
Quick check
In the run above, the masked encoder's features predicted brightness almost perfectly and the contrastive encoder's didn't. Why?
  1. The masked encoder was trained for more steps than the contrastive one
  2. The contrastive encoder was shown the brightness labels and the masked one wasn't
  3. Masked encoders are larger, so they can store more
  4. Filling in pixels requires keeping what explains them; matching views discards what differs between views
Show the answer

Filling in pixels requires keeping what explains them; matching views discards what differs between views

Each task rewards a different kind of memory. To reconstruct hidden pixels the model must hold on to lighting, since lighting changes many pixels at once. To match two views that were lit differently, the model must let go of lighting. The features show what each task asked for.

Worked example

Worked example: how many labels is the pretraining worth?

A concrete way to price pretraining: find how many labeled photos a classifier on raw pixels needs before it matches the pretrained encoder with 40.

Uses the setup above, 101 lines.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.linear_model import LogisticRegression

g0 = np.random.default_rng(0)
CLASSES = 10
centers = g0.normal(size=(CLASSES, 6)).astype(np.float32)
# how the subject of a photo shows up in its pixels
P = g0.normal(size=(6, 48)).astype(np.float32)
# how lighting, framing and color cast show up
Q = g0.normal(size=(6, 48)).astype(np.float32)

def photos(n, seed):
    """n photos, described by what is in them (content), how they were taken
    (style), and a class label."""
    g = np.random.default_rng(seed)
    label = g.integers(0, CLASSES, size=n)
    content = centers[label] + 0.4 * g.normal(size=(n, 6)).astype(np.float32)
    style = 2.0 * g.normal(size=(n, 6)).astype(np.float32)
    return content, style, label

def render(content, style, g):
    return (content @ P + style @ Q
            + 0.3 * g.normal(size=(len(content), 48)).astype(np.float32))

def probe(features, n_labels, target="class"):
    """Fit a logistic regression on n_labels labeled photos, using the given
    features. Test on 2,000 others."""
    g = np.random.default_rng(5)
    (c, s, y), (ct, st, yt) = photos(n_labels, 10), photos(2000, 11)
    X, X_test = render(c, s, g), render(ct, st, g)
    if target == "bright":  # a different question: was the photo taken in bright light?
        y, yt = (s[:, 0] > 0).astype(int), (st[:, 0] > 0).astype(int)
    model = LogisticRegression(max_iter=2000).fit(features(X), y)
    return round(float(model.score(features(X_test), yt)), 3)

def unit(v):
    return v / v.norm(dim=-1, keepdim=True)

def new_encoder():
    return nn.Sequential(nn.Linear(48, 64), nn.ReLU(), nn.Linear(64, 16))

def pretrain(loss_kind="contrastive", keep_brightness=False, steps=300, batch=128):
    """Train an encoder on 4,096 photos with no labels. Each step shows it two views of
    each photo in the batch."""
    torch.manual_seed(0)
    encoder, g = new_encoder(), np.random.default_rng(0)
    opt = torch.optim.Adam(encoder.parameters(), lr=3e-3)
    content, style, _ = photos(4096, 1)
    losses = []
    for step in range(steps):
        idx = g.permutation(4096)[:batch]
        # an augmentation: the same subject, under new lighting, framing and color
        style_a = 2.0 * g.normal(size=(batch, 6)).astype(np.float32)
        style_b = 2.0 * g.normal(size=(batch, 6)).astype(np.float32)
        if keep_brightness:    # a gentler augmentation that leaves brightness as it was
            style_a[:, 0] = style_b[:, 0] = style[idx, 0]
        a = encoder(torch.tensor(render(content[idx], style_a, g)))
        b = encoder(torch.tensor(render(content[idx], style_b, g)))
        if loss_kind == "contrastive":
            scores = unit(a) @ unit(b).T / 0.1
            # view a of photo i goes with view b of photo i
            labels = torch.arange(batch)
            loss = (F.cross_entropy(scores, labels)
                    + F.cross_entropy(scores.T, labels)) / 2
        else:  # "agree": pull the two views together, and nothing else
            loss = ((a - b) ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return encoder, losses

def encoded_by(encoder):
    def features(X):
        with torch.no_grad():
            return encoder(torch.tensor(X)).numpy()
    return features

encoder, _ = pretrain()
target = probe(encoded_by(encoder), 40)
print("pretrained encoder, 40 labels:", target)
for n in (40, 60, 80, 100, 120, 160):
    score = probe(lambda X: X, n)
    print("raw pixels,", str(n).rjust(3), "labels:", str(score).ljust(5),
          "<- catches up" if score >= target else "")
It runs right here in your browser, with nothing to install.

The pretrained encoder with 40 labels scores 0.674. A classifier on raw pixels has 0.494 at 40 labels and passes that mark at 60, with 0.755. So in this small problem the pretraining was worth about 20 labels: half again as many as were on hand. That is a modest price, for an honest reason. These photos are simple enough that a linear classifier on raw pixels learns quickly once it has seen a few examples per class. On real photographs raw pixels are a far worse starting point, and the same comparison comes out much more lopsided. The method of the comparison carries over: fix a score, and count the labels each route needs to reach it.

On the job: the practical question is how many labels you can skip. Train the probe on 40, 160 and 640 labels with and without pretrained features, as in the table above, and read where the curves meet. If pretraining only helps below 100 labels and you have 5,000, a plain model may be enough. If it helps at every size, it is worth the extra step.

In your own words
You have a large pile of unlabeled images and a small labeled set. What do you decide before pretraining, and how do you check the result?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

What makes a learning method "self-supervised"?

  1. The training signal is built from the data itself, with no labels from people
  2. The model chooses its own learning rate as it trains
  3. The model is trained with no loss function at all
  4. People label a small part of the data and the model labels the rest

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write