Index Ai Academy
Phase 5 · Beyond text

Diffusion 3: images, labels and guidance

From points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A diffusion model that only makes "an image like the training set" is a curiosity. The ones people use take a request: a label, or a sentence. This lesson adds that. It moves from points to small images, gives the denoiser a label to look at, and then builds the technique behind the "guidance scale" setting in every image generator: classifier-free guidance. You will see why a model that has been told what to draw still often doesn't draw it, how guidance fixes that, and what it breaks when it is turned up too far.

What you'll be able to do

  • Apply the same diffusion method to small images, predicting the clean image and clipping it to valid pixel values
  • Condition a denoiser on a label, so that a particular kind of image can be asked for
  • Train one network to answer with and without the label, by hiding the label some of the time
  • Combine the two answers with classifier-free guidance, and say what the guidance scale trades
  • Recognize an over-guided model: right kind, less variety, then broken images
Quick check
A guess before you start. An image generator has a setting, the guidance scale, that controls how strongly it follows the prompt. What do you expect to happen as it is raised from low to very high?
  1. Images keep matching the prompt better, with no downside
  2. Better matches first, then sameness and breakdown
  3. Nothing visible changes, since the prompt is the same
  4. Images get more varied, and match the prompt less
Show the answer

Better matches first, then sameness and breakdown

Guidance pushes each denoising step further in the direction the prompt points. A moderate push turns vague, half-formed images into clear ones of the right kind. A hard push drives every sample toward the same exaggerated answer: variety drops first, and then the images themselves fall apart. This lesson measures all three stages.

The idea: two guesses, and a dial between them

This rests on Diffusion 2: learning to denoise, and sampling for the denoising loop and Vectors: lists of numbers that point for adding and scaling a difference.

From points to images

Nothing in Diffusion 1: turning data into noise or Diffusion 2: learning to denoise, and sampling was specific to points with two numbers. An image is a point with more numbers. The images here are 6 by 6 pixels and gray, so each is 36 numbers: −1 for a dark pixel and +1 for a bright one. There are three kinds.

Setup (runs before every cell) · 116 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

NAMES = ["a flat bar", "an upright bar", "a square"]

def images(n, seed):
    """n tiny gray images, 6 by 6, of three kinds. The background
    is -1 and the shape is +1."""
    g = np.random.default_rng(seed)
    kind = g.integers(0, 3, size=n)
    X = -np.ones((n, 6, 6), dtype=np.float32)
    for i in range(n):
        if kind[i] == 0:
            X[i, g.integers(0, 6), :] = 1           # a flat bar across one row
        elif kind[i] == 1:
            X[i, :, g.integers(0, 6)] = 1           # an upright bar down one column
        else:
            r, c = g.integers(0, 4), g.integers(0, 4)
            X[i, r:r + 3, c:c + 3] = 1              # a 3 by 3 square
    return X.reshape(n, 36), kind                   # each image flattened to 36 numbers

def show(batch, titles):
    """Print a few images side by side: # where a pixel is above 0."""
    print("  ".join(title.ljust(7)[:7] for title in titles))
    for row in range(6):
        print("  ".join(
            " " + "".join("#" if v > 0 else "." for v in image.reshape(6, 6)[row])
            for image in batch))

T = 100
betas = np.linspace(0.001, 0.2, T).astype(np.float32)
alpha_bar = np.cumprod(1 - betas).astype(np.float32)
NO_LABEL = 3                        # what the network is told when no kind is asked for

def inputs(x, t, label):
    """The noisy image, the step as a few waves, and the label as four on/off numbers.
    """
    s = (np.asarray(t, dtype=np.float32) / T)[:, None]
    waves = [s, np.sin(2 * np.pi * s), np.cos(2 * np.pi * s), np.sin(4 * np.pi * s),
             np.cos(4 * np.pi * s)]
    return np.concatenate([x] + waves + [np.eye(4, dtype=np.float32)[label]],
                          axis=1).astype(np.float32)

def train(steps=1000, hide=0.15):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(45, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(),
                          nn.Linear(128, 36))
    opt, g = torch.optim.Adam(model.parameters(), lr=2e-3), np.random.default_rng(0)
    data, kind = images(4096, 1)
    for step in range(steps):
        idx, t = g.integers(0, len(data), size=64), g.integers(1, T + 1, size=64)
        eps = g.normal(size=(64, 36)).astype(np.float32)
        xt = (np.sqrt(alpha_bar[t - 1])[:, None] * data[idx]
              + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps)
        # now and then, hide the label
        label = np.where(g.random(64) < hide, NO_LABEL, kind[idx])
        # the target is the clean image
        loss = F.mse_loss(model(torch.tensor(inputs(xt, t, label))),
                          torch.tensor(data[idx]))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model

def predict_clean(model, x, t, label):
    with torch.no_grad():
        return model(torch.tensor(inputs(x, np.full(len(x), t), label))).numpy()

def sample(model, labels, seed, guidance=1.0, clip=True):
    """Generate one image per entry of labels. Guidance 0 ignores the labels,
    and 1 uses them plainly."""
    g, n = np.random.default_rng(seed), len(labels)
    x = g.normal(size=(n, 36)).astype(np.float32)
    for t in range(T, 0, -1):
        # the guess with no label
        without = predict_clean(model, x, t, np.full(n, NO_LABEL))
        clean = without if guidance == 0 else without + guidance * (
            predict_clean(model, x, t, labels) - without)
        if clip:
            clean = np.clip(clean, -1, 1)  # a pixel can't be outside -1 to 1
        # the noise this guess implies
        noise = (x - np.sqrt(alpha_bar[t - 1]) * clean) / np.sqrt(1 - alpha_bar[t - 1])
        x = ((x - betas[t - 1] / np.sqrt(1 - alpha_bar[t - 1]) * noise)
             / np.sqrt(1 - betas[t - 1]))
        if t > 1:
            x = x + np.sqrt(betas[t - 1]) * g.normal(size=x.shape).astype(np.float32)
    return x

def every_real_image():
    out, kinds = [], []
    for kind in range(3):
        for a in range(6 if kind < 2 else 4):
            for b in range(1 if kind < 2 else 4):
                image = -np.ones((6, 6), dtype=np.float32)
                if kind == 0:
                    image[a, :] = 1
                elif kind == 1:
                    image[:, a] = 1
                else:
                    image[a:a + 3, b:b + 3] = 1
                out.append(image.reshape(36))
                kinds.append(kind)
    return np.array(out), np.array(kinds)

REAL, REAL_KIND = every_real_image()                # all 28 images the data can contain

def judge(batch):
    """For each generated image: its nearest real image, that image's kind, and how
    many pixels differ from it."""
    differ = ((batch > 0)[:, None, :] != (REAL > 0)[None, :, :]).sum(axis=2)
    nearest = differ.argmin(axis=1)
    return nearest, REAL_KIND[nearest], differ.min(axis=1)

model = train()                     # trained once, here, and reused by every cell
import numpy as np

NAMES = ["a flat bar", "an upright bar", "a square"]

def images(n, seed):
    """n tiny gray images, 6 by 6, of three kinds. The background
    is -1 and the shape is +1."""
    g = np.random.default_rng(seed)
    kind = g.integers(0, 3, size=n)
    X = -np.ones((n, 6, 6), dtype=np.float32)
    for i in range(n):
        if kind[i] == 0:
            X[i, g.integers(0, 6), :] = 1           # a flat bar across one row
        elif kind[i] == 1:
            X[i, :, g.integers(0, 6)] = 1           # an upright bar down one column
        else:
            r, c = g.integers(0, 4), g.integers(0, 4)
            X[i, r:r + 3, c:c + 3] = 1              # a 3 by 3 square
    return X.reshape(n, 36), kind                   # each image flattened to 36 numbers

def show(batch, titles):
    """Print a few images side by side: # where a pixel is above 0."""
    print("  ".join(title.ljust(7)[:7] for title in titles))
    for row in range(6):
        print("  ".join(
            " " + "".join("#" if v > 0 else "." for v in image.reshape(6, 6)[row])
            for image in batch))

data, kind = images(4096, 1)
print("images:", data.shape, "| pixel values:", sorted(set(data.reshape(-1).tolist())))
print("how many of each kind:", {NAMES[k]: int((kind == k).sum()) for k in range(3)})
print("different images in the whole set:", len({tuple(row) for row in data.tolist()}))
print()
first = [int(np.where(kind == k)[0][j]) for k in range(3) for j in range(2)]
show(data[first], ["flat", "flat", "upright", "upright", "square", "square"])
It runs right here in your browser, with nothing to install.

The set holds 4,096 images, but only 28 different ones: six flat bars, six upright bars and sixteen squares. That makes it easy to judge a generated image exactly, by comparing it with all 28.

Two things change in the move to images, and both are standard practice.

The network predicts the clean image instead of the noise. You saw that the two carry the same information: from a guess of the noise you can work out the clean point, and from the clean point the noise. For images the clean picture is the easier target, since every pixel is known to lie between −1 and 1.

The guess is clipped. Before it is used, any pixel the network puts outside −1 to 1 is pulled back to the edge. A guess can't be right out there, and small excesses compound over a hundred steps.

The cell trains the denoiser and generates 300 images with no kind asked for. Each is compared with its nearest real image.

Uses the setup above, 116 lines.

# guidance 0: no kind is asked for
generated = sample(model, np.zeros(300, dtype=int), 3, guidance=0)
nearest, found, wrong = judge(generated)
print("300 images generated with no label")
print("exactly a real image:", int((wrong == 0).sum()), "| one or two pixels off:",
      int(((wrong > 0) & (wrong <= 2)).sum()), "| further off:", int((wrong > 2).sum()))
print("the exact ones, by kind:",
      {NAMES[k]: int(((found == k) & (wrong == 0)).sum()) for k in range(3)})
print()
show(generated[:6], ["no. " + str(i) for i in range(6)])

Of 300 images, 45 are exactly one of the real ones, 31 are a pixel or two off, and 224 are further away: blank, or a smudge, like most of the six printed. For a model this small, trained for a few seconds, making "some image from the set" out of pure noise is too vague a job. Each step's best guess is an average over flat bars, upright bars and squares at once, and an average of those is mud.

That weakness is the starting point for the rest of the lesson.

Quick check
From the lesson on denoising and sampling: a noisy value is 0.4 at a step where alpha bar is 0.36, and the network guesses the clean value is 2.0. What does this print: the noise that guess implies?
import math

xt, clean, alpha_bar = 0.4, 2.0, 0.36
print(round((xt - math.sqrt(alpha_bar) * clean) / math.sqrt(1 - alpha_bar), 3))
  1. 1.0
  2. -2.0
  3. -1.0
  4. -0.8
Show the answer

-1.0

The noisy value is 0.6 × clean + 0.8 × noise. With a clean guess of 2.0, the data's share is 1.2, so the noise's share is 0.4 − 1.2 = −0.8, and dividing by its weight, 0.8, gives −1.0. A guess of the clean image and a guess of the noise are two ways of writing the same answer.

Try it now

Press Next. Each arrow is a guess for the same pixel pair. Guided = no-label guess + scale × (with-label guess − no-label guess).

In your own words
Why did the model with no label produce so many blank or smudged images, and what does the label add?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: asking for a kind, and turning the dial

Asking for a kind

To ask for something, the network needs to be told what was asked. This is conditioning: an extra input that travels with the noisy image at every step. Here it is the label, written as four on/off numbers: one for each kind, and a fourth meaning "no label given".

During training each noisy image comes with its true label. The network learns "what clean image is under this noise, given that it is a square", a far narrower question than before. At sampling time you supply the label you want.

Uses the setup above, 116 lines.

asked = np.repeat(np.arange(3), 100)                        # 100 requests for each kind
generated = sample(model, asked, 3, guidance=1.0)
nearest, found, wrong = judge(generated)
print("asked for      | that kind, at most 2 pixels off | that kind exactly")
for k in range(3):
    mine = asked == k
    print(NAMES[k].ljust(14), "|",
          str(int(((found[mine] == k) & (wrong[mine] <= 2)).sum())).rjust(31), "|",
          int(((found[mine] == k) & (wrong[mine] == 0)).sum()))
print()
show(generated[[0, 1, 100, 101, 200, 201]],
     ["flat", "flat", "upright", "upright", "square", "square"])

Asking helps a great deal. With no label 45 of 300 images were exact. Now 63 of 100 flat bars are, and 67 of 100 upright bars and 57 of 100 squares.

It is still far from reliable. A third of the requests for a flat bar didn't produce anything close to one, and the printed samples show why: some stay blank. The label is one more input among forty-five, and when the picture is nearly all noise, the network's guess leans on it only weakly.

In a text-to-image model the conditioning is a set of vectors from a text encoder, such as CLIP's, in place of this label. The principle is the same, and so is the problem: a prompt that is merely supplied is often half ignored.

Quick check
What does "conditioning" a denoiser on a label mean?
  1. Giving it the label as an extra input
  2. Sorting the finished images by label afterward
  3. Training a separate denoiser for each label
  4. Adding the label's name to the image as text
Show the answer

Giving it the label as an extra input

The label is fed to the network alongside the noisy image and the step number, every time the network is called. One network handles all the labels. It learns how the right answer differs depending on what was asked for.

Guidance: turning the label up

The fix comes from an observation. The network can be asked twice at each step: once with the label, and once without. The difference between its two answers is exactly "what the label changes". If the label's effect is too weak, exaggerate it.

guess=without+w×(with−without)\text{guess} = \text{without} + w \times (\text{with} - \text{without})

In plain words: start from the answer the network gives with no label. Add the difference the label makes, multiplied by w. With w = 1 this is just the answer with the label. With w above 1 it goes past it, further in the direction the label points.

The number w is the guidance scale, and the method is classifier-free guidance. One network gives both answers, because of a detail in training you may have noticed in the code: for 15% of the training examples, chosen at random, the label was replaced by "no label given". So the network knows how to answer either way.

Uses the setup above, 116 lines.

asked = np.repeat(np.arange(3), 100)
print("guidance | the kind asked for, at most 2 pixels off | "
      "exactly a real image | different squares, of 16")
for guidance in (0, 1, 2, 4, 8, 16):
    nearest, found, wrong = judge(sample(model, asked, 3, guidance=guidance))
    # the clean squares among the 100 asked for
    squares = nearest[200:][(found[200:] == 2) & (wrong[200:] == 0)]
    print(str(guidance).rjust(8), "|",
          str(round(float(((found == asked) & (wrong <= 2)).mean()), 3)).ljust(41), "|",
          str(round(float(((found == asked) & (wrong == 0)).mean()), 3)).ljust(20), "|",
          len(set(squares.tolist())))

Read down the table.

  • 0 ignores the label. Only 0.063 of the images are an exact image of the kind asked for.
  • 1 is the plain conditioning of the last section: 0.623.
  • 2 and 4 are the sweet spot. Over 0.9 of the images are exactly right, and at least 11 of the 16 possible squares still turn up among them.
  • 8 begins to break. Exact images fall to 0.7, and only 6 different squares appear.
  • 16 is wreckage: 0.297 exact, and 2 different squares.

So guidance buys obedience with variety. A moderate scale makes the model commit to a clear image of the right kind. A high scale pushes every sample toward the same exaggerated answer, and then past anything that was ever in the data.

It also costs time. Each step now calls the network twice, so generation takes twice as long.

Quick check
At one step, the network's guess for a pixel is 0.2 without the label and 0.5 with it. What does this print: the guided guess at scales 0, 1, 2 and 4?
without, with_label = 0.2, 0.5
print([round(without + w * (with_label - without), 2) for w in (0, 1, 2, 4)])
  1. [0.2, 0.5, 0.8, 1.4]
  2. [0.2, 0.5, 1.0, 2.0]
  3. [0.2, 0.7, 1.2, 2.2]
  4. [0.0, 0.5, 1.0, 2.0]
Show the answer

[0.2, 0.5, 0.8, 1.4]

The label moves the guess by 0.3. Guidance multiplies that move: none of it at 0, all of it at 1, twice at 2 (0.2 + 0.6 = 0.8), four times at 4 (0.2 + 1.2 = 1.4). At a scale of 4 the guess is 1.4, beyond the largest value a pixel can have, which is why the guess is clipped.

Drill

Combine two guesses with a guidance scale

Write guided(without, with_label, w, low, high). without and with_label are the network's two guesses for an image, as lists of pixel values. Return the guided guess, pixel by pixel: without + w * (with_label - without), then clipped to lie between low and high, and rounded to 3 places.

Uses the setup above, 116 lines.

def guided(without, with_label, w, low, high):
    return with_label


print(guided([0.2, -0.4, 0.9], [0.5, -0.9, 1.0], 4, -1, 1))
In the lesson, your code is checked as you write it.
Quick check
In classifier-free guidance, how does a single network come to give an answer both with a label and without one?
  1. A separate classifier removes the label's effect afterward
  2. During training, the label is replaced by "no label" for some examples
  3. Two networks are trained and their weights are averaged
  4. The network is trained twice, first without labels and then with them
Show the answer

During training, the label is replaced by "no label" for some examples

A random fraction of training examples, 15% here, have their label swapped for a special "no label" input. The network therefore practices both questions, and at sampling time either can be asked. No classifier is involved, which is where the method's name comes from.

Quick check
An image is generated in 50 sampling steps with classifier-free guidance. How many times is the denoising network called?
  1. 100
  2. 50
  3. 51
  4. 2,500
Show the answer

100

Guidance needs two answers at every step, one with the condition and one without, so 50 steps make 100 calls. This is why guided generation takes about twice as long as unguided generation at the same number of steps.

Drill

Hide some labels for training

Write hide_labels(labels, draws, share, no_label). labels is a list of class numbers for a training batch. draws is a list of random numbers between 0 and 1, one per example. Return a new list in which a label is replaced by no_label wherever its draw is below share, and kept otherwise. Then write hidden_share(labels, no_label): the share of a list of labels equal to no_label, rounded to 3 places, or 0.0 for an empty list.

Uses the setup above, 116 lines.

def hide_labels(labels, draws, share, no_label):
    return labels


def hidden_share(labels, no_label):
    return 0.0


batch = hide_labels([0, 2, 1, 1, 0, 2], [0.90, 0.10, 0.50, 0.03, 0.15, 0.72], 0.15, 3)
print(batch, hidden_share(batch, 3))
In the lesson, your code is checked as you write it.
Worked example

Worked example: what the noise decides and what the label decides

Two things go into a generated image: the starting noise, which is fixed by the seed, and the label. Hold one still and change the other, at a guidance scale of 2. Then look at what clipping does at a scale of 4.

Uses the setup above, 116 lines.

print("the same starting noise, three different labels:")
show(sample(model, np.array([0, 1, 2]), 7, guidance=2.0), ["flat", "upright", "square"])
print()
print("the same label, four different starting noises:")
show(np.concatenate([
    sample(model, np.array([2]), seed, guidance=2.0)
    for seed in (1, 2, 3, 4)]), ["seed 1", "seed 2", "seed 3", "seed 4"])
print()
asked = np.repeat(np.arange(3), 100)
for clip in (True, False):
    generated = sample(model, asked, 3, guidance=4.0, clip=clip)
    nearest, found, wrong = judge(generated)
    print("guidance 4,",
          "guess clipped to -1..1 |" if clip else "guess not clipped      |",
          "exactly a real image:",
          round(float(((found == asked) & (wrong == 0)).mean()), 3),
          "| largest pixel size reached:", round(float(np.abs(generated).max()), 2))
In the lesson you can change this and run it.

With the starting noise held still and the label changed, the three images share nothing visible: a flat bar, an upright bar, a square. The label decides what is drawn. With the label held at "square" and the noise changed, the squares land in different places, and two of the four seeds happen to give the same one. The noise decides which image of that kind you get, which is why an image tool that lets you fix the seed can reproduce a picture exactly. The fourth seed is a reminder that this small model still misses: its square is ragged. The last two lines are about clipping. At a scale of 4 the share of exact images is about the same with it and without it, 0.923 and 0.937, since the judging only asks whether each pixel is above zero. The values tell the other story: without clipping, a pixel reached 15.79 on a scale where 1 is the brightest possible. In a real image those are the harsh, blown-out patches of an over-guided picture.

In your own words
A denoiser is told the label at every step and still often draws the wrong kind. Why, and what can turn up the label's influence?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: guidance in practice

Real image generators use exactly this dial. Their guidance scale is typically a number like 5 to 8, and they take a negative prompt: a second guess to push away from. The experiments below find the setting that works for this small model, then the library's version.

The cell below draws what the sweep counted. It asks for squares six times at four guidance scales, one row each, with the same starting noise in every row.

Uses the setup above, 116 lines.

import matplotlib.pyplot as plt

asked = np.full(6, 2)                              # six requests for a square
fig, axes = plt.subplots(4, 6, figsize=(6.5, 4.6))
for row, guidance in zip(axes, (0, 1, 4, 16)):
    for ax, image in zip(row, sample(model, asked, 3, guidance=guidance)):
        ax.imshow(image.reshape(6, 6), cmap="gray_r", vmin=-1, vmax=1)
        ax.axis("off")
    row[0].set_title(f"guidance {guidance}", fontsize=8, loc="left")

The sweep's counts say what to look for. With no guidance only 0.11 of the images are the kind asked for. At 4, 0.97 are. At 16 only 0.41 are, and the ones that are right come out as the same two or three squares.

On the job: guidance scale is the first knob to sweep when image quality disappoints. Fix the seed and the prompt, generate at several scales, and look at the row. Too low gives images that ignore the prompt, too high gives oversaturated, samey ones. Pick the scale where the prompt is followed and the pictures still differ, then keep it as a setting with the seed so a result can be reproduced.

In your own words
A colleague sets the guidance scale very high "so the model follows the prompt exactly". What do you tell them?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

A denoiser for images is trained to predict the clean image, and its guess is clipped to the range of valid pixel values before use. What is the clipping for?

  1. A guess outside the valid range can't be right
  2. It makes the network train faster on each batch
  3. It removes the need to add fresh noise while sampling
  4. It turns gray images into black and white ones

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write