Index Ai Academy
Phase 5 · Beyond text

Diffusion 4: latent diffusion and the whole pipeline

Why image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Everything so far would be too slow for real pictures. A denoiser that reads and writes every pixel of a large image, a hundred times over, twice per step with guidance, is an enormous amount of arithmetic. The systems people use don't do that. They squeeze each image into a much smaller grid of numbers first, run the whole diffusion process there, and turn the result back into pixels once at the end. This is latent diffusion, and it is the design behind Stable Diffusion and most of what followed. This lesson builds one in miniature, and then assembles the full pipeline, from prompt to picture.

What you'll be able to do

  • Explain why image generators run diffusion on an autoencoder's codes instead of on pixels
  • Build a latent diffusion model: compress images to codes, diffuse the codes, decode once at the end
  • Say what the autoencoder must be trained to do so that generated codes decode well
  • Name the three parts of a text-to-image generator and what each one does
  • Count the work of generating an image from its size, its steps and its guidance
Quick check
A guess before you start. A text-to-image generator makes a 512 by 512 picture in 50 denoising steps. How many times does it produce a full grid of 512 by 512 pixels along the way?
  1. A hundred times, with guidance
  2. Once, at the very end
  3. Fifty times, once per step
  4. Never: it outputs a compressed file
Show the answer

Once, at the very end

A latent diffusion model does all of its denoising on a small grid of numbers, typically 64 by 64, that an autoencoder knows how to turn into an image. Pixels appear only when the finished grid is decoded, once. That is the trick that made image generation cheap enough to run on an ordinary graphics card.

The idea: diffuse on a small code, not on pixels

This rests on Autoencoders, which squeeze a picture into a few numbers and back, and on Diffusion 3: images, labels and guidance for the denoiser and the label.

The cost of pixels

Count what the denoiser has to handle. A 512 by 512 color image is a point with three numbers per pixel. The denoiser reads all of them and writes all of them on every call.

The lesson on autoencoders described the way out. An autoencoder squeezes an image through a narrow middle, the latent, and rebuilds it on the other side. If the rebuilt image is good, then the latent holds everything that matters, in far fewer numbers.

Setup (runs before every cell) · 124 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

NAMES = ["a flat bar", "an upright bar", "a square"]

def images(n, seed):
    """The 6 by 6 images of Diffusion 3: flat bars, upright bars and
    squares, as rows of 36 numbers."""
    g = np.random.default_rng(seed)
    kind = g.integers(0, 3, size=n)
    X = -np.ones((n, 6, 6), dtype=np.float32)
    for i in range(n):
        if kind[i] == 0:
            X[i, g.integers(0, 6), :] = 1
        elif kind[i] == 1:
            X[i, :, g.integers(0, 6)] = 1
        else:
            r, c = g.integers(0, 4), g.integers(0, 4)
            X[i, r:r + 3, c:c + 3] = 1
    return X.reshape(n, 36), kind

def every_real_image():
    out, kinds = [], []
    for kind in range(3):
        for a in range(6 if kind < 2 else 4):
            for b in range(1 if kind < 2 else 4):
                image = -np.ones((6, 6), dtype=np.float32)
                if kind == 0:
                    image[a, :] = 1
                elif kind == 1:
                    image[:, a] = 1
                else:
                    image[a:a + 3, b:b + 3] = 1
                out.append(image.reshape(36))
                kinds.append(kind)
    return np.array(out), np.array(kinds)

REAL, REAL_KIND = every_real_image()

def exact_share(batch, asked):
    """The share of generated images that are exactly a real
    image of the kind asked for."""
    differ = ((batch > 0)[:, None, :] != (REAL > 0)[None, :, :]).sum(axis=2)
    exact = (REAL_KIND[differ.argmin(axis=1)] == asked) & (differ.min(axis=1) == 0)
    return round(float(exact.mean()), 3)

def train_autoencoder(size, steps=1000):
    """An encoder from 36 pixels down to `size` numbers, and a decoder back up."""
    torch.manual_seed(0)
    encoder = nn.Sequential(nn.Linear(36, 64), nn.ReLU(), nn.Linear(64, size))
    decoder = nn.Sequential(nn.Linear(size, 64), nn.ReLU(), nn.Linear(64, 36))
    opt, g = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()),
                              lr=3e-3), np.random.default_rng(0)
    data, _ = images(4096, 1)
    for step in range(steps):
        x = torch.tensor(data[g.integers(0, len(data), size=64)])
        code = encoder(x)
        # the decoder must cope with codes that are a little off
        shaken = code + 0.5 * torch.tensor(g.normal(size=(64, size)).astype(np.float32))
        # and the codes are kept from growing large
        loss = F.mse_loss(decoder(shaken), x) + 0.01 * (code ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
    return encoder, decoder

def run(network, array):
    with torch.no_grad():
        return network(torch.tensor(np.asarray(array, dtype=np.float32))).numpy()

T = 100
betas = np.linspace(0.001, 0.2, T).astype(np.float32)
alpha_bar = np.cumprod(1 - betas).astype(np.float32)
NO_LABEL = 3

def inputs(x, t, label):
    s = (np.asarray(t, dtype=np.float32) / T)[:, None]
    waves = [s, np.sin(2 * np.pi * s), np.cos(2 * np.pi * s), np.sin(4 * np.pi * s),
             np.cos(4 * np.pi * s)]
    return np.concatenate([x] + waves + [np.eye(4, dtype=np.float32)[label]],
                          axis=1).astype(np.float32)

def train_denoiser(data, kind, width, steps=1000):
    """The denoiser of Diffusion 3, for rows of any length: pixels or codes."""
    d = data.shape[1]
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(d + 9, width), nn.ReLU(), nn.Linear(width, width),
                          nn.ReLU(), nn.Linear(width, d))
    opt, g = torch.optim.Adam(model.parameters(), lr=2e-3), np.random.default_rng(0)
    for step in range(steps):
        idx, t = g.integers(0, len(data), size=64), g.integers(1, T + 1, size=64)
        eps = g.normal(size=(64, d)).astype(np.float32)
        xt = (np.sqrt(alpha_bar[t - 1])[:, None] * data[idx]
              + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps)
        label = np.where(g.random(64) < 0.15, NO_LABEL, kind[idx])
        loss = F.mse_loss(model(torch.tensor(inputs(xt, t, label))),
                          torch.tensor(data[idx]))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model

def sample(model, labels, seed, d, guidance=2.0, clip=None):
    g, n = np.random.default_rng(seed), len(labels)
    x = g.normal(size=(n, d)).astype(np.float32)
    for t in range(T, 0, -1):
        without = run(model, inputs(x, np.full(n, t), np.full(n, NO_LABEL)))
        clean = (without
                 + guidance * (run(model, inputs(x, np.full(n, t), labels)) - without))
        if clip is not None:
            clean = np.clip(clean, -clip, clip)
        noise = (x - np.sqrt(alpha_bar[t - 1]) * clean) / np.sqrt(1 - alpha_bar[t - 1])
        x = ((x - betas[t - 1] / np.sqrt(1 - alpha_bar[t - 1]) * noise)
             / np.sqrt(1 - betas[t - 1]))
        if t > 1:
            x = x + np.sqrt(betas[t - 1]) * g.normal(size=x.shape).astype(np.float32)
    return x

def weights(network):
    return sum(p.numel() for p in network.parameters())

data, kind = images(4096, 1)
for name, height, width, channels in (
    ("pixels of a 512 by 512 color image", 512, 512, 3),
    ("its latent in a typical generator", 64, 64, 4)):
    print(name.ljust(34), "|", height, "x", width, "x", channels, "=",
          f"{height * width * channels:,}", "numbers")
print("the denoiser handles", 512 * 512 * 3 // (64 * 64 * 4),
      "times fewer numbers per call in the latent")
It runs right here in your browser, with nothing to install.

The autoencoder used by a typical generator shrinks each side of the image eight times and keeps four numbers at each position. The denoiser then handles 16,384 numbers in place of 786,432: 48 times fewer, on every one of its calls.

The plan, then, has three steps.

  1. Train an autoencoder on images. Keep its encoder and its decoder.
  2. Encode every training image, and train a diffusion model on the codes.
  3. To generate: run the sampling loop to produce a code, and decode it once.

The diffusion model never sees a pixel.

Quick check
From the lesson on autoencoders: what is an autoencoder trained to do?
  1. Add noise to its input until nothing is left
  2. Predict the next input from the ones before it
  3. Sort its inputs into classes given by labels
  4. Rebuild its input from a small code
Show the answer

Rebuild its input from a small code

An autoencoder's target is its own input. An encoder compresses the input to a small code, a decoder rebuilds the input from the code, and the two are trained together to make the rebuilt version match. The code in the middle is the latent that diffusion will work on.

Quick check
An autoencoder shrinks each side of an image 8 times and keeps 4 numbers at each position. What does this print for a 256 by 256 color image: the numbers in the image, the numbers in its latent, and the ratio?
side = 256
pixels = side * side * 3
latent = (side // 8) ** 2 * 4
print(pixels, latent, pixels // latent)
  1. 196608 4096 48
  2. 65536 4096 16
  3. 196608 1024 192
  4. 196608 32768 6
Show the answer

196608 4096 48

The image has 256 × 256 × 3 = 196,608 numbers. Its latent is 32 by 32 positions with 4 numbers each: 4,096. The ratio is 48, the same as for the 512-pixel image, since both sides shrink by the same factor whatever the size.

Try it now

Press Next. The long walk happens on the small tensor; the large one is made only at the end.

In your own words
Why is it so much cheaper to run diffusion on an autoencoder's latent than on pixels?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: an autoencoder, then diffusion on the codes

An autoencoder fit for the job

Start with step 1, on the 6 by 6 images of Diffusion 3: images, labels and guidance. The autoencoder squeezes each image's 36 numbers into a code of 12.

Two details of its training matter, and both exist for the sake of the diffusion model that comes next.

  • The decoder is trained on codes that are slightly wrong. A little random noise is added to each code before decoding. A generated code will never be exactly the code of a real image, so the decoder has to turn "nearly right" into a clean picture.
  • The codes are kept small. A penalty on their size stops the encoder from spreading images far apart to make them easy to tell apart. Diffusion starts from standard noise, so the codes should live at about that scale.

Real systems get both properties from a variational autoencoder, which you met in the lesson on autoencoders, and then divide the codes by a fixed number so their spread is about 1.

Uses the setup above, 124 lines.

encoder, decoder = train_autoencoder(12)

codes = run(encoder, data)
rebuilt = run(decoder, codes)
print("images:", data.shape, "-> codes:", codes.shape)
print("images rebuilt exactly from their codes:",
      round(float(((rebuilt > 0) == (data > 0)).all(axis=1).mean()), 3))
print("one image's 12 numbers:", [round(float(v), 2) for v in codes[0]])
print("spread of the codes (standard deviation):", round(float(codes.std()), 3))

Every one of the 4,096 images is rebuilt exactly from its 12 numbers. A third of the size, and nothing lost.

The code itself is unreadable: 12 numbers with no obvious meaning. Nobody designed what each one stands for. The encoder and decoder agreed on it during training, and only the decoder can read it back.

The codes have a spread of 1.531, so they will be divided by that before diffusion, and multiplied by it again before decoding.

Quick check
Why is the decoder of a latent diffusion model trained on codes with a little noise added?
  1. The decoder would otherwise memorize the training images
  2. It replaces the noise that the diffusion model adds
  3. Generated codes are never exact, and it must still decode them cleanly
  4. The noise makes the codes shorter and cheaper to store
Show the answer

Generated codes are never exact, and it must still decode them cleanly

The diffusion model produces codes that are close to those of real images, not identical to them. A decoder that had only ever seen perfect codes might turn a slightly-off code into garbage. Training it on perturbed codes teaches it that everything near a real code should decode to the same clean image.

Quick check
Codes are divided by their spread before the diffusion model sees them, and multiplied by it again before decoding. What does this print?
codes = [3.0, -1.5, 0.0, 4.5]
scale = 1.5
scaled = [c / scale for c in codes]
print(scaled, [s * scale for s in scaled] == codes)
  1. [2.0, -1.0, 0.0, 3.0] True
  2. [2.0, -1.0, 0.0, 3.0] False
  3. [1.5, -3.0, -1.5, 3.0] True
  4. [4.5, -2.25, 0.0, 6.75] True
Show the answer

[2.0, -1.0, 0.0, 3.0] True

Dividing by 1.5 brings the codes to 2.0, −1.0, 0.0 and 3.0, the scale the diffusion model works at. Multiplying by 1.5 afterward returns exactly the original codes, which is what the decoder was trained on. Forgetting the second step is a classic bug: the decoder is handed codes of the wrong size and paints mud.

Diffusion on the codes

Step 2 needs no new ideas. A code is a row of 12 numbers, and the denoiser from Diffusion 3: images, labels and guidance handles rows of numbers. It is trained on the scaled codes, with labels and with the label hidden some of the time, exactly as before. Step 3 runs the guided sampling loop to make new codes, then decodes them.

The cell builds both systems, pixel diffusion and latent diffusion, and puts them side by side. Each generates 300 images at a guidance scale of 2.

Uses the setup above, 124 lines.

asked = np.repeat(np.arange(3), 100)

# diffusion on pixels, as in Diffusion 3
pixel_model = train_denoiser(data, kind, width=128)
pixel_images = sample(pixel_model, asked, 3, 36, clip=1.0)

encoder, decoder = train_autoencoder(12)
codes = run(encoder, data)
scale = float(codes.std())  # scale the codes to a spread of 1, like standard noise
# diffusion on codes
latent_model = train_denoiser((codes / scale).astype(np.float32), kind, width=64)
# generate codes, then decode them once
latent_images = run(decoder, sample(latent_model, asked, 3, 12) * scale)

print(
    "diffusion on | numbers per image | denoiser weights | exactly the image asked for")
print("pixels       |", str(36).ljust(17), "|", str(weights(pixel_model)).ljust(16),
      "|", exact_share(pixel_images, asked))
print("codes        |", str(12).ljust(17), "|", str(weights(latent_model)).ljust(16),
      "|", exact_share(latent_images, asked))
print("the autoencoder's own weights, used once per image:",
      weights(encoder) + weights(decoder))

The latent model matches the pixel model on quality, 0.93 against 0.903, with a denoiser of 6,348 weights in place of 27,044. That network is the one called twice at each of a hundred steps, so it is where the cost is. The autoencoder adds 6,320 weights, and its decoder runs once per image.

The scale of the saving here is modest, since 36 numbers were never expensive. On real images the same move shrinks the denoiser's input by 48 times, as you counted above.

There is a price, and the next table shows where it is paid.

Uses the setup above, 124 lines.

asked = np.repeat(np.arange(3), 100)
print("numbers in the code | rebuilt exactly | generated: exactly the image asked for")
for size in (2, 4, 12):
    encoder, decoder = train_autoencoder(size)
    codes = run(encoder, data)
    rebuilt = round(
        float(((run(decoder, codes) > 0) == (data > 0)).all(axis=1).mean()), 3)
    scale = float(codes.std())
    model = train_denoiser((codes / scale).astype(np.float32), kind, width=64)
    generated = run(decoder, sample(model, asked, 3, size) * scale)
    print(str(size).rjust(19), "|", str(rebuilt).ljust(15), "|",
          exact_share(generated, asked))

A code of 2 numbers is too small: only 0.606 of the images can even be rebuilt from it, and generation manages 0.38. With 4 numbers every image is rebuilt exactly, and generation still reaches only 0.637. With 12, it reaches 0.93.

Two limits are at work. The first is plain: what the autoencoder can't rebuild, the generator can't draw. That is why generated images used to garble small text and fingers, detail too fine for the latent grid to hold. The second is subtler. A code can be large enough to rebuild from and still be a hard space to do diffusion in, with 28 images crammed so close together that a small error lands on the wrong one. The latent has to be roomy enough for the denoiser, not only for the decoder.

Quick check
A latent diffusion model produces images in which small printed text is always garbled, however long it is trained. Which part most likely sets that limit?
  1. The guidance scale, which is set too low
  2. The autoencoder, whose latent is too coarse
  3. The noise schedule, which has too few steps
  4. The text encoder, which misreads the prompt
Show the answer

The autoencoder, whose latent is too coarse

The diffusion model only ever produces codes, and the decoder turns codes into pixels. Detail that the autoencoder can't represent in its latent, and so can't rebuild even from a real image, can't appear in a generated one. Better generators raised this ceiling with larger or richer latents.

Drill

How big is the latent?

Write latent_size(side, shrink, channels) for a square color image side pixels on a side, whose autoencoder shrinks each side shrink times and keeps channels numbers at each position. Return a tuple (pixel_numbers, latent_numbers, ratio): three numbers per pixel for the image, the count for the latent, and how many times smaller the latent is, rounded to 1 place. If the side isn't a multiple of shrink, raise ValueError.

Uses the setup above, 124 lines.

def latent_size(side, shrink, channels):
    return (0, 0, 0.0)


print(latent_size(512, 8, 4))
In the lesson, your code is checked as you write it.

The cell draws what comes out: twelve images generated by diffusion on the codes, four requests for each kind, and decoded once at the end.

Uses the setup above, 124 lines.

import matplotlib.pyplot as plt

encoder, decoder = train_autoencoder(12)
codes = run(encoder, data)
scale = float(codes.std())
model = train_denoiser((codes / scale).astype(np.float32), kind, width=64)
asked = np.repeat(np.arange(3), 4)                       # 4 flat, 4 upright, 4 square
made = run(decoder, sample(model, asked, 3, 12) * scale)
fig, axes = plt.subplots(3, 4, figsize=(5, 3.8))
for ax, image in zip(axes.ravel(), made):
    ax.imshow(image.reshape(6, 6), cmap="gray_r", vmin=-1, vmax=1)
    ax.axis("off")

Rows are flat bars, upright bars and squares. The table above says how many of such images are exactly right: 0.93 for the 12-number code.

In your own words
What limits the quality of a latent diffusion model that pixel diffusion doesn't have?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: from prompt to picture

A text-to-image generator is the system above with one substitution: the label becomes a sentence. It has three trained parts.

PartWhat it doesWhen it runs
Text encoderTurns the prompt into a sequence of vectors, one per token. Often the text half of an image-text model such as CLIP.Once per prompt
DenoiserGiven a noisy latent, the step number and the prompt's vectors, predicts the noise or the clean latent.Every step, twice with guidance
DecoderTurns the finished latent into pixels. The other half of the autoencoder.Once per image

The prompt's vectors reach the denoiser through cross-attention: at several points inside the network, each position of the latent grid attends over the prompt's tokens and takes in what is relevant to it. That is the attention you built in this phase, with queries from the image and keys and values from the text. The label in Diffusion 3: images, labels and guidance was the same thing, reduced to four numbers.

The denoiser was first a convolutional network of the shrink-and-enlarge kind you met for segmentation, with attention layers added. Newer systems use a transformer over patches of the latent.

Guidance works as before. "With no label" becomes the answer for an empty prompt. Many tools also accept a negative prompt, which takes the empty prompt's place: the guess is then pushed away from that text instead of away from nothing in particular.

Quick check
A team wants to speed up its latent diffusion generator. Which change cuts the most work from making one image?
  1. Using a smaller decoder
  2. Using a faster text encoder
  3. Sampling in 25 steps instead of 50
  4. Shortening the prompt by half
Show the answer

Sampling in 25 steps instead of 50

The denoiser is called at every step, twice with guidance, so the sampling loop is nearly all of the cost. Halving the steps halves it. The text encoder and the decoder each run once per image, so even making them free would save little.

Drill

Count the denoiser's work

Write denoiser_work(latent_numbers, steps, guided). Return a tuple (calls, numbers_processed): the number of times the denoiser is called to make one image, which is steps, doubled when guided is true, and the total numbers it handles, which is the calls times latent_numbers.

Uses the setup above, 124 lines.

def denoiser_work(latent_numbers, steps, guided):
    return (0, 0)


print(denoiser_work(16384, 50, True))
In the lesson, your code is checked as you write it.
Worked example

Worked example: where the cost of an image goes

Put the four dials of this track on one table: where diffusion runs, how many steps, guidance, and the image's size. The count is the numbers the denoiser has to handle to make one image. It leaves out the size of the network itself, so it is a way of comparing plans, not a prediction of seconds.

Uses the setup above, 124 lines.

def generation_cost(side, steps, guided, latent):
    """Count the work of generating one square color image."""
    # the autoencoder shrinks each side 8 times, into 4 channels
    numbers = (side // 8) ** 2 * 4 if latent else side * side * 3
    calls = steps * (2 if guided else 1)
    return {"numbers per denoiser call": numbers, "denoiser calls": calls,
            "numbers processed": numbers * calls, "decoder calls": 1 if latent else 0}

plans = [("pixels, 1000 steps, guided", 512, 1000, True, False),
         ("latent, 1000 steps, guided", 512, 1000, True, True),
         ("latent, 50 steps, guided", 512, 50, True, True),
         ("latent, 50 steps, guided, 1024 wide", 1024, 50, True, True),
         ("latent, 4 steps, no guidance", 512, 4, False, True)]
first = None
for name, side, steps, guided, latent in plans:
    cost = generation_cost(side, steps, guided, latent)
    first = first or cost["numbers processed"]
    print(name.ljust(36), "|", f"{cost['numbers per denoiser call']:,}".rjust(9),
          "numbers x", str(cost["denoiser calls"]).rjust(4), "calls =",
          f"{cost['numbers processed']:,}".rjust(13), "| 1 /",
          round(first / cost["numbers processed"]), "of the first")
In the lesson you can change this and run it.

The first row is the method as first described: diffusion on pixels, a thousand steps, guidance. The denoiser handles about 1.57 billion numbers for one image. Moving to a latent divides that by 48. Sampling in 50 steps instead of 1,000 divides it by 20 again, for 1/960 of the first row: these two changes are what took image generation from a research cluster to a laptop. The fourth row shows what size costs. Doubling the side quadruples the latent, and the work with it. The last row is the direction of recent systems: models trained to sample in a handful of steps without guidance, for 1/24,000 of the first row. Each dial was the subject of a lesson in this track, and they multiply.

What real generators use instead of this small network

The small denoiser here is a stack of linear layers on a row of numbers. Real latent models run a transformer over the latent. The latent is cut into patches, as in Build a vision transformer, and each patch becomes a token. Self-attention lets the patches talk to each other, and cross-attention lets each patch read the prompt: the queries come from the image patches and the keys and values from the text tokens. The cell runs that step on random numbers and prints the shapes.

Uses the setup above, 124 lines.

import torch

torch.manual_seed(0)
patches = torch.randn(1, 16, 32)                  # 16 latent patches of 32 numbers
text = torch.randn(1, 6, 32)                      # 6 text tokens from the prompt
Wq, Wk, Wv = (torch.randn(32, 32) for _ in range(3))
q, k, v = patches @ Wq, text @ Wk, text @ Wv      # the patches ask, the text answers
weights = torch.softmax(q @ k.transpose(1, 2) / 32 ** 0.5, dim=-1)
out = weights @ v
print(weights.shape, out.shape)
print([round(w, 2) for w in weights.sum(dim=-1)[0, :3].tolist()])

Each of the 16 patches gets a row of 6 weights over the text tokens, and a row adds to 1. The output has one blended text vector per patch, so the denoiser's input and output keep the latent's shape.

Flow matching is a newer way to train the same kind of model. Instead of predicting the noise at a random step, the network is trained to predict the straight-line velocity from a noise sample to a data sample. Sampling then follows those velocities in a few steps. The pieces are the same as here, a denoiser on latent patches and a text prompt read by attention, with a simpler training target.

On the job: when a latent model can't draw fine detail such as small text or faces, look at the autoencoder before the denoiser. Encode and decode a real image with the vae alone and compare: if the round trip already loses the detail, no amount of denoiser training or prompt work will bring it back. The same split tells you what to swap: a better decoder fixes detail, a better denoiser fixes composition.

In your own words
Trace what happens between typing a prompt and seeing an image, in a latent diffusion generator.

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

In a latent diffusion model, what does the diffusion process operate on?

  1. The codes produced by an autoencoder's encoder
  2. The pixels of the image, at full size
  3. The tokens of the text prompt
  4. The weights of the decoder

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write