Index Ai Academy
Phase 5 · Beyond text

Diffusion 2: learning to denoise, and sampling

Train a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Diffusion 1: turning data into noise built the easy half of a diffusion model: a formula that buries data in noise. This one builds the half that matters. You will train a small network to look at a noisy point and say what noise is in it, then run that network a hundred times in a row, starting from nothing but random numbers, and watch a ring assemble itself. The same loop, with a much larger network and images instead of points, is what runs when an image generator turns a prompt into a picture.

What you'll be able to do

  • Train a network to predict the noise in a noisy example, and read its loss step range by step range
  • Write the sampling loop that walks from pure noise back to data, one step at a time
  • Explain why a little fresh noise is put back at every step of sampling
  • Trade quality for speed by visiting fewer steps, and find where that breaks
  • Check that generated samples are new, not copies of the training data
Quick check
A guess before you start. A denoising network is trained until it predicts the noise in a noisy example as well as it can. Can its loss reach zero?
  1. No: many different clean points could lie behind one noisy point
  2. Yes: but only if the network is given the step number
  3. No: because the network is too small to memorize the data
  4. Yes: with enough training the noise can always be recovered
Show the answer

No: many different clean points could lie behind one noisy point

A noisy point doesn't determine the noise that made it. The same noisy point could be a ring point from the left with noise pushing right, or one from the right with noise pushing left. The best any network can do is predict the average of the possibilities, which leaves an error that no amount of training removes. A loss well below 1 and well above 0 is what success looks like.

The idea: one step back from noise

Now the reverse process. Start from standard noise, which is where the forward process ends, and take the steps in reverse order: 100, 99, 98, down to 1. At each one:

  1. Ask the network what noise it sees in the current point.
  2. Remove this step's share of that noise. Not all of it: the guess is rough when the point is very noisy, so only a small correction is trusted.
  3. Add a little fresh noise back, except at the very last step.

Step 2 is the forward step run backward. The forward step multiplied by 1−βt\sqrt{1 - \beta_t} and added βt\sqrt{\beta_t} of noise. The reverse step subtracts the estimated noise, scaled by βt/1−αˉt\beta_t / \sqrt{1 - \bar\alpha_t}, and divides by 1−βt\sqrt{1 - \beta_t} to undo the shrinking.

Quick check
From the first diffusion lesson: a noisy example at step t is made with one formula. What does this print, for a point at 2.0 and noise of −1.0 at a step where alpha bar is 0.36?
import math

x0, eps, alpha_bar = 2.0, -1.0, 0.36
print(round(math.sqrt(alpha_bar) * x0 + math.sqrt(1 - alpha_bar) * eps, 3))
  1. 1.2
  2. 0.4
  3. 0.08
  4. 2.0
Show the answer

0.4

The weight on the data is √0.36 = 0.6, and the weight on the noise is √0.64 = 0.8. So the noisy point is 0.6 × 2.0 + 0.8 × (−1.0) = 1.2 − 0.8 = 0.4. The network would be shown 0.4 and the step, and asked for −1.0.

Quick check
One reverse step is applied to a single number, with the fresh noise left out. What does this print?
import math

x, guess, beta, alpha_bar = 1.0, 0.5, 0.19, 0.36
print(round((x - beta / math.sqrt(1 - alpha_bar) * guess) / math.sqrt(1 - beta), 3))
  1. 0.5
  2. 0.794
  3. 1.243
  4. 0.979
Show the answer

0.979

The share of the guess to remove is 0.19 / √0.64 = 0.2375, times 0.5, which is about 0.119. That leaves 0.881, and dividing by √0.81 = 0.9 undoes the forward step's shrinking: 0.979. The point barely moves. A reverse step is a small correction, and it takes many of them to get anywhere.

Try it now

Press Next. One step is a small correction. The sampler repeats it 100 times.

In your own words
What are the three parts of one reverse step, and why is only a share of the guessed noise removed?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: train the denoiser, then sample

Train the denoiser

The network's job was set at the end of Diffusion 1: given a noisy point and its step number, predict the noise that was added. Here it is, on the ring.

The network is a small multi-layer network with two hidden layers. It takes the two coordinates of the noisy point, and the step. The step is given as a handful of sine and cosine waves of the step number instead of the bare number, which makes it easier for the network to behave differently at different steps. Position embeddings in a transformer use the same trick.

Each training step draws a fresh batch: real points, a random step for each, and the noise, which is also the target.

The helpers (the ring data, the schedule, the network, train, sample and the rest) live in the lesson's setup, folded above. The setup also trains the network once for the cells after this one. This cell runs the training again so that you can watch it.

Setup (runs before every cell) · 102 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

def ring(n, seed):
    g = np.random.default_rng(seed)
    angle, radius = g.uniform(0, 2 * np.pi, size=n), 2 + 0.1 * g.normal(size=n)
    return np.stack([radius * np.cos(angle), radius * np.sin(angle)],
                    axis=1).astype(np.float32)

T = 100
betas = np.linspace(0.001, 0.2, T).astype(np.float32)    # the schedule from Diffusion 1
alpha_bar = np.cumprod(1 - betas).astype(np.float32)

def jump(x0, t, eps):
    return (np.sqrt(alpha_bar[t - 1])[:, None] * x0
            + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps)

def inputs(x, t):
    """What the network sees: the noisy point, and the step number
    written as a few smooth waves."""
    s = (np.asarray(t, dtype=np.float32) / T)[:, None]
    waves = [s, np.sin(2 * np.pi * s), np.cos(2 * np.pi * s), np.sin(4 * np.pi * s),
             np.cos(4 * np.pi * s)]
    return np.concatenate([x] + waves, axis=1).astype(np.float32)

def new_model():
    return nn.Sequential(nn.Linear(7, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(),
                         nn.Linear(64, 2))

def train(steps):
    torch.manual_seed(0)
    model, g = new_model(), np.random.default_rng(0)
    opt = torch.optim.Adam(model.parameters(), lr=2e-3)
    data, losses = ring(4096, 1), []
    for step in range(steps):
        x0 = data[g.integers(0, len(data), size=256)]            # real points
        t = g.integers(1, T + 1, size=256)  # a random step for each
        # the noise, which is the answer
        eps = g.normal(size=x0.shape).astype(np.float32)
        loss = F.mse_loss(model(torch.tensor(inputs(jump(x0, t, eps), t))),
                          torch.tensor(eps))
        opt.zero_grad()
        loss.backward()
        opt.step()
        losses.append(loss.item())
    return model, losses

def predict_noise(model, x, t):
    with torch.no_grad():
        return model(torch.tensor(inputs(x, np.full(len(x), t)))).numpy()

def sample(model, n, seed, add_noise=True, keep=()):
    """Start from standard noise and walk back through every step. Returns the samples
    and any steps asked for."""
    g = np.random.default_rng(seed)
    x, kept = g.normal(size=(n, 2)).astype(np.float32), {}
    for t in range(T, 0, -1):
        guess = predict_noise(model, x, t)
        # remove this step's share of the noise
        x = ((x - betas[t - 1] / np.sqrt(1 - alpha_bar[t - 1]) * guess)
             / np.sqrt(1 - betas[t - 1]))
        if add_noise and t > 1:
            # and put a little fresh noise back
            x = x + np.sqrt(betas[t - 1]) * g.normal(size=x.shape).astype(np.float32)
        if t - 1 in keep:
            kept[t - 1] = x.copy()
    return x, kept

def on_ring(x):
    return round(float((np.abs(np.linalg.norm(x, axis=1) - 2) < 0.3).mean()), 3)

def describe(x):
    radius = np.linalg.norm(x, axis=1)
    return ("mean distance " + str(round(float(radius.mean()), 3)).ljust(5)
            + " | spread " + str(round(float(radius.std()), 3)).ljust(5)
            + " | share within 0.3 of the ring " + str(on_ring(x)))


def sample_fast(model, n, seed, calls):
    """Visit only some of the 100 steps, evenly spaced, and jump between them."""
    g = np.random.default_rng(seed)
    x = g.normal(size=(n, 2)).astype(np.float32)
    visit = [int(round(s)) for s in np.linspace(T, 1, calls)]
    for i, t in enumerate(visit):
        guess = predict_noise(model, x, t)
        # the clean point this guess implies
        clean = (x - np.sqrt(1 - alpha_bar[t - 1]) * guess) / np.sqrt(alpha_bar[t - 1])
        if i + 1 == len(visit):
            return clean
        s = visit[i + 1]  # jump to the next step on the list
        x = np.sqrt(alpha_bar[s - 1]) * clean + np.sqrt(1 - alpha_bar[s - 1]) * guess


def nearest(points, others):
    """For each point, the distance to the closest of the other points."""
    return np.sqrt(
        ((points[:, None, :] - others[None, :, :]) ** 2).sum(axis=2).min(axis=1))


trained, trained_losses = train(1500)    # trained once, shared by the cells below
model, losses = train(1500)
print("loss over the first 50 steps:", round(float(np.mean(losses[:50])), 3),
      "| over the last 50:", round(float(np.mean(losses[-50:])), 3))

# fresh points the model never trained on
g, x0 = np.random.default_rng(9), ring(3000, 7)
print("steps      | the model | guessing 'no noise' | guessing 'it is all noise'")
for low, high in ((1, 10), (11, 30), (31, 60), (61, 100)):
    t, eps = (g.integers(low, high + 1, size=3000),
              g.normal(size=x0.shape).astype(np.float32))
    xt = jump(x0, t, eps)
    with torch.no_grad():
        guess = model(torch.tensor(inputs(xt, t))).numpy()
    print((str(low) + " to " + str(high)).ljust(10), "|",
          str(round(float(((guess - eps) ** 2).mean()), 3)).ljust(9), "|",
          str(round(float((eps ** 2).mean()), 3)).ljust(19), "|",
          round(float(((xt - eps) ** 2).mean()), 3))

The loss falls from 0.533 over the first 50 steps to 0.263 at the end, well under the 1.0 of a model that has learned nothing. It will not go to zero, for the reason in the opening question.

The table breaks that single number up by step, on fresh points, beside the two useless guesses from Diffusion 1. It shows where the work is.

  • Late steps, 61 to 100. The model scores 0.011. So does the guess "it is all noise", since by then it nearly is. Nothing is learned here.
  • Middle steps, 31 to 60. The model scores 0.235, and beats both guesses. This is where a blob starts to become a ring.
  • Early steps, 1 to 10. The noise is faint, and finding a faint thing exactly is hard: the model scores 0.73, against 1.032 for "no noise". Its error is large, and it matters little, since the points are almost clean already.
Quick check
In the table above, the model's loss on steps 61 to 100 is tiny, and so is the loss of simply guessing that the whole input is noise. What does that tell you?
  1. The model is overfitting to the late steps
  2. The schedule should have more late steps
  3. The model has learned the most at those steps
  4. Those steps are easy and test very little
Show the answer

Those steps are easy and test very little

At late steps the input is nearly pure noise, so "the input is the noise" is almost exactly right, and no knowledge of the data is needed. A low loss there says little. The steps that test the model are the middle ones, where data and noise are mixed in comparable amounts.

Walking back, in code

Now run the loop. The code below walks 1,000 points back through all 100 steps and keeps three snapshots along the way. A real image model does the same walk on pixels or on a compressed latent.

Uses the setup above, 102 lines.

import matplotlib.pyplot as plt

model = trained
samples, kept = sample(model, 1000, 3, keep=(60, 30, 10))
print("real data            |", describe(ring(1000, 5)))
for step in (60, 30, 10):
    print(("back at step " + str(step)).ljust(20), "|", describe(kept[step]))
print("finished: step 0     |", describe(samples))

fig, axes = plt.subplots(1, 4, figsize=(12, 3.2))
panels = [("step 60", kept[60]), ("step 30", kept[30]), ("step 10", kept[10]),
          ("step 0: the samples", samples)]
for ax, (title, points) in zip(axes, panels):
    ax.scatter(points[:, 0], points[:, 1], s=2)
    ax.set_xlim(-4, 4)
    ax.set_ylim(-4, 4)
    ax.set_aspect("equal")
    ax.set_title(title)

The points begin as a blob. Back at step 60 they still are: a mean distance of 1.297 from the center, where noise sits. By step 30 they have started to move outward, and by step 10 a ring is visible. At the end, 0.91 of the samples lie within 0.3 of the ring, at a mean distance of 1.986. The real data's is 2.004.

The match isn't perfect. The samples' distances have a spread of 0.188, about twice the real data's 0.097: the generated ring is fuzzier than the true one. A longer training run narrows that.

Nothing in the loop knows what a ring is. The loop is arithmetic, the same for any dataset. Everything about rings is inside the network, learned from examples by predicting noise.

Drill

One reverse step

Write reverse_step(x, guess, beta, alpha_bar, fresh). x is the current point and guess the network's noise estimate, both lists of numbers. beta and alpha_bar are the schedule's values at this step. fresh is a list of fresh standard noise, or None at the last step. For each coordinate: subtract beta / sqrt(1 - alpha_bar) times the guess, divide by sqrt(1 - beta), and then, if fresh is given, add sqrt(beta) times the fresh noise. Return the new point as a list, rounded to 3 places.

Uses the setup above, 102 lines.

def reverse_step(x, guess, beta, alpha_bar, fresh):
    return x


print(reverse_step([1.0, -2.0], [0.5, 0.0], 0.19, 0.36, None))
In the lesson, your code is checked as you write it.

Two questions about that loop are worth settling with a measurement. How much does training matter? And what is the fresh noise in step 3 for, when the whole aim is to remove noise?

Uses the setup above, 102 lines.

for steps in (0, 200, 1500):
    model, _ = train(steps)
    print("trained for", str(steps).rjust(4), "steps |",
          describe(sample(model, 1000, 3)[0]))
print("trained for 1500 steps, no noise put back |",
      describe(sample(model, 1000, 3, add_noise=False)[0]))
print("real data              |", describe(ring(1000, 5)))

The first three lines are the same loop with three networks. Untrained, it produces nothing near the ring. After 200 training steps, 0.428 of the samples land on it, and after 1,500, 0.91. The loop is only as good as the noise estimates it is fed.

The fourth line removes the fresh noise. Every sample now lands near the ring, which sounds like an improvement, until you read the other two numbers. The mean distance is 1.856, short of the true 2.0, and the spread is 0.006, against a real spread of 0.097. The samples have collapsed onto a thin circle that is the wrong size.

The reason goes back to the opening question. The network's guess is an average over everything that could have produced a noisy point. Follow averages at every step, with nothing to restore variety, and the samples drift toward the average of the data: too smooth, too similar, too central. The fresh noise puts back, at each step, the randomness that the averaging removes. In images, leaving it out gives the washed-out, over-smooth look that early samplers were known for.

Quick check
Why does the sampling loop add a little fresh noise after each denoising step?
  1. To make the loop run faster on large images
  2. To make sure each step uses the right beta
  3. To keep the network from memorizing the training data
  4. To restore variety that averaged guesses remove
Show the answer

To restore variety that averaged guesses remove

The network predicts the average noise over all the clean points that could explain what it sees. Stepping along averages alone pulls every sample toward the middle of the data and squeezes out its natural spread. Fresh noise at each step keeps the samples as varied as the real data.

In your own words
The sampling loop contains no knowledge of the data. Where is that knowledge, and how does the loop use it?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: fewer steps, and are the samples new?

Fewer steps

Sampling calls the network once per step, for every sample. A hundred calls is nothing here. For a large image model with a thousand steps it is the whole cost of generating a picture, so a great deal of work has gone into taking fewer steps.

The simplest approach uses a second way of reading the network's guess. If the guess is right, the clean point can be worked out directly, by rearranging the jump formula:

x^0=xt−1−αˉt  ε^αˉt\hat{x}_0 = \frac{x_t - \sqrt{1 - \bar\alpha_t}\; \hat\varepsilon}{\sqrt{\bar\alpha_t}}

In plain words: take the noisy point, subtract the noise the network thinks is in it, and scale back up by the amount the data was shrunk. ε^\hat\varepsilon is the guess, and x^0\hat{x}_0 is the clean point that guess implies.

With an estimate of the clean point in hand, the jump formula can carry it to any earlier step, not only the next one. So a sampler can visit, say, every tenth step and jump between them. This version adds no fresh noise, so the same starting noise always gives the same sample.

Uses the setup above, 102 lines.

model = trained
for calls in (100, 50, 20, 10, 5, 2):
    print(str(calls).rjust(3), "calls to the network |",
          describe(sample_fast(model, 1000, 3, calls)))

Ten calls do as well as a hundred: 0.92 of the samples on the ring against 0.928. Five calls slip to 0.754. Two calls fail completely, with samples a distance of 18 from the center.

The failure at two calls is the formula above meeting pure noise. At step 100 the weight on the data, αˉt\sqrt{\bar\alpha_t}, is 0.005, and dividing by it multiplies any error in the guess two hundred times over. From pure noise the network can only guess the middle of the data, so one giant leap lands nowhere useful. Several smaller leaps let each estimate correct the last.

So the number of steps is a dial between cost and quality, with a floor below which quality collapses. Real samplers of this kind commonly run 20 to 50 steps on models trained with 1,000.

Quick check
A noisy point is 0.4, the network guesses its noise as −1.0, and alpha bar at this step is 0.36. What does this print: the clean point the guess implies?
import math

xt, guess, alpha_bar = 0.4, -1.0, 0.36
print(round((xt - math.sqrt(1 - alpha_bar) * guess) / math.sqrt(alpha_bar), 3))
  1. 0.4
  2. 3.333
  3. 1.2
  4. 2.0
Show the answer

2.0

Subtracting the noise's share, 0.8 × (−1.0) = −0.8, gives 1.2. Dividing by the data's weight, 0.6, undoes the shrinking: 2.0. This is the example from the earlier question run backward: the noisy point 0.4 was made from a clean 2.0 and noise of −1.0, and a perfect guess recovers it exactly.

Drill

Which steps does a fast sampler visit?

Write steps_to_visit(total, calls). A fast sampler makes calls calls to the network, at steps spread evenly from total down to 1, both included. Return the list of steps, highest first. Step i of the list (counting from 0) is total - i * (total - 1) / (calls - 1), rounded to the nearest whole number with round. With a single call, visit only step total. If calls is less than 1 or more than total, raise ValueError.

Uses the setup above, 102 lines.

def steps_to_visit(total, calls):
    return []


print(steps_to_visit(100, 5))
In the lesson, your code is checked as you write it.
Worked example

Worked example: are the samples new?

A model that scored well by handing back its training data would be useless, and for a model trained on other people's images it would be a problem. Check directly: for each generated point, find the nearest of the 4,096 training points. Do the same for 1,000 real points the model never saw, as a yardstick. Then check that the samples cover the whole ring.

Uses the setup above, 102 lines.

model = trained
generated = sample(model, 1000, 3)[0]
# the training set, and real points the model never saw
training, fresh = ring(4096, 1), ring(1000, 99)

for name, points in (("generated points", generated), ("fresh real points", fresh)):
    d = nearest(points, training)
    print(name.ljust(17), "| distance to the nearest training point: median",
          round(float(np.median(d)), 4), "| exact copies of a training point:",
          int((d < 1e-4).sum()))

angle = np.arctan2(generated[:, 1], generated[:, 0])
print("generated points in each quarter of the ring:", [
    int(((angle >= a) & (angle < a + np.pi / 2)).sum())
    for a in (-np.pi, -np.pi / 2, 0, np.pi / 2)])
In the lesson you can change this and run it.

A generated point's nearest training point is a median of 0.0204 away. For fresh real points, which are certainly not copies, the figure is 0.0161. The two are of the same order: generated points sit about as close to the training set as new real data does, no closer, and none is an exact copy. The model has learned where rings are, not where its 4,096 examples were. The last line checks coverage. A model can also fail by producing only part of the data, which is called mode collapse. Here the four quarters of the ring get 231, 239, 236 and 294 of the 1,000 samples: all of the ring is covered, with one quarter somewhat favored. With this little training the model isn't perfectly even, and this is the kind of flaw a single quality score would hide.

On the job: the number of sampling steps is the first dial you turn when an image or audio model is too slow, because each step is one full network call. Measure quality at 50, 20 and 10 steps with a fixed seed before you pick one. Run the nearest-neighbor check on a sample of the outputs too: a model that hands back its training data is a legal problem, not only a quality one.

Goes deeper in

In your own words
You are handed a trained diffusion model and asked whether it is good. Name three separate things you would check.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

A denoising network's loss settles at 0.26 and won't go lower, however long it trains. What is the right reading?

  1. That may be near the best possible: the noise can't be fully recovered
  2. The network is broken, since a good loss is close to zero
  3. The learning rate must be too high for the loss to settle
  4. The network has memorized the training data

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write