Index Ai Academy
Phase 5 · Beyond text

Diffusion 1: turning data into noise

The easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Image generators such as Stable Diffusion start from a field of random noise and remove it a little at a time until a picture is left. The lesson on autoencoders mentioned that. This track builds it. The method, diffusion, rests on an odd trade: making a realistic image from nothing is very hard, and ruining one with noise is trivial. So diffusion ruins data on purpose, in many small steps, and trains a network to undo a single step. This first lesson is the easy half, the ruining, and it needs no network at all. Everything the hard half depends on is decided here.

What you'll be able to do

  • Describe the idea behind diffusion models: destroy data in small steps, and learn to undo one step
  • Apply one step of noise, and say why the data is shrunk as the noise is added
  • Jump straight to any step with the cumulative product of the schedule
  • Read a noise schedule: how much signal and how much noise there is at each step
  • Build the training examples a denoiser learns from, and say what counts as a good loss
Quick check
A guess before you start. When a diffusion model generates a new image, what does it begin with?
  1. A blank white canvas, which it paints on in strokes
  2. A rough sketch drawn by a second, smaller model
  3. Pure random noise, which it cleans up step by step
  4. A similar image from its training set, which it edits
Show the answer

Pure random noise, which it cleans up step by step

Generation starts from noise: an array of random numbers the size of the image. The model then removes a little noise at a time, perhaps over 50 or 1,000 steps, and structure emerges. To be able to do that, it was trained on the reverse journey, real images being buried in noise, which is what this lesson builds.

The idea: destroy a little at a time

This rests on Bernoulli, binomial, uniform and normal (the normal noise) and Exponentials: growth and decay (repeated shrinking).

A generative model must produce new examples like its training data. Doing that in one leap, from nothing to a finished image, is a hard thing to learn. Diffusion breaks the leap into a long staircase.

  • The forward process takes a real example and adds a little random noise, again and again, until nothing of it is left. This is done by a formula. Nothing is learned.
  • The reverse process is a network trained to undo one small step: given a slightly noisy example, make it slightly cleaner. Run it many times, starting from pure noise, and it walks back up the staircase to something that looks like data.

To see every part of this clearly, the track starts with the smallest possible "images": points with two numbers each. The dataset is a ring. A real image is the same thing with more numbers: three for every pixel, instead of two for the whole example.

Setup (runs before every cell) · 9 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import matplotlib.pyplot as plt

def ring(n, seed):
    """n points on a slightly fuzzy ring of radius 2: this track's stand-in
    for a dataset of images."""
    g = np.random.default_rng(seed)
    angle, radius = g.uniform(0, 2 * np.pi, size=n), 2 + 0.1 * g.normal(size=n)
    return np.stack([radius * np.cos(angle), radius * np.sin(angle)], axis=1)
data = ring(2000, 0)
noise = np.random.default_rng(1).normal(size=(2000, 2))

def describe(name, points):
    radius = np.linalg.norm(points, axis=1)      # each point's distance from the center
    print(name.ljust(10), "| mean distance from the center",
          round(float(radius.mean()), 3), "| spread of that distance",
          round(float(radius.std()), 3))

describe("the data", data)
describe("pure noise", noise)

fig, axes = plt.subplots(1, 2, figsize=(7, 3.4))
for ax, points, title in ((axes[0], data, "the data: a ring"),
                          (axes[1], noise, "pure noise")):
    ax.scatter(points[:, 0], points[:, 1], s=2)
    ax.set_xlim(-4, 4)
    ax.set_ylim(-4, 4)
    ax.set_aspect("equal")
    ax.set_title(title)

These are the two ends of the staircase. The data sits at a distance of about 2 from the center, with hardly any spread. Pure noise is a blob around the center. Each of its two numbers is drawn from a normal distribution with mean 0 and standard deviation 1, which this track calls standard noise.

One number will track progress between them all lesson: the mean distance from the center. It is 2.0 for the data and 1.252 for this draw of noise.

Quick check
The data sits about 2.0 from the center, with hardly any spread. Pure standard noise sits about 1.252 from it on average, with a wide spread. What can a point's distance tell you at this stage?
  1. Nothing: both are the same cloud
  2. Not much alone: noise can land at 2.0 too
  3. That noise is always closer than 1
  4. Everything: any point at 2.0 is data
Show the answer

Not much alone: noise can land at 2.0 too

The ring has a mean of 2.0 with a spread of 0.1; noise has a mean of 1.252 with a spread of 0.667. A noise point can land at 2.0, so one distance isn't proof, but the two clouds differ in mean and in spread.

One step of noise

A single forward step takes the current point and does two things: shrinks it slightly toward the center, and adds a little standard noise.

xt=1−βt  xt−1+βt  εx_t = \sqrt{1 - \beta_t}\; x_{t-1} + \sqrt{\beta_t}\; \varepsilon

In plain words: xt−1x_{t-1} is the point before the step and xtx_t the point after. ε\varepsilon (epsilon) is fresh standard noise. βt\beta_t (beta) is a small number between 0 and 1 that says how much of this step is noise. Keep most of the old point, mix in a little noise.

The list of betas, one per step, is the noise schedule. This track uses 100 steps, with beta rising in a straight line from 0.001 to 0.2: gentle at first, when there is fine detail to lose, and rough at the end.

Why shrink the point as well as adding noise? Because of what happens to the spread. The two weights are chosen so that their squares add to 1: (1−β)+β=1(1 - \beta) + \beta = 1. If the point already has a variance of 1, it still has a variance of 1 after the step. Without the shrinking, every step would add variance, and the points would drift outward forever instead of settling into standard noise.

Uses the setup above, 9 lines.

T = 100
betas = np.linspace(0.001, 0.2, T)         # the noise schedule: one beta per step

g = np.random.default_rng(0)
shrunk = ring(2000, 0)                     # the real rule
unshrunk = ring(2000, 0)                   # the same noise, with the shrinking left out
print("step | mean distance from the center | variance of "
      "each coordinate | variance without the shrinking")
for t in range(1, T + 1):
    eps = g.normal(size=shrunk.shape)
    shrunk = np.sqrt(1 - betas[t - 1]) * shrunk + np.sqrt(betas[t - 1]) * eps
    unshrunk = unshrunk + np.sqrt(betas[t - 1]) * eps
    if t in (1, 10, 25, 50, 100):
        print(str(t).rjust(4), "|",
              str(round(float(np.linalg.norm(shrunk, axis=1).mean()), 3)).ljust(29),
              "|", str(round(float(shrunk.var()), 3)).ljust(27), "|",
              round(float(unshrunk.var()), 3))

After one step the ring is barely touched: its mean distance from the center is 1.999. By step 25 it is 1.619, by step 50 it is 1.309, and after all 100 steps it is 1.254, which is what pure noise measured in the first cell. The ring is gone.

The variance columns show what the shrinking is for. This data starts with a variance of about 2 in each coordinate. With the shrinking, every step pulls that toward 1, and it ends at 1.014: standard noise. Without it, the same noise piles up on top of the data, and the variance climbs to 12.257 and would keep climbing. Generation has to start from the place where the forward process ends, so that place has to be known in advance.

Quick check
One forward step is applied to a single number. What does this print?
import math

x, beta, eps = 2.0, 0.19, -1.0
print(round(math.sqrt(1 - beta) * x + math.sqrt(beta) * eps, 3))
  1. 1.62
  2. 2.436
  3. 1.364
  4. 1.81
Show the answer

1.364

The square root of 1 − 0.19 = 0.81 is 0.9, so the old value is shrunk to 1.8. The square root of 0.19 is about 0.436, and the noise drawn was −1, so 0.436 is subtracted: 1.364. Both weights are square roots, which is what makes their squares add to 1.

Quick check
Why does each forward step multiply the point by a number slightly below 1 before adding noise?
  1. So fewer steps are needed to reach pure noise
  2. So the total variance stays at 1 instead of growing with every step
  3. So the points move toward the nearest training example
  4. So the noise added at each step can be removed exactly
Show the answer

So the total variance stays at 1 instead of growing with every step

Adding noise adds variance. Shrinking by the square root of 1 − beta removes exactly as much as the noise adds, when the data starts with a variance of 1. The process then ends at standard noise, a known distribution that generation can start from.

Try it now

Press Next through the steps 1, 10, 25, 50 and 100.

In your own words
What is a noise schedule, and why does this one start with small betas and end with large ones?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: jumping straight to step t

Jumping straight to step t

Training needs noisy examples from every step of the staircase, millions of them. Walking there one step at a time would be wasteful, and it isn't necessary. Because each step only scales and adds normal noise, any number of steps collapses into a single one.

Write αt=1−βt\alpha_t = 1 - \beta_t (alpha) for the share of signal each step keeps, and αˉt\bar\alpha_t ("alpha bar") for the product of all the alphas up to step t.

xt=αˉt  x0+1−αˉt  εwhereαˉt=α1×α2×⋯×αtx_t = \sqrt{\bar\alpha_t}\; x_0 + \sqrt{1 - \bar\alpha_t}\; \varepsilon \qquad \text{where} \quad \bar\alpha_t = \alpha_1 \times \alpha_2 \times \cdots \times \alpha_t

In plain words: to get the point at step t, take the original point x0x_0, scale it by the square root of alpha bar, and add standard noise scaled by the square root of what is left. One draw of noise does the work of t draws.

Alpha bar is the fraction of the original signal's variance that survives to step t. It starts near 1 and falls toward 0.

Uses the setup above, 9 lines.

T = 100
betas = np.linspace(0.001, 0.2, T)
alpha_bar = np.cumprod(1 - betas)                   # alpha_bar[t - 1] belongs to step t

x0 = ring(2000, 0)
g = np.random.default_rng(3)
print("step | alpha bar | weight on the data | "
      "weight on the noise | mean distance from the center")
for t in (1, 10, 25, 50, 100):
    signal, noise = np.sqrt(alpha_bar[t - 1]), np.sqrt(1 - alpha_bar[t - 1])
    xt = signal * x0 + noise * g.normal(size=x0.shape)    # the jump: no loop over steps
    print(str(t).rjust(4), "|", str(round(float(alpha_bar[t - 1]), 4)).ljust(9), "|",
          str(round(float(signal), 3)).ljust(18), "|",
          str(round(float(noise), 3)).ljust(19), "|",
          round(float(np.linalg.norm(xt, axis=1).mean()), 3))

The last column agrees with the step-by-step walk above, to within the randomness of the draws: 1.614 at step 25 against 1.619, and 1.248 at step 100 against 1.254. One calculation has replaced t of them.

The middle columns are the schedule, read as a recipe. At step 10 a noisy point is 0.951 parts data to 0.31 parts noise. By step 25 the two are nearly equal, 0.727 and 0.687. At step 50 the data's weight is down to 0.272, and at step 100 it is 0.005. The squares of the two weights always add to 1.

Quick check
A schedule has only three steps, with betas of 0.1, 0.2 and 0.5. What does this print: alpha bar after each step?
betas = [0.1, 0.2, 0.5]
alpha_bar, product = [], 1.0
for beta in betas:
    product *= 1 - beta
    alpha_bar.append(round(product, 2))
print(alpha_bar)
  1. [0.9, 0.8, 0.5]
  2. [0.9, 0.72, 0.36]
  3. [0.9, 0.7, 0.2]
  4. [0.1, 0.02, 0.01]
Show the answer

[0.9, 0.72, 0.36]

Each step keeps 1 − beta of the signal's variance: 0.9, then 0.8, then 0.5. Alpha bar is the running product: 0.9, then 0.9 × 0.8 = 0.72, then 0.72 × 0.5 = 0.36. After three steps, 36% of the original signal's variance is left.

Drill

Alpha bar from a schedule

Write alpha_bars(betas). Given the list of betas of a noise schedule, return a list of the same length holding alpha bar after each step: the running product of 1 - beta, each rounded to 4 places. Round only what you return, not the running product.

Uses the setup above, 9 lines.

def alpha_bars(betas):
    return []


print(alpha_bars([0.1, 0.2, 0.5]))
In the lesson, your code is checked as you write it.
Quick check
At some step of a schedule, alpha bar is 0.5. How much of a noisy point at that step is data, and how much is noise?
  1. Mostly data: 0.75 on the data and 0.25 on the noise
  2. Half each: 0.5 on the data and 0.5 on the noise
  3. Equal weights: about 0.707 on the data and 0.707 on the noise
  4. Mostly noise: 0.25 on the data and 0.75 on the noise
Show the answer

Equal weights: about 0.707 on the data and 0.707 on the noise

The weights are square roots: √0.5 ≈ 0.707 on the data and √(1 − 0.5) ≈ 0.707 on the noise. Their squares add to 1, as always. This is the step where signal and noise are equally strong, and in the schedule above it comes at about step 26 of 100.

Drill

Jump to a step

Write jump(x0, eps, alpha_bar). x0 is a point as a list of numbers, eps is a list of noise values of the same length, and alpha_bar is alpha bar at the step you are jumping to. Return the noisy point as a list: each coordinate is the square root of alpha bar times the coordinate of x0, plus the square root of one minus alpha bar times the matching noise value, rounded to 3 places.

Uses the setup above, 9 lines.

def jump(x0, eps, alpha_bar):
    return x0


print(jump([2.0, 0.0], [1.0, -1.0], 0.36))
In the lesson, your code is checked as you write it.

What the network will be asked

The forward process makes a training example out of nothing more than a real data point.

  1. Pick a real point x0x_0 from the dataset.
  2. Pick a step t at random.
  3. Draw standard noise ε\varepsilon, and jump: xt=αˉt x0+1−αˉt εx_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon.

The network is shown the noisy point xtx_t and the step number t, and is asked: what noise was added? Its answer is compared with the ε\varepsilon that was really used, by mean squared error. Knowing the noise is as good as knowing the clean point, since the formula can be rearranged to recover one from the other.

No labels are involved. The "right answer" is the noise you drew yourself, so this is self-supervised learning, in the family of tasks that hide something and ask for it back.

Before training anything, it helps to know what a useless model scores.

Uses the setup above, 9 lines.

T = 100
alpha_bar = np.cumprod(1 - np.linspace(0.001, 0.2, T))
g = np.random.default_rng(0)

x0 = ring(5000, 0)                              # 1. real points
t = g.integers(1, T + 1, size=5000)             # 2. a random step for each
eps = g.normal(size=x0.shape)                   # 3. the noise, which is also the answer
xt = (np.sqrt(alpha_bar[t - 1])[:, None] * x0
      + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps)

print("one training example | step", int(t[0]), "| noisy point", np.round(xt[0], 3),
      "| the noise to predict", np.round(eps[0], 3))
print("loss of a model that always answers 'no noise':      ",
      round(float((eps ** 2).mean()), 3))
print("loss of a model that answers 'the point is all noise':",
      round(float(((xt - eps) ** 2).mean()), 3))
late = t > 50
print("the same second model, on steps after 50 only:        ",
      round(float(((xt[late] - eps[late]) ** 2).mean()), 3))

The example printed first happens to be at step 86, where almost no data is left, and the noisy point and the noise are nearly the same numbers.

A model that always answers "no noise" scores 0.993, since standard noise has a mean square of 1. That is the number to beat: a loss near 1 means the network has learned nothing.

The second guess, "the whole point is noise", scores 0.651 overall and 0.023 on the late steps alone. The late steps are easy, since there the input nearly is the answer. All of the difficulty, and everything the network has to know about the data, is in the early and middle steps, where a point is part ring and part noise and the two have to be told apart. That is Diffusion 2: learning to denoise, and sampling.

Quick check
From the lesson on self-supervised learning: what makes a training task self-supervised?
  1. The model picks its own learning rate while it trains
  2. Its right answers come from the data itself, with no labels from people
  3. The model is trained with no loss function
  4. People label a few examples and the model labels the rest
Show the answer

Its right answers come from the data itself, with no labels from people

A self-supervised task builds its targets from the data: the other view of an image, a hidden patch, or here the noise that was just added. Nobody annotates anything, so a diffusion model can train on any pile of images.

In your own words
Why can a noisy example for any step be made in one calculation, and why does that matter for training?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: schedules and the library

The schedule is a design choice with a narrow target, and real libraries ship the common ones. Three schedules next, then the library call that makes a noisy example for you.

The cell draws the shares of data and noise against the step for each schedule. The crossing point, where the two curves meet, is the step where noise overtakes signal.

Uses the setup above, 9 lines.

T = 100
schedules = {"too gentle": np.full(T, 0.001), "too rough": np.full(T, 0.2),
             "rising": np.linspace(0.001, 0.2, T)}
fig, axes = plt.subplots(1, 3, figsize=(9, 2.8), sharey=True)
for ax, (name, betas) in zip(axes, schedules.items()):
    alpha_bar = np.cumprod(1 - betas)
    ax.plot(np.sqrt(alpha_bar), label="data")
    ax.plot(np.sqrt(1 - alpha_bar), label="noise")
    ax.set_title(name)
    ax.set_xlabel("step")
axes[0].legend()

The first schedule never lets the noise take over, so a sampler starting from pure noise would begin somewhere the network never trained. The second destroys the data within a few steps and wastes the rest. The rising one crosses mid-way and ends in pure noise.

Worked example

Worked example: three schedules, one of them usable

The schedule is a design choice with a narrow target. It must destroy the data completely by the last step, since generation will start from pure noise, and it must not destroy it too early, since steps where nothing is left teach nothing. Compare three ways of spending 100 steps.

Uses the setup above, 9 lines.

import numpy as np

T = 100
schedules = {
    "too gentle: beta 0.001 throughout": np.full(T, 0.001),
    "too rough: beta 0.2 throughout": np.full(T, 0.2),
    "rising from 0.001 to 0.2": np.linspace(0.001, 0.2, T),
}
for name, betas in schedules.items():
    alpha_bar = np.cumprod(1 - betas)
    # first step with more noise than signal
    even = int(np.argmax(alpha_bar < 0.5)) + 1 if (alpha_bar < 0.5).any() else None
    # first step with under 1% of the signal left
    gone = int(np.argmax(alpha_bar < 0.01)) + 1 if (alpha_bar < 0.01).any() else None
    print(name.ljust(34), "| signal left at the end:",
          str(round(float(alpha_bar[-1]), 4)).ljust(6),
          "| noise overtakes signal at step:", str(even).ljust(4),
          "| signal under 1% from step:", gone)
It runs right here in your browser, with nothing to install.

With a beta of 0.001 throughout, 0.9048 of the signal is still there after 100 steps. The data never becomes noise, so a sampler that starts from noise would begin somewhere the model was never trained. With a beta of 0.2 throughout, noise overtakes the signal at step 4, and under 1% of the signal is left from step 21 on, so about eighty of the hundred steps are spent on inputs that are pure noise and teach nothing. The rising schedule also ends with no signal, and spends its steps better: the crossover comes at step 27, and some signal lasts until step 67. Real systems use around 1,000 steps and gentler curves, and the test of a schedule is the same: where does the signal run out, and how many steps are spent where there is something to learn?

On the job: when a diffusion model's samples look washed out or muddy, look at the schedule first. Print the alpha bar at the last step: it should be near 0. If a fifth of the data is still there at the end, the sampler starts from a place the network never trained, and no amount of extra training steps fixes that. Libraries expose the schedule as a few settings, so the check is one line.

In your own words
A colleague's diffusion model produces washed-out, shapeless samples. You find its schedule leaves 40% of the signal at the last step. Why would that cause trouble?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Which part of a diffusion model involves no learning at all?

  1. Adding noise to data according to the schedule
  2. Estimating the noise in a noisy example
  3. Turning pure noise into a realistic sample
  4. Knowing what the training data looks like

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

The math behind this

The Phase 2 lessons this one rests on, from the course map.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write