Diffusion 3: images, labels and guidance
From points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.
What it covers6
Why it matters
A diffusion model that only makes "an image like the training set" is a curiosity. The ones people use take a request: a label, or a sentence. This lesson adds that. It moves from points to small images, gives the denoiser a label to look at, and then builds the technique behind the "guidance scale" setting in every image generator: classifier-free guidance. You will see why a model that has been told what to draw still often doesn't draw it, how guidance fixes that, and what it breaks when it is turned up too far.
What you'll be able to do
- Apply the same diffusion method to small images, predicting the clean image and clipping it to valid pixel values
- Condition a denoiser on a label, so that a particular kind of image can be asked for
- Train one network to answer with and without the label, by hiding the label some of the time
- Combine the two answers with classifier-free guidance, and say what the guidance scale trades
- Recognize an over-guided model: right kind, less variety, then broken images
Show the answer
Better matches first, then sameness and breakdown
Guidance pushes each denoising step further in the direction the prompt points. A moderate push turns vague, half-formed images into clear ones of the right kind. A hard push drives every sample toward the same exaggerated answer: variety drops first, and then the images themselves fall apart. This lesson measures all three stages.
The idea: two guesses, and a dial between them
This rests on Diffusion 2: learning to denoise, and sampling for the denoising loop and Vectors: lists of numbers that point for adding and scaling a difference.
From points to images
Nothing in Diffusion 1: turning data into noise or Diffusion 2: learning to denoise, and sampling was specific to points with two numbers. An image is a point with more numbers. The images here are 6 by 6 pixels and gray, so each is 36 numbers: −1 for a dark pixel and +1 for a bright one. There are three kinds.
Setup (runs before every cell) · 116 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F NAMES = ["a flat bar", "an upright bar", "a square"] def images(n, seed): """n tiny gray images, 6 by 6, of three kinds. The background is -1 and the shape is +1.""" g = np.random.default_rng(seed) kind = g.integers(0, 3, size=n) X = -np.ones((n, 6, 6), dtype=np.float32) for i in range(n): if kind[i] == 0: X[i, g.integers(0, 6), :] = 1 # a flat bar across one row elif kind[i] == 1: X[i, :, g.integers(0, 6)] = 1 # an upright bar down one column else: r, c = g.integers(0, 4), g.integers(0, 4) X[i, r:r + 3, c:c + 3] = 1 # a 3 by 3 square return X.reshape(n, 36), kind # each image flattened to 36 numbers def show(batch, titles): """Print a few images side by side: # where a pixel is above 0.""" print(" ".join(title.ljust(7)[:7] for title in titles)) for row in range(6): print(" ".join( " " + "".join("#" if v > 0 else "." for v in image.reshape(6, 6)[row]) for image in batch)) T = 100 betas = np.linspace(0.001, 0.2, T).astype(np.float32) alpha_bar = np.cumprod(1 - betas).astype(np.float32) NO_LABEL = 3 # what the network is told when no kind is asked for def inputs(x, t, label): """The noisy image, the step as a few waves, and the label as four on/off numbers. """ s = (np.asarray(t, dtype=np.float32) / T)[:, None] waves = [s, np.sin(2 * np.pi * s), np.cos(2 * np.pi * s), np.sin(4 * np.pi * s), np.cos(4 * np.pi * s)] return np.concatenate([x] + waves + [np.eye(4, dtype=np.float32)[label]], axis=1).astype(np.float32) def train(steps=1000, hide=0.15): torch.manual_seed(0) model = nn.Sequential(nn.Linear(45, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 36)) opt, g = torch.optim.Adam(model.parameters(), lr=2e-3), np.random.default_rng(0) data, kind = images(4096, 1) for step in range(steps): idx, t = g.integers(0, len(data), size=64), g.integers(1, T + 1, size=64) eps = g.normal(size=(64, 36)).astype(np.float32) xt = (np.sqrt(alpha_bar[t - 1])[:, None] * data[idx] + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps) # now and then, hide the label label = np.where(g.random(64) < hide, NO_LABEL, kind[idx]) # the target is the clean image loss = F.mse_loss(model(torch.tensor(inputs(xt, t, label))), torch.tensor(data[idx])) opt.zero_grad() loss.backward() opt.step() return model def predict_clean(model, x, t, label): with torch.no_grad(): return model(torch.tensor(inputs(x, np.full(len(x), t), label))).numpy() def sample(model, labels, seed, guidance=1.0, clip=True): """Generate one image per entry of labels. Guidance 0 ignores the labels, and 1 uses them plainly.""" g, n = np.random.default_rng(seed), len(labels) x = g.normal(size=(n, 36)).astype(np.float32) for t in range(T, 0, -1): # the guess with no label without = predict_clean(model, x, t, np.full(n, NO_LABEL)) clean = without if guidance == 0 else without + guidance * ( predict_clean(model, x, t, labels) - without) if clip: clean = np.clip(clean, -1, 1) # a pixel can't be outside -1 to 1 # the noise this guess implies noise = (x - np.sqrt(alpha_bar[t - 1]) * clean) / np.sqrt(1 - alpha_bar[t - 1]) x = ((x - betas[t - 1] / np.sqrt(1 - alpha_bar[t - 1]) * noise) / np.sqrt(1 - betas[t - 1])) if t > 1: x = x + np.sqrt(betas[t - 1]) * g.normal(size=x.shape).astype(np.float32) return x def every_real_image(): out, kinds = [], [] for kind in range(3): for a in range(6 if kind < 2 else 4): for b in range(1 if kind < 2 else 4): image = -np.ones((6, 6), dtype=np.float32) if kind == 0: image[a, :] = 1 elif kind == 1: image[:, a] = 1 else: image[a:a + 3, b:b + 3] = 1 out.append(image.reshape(36)) kinds.append(kind) return np.array(out), np.array(kinds) REAL, REAL_KIND = every_real_image() # all 28 images the data can contain def judge(batch): """For each generated image: its nearest real image, that image's kind, and how many pixels differ from it.""" differ = ((batch > 0)[:, None, :] != (REAL > 0)[None, :, :]).sum(axis=2) nearest = differ.argmin(axis=1) return nearest, REAL_KIND[nearest], differ.min(axis=1) model = train() # trained once, here, and reused by every cell
import numpy as np NAMES = ["a flat bar", "an upright bar", "a square"] def images(n, seed): """n tiny gray images, 6 by 6, of three kinds. The background is -1 and the shape is +1.""" g = np.random.default_rng(seed) kind = g.integers(0, 3, size=n) X = -np.ones((n, 6, 6), dtype=np.float32) for i in range(n): if kind[i] == 0: X[i, g.integers(0, 6), :] = 1 # a flat bar across one row elif kind[i] == 1: X[i, :, g.integers(0, 6)] = 1 # an upright bar down one column else: r, c = g.integers(0, 4), g.integers(0, 4) X[i, r:r + 3, c:c + 3] = 1 # a 3 by 3 square return X.reshape(n, 36), kind # each image flattened to 36 numbers def show(batch, titles): """Print a few images side by side: # where a pixel is above 0.""" print(" ".join(title.ljust(7)[:7] for title in titles)) for row in range(6): print(" ".join( " " + "".join("#" if v > 0 else "." for v in image.reshape(6, 6)[row]) for image in batch)) data, kind = images(4096, 1) print("images:", data.shape, "| pixel values:", sorted(set(data.reshape(-1).tolist()))) print("how many of each kind:", {NAMES[k]: int((kind == k).sum()) for k in range(3)}) print("different images in the whole set:", len({tuple(row) for row in data.tolist()})) print() first = [int(np.where(kind == k)[0][j]) for k in range(3) for j in range(2)] show(data[first], ["flat", "flat", "upright", "upright", "square", "square"])
The set holds 4,096 images, but only 28 different ones: six flat bars, six upright bars and sixteen squares. That makes it easy to judge a generated image exactly, by comparing it with all 28.
Two things change in the move to images, and both are standard practice.
The network predicts the clean image instead of the noise. You saw that the two carry the same information: from a guess of the noise you can work out the clean point, and from the clean point the noise. For images the clean picture is the easier target, since every pixel is known to lie between −1 and 1.
The guess is clipped. Before it is used, any pixel the network puts outside −1 to 1 is pulled back to the edge. A guess can't be right out there, and small excesses compound over a hundred steps.
The cell trains the denoiser and generates 300 images with no kind asked for. Each is compared with its nearest real image.
Uses the setup above, 116 lines.
# guidance 0: no kind is asked for generated = sample(model, np.zeros(300, dtype=int), 3, guidance=0) nearest, found, wrong = judge(generated) print("300 images generated with no label") print("exactly a real image:", int((wrong == 0).sum()), "| one or two pixels off:", int(((wrong > 0) & (wrong <= 2)).sum()), "| further off:", int((wrong > 2).sum())) print("the exact ones, by kind:", {NAMES[k]: int(((found == k) & (wrong == 0)).sum()) for k in range(3)}) print() show(generated[:6], ["no. " + str(i) for i in range(6)])
Of 300 images, 45 are exactly one of the real ones, 31 are a pixel or two off, and 224 are further away: blank, or a smudge, like most of the six printed. For a model this small, trained for a few seconds, making "some image from the set" out of pure noise is too vague a job. Each step's best guess is an average over flat bars, upright bars and squares at once, and an average of those is mud.
That weakness is the starting point for the rest of the lesson.
import math xt, clean, alpha_bar = 0.4, 2.0, 0.36 print(round((xt - math.sqrt(alpha_bar) * clean) / math.sqrt(1 - alpha_bar), 3))
Show the answer
-1.0
The noisy value is 0.6 × clean + 0.8 × noise. With a clean guess of 2.0, the data's share is 1.2, so the noise's share is 0.4 − 1.2 = −0.8, and dividing by its weight, 0.8, gives −1.0. A guess of the clean image and a guess of the noise are two ways of writing the same answer.
Press Next. Each arrow is a guess for the same pixel pair. Guided = no-label guess + scale × (with-label guess − no-label guess).
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: asking for a kind, and turning the dial
Asking for a kind
To ask for something, the network needs to be told what was asked. This is conditioning: an extra input that travels with the noisy image at every step. Here it is the label, written as four on/off numbers: one for each kind, and a fourth meaning "no label given".
During training each noisy image comes with its true label. The network learns "what clean image is under this noise, given that it is a square", a far narrower question than before. At sampling time you supply the label you want.
Uses the setup above, 116 lines.
asked = np.repeat(np.arange(3), 100) # 100 requests for each kind generated = sample(model, asked, 3, guidance=1.0) nearest, found, wrong = judge(generated) print("asked for | that kind, at most 2 pixels off | that kind exactly") for k in range(3): mine = asked == k print(NAMES[k].ljust(14), "|", str(int(((found[mine] == k) & (wrong[mine] <= 2)).sum())).rjust(31), "|", int(((found[mine] == k) & (wrong[mine] == 0)).sum())) print() show(generated[[0, 1, 100, 101, 200, 201]], ["flat", "flat", "upright", "upright", "square", "square"])
Asking helps a great deal. With no label 45 of 300 images were exact. Now 63 of 100 flat bars are, and 67 of 100 upright bars and 57 of 100 squares.
It is still far from reliable. A third of the requests for a flat bar didn't produce anything close to one, and the printed samples show why: some stay blank. The label is one more input among forty-five, and when the picture is nearly all noise, the network's guess leans on it only weakly.
In a text-to-image model the conditioning is a set of vectors from a text encoder, such as CLIP's, in place of this label. The principle is the same, and so is the problem: a prompt that is merely supplied is often half ignored.
Show the answer
Giving it the label as an extra input
The label is fed to the network alongside the noisy image and the step number, every time the network is called. One network handles all the labels. It learns how the right answer differs depending on what was asked for.
Guidance: turning the label up
The fix comes from an observation. The network can be asked twice at each step: once with the label, and once without. The difference between its two answers is exactly "what the label changes". If the label's effect is too weak, exaggerate it.
In plain words: start from the answer the network gives with no label. Add the difference the label makes, multiplied by w. With w = 1 this is just the answer with the label. With w above 1 it goes past it, further in the direction the label points.
The number w is the guidance scale, and the method is classifier-free guidance. One network gives both answers, because of a detail in training you may have noticed in the code: for 15% of the training examples, chosen at random, the label was replaced by "no label given". So the network knows how to answer either way.
Uses the setup above, 116 lines.
asked = np.repeat(np.arange(3), 100) print("guidance | the kind asked for, at most 2 pixels off | " "exactly a real image | different squares, of 16") for guidance in (0, 1, 2, 4, 8, 16): nearest, found, wrong = judge(sample(model, asked, 3, guidance=guidance)) # the clean squares among the 100 asked for squares = nearest[200:][(found[200:] == 2) & (wrong[200:] == 0)] print(str(guidance).rjust(8), "|", str(round(float(((found == asked) & (wrong <= 2)).mean()), 3)).ljust(41), "|", str(round(float(((found == asked) & (wrong == 0)).mean()), 3)).ljust(20), "|", len(set(squares.tolist())))
Read down the table.
- 0 ignores the label. Only 0.063 of the images are an exact image of the kind asked for.
- 1 is the plain conditioning of the last section: 0.623.
- 2 and 4 are the sweet spot. Over 0.9 of the images are exactly right, and at least 11 of the 16 possible squares still turn up among them.
- 8 begins to break. Exact images fall to 0.7, and only 6 different squares appear.
- 16 is wreckage: 0.297 exact, and 2 different squares.
So guidance buys obedience with variety. A moderate scale makes the model commit to a clear image of the right kind. A high scale pushes every sample toward the same exaggerated answer, and then past anything that was ever in the data.
It also costs time. Each step now calls the network twice, so generation takes twice as long.
without, with_label = 0.2, 0.5 print([round(without + w * (with_label - without), 2) for w in (0, 1, 2, 4)])
Show the answer
[0.2, 0.5, 0.8, 1.4]
The label moves the guess by 0.3. Guidance multiplies that move: none of it at 0, all of it at 1, twice at 2 (0.2 + 0.6 = 0.8), four times at 4 (0.2 + 1.2 = 1.4). At a scale of 4 the guess is 1.4, beyond the largest value a pixel can have, which is why the guess is clipped.
Combine two guesses with a guidance scale
Write guided(without, with_label, w, low, high). without and with_label are the network's two guesses for an image, as lists of pixel values. Return the guided guess, pixel by pixel: without + w * (with_label - without), then clipped to lie between low and high, and rounded to 3 places.
Uses the setup above, 116 lines.
def guided(without, with_label, w, low, high): return with_label print(guided([0.2, -0.4, 0.9], [0.5, -0.9, 1.0], 4, -1, 1))
Show the answer
During training, the label is replaced by "no label" for some examples
A random fraction of training examples, 15% here, have their label swapped for a special "no label" input. The network therefore practices both questions, and at sampling time either can be asked. No classifier is involved, which is where the method's name comes from.
Show the answer
100
Guidance needs two answers at every step, one with the condition and one without, so 50 steps make 100 calls. This is why guided generation takes about twice as long as unguided generation at the same number of steps.
Hide some labels for training
Write hide_labels(labels, draws, share, no_label). labels is a list of class numbers for a training batch. draws is a list of random numbers between 0 and 1, one per example. Return a new list in which a label is replaced by no_label wherever its draw is below share, and kept otherwise. Then write hidden_share(labels, no_label): the share of a list of labels equal to no_label, rounded to 3 places, or 0.0 for an empty list.
Uses the setup above, 116 lines.
def hide_labels(labels, draws, share, no_label): return labels def hidden_share(labels, no_label): return 0.0 batch = hide_labels([0, 2, 1, 1, 0, 2], [0.90, 0.10, 0.50, 0.03, 0.15, 0.72], 0.15, 3) print(batch, hidden_share(batch, 3))
Worked example: what the noise decides and what the label decides
Two things go into a generated image: the starting noise, which is fixed by the seed, and the label. Hold one still and change the other, at a guidance scale of 2. Then look at what clipping does at a scale of 4.
Uses the setup above, 116 lines.
print("the same starting noise, three different labels:") show(sample(model, np.array([0, 1, 2]), 7, guidance=2.0), ["flat", "upright", "square"]) print() print("the same label, four different starting noises:") show(np.concatenate([ sample(model, np.array([2]), seed, guidance=2.0) for seed in (1, 2, 3, 4)]), ["seed 1", "seed 2", "seed 3", "seed 4"]) print() asked = np.repeat(np.arange(3), 100) for clip in (True, False): generated = sample(model, asked, 3, guidance=4.0, clip=clip) nearest, found, wrong = judge(generated) print("guidance 4,", "guess clipped to -1..1 |" if clip else "guess not clipped |", "exactly a real image:", round(float(((found == asked) & (wrong == 0)).mean()), 3), "| largest pixel size reached:", round(float(np.abs(generated).max()), 2))
With the starting noise held still and the label changed, the three images share nothing visible: a flat bar, an upright bar, a square. The label decides what is drawn. With the label held at "square" and the noise changed, the squares land in different places, and two of the four seeds happen to give the same one. The noise decides which image of that kind you get, which is why an image tool that lets you fix the seed can reproduce a picture exactly. The fourth seed is a reminder that this small model still misses: its square is ragged. The last two lines are about clipping. At a scale of 4 the share of exact images is about the same with it and without it, 0.923 and 0.937, since the judging only asks whether each pixel is above zero. The values tell the other story: without clipping, a pixel reached 15.79 on a scale where 1 is the brightest possible. In a real image those are the harsh, blown-out patches of an over-guided picture.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: guidance in practice
Real image generators use exactly this dial. Their guidance scale is typically a number like 5 to 8, and they take a negative prompt: a second guess to push away from. The experiments below find the setting that works for this small model, then the library's version.
The cell below draws what the sweep counted. It asks for squares six times at four guidance scales, one row each, with the same starting noise in every row.
Uses the setup above, 116 lines.
import matplotlib.pyplot as plt asked = np.full(6, 2) # six requests for a square fig, axes = plt.subplots(4, 6, figsize=(6.5, 4.6)) for row, guidance in zip(axes, (0, 1, 4, 16)): for ax, image in zip(row, sample(model, asked, 3, guidance=guidance)): ax.imshow(image.reshape(6, 6), cmap="gray_r", vmin=-1, vmax=1) ax.axis("off") row[0].set_title(f"guidance {guidance}", fontsize=8, loc="left")
The sweep's counts say what to look for. With no guidance only 0.11 of the images are the kind asked for. At 4, 0.97 are. At 16 only 0.41 are, and the ones that are right come out as the same two or three squares.
On the job: guidance scale is the first knob to sweep when image quality disappoints. Fix the seed and the prompt, generate at several scales, and look at the row. Too low gives images that ignore the prompt, too high gives oversaturated, samey ones. Pick the scale where the prompt is followed and the pictures still differ, then keep it as a setting with the seed so a result can be reproduced.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Diffusion 4: latent diffusion and the whole pipeline: the whole pipeline, with a text encoder, a latent denoiser and a decoder.
- Diffusion 2: learning to denoise, and sampling: the denoising loop this lesson conditions.
- Alignment and safety: why a model's outputs are steered, and what negative prompts can and can't do.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
A denoiser for images is trained to predict the clean image, and its guess is clipped to the range of valid pixel values before use. What is the clipping for?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write