Diffusion 4: latent diffusion and the whole pipeline
Why image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.
What it covers6
Why it matters
Everything so far would be too slow for real pictures. A denoiser that reads and writes every pixel of a large image, a hundred times over, twice per step with guidance, is an enormous amount of arithmetic. The systems people use don't do that. They squeeze each image into a much smaller grid of numbers first, run the whole diffusion process there, and turn the result back into pixels once at the end. This is latent diffusion, and it is the design behind Stable Diffusion and most of what followed. This lesson builds one in miniature, and then assembles the full pipeline, from prompt to picture.
What you'll be able to do
- Explain why image generators run diffusion on an autoencoder's codes instead of on pixels
- Build a latent diffusion model: compress images to codes, diffuse the codes, decode once at the end
- Say what the autoencoder must be trained to do so that generated codes decode well
- Name the three parts of a text-to-image generator and what each one does
- Count the work of generating an image from its size, its steps and its guidance
Show the answer
Once, at the very end
A latent diffusion model does all of its denoising on a small grid of numbers, typically 64 by 64, that an autoencoder knows how to turn into an image. Pixels appear only when the finished grid is decoded, once. That is the trick that made image generation cheap enough to run on an ordinary graphics card.
The idea: diffuse on a small code, not on pixels
This rests on Autoencoders, which squeeze a picture into a few numbers and back, and on Diffusion 3: images, labels and guidance for the denoiser and the label.
The cost of pixels
Count what the denoiser has to handle. A 512 by 512 color image is a point with three numbers per pixel. The denoiser reads all of them and writes all of them on every call.
The lesson on autoencoders described the way out. An autoencoder squeezes an image through a narrow middle, the latent, and rebuilds it on the other side. If the rebuilt image is good, then the latent holds everything that matters, in far fewer numbers.
Setup (runs before every cell) · 124 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F NAMES = ["a flat bar", "an upright bar", "a square"] def images(n, seed): """The 6 by 6 images of Diffusion 3: flat bars, upright bars and squares, as rows of 36 numbers.""" g = np.random.default_rng(seed) kind = g.integers(0, 3, size=n) X = -np.ones((n, 6, 6), dtype=np.float32) for i in range(n): if kind[i] == 0: X[i, g.integers(0, 6), :] = 1 elif kind[i] == 1: X[i, :, g.integers(0, 6)] = 1 else: r, c = g.integers(0, 4), g.integers(0, 4) X[i, r:r + 3, c:c + 3] = 1 return X.reshape(n, 36), kind def every_real_image(): out, kinds = [], [] for kind in range(3): for a in range(6 if kind < 2 else 4): for b in range(1 if kind < 2 else 4): image = -np.ones((6, 6), dtype=np.float32) if kind == 0: image[a, :] = 1 elif kind == 1: image[:, a] = 1 else: image[a:a + 3, b:b + 3] = 1 out.append(image.reshape(36)) kinds.append(kind) return np.array(out), np.array(kinds) REAL, REAL_KIND = every_real_image() def exact_share(batch, asked): """The share of generated images that are exactly a real image of the kind asked for.""" differ = ((batch > 0)[:, None, :] != (REAL > 0)[None, :, :]).sum(axis=2) exact = (REAL_KIND[differ.argmin(axis=1)] == asked) & (differ.min(axis=1) == 0) return round(float(exact.mean()), 3) def train_autoencoder(size, steps=1000): """An encoder from 36 pixels down to `size` numbers, and a decoder back up.""" torch.manual_seed(0) encoder = nn.Sequential(nn.Linear(36, 64), nn.ReLU(), nn.Linear(64, size)) decoder = nn.Sequential(nn.Linear(size, 64), nn.ReLU(), nn.Linear(64, 36)) opt, g = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()), lr=3e-3), np.random.default_rng(0) data, _ = images(4096, 1) for step in range(steps): x = torch.tensor(data[g.integers(0, len(data), size=64)]) code = encoder(x) # the decoder must cope with codes that are a little off shaken = code + 0.5 * torch.tensor(g.normal(size=(64, size)).astype(np.float32)) # and the codes are kept from growing large loss = F.mse_loss(decoder(shaken), x) + 0.01 * (code ** 2).mean() opt.zero_grad() loss.backward() opt.step() return encoder, decoder def run(network, array): with torch.no_grad(): return network(torch.tensor(np.asarray(array, dtype=np.float32))).numpy() T = 100 betas = np.linspace(0.001, 0.2, T).astype(np.float32) alpha_bar = np.cumprod(1 - betas).astype(np.float32) NO_LABEL = 3 def inputs(x, t, label): s = (np.asarray(t, dtype=np.float32) / T)[:, None] waves = [s, np.sin(2 * np.pi * s), np.cos(2 * np.pi * s), np.sin(4 * np.pi * s), np.cos(4 * np.pi * s)] return np.concatenate([x] + waves + [np.eye(4, dtype=np.float32)[label]], axis=1).astype(np.float32) def train_denoiser(data, kind, width, steps=1000): """The denoiser of Diffusion 3, for rows of any length: pixels or codes.""" d = data.shape[1] torch.manual_seed(0) model = nn.Sequential(nn.Linear(d + 9, width), nn.ReLU(), nn.Linear(width, width), nn.ReLU(), nn.Linear(width, d)) opt, g = torch.optim.Adam(model.parameters(), lr=2e-3), np.random.default_rng(0) for step in range(steps): idx, t = g.integers(0, len(data), size=64), g.integers(1, T + 1, size=64) eps = g.normal(size=(64, d)).astype(np.float32) xt = (np.sqrt(alpha_bar[t - 1])[:, None] * data[idx] + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps) label = np.where(g.random(64) < 0.15, NO_LABEL, kind[idx]) loss = F.mse_loss(model(torch.tensor(inputs(xt, t, label))), torch.tensor(data[idx])) opt.zero_grad() loss.backward() opt.step() return model def sample(model, labels, seed, d, guidance=2.0, clip=None): g, n = np.random.default_rng(seed), len(labels) x = g.normal(size=(n, d)).astype(np.float32) for t in range(T, 0, -1): without = run(model, inputs(x, np.full(n, t), np.full(n, NO_LABEL))) clean = (without + guidance * (run(model, inputs(x, np.full(n, t), labels)) - without)) if clip is not None: clean = np.clip(clean, -clip, clip) noise = (x - np.sqrt(alpha_bar[t - 1]) * clean) / np.sqrt(1 - alpha_bar[t - 1]) x = ((x - betas[t - 1] / np.sqrt(1 - alpha_bar[t - 1]) * noise) / np.sqrt(1 - betas[t - 1])) if t > 1: x = x + np.sqrt(betas[t - 1]) * g.normal(size=x.shape).astype(np.float32) return x def weights(network): return sum(p.numel() for p in network.parameters()) data, kind = images(4096, 1)
for name, height, width, channels in ( ("pixels of a 512 by 512 color image", 512, 512, 3), ("its latent in a typical generator", 64, 64, 4)): print(name.ljust(34), "|", height, "x", width, "x", channels, "=", f"{height * width * channels:,}", "numbers") print("the denoiser handles", 512 * 512 * 3 // (64 * 64 * 4), "times fewer numbers per call in the latent")
The autoencoder used by a typical generator shrinks each side of the image eight times and keeps four numbers at each position. The denoiser then handles 16,384 numbers in place of 786,432: 48 times fewer, on every one of its calls.
The plan, then, has three steps.
- Train an autoencoder on images. Keep its encoder and its decoder.
- Encode every training image, and train a diffusion model on the codes.
- To generate: run the sampling loop to produce a code, and decode it once.
The diffusion model never sees a pixel.
Show the answer
Rebuild its input from a small code
An autoencoder's target is its own input. An encoder compresses the input to a small code, a decoder rebuilds the input from the code, and the two are trained together to make the rebuilt version match. The code in the middle is the latent that diffusion will work on.
side = 256 pixels = side * side * 3 latent = (side // 8) ** 2 * 4 print(pixels, latent, pixels // latent)
Show the answer
196608 4096 48
The image has 256 × 256 × 3 = 196,608 numbers. Its latent is 32 by 32 positions with 4 numbers each: 4,096. The ratio is 48, the same as for the 512-pixel image, since both sides shrink by the same factor whatever the size.
Press Next. The long walk happens on the small tensor; the large one is made only at the end.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: an autoencoder, then diffusion on the codes
An autoencoder fit for the job
Start with step 1, on the 6 by 6 images of Diffusion 3: images, labels and guidance. The autoencoder squeezes each image's 36 numbers into a code of 12.
Two details of its training matter, and both exist for the sake of the diffusion model that comes next.
- The decoder is trained on codes that are slightly wrong. A little random noise is added to each code before decoding. A generated code will never be exactly the code of a real image, so the decoder has to turn "nearly right" into a clean picture.
- The codes are kept small. A penalty on their size stops the encoder from spreading images far apart to make them easy to tell apart. Diffusion starts from standard noise, so the codes should live at about that scale.
Real systems get both properties from a variational autoencoder, which you met in the lesson on autoencoders, and then divide the codes by a fixed number so their spread is about 1.
Uses the setup above, 124 lines.
encoder, decoder = train_autoencoder(12) codes = run(encoder, data) rebuilt = run(decoder, codes) print("images:", data.shape, "-> codes:", codes.shape) print("images rebuilt exactly from their codes:", round(float(((rebuilt > 0) == (data > 0)).all(axis=1).mean()), 3)) print("one image's 12 numbers:", [round(float(v), 2) for v in codes[0]]) print("spread of the codes (standard deviation):", round(float(codes.std()), 3))
Every one of the 4,096 images is rebuilt exactly from its 12 numbers. A third of the size, and nothing lost.
The code itself is unreadable: 12 numbers with no obvious meaning. Nobody designed what each one stands for. The encoder and decoder agreed on it during training, and only the decoder can read it back.
The codes have a spread of 1.531, so they will be divided by that before diffusion, and multiplied by it again before decoding.
Show the answer
Generated codes are never exact, and it must still decode them cleanly
The diffusion model produces codes that are close to those of real images, not identical to them. A decoder that had only ever seen perfect codes might turn a slightly-off code into garbage. Training it on perturbed codes teaches it that everything near a real code should decode to the same clean image.
codes = [3.0, -1.5, 0.0, 4.5] scale = 1.5 scaled = [c / scale for c in codes] print(scaled, [s * scale for s in scaled] == codes)
Show the answer
[2.0, -1.0, 0.0, 3.0] True
Dividing by 1.5 brings the codes to 2.0, −1.0, 0.0 and 3.0, the scale the diffusion model works at. Multiplying by 1.5 afterward returns exactly the original codes, which is what the decoder was trained on. Forgetting the second step is a classic bug: the decoder is handed codes of the wrong size and paints mud.
Diffusion on the codes
Step 2 needs no new ideas. A code is a row of 12 numbers, and the denoiser from Diffusion 3: images, labels and guidance handles rows of numbers. It is trained on the scaled codes, with labels and with the label hidden some of the time, exactly as before. Step 3 runs the guided sampling loop to make new codes, then decodes them.
The cell builds both systems, pixel diffusion and latent diffusion, and puts them side by side. Each generates 300 images at a guidance scale of 2.
Uses the setup above, 124 lines.
asked = np.repeat(np.arange(3), 100) # diffusion on pixels, as in Diffusion 3 pixel_model = train_denoiser(data, kind, width=128) pixel_images = sample(pixel_model, asked, 3, 36, clip=1.0) encoder, decoder = train_autoencoder(12) codes = run(encoder, data) scale = float(codes.std()) # scale the codes to a spread of 1, like standard noise # diffusion on codes latent_model = train_denoiser((codes / scale).astype(np.float32), kind, width=64) # generate codes, then decode them once latent_images = run(decoder, sample(latent_model, asked, 3, 12) * scale) print( "diffusion on | numbers per image | denoiser weights | exactly the image asked for") print("pixels |", str(36).ljust(17), "|", str(weights(pixel_model)).ljust(16), "|", exact_share(pixel_images, asked)) print("codes |", str(12).ljust(17), "|", str(weights(latent_model)).ljust(16), "|", exact_share(latent_images, asked)) print("the autoencoder's own weights, used once per image:", weights(encoder) + weights(decoder))
The latent model matches the pixel model on quality, 0.93 against 0.903, with a denoiser of 6,348 weights in place of 27,044. That network is the one called twice at each of a hundred steps, so it is where the cost is. The autoencoder adds 6,320 weights, and its decoder runs once per image.
The scale of the saving here is modest, since 36 numbers were never expensive. On real images the same move shrinks the denoiser's input by 48 times, as you counted above.
There is a price, and the next table shows where it is paid.
Uses the setup above, 124 lines.
asked = np.repeat(np.arange(3), 100) print("numbers in the code | rebuilt exactly | generated: exactly the image asked for") for size in (2, 4, 12): encoder, decoder = train_autoencoder(size) codes = run(encoder, data) rebuilt = round( float(((run(decoder, codes) > 0) == (data > 0)).all(axis=1).mean()), 3) scale = float(codes.std()) model = train_denoiser((codes / scale).astype(np.float32), kind, width=64) generated = run(decoder, sample(model, asked, 3, size) * scale) print(str(size).rjust(19), "|", str(rebuilt).ljust(15), "|", exact_share(generated, asked))
A code of 2 numbers is too small: only 0.606 of the images can even be rebuilt from it, and generation manages 0.38. With 4 numbers every image is rebuilt exactly, and generation still reaches only 0.637. With 12, it reaches 0.93.
Two limits are at work. The first is plain: what the autoencoder can't rebuild, the generator can't draw. That is why generated images used to garble small text and fingers, detail too fine for the latent grid to hold. The second is subtler. A code can be large enough to rebuild from and still be a hard space to do diffusion in, with 28 images crammed so close together that a small error lands on the wrong one. The latent has to be roomy enough for the denoiser, not only for the decoder.
Show the answer
The autoencoder, whose latent is too coarse
The diffusion model only ever produces codes, and the decoder turns codes into pixels. Detail that the autoencoder can't represent in its latent, and so can't rebuild even from a real image, can't appear in a generated one. Better generators raised this ceiling with larger or richer latents.
How big is the latent?
Write latent_size(side, shrink, channels) for a square color image side pixels on a side, whose autoencoder shrinks each side shrink times and keeps channels numbers at each position. Return a tuple (pixel_numbers, latent_numbers, ratio): three numbers per pixel for the image, the count for the latent, and how many times smaller the latent is, rounded to 1 place. If the side isn't a multiple of shrink, raise ValueError.
Uses the setup above, 124 lines.
def latent_size(side, shrink, channels): return (0, 0, 0.0) print(latent_size(512, 8, 4))
The cell draws what comes out: twelve images generated by diffusion on the codes, four requests for each kind, and decoded once at the end.
Uses the setup above, 124 lines.
import matplotlib.pyplot as plt encoder, decoder = train_autoencoder(12) codes = run(encoder, data) scale = float(codes.std()) model = train_denoiser((codes / scale).astype(np.float32), kind, width=64) asked = np.repeat(np.arange(3), 4) # 4 flat, 4 upright, 4 square made = run(decoder, sample(model, asked, 3, 12) * scale) fig, axes = plt.subplots(3, 4, figsize=(5, 3.8)) for ax, image in zip(axes.ravel(), made): ax.imshow(image.reshape(6, 6), cmap="gray_r", vmin=-1, vmax=1) ax.axis("off")
Rows are flat bars, upright bars and squares. The table above says how many of such images are exactly right: 0.93 for the 12-number code.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: from prompt to picture
A text-to-image generator is the system above with one substitution: the label becomes a sentence. It has three trained parts.
| Part | What it does | When it runs |
|---|---|---|
| Text encoder | Turns the prompt into a sequence of vectors, one per token. Often the text half of an image-text model such as CLIP. | Once per prompt |
| Denoiser | Given a noisy latent, the step number and the prompt's vectors, predicts the noise or the clean latent. | Every step, twice with guidance |
| Decoder | Turns the finished latent into pixels. The other half of the autoencoder. | Once per image |
The prompt's vectors reach the denoiser through cross-attention: at several points inside the network, each position of the latent grid attends over the prompt's tokens and takes in what is relevant to it. That is the attention you built in this phase, with queries from the image and keys and values from the text. The label in Diffusion 3: images, labels and guidance was the same thing, reduced to four numbers.
The denoiser was first a convolutional network of the shrink-and-enlarge kind you met for segmentation, with attention layers added. Newer systems use a transformer over patches of the latent.
Guidance works as before. "With no label" becomes the answer for an empty prompt. Many tools also accept a negative prompt, which takes the empty prompt's place: the guess is then pushed away from that text instead of away from nothing in particular.
Show the answer
Sampling in 25 steps instead of 50
The denoiser is called at every step, twice with guidance, so the sampling loop is nearly all of the cost. Halving the steps halves it. The text encoder and the decoder each run once per image, so even making them free would save little.
Count the denoiser's work
Write denoiser_work(latent_numbers, steps, guided). Return a tuple (calls, numbers_processed): the number of times the denoiser is called to make one image, which is steps, doubled when guided is true, and the total numbers it handles, which is the calls times latent_numbers.
Uses the setup above, 124 lines.
def denoiser_work(latent_numbers, steps, guided): return (0, 0) print(denoiser_work(16384, 50, True))
Worked example: where the cost of an image goes
Put the four dials of this track on one table: where diffusion runs, how many steps, guidance, and the image's size. The count is the numbers the denoiser has to handle to make one image. It leaves out the size of the network itself, so it is a way of comparing plans, not a prediction of seconds.
Uses the setup above, 124 lines.
def generation_cost(side, steps, guided, latent): """Count the work of generating one square color image.""" # the autoencoder shrinks each side 8 times, into 4 channels numbers = (side // 8) ** 2 * 4 if latent else side * side * 3 calls = steps * (2 if guided else 1) return {"numbers per denoiser call": numbers, "denoiser calls": calls, "numbers processed": numbers * calls, "decoder calls": 1 if latent else 0} plans = [("pixels, 1000 steps, guided", 512, 1000, True, False), ("latent, 1000 steps, guided", 512, 1000, True, True), ("latent, 50 steps, guided", 512, 50, True, True), ("latent, 50 steps, guided, 1024 wide", 1024, 50, True, True), ("latent, 4 steps, no guidance", 512, 4, False, True)] first = None for name, side, steps, guided, latent in plans: cost = generation_cost(side, steps, guided, latent) first = first or cost["numbers processed"] print(name.ljust(36), "|", f"{cost['numbers per denoiser call']:,}".rjust(9), "numbers x", str(cost["denoiser calls"]).rjust(4), "calls =", f"{cost['numbers processed']:,}".rjust(13), "| 1 /", round(first / cost["numbers processed"]), "of the first")
The first row is the method as first described: diffusion on pixels, a thousand steps, guidance. The denoiser handles about 1.57 billion numbers for one image. Moving to a latent divides that by 48. Sampling in 50 steps instead of 1,000 divides it by 20 again, for 1/960 of the first row: these two changes are what took image generation from a research cluster to a laptop. The fourth row shows what size costs. Doubling the side quadruples the latent, and the work with it. The last row is the direction of recent systems: models trained to sample in a handful of steps without guidance, for 1/24,000 of the first row. Each dial was the subject of a lesson in this track, and they multiply.
What real generators use instead of this small network
The small denoiser here is a stack of linear layers on a row of numbers. Real latent models run a transformer over the latent. The latent is cut into patches, as in Build a vision transformer, and each patch becomes a token. Self-attention lets the patches talk to each other, and cross-attention lets each patch read the prompt: the queries come from the image patches and the keys and values from the text tokens. The cell runs that step on random numbers and prints the shapes.
Uses the setup above, 124 lines.
import torch torch.manual_seed(0) patches = torch.randn(1, 16, 32) # 16 latent patches of 32 numbers text = torch.randn(1, 6, 32) # 6 text tokens from the prompt Wq, Wk, Wv = (torch.randn(32, 32) for _ in range(3)) q, k, v = patches @ Wq, text @ Wk, text @ Wv # the patches ask, the text answers weights = torch.softmax(q @ k.transpose(1, 2) / 32 ** 0.5, dim=-1) out = weights @ v print(weights.shape, out.shape) print([round(w, 2) for w in weights.sum(dim=-1)[0, :3].tolist()])
Each of the 16 patches gets a row of 6 weights over the text tokens, and a row adds to 1. The output has one blended text vector per patch, so the denoiser's input and output keep the latent's shape.
Flow matching is a newer way to train the same kind of model. Instead of predicting the noise at a random step, the network is trained to predict the straight-line velocity from a noise sample to a data sample. Sampling then follows those velocities in a few steps. The pieces are the same as here, a denoiser on latent patches and a text prompt read by attention, with a simpler training target.
On the job: when a latent model can't draw fine detail such as small text or faces, look at the autoencoder before the denoiser. Encode and decode a real image with the vae alone and compare: if the round trip already loses the detail, no amount of denoiser training or prompt work will bring it back. The same split tells you what to swap: a better decoder fixes detail, a better denoiser fixes composition.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Diffusion 3: images, labels and guidance: the guidance scale, and how a prompt steers every step.
- Autoencoders: the squeeze-and-rebuild network behind the latent.
- Multimodal transformers: how text and images share one space in other model families.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
In a latent diffusion model, what does the diffusion process operate on?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write