Diffusion 1: turning data into noise
The easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.
What it covers6
Why it matters
Image generators such as Stable Diffusion start from a field of random noise and remove it a little at a time until a picture is left. The lesson on autoencoders mentioned that. This track builds it. The method, diffusion, rests on an odd trade: making a realistic image from nothing is very hard, and ruining one with noise is trivial. So diffusion ruins data on purpose, in many small steps, and trains a network to undo a single step. This first lesson is the easy half, the ruining, and it needs no network at all. Everything the hard half depends on is decided here.
What you'll be able to do
- Describe the idea behind diffusion models: destroy data in small steps, and learn to undo one step
- Apply one step of noise, and say why the data is shrunk as the noise is added
- Jump straight to any step with the cumulative product of the schedule
- Read a noise schedule: how much signal and how much noise there is at each step
- Build the training examples a denoiser learns from, and say what counts as a good loss
Show the answer
Pure random noise, which it cleans up step by step
Generation starts from noise: an array of random numbers the size of the image. The model then removes a little noise at a time, perhaps over 50 or 1,000 steps, and structure emerges. To be able to do that, it was trained on the reverse journey, real images being buried in noise, which is what this lesson builds.
The idea: destroy a little at a time
This rests on Bernoulli, binomial, uniform and normal (the normal noise) and Exponentials: growth and decay (repeated shrinking).
A generative model must produce new examples like its training data. Doing that in one leap, from nothing to a finished image, is a hard thing to learn. Diffusion breaks the leap into a long staircase.
- The forward process takes a real example and adds a little random noise, again and again, until nothing of it is left. This is done by a formula. Nothing is learned.
- The reverse process is a network trained to undo one small step: given a slightly noisy example, make it slightly cleaner. Run it many times, starting from pure noise, and it walks back up the staircase to something that looks like data.
To see every part of this clearly, the track starts with the smallest possible "images": points with two numbers each. The dataset is a ring. A real image is the same thing with more numbers: three for every pixel, instead of two for the whole example.
Setup (runs before every cell) · 9 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import matplotlib.pyplot as plt def ring(n, seed): """n points on a slightly fuzzy ring of radius 2: this track's stand-in for a dataset of images.""" g = np.random.default_rng(seed) angle, radius = g.uniform(0, 2 * np.pi, size=n), 2 + 0.1 * g.normal(size=n) return np.stack([radius * np.cos(angle), radius * np.sin(angle)], axis=1)
data = ring(2000, 0) noise = np.random.default_rng(1).normal(size=(2000, 2)) def describe(name, points): radius = np.linalg.norm(points, axis=1) # each point's distance from the center print(name.ljust(10), "| mean distance from the center", round(float(radius.mean()), 3), "| spread of that distance", round(float(radius.std()), 3)) describe("the data", data) describe("pure noise", noise) fig, axes = plt.subplots(1, 2, figsize=(7, 3.4)) for ax, points, title in ((axes[0], data, "the data: a ring"), (axes[1], noise, "pure noise")): ax.scatter(points[:, 0], points[:, 1], s=2) ax.set_xlim(-4, 4) ax.set_ylim(-4, 4) ax.set_aspect("equal") ax.set_title(title)
These are the two ends of the staircase. The data sits at a distance of about 2 from the center, with hardly any spread. Pure noise is a blob around the center. Each of its two numbers is drawn from a normal distribution with mean 0 and standard deviation 1, which this track calls standard noise.
One number will track progress between them all lesson: the mean distance from the center. It is 2.0 for the data and 1.252 for this draw of noise.
Show the answer
Not much alone: noise can land at 2.0 too
The ring has a mean of 2.0 with a spread of 0.1; noise has a mean of 1.252 with a spread of 0.667. A noise point can land at 2.0, so one distance isn't proof, but the two clouds differ in mean and in spread.
One step of noise
A single forward step takes the current point and does two things: shrinks it slightly toward the center, and adds a little standard noise.
In plain words: is the point before the step and the point after. (epsilon) is fresh standard noise. (beta) is a small number between 0 and 1 that says how much of this step is noise. Keep most of the old point, mix in a little noise.
The list of betas, one per step, is the noise schedule. This track uses 100 steps, with beta rising in a straight line from 0.001 to 0.2: gentle at first, when there is fine detail to lose, and rough at the end.
Why shrink the point as well as adding noise? Because of what happens to the spread. The two weights are chosen so that their squares add to 1: . If the point already has a variance of 1, it still has a variance of 1 after the step. Without the shrinking, every step would add variance, and the points would drift outward forever instead of settling into standard noise.
Uses the setup above, 9 lines.
T = 100 betas = np.linspace(0.001, 0.2, T) # the noise schedule: one beta per step g = np.random.default_rng(0) shrunk = ring(2000, 0) # the real rule unshrunk = ring(2000, 0) # the same noise, with the shrinking left out print("step | mean distance from the center | variance of " "each coordinate | variance without the shrinking") for t in range(1, T + 1): eps = g.normal(size=shrunk.shape) shrunk = np.sqrt(1 - betas[t - 1]) * shrunk + np.sqrt(betas[t - 1]) * eps unshrunk = unshrunk + np.sqrt(betas[t - 1]) * eps if t in (1, 10, 25, 50, 100): print(str(t).rjust(4), "|", str(round(float(np.linalg.norm(shrunk, axis=1).mean()), 3)).ljust(29), "|", str(round(float(shrunk.var()), 3)).ljust(27), "|", round(float(unshrunk.var()), 3))
After one step the ring is barely touched: its mean distance from the center is 1.999. By step 25 it is 1.619, by step 50 it is 1.309, and after all 100 steps it is 1.254, which is what pure noise measured in the first cell. The ring is gone.
The variance columns show what the shrinking is for. This data starts with a variance of about 2 in each coordinate. With the shrinking, every step pulls that toward 1, and it ends at 1.014: standard noise. Without it, the same noise piles up on top of the data, and the variance climbs to 12.257 and would keep climbing. Generation has to start from the place where the forward process ends, so that place has to be known in advance.
import math x, beta, eps = 2.0, 0.19, -1.0 print(round(math.sqrt(1 - beta) * x + math.sqrt(beta) * eps, 3))
Show the answer
1.364
The square root of 1 − 0.19 = 0.81 is 0.9, so the old value is shrunk to 1.8. The square root of 0.19 is about 0.436, and the noise drawn was −1, so 0.436 is subtracted: 1.364. Both weights are square roots, which is what makes their squares add to 1.
Show the answer
So the total variance stays at 1 instead of growing with every step
Adding noise adds variance. Shrinking by the square root of 1 − beta removes exactly as much as the noise adds, when the data starts with a variance of 1. The process then ends at standard noise, a known distribution that generation can start from.
Press Next through the steps 1, 10, 25, 50 and 100.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: jumping straight to step t
Jumping straight to step t
Training needs noisy examples from every step of the staircase, millions of them. Walking there one step at a time would be wasteful, and it isn't necessary. Because each step only scales and adds normal noise, any number of steps collapses into a single one.
Write (alpha) for the share of signal each step keeps, and ("alpha bar") for the product of all the alphas up to step t.
In plain words: to get the point at step t, take the original point , scale it by the square root of alpha bar, and add standard noise scaled by the square root of what is left. One draw of noise does the work of t draws.
Alpha bar is the fraction of the original signal's variance that survives to step t. It starts near 1 and falls toward 0.
Uses the setup above, 9 lines.
T = 100 betas = np.linspace(0.001, 0.2, T) alpha_bar = np.cumprod(1 - betas) # alpha_bar[t - 1] belongs to step t x0 = ring(2000, 0) g = np.random.default_rng(3) print("step | alpha bar | weight on the data | " "weight on the noise | mean distance from the center") for t in (1, 10, 25, 50, 100): signal, noise = np.sqrt(alpha_bar[t - 1]), np.sqrt(1 - alpha_bar[t - 1]) xt = signal * x0 + noise * g.normal(size=x0.shape) # the jump: no loop over steps print(str(t).rjust(4), "|", str(round(float(alpha_bar[t - 1]), 4)).ljust(9), "|", str(round(float(signal), 3)).ljust(18), "|", str(round(float(noise), 3)).ljust(19), "|", round(float(np.linalg.norm(xt, axis=1).mean()), 3))
The last column agrees with the step-by-step walk above, to within the randomness of the draws: 1.614 at step 25 against 1.619, and 1.248 at step 100 against 1.254. One calculation has replaced t of them.
The middle columns are the schedule, read as a recipe. At step 10 a noisy point is 0.951 parts data to 0.31 parts noise. By step 25 the two are nearly equal, 0.727 and 0.687. At step 50 the data's weight is down to 0.272, and at step 100 it is 0.005. The squares of the two weights always add to 1.
betas = [0.1, 0.2, 0.5] alpha_bar, product = [], 1.0 for beta in betas: product *= 1 - beta alpha_bar.append(round(product, 2)) print(alpha_bar)
Show the answer
[0.9, 0.72, 0.36]
Each step keeps 1 − beta of the signal's variance: 0.9, then 0.8, then 0.5. Alpha bar is the running product: 0.9, then 0.9 × 0.8 = 0.72, then 0.72 × 0.5 = 0.36. After three steps, 36% of the original signal's variance is left.
Alpha bar from a schedule
Write alpha_bars(betas). Given the list of betas of a noise schedule, return a list of the same length holding alpha bar after each step: the running product of 1 - beta, each rounded to 4 places. Round only what you return, not the running product.
Uses the setup above, 9 lines.
def alpha_bars(betas): return [] print(alpha_bars([0.1, 0.2, 0.5]))
Show the answer
Equal weights: about 0.707 on the data and 0.707 on the noise
The weights are square roots: √0.5 ≈ 0.707 on the data and √(1 − 0.5) ≈ 0.707 on the noise. Their squares add to 1, as always. This is the step where signal and noise are equally strong, and in the schedule above it comes at about step 26 of 100.
Jump to a step
Write jump(x0, eps, alpha_bar). x0 is a point as a list of numbers, eps is a list of noise values of the same length, and alpha_bar is alpha bar at the step you are jumping to. Return the noisy point as a list: each coordinate is the square root of alpha bar times the coordinate of x0, plus the square root of one minus alpha bar times the matching noise value, rounded to 3 places.
Uses the setup above, 9 lines.
def jump(x0, eps, alpha_bar): return x0 print(jump([2.0, 0.0], [1.0, -1.0], 0.36))
What the network will be asked
The forward process makes a training example out of nothing more than a real data point.
- Pick a real point from the dataset.
- Pick a step t at random.
- Draw standard noise , and jump: .
The network is shown the noisy point and the step number t, and is asked: what noise was added? Its answer is compared with the that was really used, by mean squared error. Knowing the noise is as good as knowing the clean point, since the formula can be rearranged to recover one from the other.
No labels are involved. The "right answer" is the noise you drew yourself, so this is self-supervised learning, in the family of tasks that hide something and ask for it back.
Before training anything, it helps to know what a useless model scores.
Uses the setup above, 9 lines.
T = 100 alpha_bar = np.cumprod(1 - np.linspace(0.001, 0.2, T)) g = np.random.default_rng(0) x0 = ring(5000, 0) # 1. real points t = g.integers(1, T + 1, size=5000) # 2. a random step for each eps = g.normal(size=x0.shape) # 3. the noise, which is also the answer xt = (np.sqrt(alpha_bar[t - 1])[:, None] * x0 + np.sqrt(1 - alpha_bar[t - 1])[:, None] * eps) print("one training example | step", int(t[0]), "| noisy point", np.round(xt[0], 3), "| the noise to predict", np.round(eps[0], 3)) print("loss of a model that always answers 'no noise': ", round(float((eps ** 2).mean()), 3)) print("loss of a model that answers 'the point is all noise':", round(float(((xt - eps) ** 2).mean()), 3)) late = t > 50 print("the same second model, on steps after 50 only: ", round(float(((xt[late] - eps[late]) ** 2).mean()), 3))
The example printed first happens to be at step 86, where almost no data is left, and the noisy point and the noise are nearly the same numbers.
A model that always answers "no noise" scores 0.993, since standard noise has a mean square of 1. That is the number to beat: a loss near 1 means the network has learned nothing.
The second guess, "the whole point is noise", scores 0.651 overall and 0.023 on the late steps alone. The late steps are easy, since there the input nearly is the answer. All of the difficulty, and everything the network has to know about the data, is in the early and middle steps, where a point is part ring and part noise and the two have to be told apart. That is Diffusion 2: learning to denoise, and sampling.
Show the answer
Its right answers come from the data itself, with no labels from people
A self-supervised task builds its targets from the data: the other view of an image, a hidden patch, or here the noise that was just added. Nobody annotates anything, so a diffusion model can train on any pile of images.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: schedules and the library
The schedule is a design choice with a narrow target, and real libraries ship the common ones. Three schedules next, then the library call that makes a noisy example for you.
The cell draws the shares of data and noise against the step for each schedule. The crossing point, where the two curves meet, is the step where noise overtakes signal.
Uses the setup above, 9 lines.
T = 100 schedules = {"too gentle": np.full(T, 0.001), "too rough": np.full(T, 0.2), "rising": np.linspace(0.001, 0.2, T)} fig, axes = plt.subplots(1, 3, figsize=(9, 2.8), sharey=True) for ax, (name, betas) in zip(axes, schedules.items()): alpha_bar = np.cumprod(1 - betas) ax.plot(np.sqrt(alpha_bar), label="data") ax.plot(np.sqrt(1 - alpha_bar), label="noise") ax.set_title(name) ax.set_xlabel("step") axes[0].legend()
The first schedule never lets the noise take over, so a sampler starting from pure noise would begin somewhere the network never trained. The second destroys the data within a few steps and wastes the rest. The rising one crosses mid-way and ends in pure noise.
Worked example: three schedules, one of them usable
The schedule is a design choice with a narrow target. It must destroy the data completely by the last step, since generation will start from pure noise, and it must not destroy it too early, since steps where nothing is left teach nothing. Compare three ways of spending 100 steps.
Uses the setup above, 9 lines.
import numpy as np T = 100 schedules = { "too gentle: beta 0.001 throughout": np.full(T, 0.001), "too rough: beta 0.2 throughout": np.full(T, 0.2), "rising from 0.001 to 0.2": np.linspace(0.001, 0.2, T), } for name, betas in schedules.items(): alpha_bar = np.cumprod(1 - betas) # first step with more noise than signal even = int(np.argmax(alpha_bar < 0.5)) + 1 if (alpha_bar < 0.5).any() else None # first step with under 1% of the signal left gone = int(np.argmax(alpha_bar < 0.01)) + 1 if (alpha_bar < 0.01).any() else None print(name.ljust(34), "| signal left at the end:", str(round(float(alpha_bar[-1]), 4)).ljust(6), "| noise overtakes signal at step:", str(even).ljust(4), "| signal under 1% from step:", gone)
With a beta of 0.001 throughout, 0.9048 of the signal is still there after 100 steps. The data never becomes noise, so a sampler that starts from noise would begin somewhere the model was never trained. With a beta of 0.2 throughout, noise overtakes the signal at step 4, and under 1% of the signal is left from step 21 on, so about eighty of the hundred steps are spent on inputs that are pure noise and teach nothing. The rising schedule also ends with no signal, and spends its steps better: the crossover comes at step 27, and some signal lasts until step 67. Real systems use around 1,000 steps and gentler curves, and the test of a schedule is the same: where does the signal run out, and how many steps are spent where there is something to learn?
On the job: when a diffusion model's samples look washed out or muddy, look at the schedule first. Print the alpha bar at the last step: it should be near 0. If a fifth of the data is still there at the end, the sampler starts from a place the network never trained, and no amount of extra training steps fixes that. Libraries expose the schedule as a few settings, so the check is one line.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Diffusion 2: learning to denoise, and sampling: training a network to undo the noise, and sampling by walking back.
- Diffusion 3: images, labels and guidance: steering what a diffusion model draws.
- Self-supervised learning: labels from the data itself: another way to learn from data with no labels.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
Which part of a diffusion model involves no learning at all?
Before this lesson
The math behind this
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write