Beam search and constrained decoding
Why the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.
What it covers6
Why it matters
A language model gives a probability for every possible next token. Turning that into text is a separate decision, called decoding, and you have met one family of answers: sampling, with temperature and top-p. This lesson is about the other family. Search looks for the most probable whole sequence, and is what translation and speech recognition use. Constrained decoding goes further and forbids every output that doesn't fit a grammar, and is how "this model always returns valid JSON" is made true. Both are small algorithms, and both have a catch that only shows when you run them.
What you'll be able to do
- Show that choosing the likeliest word at each step doesn't give the likeliest sentence
- Run beam search by hand: keep the best few unfinished sentences, scored with log probabilities
- Explain why search prefers short and repetitive text, and when to search and when to sample
- Force a model's output into a required shape by masking the tokens a grammar forbids
- Say what constrained decoding guarantees, and the two things it doesn't
Show the answer
No: a likely first word can lead only to unlikely second words
A sentence's probability is the product of its steps. After "The" the model may be spread thinly over many continuations, and after "A" it may be nearly certain of one. Then the best sentence starts with the less likely word. Taking the best word at each step never looks ahead, so it can't know.
The idea: the likeliest word isn't the likeliest sentence
This rests on Sampling strategies, where greedy decoding is a temperature of zero, and on Logarithms: the power question, where adding logs stands in for multiplying chances.
Greedy goes wrong
The model in this lesson is small enough to print: the chance of each next word depends only on the word before. <s> starts a sentence and </s> ends one.
Greedy decoding takes the single likeliest word at every step. The cell runs it, and then does what only a toy model allows: lists every sentence the model can say, with its probability.
Setup (runs before every cell) · 34 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math, random # A toy language model: the chance of each next word depends only on the word before it. MODEL = { "<s>": {"The": 0.55, "A": 0.45}, "The": {"cat": 0.30, "dog": 0.28, "bird": 0.22, "fish": 0.20}, "A": {"storm": 0.90, "cat": 0.10}, "cat": {"sat": 0.50, "ran": 0.30, "slept": 0.20}, "dog": {"ran": 0.50, "sat": 0.30, "barked": 0.20}, "bird": {"sang": 0.60, "sat": 0.40}, "fish": {"swam": 0.70, "slept": 0.30}, "storm": {"hit": 0.85, "passed": 0.15}, "sat": {"</s>": 1.0}, "ran": {"</s>": 1.0}, "slept": {"</s>": 1.0}, "barked": {"</s>": 1.0}, "sang": {"</s>": 1.0}, "swam": {"</s>": 1.0}, "hit": {"</s>": 1.0}, "passed": {"</s>": 1.0}, } def beam_search(model, width, steps=12): """Keep the `width` most probable unfinished sentences at every step. Scores are log probabilities, so they add.""" beams = [(0.0, ["<s>"])] for step in range(steps): candidates = [] for score, words in beams: if words[-1] == "</s>": candidates.append((score, words)) # a finished sentence stays as it is continue for word, p in model[words[-1]].items(): candidates.append((score + math.log(p), words + [word])) beams = sorted(candidates, key=lambda c: -c[0])[:width] if all(words[-1] == "</s>" for score, words in beams): break return beams
def greedy(model): words, p = ["<s>"], 1.0 while words[-1] != "</s>": # the single likeliest next word best = max(model[words[-1]], key=model[words[-1]].get) p *= model[words[-1]][best] words.append(best) return p, words def every_sentence(model, words=("<s>",), p=1.0): if words[-1] == "</s>": return [(p, list(words))] return [s for word, q in model[words[-1]].items() for s in every_sentence(model, words + (word,), p * q)] p, words = greedy(MODEL) print("greedy:", " ".join(words[1:-1]), "| probability", round(p, 4)) ranked = sorted(every_sentence(MODEL), reverse=True) print("the model can say", len(ranked), "sentences. The three most probable:") for p, words in ranked[:3]: print(" ", f"{p:.4f}", " ".join(words[1:-1]))
Greedy writes "The cat sat", with a probability of 0.0825. The most probable sentence is "A storm hit", at 0.3443: more than four times as likely.
Greedy went wrong at the first word. "The" beats "A", 0.55 to 0.45. After "The", the model spreads its belief over four animals, none above 0.30. After "A", it is almost sure of "storm". The better sentence began with the worse word, and greedy, which never looks ahead, had already left it behind.
Listing all 15 sentences found the best one here. A real model has tens of thousands of tokens to choose from at every step, so a 20-token output has more possibilities than could ever be listed. What is needed is something between greedy's single path and trying them all.
Show the answer
It always takes the likeliest token, like a temperature of zero
Greedy decoding always picks the single most probable next token, which is what sampling becomes as the temperature goes to zero. It is repeatable and simple. This lesson shows its weakness: the best token at each step doesn't add up to the best sequence.
Keep the best few
Beam search keeps the best few unfinished sentences at every step, where greedy keeps one. The number kept is the beam width.
- Start with one empty sentence.
- Extend every kept sentence by every possible next word.
- Score each sentence by multiplying the chances along its path, and keep the best
width. - Repeat until the kept sentences have ended.
A width of 1 keeps a single sentence, which is greedy decoding. Watch a width of 2 on the same toy model.
Press Next three times. The lit boxes are what the beam keeps; the pale ones are thrown away.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: beam search in code
The four steps are one function, beam_search, kept in this lesson's setup so every cell can call it. The cell runs it with widths 1, 2 and 3, then shows why the scores are added logarithms.
Uses the setup above, 34 lines.
for width in (1, 2, 3): beams = beam_search(MODEL, width) score, words = beams[0] print("width", width, "| best:", " ".join(words[1:-1]).ljust(12), "probability", f"{math.exp(score):.4f}", "| kept:", [" ".join(w[1:-1]) for s, w in beams]) print() p = 0.55 * 0.30 * 0.50 * 1.0 print("The cat sat: multiplied", round(p, 4), "| logs added", round(math.log(0.55) + math.log(0.30) + math.log(0.50) + math.log(1.0), 4), "| back again", round(math.exp(math.log(0.55) + math.log(0.30) + math.log(0.50)), 4)) print("a sentence of 200 words at 0.1 a word: multiplied", 0.1 ** 200, "| 400 words:", 0.1 ** 400, "| logs added:", round(400 * math.log(0.1), 1))
With a width of 1, beam search is greedy, and finds "The cat sat". With a width of 2, "A" survives the first step in second place, and one step later "A storm" is far ahead. The best sentence is found, at twice the work of greedy and a fraction of the work of listing everything.
Beam search is still a shortcut. A wider beam costs more and finds a sentence at least as probable, and no width short of everything guarantees the best. In practice widths of 4 to 10 are common.
The last two lines explain a detail in the code: scores are added logarithms. Multiplying probabilities and adding their logs give the same ranking, and the first breaks on a computer. A 200-word sentence at 0.1 a word still multiplies out, to a number with 200 zeros. At 400 words it underflows to exactly 0.0, and every long sentence would tie. Their logs just add up to −921.
import math steps = [0.5, 0.5, 0.25] score = sum(math.log2(p) for p in steps) print(score, 2 ** score)
Show the answer
-4.0 0.0625
The logs, in base 2, are −1, −1 and −2, and they add to −4. Raising 2 to that power gives 0.0625, which is 0.5 × 0.5 × 0.25. Adding logs is multiplying probabilities. Log probabilities are negative, and the one nearer zero is the more probable.
width, vocabulary, steps = 3, 50000, 20 print(width * vocabulary, vocabulary ** 2 > width * vocabulary * steps)
Show the answer
150000 True
Each of the 3 kept sequences is extended by every one of 50,000 tokens: 150,000 candidates a step, and 3 million over 20 steps. Listing all sequences of just two tokens is 50,000 × 50,000, which is 2.5 billion. Exhaustive search is hopeless after a couple of tokens, and beam search grows only with the width and the length.
Real code does this over a whole batch of tensors at once. The scores of every kept sentence plus every next-token log chance form one table, and topk picks the survivors from it:
Uses the setup above, 34 lines.
import torch logp = torch.log_softmax(torch.tensor([[2.0, 1.0, 0.0], [0.5, 0.5, 0.5]]), dim=-1) beam_scores = torch.tensor([-0.6, -0.8]) # two kept sentences total = (beam_scores[:, None] + logp).view(-1) # 2 sentences x 3 next tokens = 6 top = total.topk(2) print(total.shape, [round(v, 3) for v in top.values.tolist()]) print("from beam", (top.indices // 3).tolist(), "token", (top.indices % 3).tolist())
The table has 2 × 3 = 6 entries. The best two are −1.008, from beam 0 with token 0, and −1.899, from beam 1 with token 0. topk returns where they came from as one flat index, and dividing by the vocabulary size (3 here) recovers which sentence each belongs to. Real models have a vocabulary of tens of thousands, so the table has width × vocabulary entries, and this one topk call is the whole cut.
Score a sentence
Write log_prob(model, words). model maps each word to a dictionary of the words that can follow it and their probabilities. words is a sentence without the start marker. Return the sum of the natural logs of each step's probability, starting from "<s>". If any step isn't in the model, the sentence is impossible: return float("-inf").
Uses the setup above, 34 lines.
import math def log_prob(model, words): return 0.0 model = {"<s>": {"A": 0.45, "The": 0.55}, "A": {"storm": 0.9, "cat": 0.1}, "storm": {"hit": 0.85}} print(round(log_prob(model, ["A", "storm", "hit"]), 4))
Short, and stuck in a loop
Finding the most probable sequence sounds like the goal. Two things go wrong with it, and a second toy model shows both.
Uses the setup above, 34 lines.
import math, random LOOP = { # another toy model, with a habit "<s>": {"I": 1.0}, "I": {"think": 0.6, "know": 0.25, "hope": 0.15}, "think": {"that": 0.7, "so": 0.2, "</s>": 0.1}, "that": {"I": 0.5, "it": 0.3, "we": 0.2}, "know": {"that": 0.6, "</s>": 0.4}, "hope": {"so": 0.7, "that": 0.3}, "it": {"works": 0.6, "helps": 0.4}, "we": {"can": 1.0}, "can": {"</s>": 1.0}, "so": {"</s>": 1.0}, "works": {"</s>": 1.0}, "helps": {"</s>": 1.0}, } words = ["<s>"] while words[-1] != "</s>" and len(words) < 13: words.append(max(LOOP[words[-1]], key=LOOP[words[-1]].get)) print("greedy:", " ".join(words[1:]), "...") print("beam search, width 3:") for score, words in beam_search(LOOP, 3): steps = len(words) - 1 print(" ", " ".join(words[1:]).ljust(52), "| total", f"{score:6.2f}", "| per word", f"{score / steps:5.2f}") rng = random.Random(0) print("sampling, three times:") for i in range(3): words = ["<s>"] while words[-1] != "</s>" and len(words) < 14: options, chances = zip(*LOOP[words[-1]].items()) words.append(rng.choices(options, chances)[0]) print(" ", " ".join(words[1:-1]))
Greedy falls into a loop: "I think that I think that...". Nothing is broken. After "that", the likeliest word is "I", and after "I" it is "think". The most probable next step, taken forever, goes in circles. Real models do the same, and repeated phrases are the signature of text decoded for maximum probability.
Beam search escapes the loop and shows the second problem. Its favorite is "I think so", with a total of −2.12. The longer and more useful "I think that it works" scores −2.58. Every extra word multiplies in another number below 1, so longer sentences always score lower, and search drifts toward the shortest thing that can be said.
The usual repair is in the last column: divide the score by the length. Per word, "I think that it works" is the best of the three, at −0.43. Real systems use this length normalization, often with a tunable penalty.
The three samples at the bottom are none of these. They are different each time, and none is the most probable sentence. That is the trade.
- Search suits tasks with one right output: translating a sentence, transcribing speech, filling a form. There, the most probable answer is the best guess at the correct one.
- Sampling suits open writing, where the most probable text is the blandest, and people want one good answer and not the average of all of them. This is why chat models sample.
Show the answer
Every extra token multiplies in a probability below 1
A sequence's probability is a product of numbers below 1, so each added token makes it smaller, however sensible the token is. Comparing totals favors whatever ends soonest. Dividing the log probability by the length, or adding a bonus per token, puts short and long candidates on equal terms.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: constrained decoding and the library calls
Two features you will meet in real tools are the two ideas of this lesson, packaged. A library's generate(num_beams=4) is beam search. A hosted model's structured-output mode is a mask over the next token. The mask first.
Constrained decoding
Sometimes the output must have an exact shape: a date, a number, valid JSON, one of five labels. Asking nicely in the prompt works most of the time. Constrained decoding makes it work every time.
The idea is one line. At each step, a grammar says which tokens are allowed next, given what has been written. Every other token's probability is set to zero, the rest are rescaled to add up to 1, and decoding carries on as usual. The model still chooses. It just can't choose anything illegal.
The cell uses a character-level model that has seen dates written every which way, and a grammar for one format: four digits, a dash, two digits, a dash, two digits.
Uses the setup above, 34 lines.
import random, re # A character-level model that has seen dates written every which way. SEEN = ["2024-03-05", "March 5", "05/03/2024", "5 Mar 2024", "2023-11-20", "Nov 20", "20/11/2023", "2025-01-09", "9 Jan", "Jan 9 2025", "2024-12-31", "12/31/24"] counts = {} for text in SEEN: chars = ["^"] + list(text) + ["$"] # ^ starts a string and $ ends it for a, b in zip(chars, chars[1:]): counts.setdefault(a, {}) counts[a][b] = counts[a].get(b, 0) + 1 MODEL = {a: {b: n / sum(nexts.values()) for b, n in nexts.items()} for a, nexts in counts.items()} def allowed(so_far): """The grammar for YYYY-MM-DD: which characters may come next, given what has been written.""" if len(so_far) in (4, 7): return {"-"} return set("0123456789") if len(so_far) < 10 else {"$"} def write(rng, constrain): out, last = "", "^" for i in range(20): options = dict(MODEL.get(last, {})) if constrain: ok = allowed(out) # mask, and if nothing is left, any allowed character options = ({c: p for c, p in options.items() if c in ok} or {c: 1.0 for c in sorted(ok)}) chars, chances = zip(*options.items()) # choices() rescales the chances that are left c = rng.choices(chars, chances)[0] if c == "$": break out, last = out + c, c return out shape = re.compile(r"\d{4}-\d{2}-\d{2}") for constrain in (False, True): rng = random.Random(0) texts = [write(rng, constrain) for i in range(1000)] right_shape = [t for t in texts if shape.fullmatch(t)] possible = [ t for t in right_shape if 1 <= int(t[5:7]) <= 12 and 1 <= int(t[8:10]) <= 31] print("constrained" if constrain else "free ", "| first four:", texts[:4]) print(" | right shape:", len(right_shape), "of 1000 | and a date that could exist:", len(possible), "of 1000")
Left alone, the model writes plausible mush, "Jarch Jar 9 Man", and gets the required shape 0 times in 1,000. With the grammar, it is right 1,000 times in 1,000. That is a guarantee, not a statistic: an illegal character has no chance at any step.
Now read the constrained outputs. "2056-31-31" has the right shape and a thirty-first month. Only 776 of the 1,000 are dates that could exist. The grammar guarantees the form. It knows nothing about the truth. A model forced to produce valid JSON will produce valid JSON when it has no idea of the answer, too, and the result looks exactly as trustworthy as when it does.
This is how "structured output" features work. A JSON schema is compiled into a grammar over the model's tokens, and the mask is applied at every step. It costs a little speed, removes a whole class of parsing failures, and still needs the checks you would run on any model output.
chances = {"7": 0.2, "seven": 0.5, "3": 0.05, "about": 0.25}
allowed = {t: p for t, p in chances.items() if t.isdigit()}
total = sum(allowed.values())
print({t: round(p / total, 2) for t, p in allowed.items()})Show the answer
{'7': 0.8, '3': 0.2}
The mask removes "seven" and "about", leaving 0.2 and 0.05. Dividing each by their total, 0.25, gives 0.8 and 0.2. The model's preference between the legal tokens is kept. Notice what was discarded: the model put 0.5 on "seven". The constraint didn't change its mind. It overruled it.
A grammar for a fixed list of answers
The simplest grammar is a list of allowed outputs. Write allowed_next(written, choices). choices is a list of allowed outputs, each a list of tokens. written is the list of tokens produced so far. Return the sorted list of distinct tokens that can come next: the next token of every choice that starts with written. If written is itself a complete choice, include "</s>", since the output may end there.
Uses the setup above, 34 lines.
def allowed_next(written, choices): return [] choices = [["pos", "itive"], ["neg", "ative"], ["neu", "tral"], ["neg"]] print(allowed_next([], choices), allowed_next(["neg"], choices))
Worked example: a classifier that can only say three things
A model is asked for the sentiment of a review and must reply with one of three labels. It writes in pieces, as real models do: "positive" is the tokens pos and itive. A small grammar, built from the list of labels, masks every piece that leads nowhere.
Uses the setup above, 34 lines.
# A classifier built on a language model must answer with exactly one of three labels. # The model writes in pieces. LABELS = {"positive": ["pos", "itive"], "negative": ["neg", "ative"], "neutral": ["neu", "tral"]} # the model's chances for the first piece FIRST = {"pos": 0.35, "mixed": 0.20, "neg": 0.20, "neu": 0.15, "I": 0.10} SECOND = {"pos": {"itive": 0.80, "sible": 0.20}, "neg": {"ative": 0.90, "lect": 0.10}, "neu": {"tral": 0.70, "ron": 0.30}} def allowed_next(pieces): """A trie over the labels: the pieces that still lead to one of them.""" return sorted({label[len(pieces)] for label in LABELS.values() if label[:len(pieces)] == pieces and len(label) > len(pieces)}) off_list = 1 - sum(FIRST[label[0]] * SECOND[label[0]][label[1]] for label in LABELS.values()) print("left alone, the model writes something that isn't one of the labels", f"{100 * off_list:.1f}%", "of the time") print("allowed first pieces:", allowed_next([]), "| after 'neg':", allowed_next(["neg"]), "| after 'neg', 'ative':", allowed_next(["neg", "ative"])) print() print("label | mask at each step | rescale whole labels") step1 = {piece: FIRST[piece] for piece in allowed_next([])} whole = {name: FIRST[a] * SECOND[a][b] for name, (a, b) in LABELS.items()} for name, (a, b) in LABELS.items(): masked = step1[a] / sum(step1.values()) # then only one second piece is allowed print(name.ljust(8), "|", f"{masked:.3f}".rjust(17), "|", f"{whole[name] / sum(whole.values()):.3f}")
Left alone, the model wanders off the list 43.5% of the time: it starts with "mixed" or "I", or writes "pos" and then "sible". The grammar stops all of that. At the start only the three first pieces are allowed, and after "neg" only "ative". The table at the end is a subtlety worth knowing. Masking at each step gives "negative" a chance of 0.286. Working out the chance of each whole label and rescaling gives 0.319. They differ, since the step-by-step mask rescales at the first piece, before it knows that the model would often have left the list at the second: after "neu" it writes "ron" 30% of the time, and after "neg" it goes wrong only 10%. So constrained decoding guarantees a legal output, and does not quite give the legal output the model thinks most likely. For a classifier the difference is small. It is one more reason to treat a constrained answer as a decision to check, not as the model's considered view.
On the job: the choice is a setting, so it is yours to make per task. Translation and transcription take beam search with a width of 4 or 5. Chat and creative writing sample. Anything a program will parse, such as an order as JSON or a label from a fixed list, takes the provider's structured-output mode, then your own validation, because the mask guarantees the shape and nothing about the value.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Sampling strategies: temperature, top-k and top-p, the other family of decoding.
- Structured output: asking a model API for JSON and checking what comes back.
- Recognition and synthesis as a pipeline: search over sound guesses, where beam search is the standard decoder.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
What does beam search keep at each step?
Before this lesson
The math behind this
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write