Index Ai Academy
Phase 5 · Language modeling

Beam search and constrained decoding

Why the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A language model gives a probability for every possible next token. Turning that into text is a separate decision, called decoding, and you have met one family of answers: sampling, with temperature and top-p. This lesson is about the other family. Search looks for the most probable whole sequence, and is what translation and speech recognition use. Constrained decoding goes further and forbids every output that doesn't fit a grammar, and is how "this model always returns valid JSON" is made true. Both are small algorithms, and both have a catch that only shows when you run them.

What you'll be able to do

  • Show that choosing the likeliest word at each step doesn't give the likeliest sentence
  • Run beam search by hand: keep the best few unfinished sentences, scored with log probabilities
  • Explain why search prefers short and repetitive text, and when to search and when to sample
  • Force a model's output into a required shape by masking the tokens a grammar forbids
  • Say what constrained decoding guarantees, and the two things it doesn't
Quick check
A guess before you start. A model is writing a two-word answer. For the first word it gives "The" a chance of 0.55 and "A" a chance of 0.45. If you always take the likeliest word at each step, will you always end up with the likeliest two-word answer?
  1. Yes: the likeliest words make the likeliest sentence
  2. Yes, as long as the temperature is set to zero
  3. No: a likely first word can lead only to unlikely second words
  4. No: the likeliest sentence can't be found at all
Show the answer

No: a likely first word can lead only to unlikely second words

A sentence's probability is the product of its steps. After "The" the model may be spread thinly over many continuations, and after "A" it may be nearly certain of one. Then the best sentence starts with the less likely word. Taking the best word at each step never looks ahead, so it can't know.

The idea: the likeliest word isn't the likeliest sentence

This rests on Sampling strategies, where greedy decoding is a temperature of zero, and on Logarithms: the power question, where adding logs stands in for multiplying chances.

Greedy goes wrong

The model in this lesson is small enough to print: the chance of each next word depends only on the word before. <s> starts a sentence and </s> ends one.

Greedy decoding takes the single likeliest word at every step. The cell runs it, and then does what only a toy model allows: lists every sentence the model can say, with its probability.

Setup (runs before every cell) · 34 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math, random

# A toy language model: the chance of each next word depends only on the word before it.
MODEL = {
    "<s>":   {"The": 0.55, "A": 0.45},
    "The":   {"cat": 0.30, "dog": 0.28, "bird": 0.22, "fish": 0.20},
    "A":     {"storm": 0.90, "cat": 0.10},
    "cat":   {"sat": 0.50, "ran": 0.30, "slept": 0.20},
    "dog":   {"ran": 0.50, "sat": 0.30, "barked": 0.20},
    "bird":  {"sang": 0.60, "sat": 0.40},
    "fish":  {"swam": 0.70, "slept": 0.30},
    "storm": {"hit": 0.85, "passed": 0.15},
    "sat": {"</s>": 1.0}, "ran": {"</s>": 1.0}, "slept": {"</s>": 1.0},
    "barked": {"</s>": 1.0},
    "sang": {"</s>": 1.0}, "swam": {"</s>": 1.0}, "hit": {"</s>": 1.0},
    "passed": {"</s>": 1.0},
}

def beam_search(model, width, steps=12):
    """Keep the `width` most probable unfinished sentences at every step. Scores are log
    probabilities, so they add."""
    beams = [(0.0, ["<s>"])]
    for step in range(steps):
        candidates = []
        for score, words in beams:
            if words[-1] == "</s>":
                candidates.append((score, words))   # a finished sentence stays as it is
                continue
            for word, p in model[words[-1]].items():
                candidates.append((score + math.log(p), words + [word]))
        beams = sorted(candidates, key=lambda c: -c[0])[:width]
        if all(words[-1] == "</s>" for score, words in beams):
            break
    return beams
def greedy(model):
    words, p = ["<s>"], 1.0
    while words[-1] != "</s>":
        # the single likeliest next word
        best = max(model[words[-1]], key=model[words[-1]].get)
        p *= model[words[-1]][best]
        words.append(best)
    return p, words

def every_sentence(model, words=("<s>",), p=1.0):
    if words[-1] == "</s>":
        return [(p, list(words))]
    return [s for word, q in model[words[-1]].items()
            for s in every_sentence(model, words + (word,), p * q)]

p, words = greedy(MODEL)
print("greedy:", " ".join(words[1:-1]), "| probability", round(p, 4))

ranked = sorted(every_sentence(MODEL), reverse=True)
print("the model can say", len(ranked), "sentences. The three most probable:")
for p, words in ranked[:3]:
    print("  ", f"{p:.4f}", " ".join(words[1:-1]))

Greedy writes "The cat sat", with a probability of 0.0825. The most probable sentence is "A storm hit", at 0.3443: more than four times as likely.

Greedy went wrong at the first word. "The" beats "A", 0.55 to 0.45. After "The", the model spreads its belief over four animals, none above 0.30. After "A", it is almost sure of "storm". The better sentence began with the worse word, and greedy, which never looks ahead, had already left it behind.

Listing all 15 sentences found the best one here. A real model has tens of thousands of tokens to choose from at every step, so a 20-token output has more possibilities than could ever be listed. What is needed is something between greedy's single path and trying them all.

Quick check
From the lesson on sampling strategies: what does greedy decoding do, and what is it the same as?
  1. It always takes the likeliest token, like a temperature of zero
  2. It samples from the ten likeliest tokens, like top-k
  3. It samples from the whole distribution, like a temperature of one
  4. It takes the least likely token, to keep the text varied
Show the answer

It always takes the likeliest token, like a temperature of zero

Greedy decoding always picks the single most probable next token, which is what sampling becomes as the temperature goes to zero. It is repeatable and simple. This lesson shows its weakness: the best token at each step doesn't add up to the best sequence.

Keep the best few

Beam search keeps the best few unfinished sentences at every step, where greedy keeps one. The number kept is the beam width.

  1. Start with one empty sentence.
  2. Extend every kept sentence by every possible next word.
  3. Score each sentence by multiplying the chances along its path, and keep the best width.
  4. Repeat until the kept sentences have ended.

A width of 1 keeps a single sentence, which is greedy decoding. Watch a width of 2 on the same toy model.

Try it now

Press Next three times. The lit boxes are what the beam keeps; the pale ones are thrown away.

In your own words
Why did always taking the likeliest word produce a sentence four times less likely than the best one?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: beam search in code

The four steps are one function, beam_search, kept in this lesson's setup so every cell can call it. The cell runs it with widths 1, 2 and 3, then shows why the scores are added logarithms.

Uses the setup above, 34 lines.

for width in (1, 2, 3):
    beams = beam_search(MODEL, width)
    score, words = beams[0]
    print("width", width, "| best:", " ".join(words[1:-1]).ljust(12), "probability",
          f"{math.exp(score):.4f}", "| kept:", [" ".join(w[1:-1]) for s, w in beams])

print()
p = 0.55 * 0.30 * 0.50 * 1.0
print("The cat sat: multiplied", round(p, 4), "| logs added",
      round(math.log(0.55) + math.log(0.30) + math.log(0.50) + math.log(1.0), 4),
      "| back again",
      round(math.exp(math.log(0.55) + math.log(0.30) + math.log(0.50)), 4))
print("a sentence of 200 words at 0.1 a word: multiplied", 0.1 ** 200, "| 400 words:",
      0.1 ** 400, "| logs added:", round(400 * math.log(0.1), 1))

With a width of 1, beam search is greedy, and finds "The cat sat". With a width of 2, "A" survives the first step in second place, and one step later "A storm" is far ahead. The best sentence is found, at twice the work of greedy and a fraction of the work of listing everything.

Beam search is still a shortcut. A wider beam costs more and finds a sentence at least as probable, and no width short of everything guarantees the best. In practice widths of 4 to 10 are common.

The last two lines explain a detail in the code: scores are added logarithms. Multiplying probabilities and adding their logs give the same ranking, and the first breaks on a computer. A 200-word sentence at 0.1 a word still multiplies out, to a number with 200 zeros. At 400 words it underflows to exactly 0.0, and every long sentence would tie. Their logs just add up to −921.

Quick check
A sentence's words have probabilities 0.5, 0.5 and 0.25. What does this print: its log probability, and the probability that comes back from it?
import math

steps = [0.5, 0.5, 0.25]
score = sum(math.log2(p) for p in steps)
print(score, 2 ** score)
  1. 1.25 2.38
  2. -4.0 -4.0
  3. -4.0 0.0625
  4. 0.0625 -4.0
Show the answer

-4.0 0.0625

The logs, in base 2, are −1, −1 and −2, and they add to −4. Raising 2 to that power gives 0.0625, which is 0.5 × 0.5 × 0.25. Adding logs is multiplying probabilities. Log probabilities are negative, and the one nearer zero is the more probable.

Quick check
A real model chooses among 50,000 tokens. Beam search with a width of 3 runs for 20 steps. What does this print: the candidates scored at each step, and whether listing every two-token sequence would already be more work than the whole search?
width, vocabulary, steps = 3, 50000, 20
print(width * vocabulary, vocabulary ** 2 > width * vocabulary * steps)
  1. 3000000 False
  2. 50003 True
  3. 150000 True
  4. 150000 False
Show the answer

150000 True

Each of the 3 kept sequences is extended by every one of 50,000 tokens: 150,000 candidates a step, and 3 million over 20 steps. Listing all sequences of just two tokens is 50,000 × 50,000, which is 2.5 billion. Exhaustive search is hopeless after a couple of tokens, and beam search grows only with the width and the length.

Real code does this over a whole batch of tensors at once. The scores of every kept sentence plus every next-token log chance form one table, and topk picks the survivors from it:

Uses the setup above, 34 lines.

import torch

logp = torch.log_softmax(torch.tensor([[2.0, 1.0, 0.0], [0.5, 0.5, 0.5]]), dim=-1)
beam_scores = torch.tensor([-0.6, -0.8])          # two kept sentences
total = (beam_scores[:, None] + logp).view(-1)    # 2 sentences x 3 next tokens = 6
top = total.topk(2)
print(total.shape, [round(v, 3) for v in top.values.tolist()])
print("from beam", (top.indices // 3).tolist(), "token", (top.indices % 3).tolist())
It runs right here in your browser, with nothing to install.

The table has 2 × 3 = 6 entries. The best two are −1.008, from beam 0 with token 0, and −1.899, from beam 1 with token 0. topk returns where they came from as one flat index, and dividing by the vocabulary size (3 here) recovers which sentence each belongs to. Real models have a vocabulary of tens of thousands, so the table has width × vocabulary entries, and this one topk call is the whole cut.

Drill

Score a sentence

Write log_prob(model, words). model maps each word to a dictionary of the words that can follow it and their probabilities. words is a sentence without the start marker. Return the sum of the natural logs of each step's probability, starting from "<s>". If any step isn't in the model, the sentence is impossible: return float("-inf").

Uses the setup above, 34 lines.

import math


def log_prob(model, words):
    return 0.0


model = {"<s>": {"A": 0.45, "The": 0.55}, "A": {"storm": 0.9, "cat": 0.1},
         "storm": {"hit": 0.85}}
print(round(log_prob(model, ["A", "storm", "hit"]), 4))
In the lesson, your code is checked as you write it.

Short, and stuck in a loop

Finding the most probable sequence sounds like the goal. Two things go wrong with it, and a second toy model shows both.

Uses the setup above, 34 lines.

import math, random

LOOP = {  # another toy model, with a habit
    "<s>": {"I": 1.0},
    "I": {"think": 0.6, "know": 0.25, "hope": 0.15},
    "think": {"that": 0.7, "so": 0.2, "</s>": 0.1},
    "that": {"I": 0.5, "it": 0.3, "we": 0.2},
    "know": {"that": 0.6, "</s>": 0.4}, "hope": {"so": 0.7, "that": 0.3},
    "it": {"works": 0.6, "helps": 0.4}, "we": {"can": 1.0}, "can": {"</s>": 1.0},
    "so": {"</s>": 1.0}, "works": {"</s>": 1.0}, "helps": {"</s>": 1.0},
}

words = ["<s>"]
while words[-1] != "</s>" and len(words) < 13:
    words.append(max(LOOP[words[-1]], key=LOOP[words[-1]].get))
print("greedy:", " ".join(words[1:]), "...")

print("beam search, width 3:")
for score, words in beam_search(LOOP, 3):
    steps = len(words) - 1
    print("  ", " ".join(words[1:]).ljust(52), "| total", f"{score:6.2f}", "| per word",
          f"{score / steps:5.2f}")

rng = random.Random(0)
print("sampling, three times:")
for i in range(3):
    words = ["<s>"]
    while words[-1] != "</s>" and len(words) < 14:
        options, chances = zip(*LOOP[words[-1]].items())
        words.append(rng.choices(options, chances)[0])
    print("  ", " ".join(words[1:-1]))

Greedy falls into a loop: "I think that I think that...". Nothing is broken. After "that", the likeliest word is "I", and after "I" it is "think". The most probable next step, taken forever, goes in circles. Real models do the same, and repeated phrases are the signature of text decoded for maximum probability.

Beam search escapes the loop and shows the second problem. Its favorite is "I think so", with a total of −2.12. The longer and more useful "I think that it works" scores −2.58. Every extra word multiplies in another number below 1, so longer sentences always score lower, and search drifts toward the shortest thing that can be said.

The usual repair is in the last column: divide the score by the length. Per word, "I think that it works" is the best of the three, at −0.43. Real systems use this length normalization, often with a tunable penalty.

The three samples at the bottom are none of these. They are different each time, and none is the most probable sentence. That is the trade.

  • Search suits tasks with one right output: translating a sentence, transcribing speech, filling a form. There, the most probable answer is the best guess at the correct one.
  • Sampling suits open writing, where the most probable text is the blandest, and people want one good answer and not the average of all of them. This is why chat models sample.
Quick check
Why does beam search, left uncorrected, prefer short outputs?
  1. Short outputs are checked first, so they win ties
  2. Every extra token multiplies in a probability below 1
  3. The beam width shrinks as the output grows
  4. Long outputs are removed to save memory
Show the answer

Every extra token multiplies in a probability below 1

A sequence's probability is a product of numbers below 1, so each added token makes it smaller, however sensible the token is. Comparing totals favors whatever ends soonest. Dividing the log probability by the length, or adding a bonus per token, puts short and long candidates on equal terms.

In your own words
A translation system uses beam search, and a story-writing assistant uses sampling. Why is each choice right for its task?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: constrained decoding and the library calls

Two features you will meet in real tools are the two ideas of this lesson, packaged. A library's generate(num_beams=4) is beam search. A hosted model's structured-output mode is a mask over the next token. The mask first.

Constrained decoding

Sometimes the output must have an exact shape: a date, a number, valid JSON, one of five labels. Asking nicely in the prompt works most of the time. Constrained decoding makes it work every time.

The idea is one line. At each step, a grammar says which tokens are allowed next, given what has been written. Every other token's probability is set to zero, the rest are rescaled to add up to 1, and decoding carries on as usual. The model still chooses. It just can't choose anything illegal.

The cell uses a character-level model that has seen dates written every which way, and a grammar for one format: four digits, a dash, two digits, a dash, two digits.

Uses the setup above, 34 lines.

import random, re

# A character-level model that has seen dates written every which way.
SEEN = ["2024-03-05", "March 5", "05/03/2024", "5 Mar 2024", "2023-11-20", "Nov 20",
        "20/11/2023", "2025-01-09", "9 Jan", "Jan 9 2025", "2024-12-31", "12/31/24"]
counts = {}
for text in SEEN:
    chars = ["^"] + list(text) + ["$"]                 # ^ starts a string and $ ends it
    for a, b in zip(chars, chars[1:]):
        counts.setdefault(a, {})
        counts[a][b] = counts[a].get(b, 0) + 1
MODEL = {a: {b: n / sum(nexts.values()) for b, n in nexts.items()}
         for a, nexts in counts.items()}

def allowed(so_far):
    """The grammar for YYYY-MM-DD: which characters may come next,
    given what has been written."""
    if len(so_far) in (4, 7):
        return {"-"}
    return set("0123456789") if len(so_far) < 10 else {"$"}

def write(rng, constrain):
    out, last = "", "^"
    for i in range(20):
        options = dict(MODEL.get(last, {}))
        if constrain:
            ok = allowed(out)
            # mask, and if nothing is left, any allowed character
            options = ({c: p for c, p in options.items() if c in ok}
                       or {c: 1.0 for c in sorted(ok)})
        chars, chances = zip(*options.items())
        # choices() rescales the chances that are left
        c = rng.choices(chars, chances)[0]
        if c == "$":
            break
        out, last = out + c, c
    return out

shape = re.compile(r"\d{4}-\d{2}-\d{2}")
for constrain in (False, True):
    rng = random.Random(0)
    texts = [write(rng, constrain) for i in range(1000)]
    right_shape = [t for t in texts if shape.fullmatch(t)]
    possible = [
        t for t in right_shape if 1 <= int(t[5:7]) <= 12 and 1 <= int(t[8:10]) <= 31]
    print("constrained" if constrain else "free       ", "| first four:", texts[:4])
    print("            | right shape:", len(right_shape),
          "of 1000 | and a date that could exist:", len(possible), "of 1000")

Left alone, the model writes plausible mush, "Jarch Jar 9 Man", and gets the required shape 0 times in 1,000. With the grammar, it is right 1,000 times in 1,000. That is a guarantee, not a statistic: an illegal character has no chance at any step.

Now read the constrained outputs. "2056-31-31" has the right shape and a thirty-first month. Only 776 of the 1,000 are dates that could exist. The grammar guarantees the form. It knows nothing about the truth. A model forced to produce valid JSON will produce valid JSON when it has no idea of the answer, too, and the result looks exactly as trustworthy as when it does.

This is how "structured output" features work. A JSON schema is compiled into a grammar over the model's tokens, and the mask is applied at every step. It costs a little speed, removes a whole class of parsing failures, and still needs the checks you would run on any model output.

Quick check
A model's chances for the next token are below, and a grammar allows only digits at this point. What does this print: the allowed tokens with their rescaled chances?
chances = {"7": 0.2, "seven": 0.5, "3": 0.05, "about": 0.25}
allowed = {t: p for t, p in chances.items() if t.isdigit()}
total = sum(allowed.values())
print({t: round(p / total, 2) for t, p in allowed.items()})
  1. {'7': 0.5, '3': 0.5}
  2. {'7': 0.8, '3': 0.2}
  3. {'seven': 0.67, 'about': 0.33}
  4. {'7': 0.2, '3': 0.05}
Show the answer

{'7': 0.8, '3': 0.2}

The mask removes "seven" and "about", leaving 0.2 and 0.05. Dividing each by their total, 0.25, gives 0.8 and 0.2. The model's preference between the legal tokens is kept. Notice what was discarded: the model put 0.5 on "seven". The constraint didn't change its mind. It overruled it.

Drill

A grammar for a fixed list of answers

The simplest grammar is a list of allowed outputs. Write allowed_next(written, choices). choices is a list of allowed outputs, each a list of tokens. written is the list of tokens produced so far. Return the sorted list of distinct tokens that can come next: the next token of every choice that starts with written. If written is itself a complete choice, include "</s>", since the output may end there.

Uses the setup above, 34 lines.

def allowed_next(written, choices):
    return []


choices = [["pos", "itive"], ["neg", "ative"], ["neu", "tral"], ["neg"]]
print(allowed_next([], choices), allowed_next(["neg"], choices))
In the lesson, your code is checked as you write it.
Worked example

Worked example: a classifier that can only say three things

A model is asked for the sentiment of a review and must reply with one of three labels. It writes in pieces, as real models do: "positive" is the tokens pos and itive. A small grammar, built from the list of labels, masks every piece that leads nowhere.

Uses the setup above, 34 lines.

# A classifier built on a language model must answer with exactly one of three labels.
# The model writes in pieces.
LABELS = {"positive": ["pos", "itive"], "negative": ["neg", "ative"],
          "neutral": ["neu", "tral"]}
# the model's chances for the first piece
FIRST = {"pos": 0.35, "mixed": 0.20, "neg": 0.20, "neu": 0.15, "I": 0.10}
SECOND = {"pos": {"itive": 0.80, "sible": 0.20}, "neg": {"ative": 0.90, "lect": 0.10},
          "neu": {"tral": 0.70, "ron": 0.30}}

def allowed_next(pieces):
    """A trie over the labels: the pieces that still lead to one of them."""
    return sorted({label[len(pieces)] for label in LABELS.values()
                   if label[:len(pieces)] == pieces and len(label) > len(pieces)})

off_list = 1 - sum(FIRST[label[0]] * SECOND[label[0]][label[1]]
                   for label in LABELS.values())
print("left alone, the model writes something that isn't one of the labels",
      f"{100 * off_list:.1f}%", "of the time")
print("allowed first pieces:", allowed_next([]), "| after 'neg':",
      allowed_next(["neg"]), "| after 'neg', 'ative':", allowed_next(["neg", "ative"]))
print()
print("label    | mask at each step | rescale whole labels")
step1 = {piece: FIRST[piece] for piece in allowed_next([])}
whole = {name: FIRST[a] * SECOND[a][b] for name, (a, b) in LABELS.items()}
for name, (a, b) in LABELS.items():
    masked = step1[a] / sum(step1.values())  # then only one second piece is allowed
    print(name.ljust(8), "|", f"{masked:.3f}".rjust(17), "|",
          f"{whole[name] / sum(whole.values()):.3f}")
In the lesson you can change this and run it.

Left alone, the model wanders off the list 43.5% of the time: it starts with "mixed" or "I", or writes "pos" and then "sible". The grammar stops all of that. At the start only the three first pieces are allowed, and after "neg" only "ative". The table at the end is a subtlety worth knowing. Masking at each step gives "negative" a chance of 0.286. Working out the chance of each whole label and rescaling gives 0.319. They differ, since the step-by-step mask rescales at the first piece, before it knows that the model would often have left the list at the second: after "neu" it writes "ron" 30% of the time, and after "neg" it goes wrong only 10%. So constrained decoding guarantees a legal output, and does not quite give the legal output the model thinks most likely. For a classifier the difference is small. It is one more reason to treat a constrained answer as a decision to check, not as the model's considered view.

On the job: the choice is a setting, so it is yours to make per task. Translation and transcription take beam search with a width of 4 or 5. Chat and creative writing sample. Anything a program will parse, such as an order as JSON or a label from a fixed list, takes the provider's structured-output mode, then your own validation, because the mask guarantees the shape and nothing about the value.

In your own words
A colleague says: "We use constrained decoding, so the model's JSON is always valid, so we can skip validation." What would you tell them?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

What does beam search keep at each step?

  1. The few most probable unfinished sequences
  2. Only the single most probable next token
  3. Every possible sequence of that length
  4. A random sample of sequences, by temperature

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

The math behind this

The Phase 2 lessons this one rests on, from the course map.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write