Index Ai Academy
Phase 5 · Beyond text

Image-text models in depth: retrieval, negatives and limits

How to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

In the lesson on multimodal transformers you trained a small CLIP: two encoders pulled matching images and captions together until they shared one space. That space now sits under image search, zero-shot classifiers and the image side of most vision-language models. This lesson asks the questions an engineer has to ask before relying on it. How do you measure whether the space is any good? What decides how fine its distinctions are? And what can a single similarity score never tell you, however well it is trained?

What you'll be able to do

  • Test a shared image-text space by retrieval, and report recall at k in both directions
  • Explain what the other pairs in a batch do, and why a larger batch teaches finer distinctions
  • Say what happens when every negative is hard, and what a false negative is
  • Describe what the temperature does to the loss
  • Show why a score built from summed word vectors can't see word order, and name tasks that follow from it
Quick check
A guess before you start. An image-text model scores how well a caption matches a photo. It is shown a photo of a red square beside a blue circle, and two captions: "a red square and a blue circle" and "a blue square and a red circle". What do you expect?
  1. It will strongly prefer the second, since it is listed last
  2. It can't score either, since neither caption was in its training data
  3. It will strongly prefer the first, as any person would
  4. It may score both about the same, since they use the same words
Show the answer

It may score both about the same, since they use the same words

The two captions contain exactly the same words. A model whose text side works mostly from which words are present, and less from how they are arranged, sees them as nearly the same sentence. Large image-text models have been measured doing poorly on exactly this kind of test. The last part of this lesson shows where the weakness comes from.

The idea: a table of scores, and the test of it

This rests on Multimodal transformers, where the similarity table first appeared, and on Angle, cosine similarity and projection for the similarity itself.

Retrieval: the test of a shared space

Contrastive training gives each image and each caption a vector, placed so that a matching pair points the same way. The direct test of such a space is retrieval: take a caption, rank every image in a collection by similarity to it, and see where the right image lands.

The standard score is recall at k, written R@k: the share of queries whose right answer is among the top k results. R@1 asks for the exact match in first place. R@5 forgives a near miss.

This lesson's data is a little richer than last time's. There are 20 kinds of scene, and every scene also has details of its own, so two dog photos are similar and not the same. Telling a dog from a car is the coarse distinction. Telling this dog photo from that one is the fine one.

Setup (runs before every cell) · 67 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

g0 = np.random.default_rng(0)
KINDS = 20
# 20 kinds of scene: dog, car, beach...
centers = g0.normal(size=(KINDS, 4)).astype(np.float32)
to_pixels, to_words = (g0.normal(size=(12, 40)).astype(np.float32),
                       g0.normal(size=(12, 20)).astype(np.float32))

def pairs(n, seed):
    """n matching (image, caption) pairs. A scene is its kind (4 numbers) plus its own
    details (8 small numbers)."""
    g = np.random.default_rng(seed)
    kind = g.integers(0, KINDS, size=n)
    scene = np.concatenate(
        [centers[kind], 0.3 * g.normal(size=(n, 8)).astype(np.float32)], axis=1)
    images = scene @ to_pixels + 0.5 * g.normal(size=(n, 40)).astype(np.float32)
    captions = scene @ to_words + 0.5 * g.normal(size=(n, 20)).astype(np.float32)
    return images, captions, kind

def unit(v):
    return v / v.norm(dim=-1, keepdim=True)

def train(batch, steps=300, temperature=0.1, one_kind=False, seed=0):
    torch.manual_seed(seed)
    image_encoder, text_encoder = nn.Linear(40, 16), nn.Linear(20, 16)
    opt = torch.optim.Adam(
        list(image_encoder.parameters()) + list(text_encoder.parameters()), lr=1e-2)
    images, captions, kind = pairs(4096, 1)
    g = np.random.default_rng(seed)
    for step in range(steps):
        if one_kind:                 # every pair in the batch is the same kind of scene
            idx = g.choice(np.where(kind == g.integers(0, KINDS))[0], size=batch,
                           replace=False)
        else:
            idx = g.permutation(4096)[:batch]
        scores = (unit(image_encoder(torch.tensor(images[idx])))
                  @ unit(text_encoder(torch.tensor(captions[idx]))).T / temperature)
        labels = torch.arange(batch)                       # image i goes with caption i
        loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2
        opt.zero_grad()
        loss.backward()
        opt.step()
    return image_encoder, text_encoder

def similarities(image_encoder, text_encoder):
    """500 unseen pairs: the similarity of every image (rows) with
    every caption (columns)."""
    images, captions, kind = pairs(500, 2)
    with torch.no_grad():
        sims = (unit(image_encoder(torch.tensor(images)))
                @ unit(text_encoder(torch.tensor(captions))).T).numpy()
    return sims, kind

def retrieval(image_encoder, text_encoder):
    """Search the 500 images with each caption. Returns recall@1, recall@5, and how
    often the top image is the right kind."""
    sims, kind = similarities(image_encoder, text_encoder)
    # images ranked above the right one
    better = np.array([(sims[:, j] > sims[j, j]).sum() for j in range(500)])
    right_kind = (kind[sims.argmax(axis=0)] == kind).mean()
    return (round(float((better == 0).mean()), 3), round(float((better < 5).mean()), 3),
            round(float(right_kind), 3))
torch.manual_seed(0)
print("untrained | recall@1, recall@5, top image is the right kind:",
      retrieval(nn.Linear(40, 16), nn.Linear(20, 16)))
print("trained   | recall@1, recall@5, top image is the right kind:",
      retrieval(*train(64)))

Before training, a caption's right image is never in first place among 500, and almost never even the right kind. After 300 steps on batches of 64, the exact image comes first for 0.832 of captions and is in the top five for 0.974.

The third number separates the two kinds of mistake. For 0.948 of captions the top image is at least the right kind of scene. So most of the misses are near misses: a dog photo was wanted and a different dog photo came back. Whether that matters depends on the product. For "show me pictures of dogs" it is a success. For "find the photo this caption was written for" it is a failure.

Quick check
From the lesson on multimodal transformers: in contrastive training, what is pushed up and what is pushed down for each image in a batch?
  1. Its own caption's score goes up, the others' go down
  2. Its pixel values are pushed toward the batch average
  3. Its score with the most similar other image goes up
  4. Its score with every caption in the batch goes up together
Show the answer

Its own caption's score goes up, the others' go down

Each image is scored against every caption in the batch, and a cross-entropy loss treats its own caption as the right answer. Raising that score and lowering the rest is what moves matching pairs together and everything else apart. The same is done from each caption's side.

Quick check
Three images (rows) are scored against three captions (columns), and image i belongs with caption i. What does this print: for each caption, the position of its right image when the images are ranked, then R@1?
sims = [[0.9, 0.8, 0.1],
        [0.2, 0.7, 0.3],
        [0.4, 0.6, 0.5]]
ranks = [1 + sum(sims[i][j] > sims[j][j] for i in range(3)) for j in range(3)]
print(ranks, round(sum(r == 1 for r in ranks) / 3, 3))
  1. [1, 1, 2] 0.667
  2. [1, 2, 1] 0.667
  3. [1, 1, 1] 1.0
  4. [1, 2, 1] 0.333
Show the answer

[1, 2, 1] 0.667

For caption 1, read down column 1: 0.8, 0.7, 0.6. Its own image, image 1, scores 0.7, and image 0 scores higher, so the right image is in second place. Captions 0 and 2 have their own image on top. Two of three are exact: R@1 is 0.667.

Drill

Rank of the right image for each caption

Write match_ranks(sims). sims is a square list of rows: sims[i][j] is the similarity of image i with caption j, and image j is the right one for caption j. For each caption, return the position of its right image when all the images are ranked by similarity to that caption: 1 plus the number of images that score strictly higher than the right one. Return the list of positions, one per caption.

Uses the setup above, 67 lines.

def match_ranks(sims):
    return []


print(match_ranks([[0.9, 0.8, 0.1], [0.2, 0.7, 0.3], [0.4, 0.6, 0.5]]))
In the lesson, your code is checked as you write it.
Try it now

Press Next. Retrieval reads a column; the loss pushes the diagonal up and everything else down.

In your own words
A model has R@1 of 0.83 and R@5 of 0.97 on caption-to-image search. What do those two numbers say, in plain words?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: what the rest of the batch is for

In contrastive training, an image's own caption is its positive. Every other caption in the batch is a negative: a wrong answer the model must rank below the right one. With 64 pairs in a batch, each image faces a 64-way choice.

The negatives decide what the model has to learn. If the only wrong answers on offer are about cars and beaches, a dog photo can find its caption by knowing "dog", and the details never matter. If another dog caption is in the batch, "dog" isn't enough any more. So a bigger batch does more than average the gradient: it raises the odds that each pair meets a look-alike.

Uses the setup above, 67 lines.

import matplotlib.pyplot as plt

print("pairs per batch | recall@1 | recall@5 | top image is the right kind")
batches, recall1, recall5 = (4, 16, 64, 256), [], []
for batch in batches:
    r1, r5, right_kind = retrieval(*train(batch))
    recall1.append(r1)
    recall5.append(r5)
    print(str(batch).rjust(15), "|", str(r1).ljust(8), "|", str(r5).ljust(8), "|",
          right_kind)
plt.plot(batches, recall1, "o-", label="recall@1")
plt.plot(batches, recall5, "s-", label="recall@5")
plt.xscale("log")
plt.xlabel("pairs per batch")
plt.ylabel("recall")
plt.legend()
print("64, all of one kind:", retrieval(*train(64, one_kind=True)))

With 4 pairs per batch, R@1 is 0.436. With 256 it is 0.858. The gain is steep at first and flattens. With 20 kinds of scene, a batch of 4 almost never holds two of a kind, a batch of 64 holds about three of each, and by then most of what there is to learn from look-alikes is being learned. Real image-text models face millions of fine distinctions, and were trained with batches in the tens of thousands.

The last line is a warning against taking that too far. Negatives that resemble the positive are called hard negatives, and if look-alikes are what teach detail, a batch made only of one kind of scene sounds ideal. It scores 0.43, and its top image is the right kind for fewer than half the captions. A model that never sees a dog next to a car is never asked to tell them apart, and it doesn't.

One more hazard hides in the batch. Two photos of the same event may each fit the other's caption. The loss still treats the other caption as wrong and pushes it away. That is a false negative, and web-scale data is full of them.

Quick check
A batch holds 128 image-caption pairs. For one image in it, how many captions is the model being told to rank below the image's own?
  1. 64
  2. 127
  3. 1
  4. 128
Show the answer

127

The image's own caption is the one right answer. The other 127 captions in the batch are its negatives. No one chose them as wrong answers: they are simply the captions of the other images that happened to be in the batch.

Quick check
In the run above, training only on batches drawn from a single kind of scene gave the worst model in the table. Why?
  1. It saw fewer training images in total
  2. It was never asked to tell one kind of scene from another
  3. Hard negatives always make contrastive training worse
  4. Its batches held fewer pairs than the other runs
Show the answer

It was never asked to tell one kind of scene from another

Every negative it met was the same kind of scene as the positive, so the loss only ever rewarded telling details apart. Nothing pushed dogs away from cars. At test time the collection holds every kind, and the model's coarse sense of kind is weak. Negatives need variety: some easy, some hard.

The other dial on the loss is the temperature, the small number the similarities are divided by before the softmax. You saw in Multimodal transformers that dividing by 0.1 stretches the scores apart. Here is what that does to training, and to a single comparison.

Uses the setup above, 67 lines.

for temperature in (1.0, 0.1, 0.02):
    print("temperature", str(temperature).ljust(4), "| recall@1, recall@5, right kind:",
          retrieval(*train(64, temperature=temperature)))

print("two captions score 0.9 and 0.8 against an image. "
      "Probability given to the better one:")
for temperature in (1.0, 0.1, 0.02):
    a, b = math.exp(0.9 / temperature), math.exp(0.8 / temperature)
    print("  temperature", str(temperature).ljust(4), "->", round(a / (a + b), 3))

At a temperature of 1.0, a caption scoring 0.9 gets a probability of 0.525 against one scoring 0.8. The model is already about as right as the loss can see, so there is little pressure to separate close candidates, and R@1 is 0.79. At 0.02 the same gap becomes 0.993 to 0.007: small differences in similarity now matter a great deal, and R@1 rises to 0.852.

A lower temperature makes the model care about its closest rivals, which is the same thing a larger batch buys by a different route. It isn't free in real training, where a very low temperature makes the loss jumpy and punishes false negatives hard, so models such as CLIP let training choose it.

Quick check
Two captions score 0.6 and 0.5 against an image. What does this print: the probability the softmax gives the better caption at a temperature of 1, then at 0.05?
import math

for temperature in (1.0, 0.05):
    a, b = math.exp(0.6 / temperature), math.exp(0.5 / temperature)
    print(round(a / (a + b), 3), end=" ")
  1. 0.545 0.881
  2. 0.525 0.881
  3. 0.881 0.525
  4. 0.525 0.525
Show the answer

0.525 0.881

At a temperature of 1 the gap between the scores is 0.1, and the softmax splits 0.525 to 0.475. Dividing by 0.05 turns the gap into 2, and the split becomes 0.881 to 0.119. The scores didn't change. The temperature decides how much a small lead counts.

In your own words
Why does a larger batch tend to give a contrastive model finer distinctions, and why isn't a batch of look-alikes alone the answer?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: what a similarity score can't say

Everything above ends in one number per image and caption. That number is good at "what is this about". It is weaker at "how are the parts arranged", and the reason can be shown without training anything.

Take the simplest possible text encoder: give each word a vector, and add up the vectors of the words in the caption. This is called a bag of words, since it keeps the words and throws away their order.

Uses the setup above, 67 lines.

import numpy as np

g = np.random.default_rng(0)
vocabulary = "a red blue square circle and dog street with no on the".split()
vector = {word: g.normal(size=16) for word in vocabulary}  # one vector per word

def encode(caption):
    # add up the words' vectors: order is lost here
    total = sum(vector[word] for word in caption.split())
    return total / np.linalg.norm(total)

def similarity(a, b):
    return round(float(encode(a) @ encode(b)), 3)

tests = [("a red square and a blue circle", "a blue square and a red circle"),
         ("a dog on the street", "the street on a dog"),
         ("a street with a dog", "a street with no dog"),
         ("a street with a dog", "a red square on the street")]
for a, b in tests:
    print(similarity(a, b), "|", a, "|", b)
It runs right here in your browser, with nothing to install.

The first two pairs score 1.0. To this encoder, "a red square and a blue circle" and "a blue square and a red circle" are the same caption: the same words, added up in a different order, give the same sum. So do "a dog on the street" and "the street on a dog". Whatever image you show, the two captions get identical scores, and the model can't prefer the right one.

The third pair shows a quieter failure. "A street with a dog" and "a street with no dog" score 0.904. One word differs, and it reverses the meaning. For comparison, a caption about something else entirely scores 0.627.

Real text encoders are transformers, which can represent word order. The trouble is the training game. With random negatives, a caption almost never meets its own words in a different arrangement, so the model can win nearly every comparison by knowing which things are present. Tests built from reordered captions have found large image-text models scoring close to chance on them.

What follows for an engineer: use these models for what a bag of things can answer, and check before trusting them with which attribute belongs to which object, relations such as "on" or "left of", counts, and negation.

Quick check
A text encoder represents a caption by adding up one vector per word. Which pair of captions will it give exactly the same vector?
  1. "a cat chasing a dog" and "a cat chasing a bird"
  2. "a cat" and "a cat chasing a dog"
  3. "a small cat" and "a large cat"
  4. "a cat chasing a dog" and "a dog chasing a cat"
Show the answer

"a cat chasing a dog" and "a dog chasing a cat"

Those two captions contain the same five words, each the same number of times, so their sums are equal, though they describe opposite events. Each of the other pairs differs in at least one word, which changes the sum.

Drill

Captions a bag of words can't tell apart

Write same_bag(a, b). It returns True when the two captions are different sentences that a bag-of-words encoder would see as identical: they contain exactly the same words, each the same number of times, and the words are not in the same order. Compare words in lower case, and split on any whitespace. Two captions with the same words in the same order are the same sentence: return False.

Uses the setup above, 67 lines.

def same_bag(a, b):
    return False


print(same_bag("a red square and a blue circle", "a blue square and a red circle"))
In the lesson, your code is checked as you write it.
Worked example

Worked example: what kind of mistakes does the model make?

R@1 says how often the top result is exactly right. Before deciding whether a model is good enough, look at what it returns when it is wrong. Here two models, trained with 4 and with 256 pairs per batch, each answer 500 captions, and every answer is put in one of three bins.

Uses the setup above, 67 lines.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

g0 = np.random.default_rng(0)
KINDS = 20
# 20 kinds of scene: dog, car, beach...
centers = g0.normal(size=(KINDS, 4)).astype(np.float32)
to_pixels, to_words = (g0.normal(size=(12, 40)).astype(np.float32),
                       g0.normal(size=(12, 20)).astype(np.float32))

def pairs(n, seed):
    """n matching (image, caption) pairs. A scene is its kind (4 numbers) plus its own
    details (8 small numbers)."""
    g = np.random.default_rng(seed)
    kind = g.integers(0, KINDS, size=n)
    scene = np.concatenate(
        [centers[kind], 0.3 * g.normal(size=(n, 8)).astype(np.float32)], axis=1)
    images = scene @ to_pixels + 0.5 * g.normal(size=(n, 40)).astype(np.float32)
    captions = scene @ to_words + 0.5 * g.normal(size=(n, 20)).astype(np.float32)
    return images, captions, kind

def unit(v):
    return v / v.norm(dim=-1, keepdim=True)

def train(batch, steps=300, temperature=0.1, one_kind=False, seed=0):
    torch.manual_seed(seed)
    image_encoder, text_encoder = nn.Linear(40, 16), nn.Linear(20, 16)
    opt = torch.optim.Adam(
        list(image_encoder.parameters()) + list(text_encoder.parameters()), lr=1e-2)
    images, captions, kind = pairs(4096, 1)
    g = np.random.default_rng(seed)
    for step in range(steps):
        if one_kind:                 # every pair in the batch is the same kind of scene
            idx = g.choice(np.where(kind == g.integers(0, KINDS))[0], size=batch,
                           replace=False)
        else:
            idx = g.permutation(4096)[:batch]
        scores = (unit(image_encoder(torch.tensor(images[idx])))
                  @ unit(text_encoder(torch.tensor(captions[idx]))).T / temperature)
        labels = torch.arange(batch)                       # image i goes with caption i
        loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2
        opt.zero_grad()
        loss.backward()
        opt.step()
    return image_encoder, text_encoder

def similarities(image_encoder, text_encoder):
    """500 unseen pairs: the similarity of every image (rows) with
    every caption (columns)."""
    images, captions, kind = pairs(500, 2)
    with torch.no_grad():
        sims = (unit(image_encoder(torch.tensor(images)))
                @ unit(text_encoder(torch.tensor(captions))).T).numpy()
    return sims, kind

def retrieval(image_encoder, text_encoder):
    """Search the 500 images with each caption. Returns recall@1, recall@5, and how
    often the top image is the right kind."""
    sims, kind = similarities(image_encoder, text_encoder)
    # images ranked above the right one
    better = np.array([(sims[:, j] > sims[j, j]).sum() for j in range(500)])
    right_kind = (kind[sims.argmax(axis=0)] == kind).mean()
    return (round(float((better == 0).mean()), 3), round(float((better < 5).mean()), 3),
            round(float(right_kind), 3))

print("pairs per batch | exact match | right kind, wrong scene | wrong kind")
for batch in (4, 256):
    sims, kind = similarities(*train(batch))
    top = sims.argmax(axis=0)                          # the best image for each caption
    exact = int((top == np.arange(500)).sum())
    near = int(((top != np.arange(500)) & (kind[top] == kind)).sum())
    print(str(batch).rjust(15), "|", str(exact).rjust(11), "|", str(near).rjust(23),
          "|", 500 - exact - near)
In the lesson you can change this and run it.

The model trained on batches of 4 returns the exact image for 218 captions of 500. Its misses are mostly of one type: 220 times it returned the right kind of scene and the wrong one of that kind, against 62 times the wrong kind altogether. It learned kinds and not details, which is what a batch too small to contain look-alikes teaches. With 256 pairs per batch the exact matches rise to 429, and the near misses fall from 220 to 49, far more than the outright misses fall, from 62 to 22. The larger batch mostly bought fine distinctions. This breakdown also tells you whether a model fits a product: for a search box that only needs the right kind of picture, the first model is right 438 times in 500, and for matching a caption to its own photo it is right 218 times.

On the job: an image-text model is a retrieval engine first. To build search over a photo library, embed every photo once, store the vectors, and embed each query as it arrives, then rank by cosine similarity. Judge it with the numbers of this lesson, R@1 and R@5 on a set of your own queries with known right answers, and read the misses: near misses and wrong kinds need different fixes.

In your own words
You are asked whether an image-text embedding model can power a search feature for a photo library. What would you measure, and what would you warn about?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

A caption-to-image search model reports R@1 = 0.40 and R@10 = 0.95. What does that combination suggest?

  1. The right image is usually near the top, and often not first
  2. The model is right 40% of the time and wrong the rest
  3. The model returns ten results for 95% of queries
  4. The model works on 95% of images and fails on 40%

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write