Index Ai Academy
Phase 5 · Beyond text

Build a vision transformer

Patch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

You have built a transformer that reads text, and seen that an image can be cut into patches and fed to the same machine. This lesson builds that machine for images and trains it: a complete vision transformer, small enough to train in the page in a few seconds. Building it shows what the pieces are for. Breaking it shows more: take away its position embeddings and it can no longer say where anything is, and give it few images and it loses badly to a convolutional network a small fraction of its size. Both failures come from the same fact, which is the main idea of this lesson: a transformer assumes almost nothing about images, and has to learn the rest.

What you'll be able to do

  • Assemble a vision transformer from patch embeddings, a class token, position embeddings and a transformer block
  • Train one to classify small images, and read which patches its class token attends to
  • Show that without position embeddings the model can't tell where anything is
  • Explain what a convolutional network assumes about images, and why a vision transformer needs more data without those assumptions
  • Work out how the number of tokens, and the cost of attention, grow as patches shrink
Quick check
A guess before you start. A vision transformer is built without position embeddings, so each patch arrives with no note of where it came from. Which question about a photo can it no longer answer?
  1. Whether the photo was taken in bright light
  2. Whether the dog is at the top or the bottom
  3. Whether the photo contains a dog at all, or none
  4. Whether the dog in the photo is brown or black
Show the answer

Whether the dog is at the top or the bottom

Attention treats its inputs as a set. Without positions, the patches of a photo are a bag of pieces, and the same bag is produced whether the dog is at the top or at the bottom. What is in the photo survives. Where it is doesn't. You will measure this below.

The idea: patches, a class token and positions

This rests on Multimodal transformers, where an image first became a sequence of tokens, and on The transformer block for the block that reads them.

Patches as tokens

A transformer reads a sequence of vectors. In Multimodal transformers you turned an image into such a sequence: cut it into square patches, flatten each into a row of numbers, and pass each row through a linear layer, the patch embedding.

This lesson needs a task small enough to train on in the page. Each image is 12 by 12 gray pixels of noise with one small shape drawn somewhere in it, and the model has to say which of four shapes it is.

Setup (runs before every cell) · 68 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

SHAPES = [[(1, 0), (1, 1), (1, 2)], [(0, 1), (1, 1), (2, 1)], [(0, 0), (1, 1), (2, 2)],
          [(0, 2), (1, 1), (2, 0)]]
NAMES = ["a flat bar", "an upright bar", "a diagonal going down", "a diagonal going up"]

def make(n, seed):
    """n gray images, 12 by 12: noise, plus one 3-pixel shape at a random place."""
    g = np.random.default_rng(seed)
    X = g.normal(0, 0.25, size=(n, 12, 12)).astype(np.float32)
    shape, top, left = (g.integers(0, 4, size=n), g.integers(0, 10, size=n),
                        g.integers(0, 10, size=n))
    for i in range(n):
        for dy, dx in SHAPES[shape[i]]:
            X[i, top[i] + dy, left[i] + dx] += 1.0
    return X, shape, top, left

def patches(X, p=4):
    n = len(X)
    return X.reshape(n, 12 // p, p, 12 // p, p).transpose(0, 1, 3, 2, 4).reshape(
        n, (12 // p) ** 2, p * p)

class ViT(nn.Module):
    def __init__(self, n_patches=9, patch_dim=16, d=16, classes=4, use_position=True):
        super().__init__()
        self.embed = nn.Linear(patch_dim, d)                       # the patch embedding
        self.cls = nn.Parameter(torch.randn(1, 1, d) * 0.02)       # the class token
        self.position = nn.Parameter(torch.randn(1, n_patches + 1, d) * 0.02)
        self.use_position = use_position
        self.ln1, self.ln2, self.ln3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
        self.q, self.k, self.v, self.proj = (nn.Linear(d, d), nn.Linear(d, d),
                                             nn.Linear(d, d), nn.Linear(d, d))
        self.mlp = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))
        self.head = nn.Linear(d, classes)

    def forward(self, x):
        # the class token goes first
        x = torch.cat([self.cls.expand(len(x), 1, -1), self.embed(x)], dim=1)
        if self.use_position:        # the line the position experiment switches off
            x = x + self.position
        h = self.ln1(x)
        self.attention = torch.softmax(
            self.q(h) @ self.k(h).transpose(1, 2) / math.sqrt(h.shape[-1]), dim=-1)
        # no mask: every token sees every token
        x = x + self.proj(self.attention @ self.v(h))
        x = x + self.mlp(self.ln2(x))
        return self.head(self.ln3(x)[:, 0])  # the answer is read off the class token

def train(model, inputs, labels, steps, seed=0):
    torch.manual_seed(seed)
    opt, g = torch.optim.Adam(model.parameters(), lr=3e-3), np.random.default_rng(seed)
    for step in range(steps):
        idx = g.integers(0, len(inputs), size=64)
        loss = F.cross_entropy(model(torch.tensor(inputs[idx])),
                               torch.tensor(labels[idx]))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model

def accuracy(model, inputs, labels):
    with torch.no_grad():
        return round(float(
            (model(torch.tensor(inputs)).argmax(dim=-1).numpy() == labels).mean()), 3)
X, shape, top, left = make(3, 5)
for y in range(12):                        # the first image, with a gap between patches
    print("  ".join(" ".join("#" if X[0, y, x] > 0.5 else "." for x in range(c, c + 4))
                    for c in (0, 4, 8)))
    if y in (3, 7):
        print()
print("the shape:", NAMES[shape[0]], "| its top left corner: row", int(top[0]),
      "column", int(left[0]))
print("images:", X.shape, "-> patch tokens:", patches(X).shape)

The image is cut into a 3 by 3 grid of patches, 4 pixels on a side. Each becomes a row of 16 numbers, so an image is 9 tokens of 16 numbers each. The shape can fall anywhere, and often straddles two patches, as it does here. Nothing tells the model that the pixels along a patch's edge continue into the next patch. (The lone mark on the right isn't part of the shape. It is noise that happened to be bright.)

Quick check
From the lesson on multimodal transformers: an image of 224 by 224 pixels is cut into patches 16 pixels on a side. How many patch tokens does it become?
  1. 14
  2. 50,176
  3. 3,136
  4. 196
Show the answer

196

224 / 16 = 14 patches fit across and 14 down, and 14 × 14 = 196. The count depends on the square of the image's side divided by the patch's side, which is why small patches are expensive.

Quick check
The same 12 by 12 images are cut into patches 3 pixels on a side instead. What does this print: the shape of the tokens for a batch of 5 images?
import numpy as np

X = np.zeros((5, 12, 12))
p = 3
tokens = X.reshape(5, 12 // p, p, 12 // p, p).transpose(0, 1, 3, 2, 4).reshape(
    5, (12 // p) ** 2, p * p)
print(tokens.shape)
  1. (5, 16, 9)
  2. (5, 9, 16)
  3. (5, 4, 9)
  4. (5, 144)
Show the answer

(5, 16, 9)

With 3-pixel patches, 4 fit across and 4 down: 16 patches. Each holds 3 × 3 = 9 pixels. Smaller patches give more tokens, each carrying less.

Try it now

Press Next. Every box is a tensor, and its shape is written under its name.

In your own words
In your own words: how does an image become a sequence a transformer can read?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the whole model

A vision transformer adds three things to the patch tokens, then reuses the transformer block you already know.

  • A class token: one extra learned vector placed at the front of every sequence. It stands for no patch. Its job is to collect, through attention, whatever the model needs to decide, and the answer is read from it at the end.
  • Position embeddings: one learned vector per position, added to the tokens, so that each carries a note of where it sat.
  • A classification head: a linear layer from the class token's final vector to one score per class.

The block in between is the one from this phase, with one change. A language model masks the future, so that a word can't look at the words after it. An image has no future. Every patch may attend to every other patch, so the mask is gone.

The cell below builds the model, trains it on 6,400 images for 600 steps, and then prints, for three test images, how the class token spread its attention over the 3 by 3 grid of patches.

Uses the setup above, 68 lines.

X, shape, _, _ = make(6400, 0)
X_test, shape_test, top_test, left_test = make(1000, 1)

torch.manual_seed(0)
vit = ViT()
print("weights:", sum(p.numel() for p in vit.parameters()))
print("test accuracy before training:", accuracy(vit, patches(X_test), shape_test))
train(vit, patches(X), shape, 600)
print("test accuracy after 600 steps:", accuracy(vit, patches(X_test), shape_test))

with torch.no_grad():
    vit(torch.tensor(patches(X_test[:3])))
for i in range(3):
    print("image", i, "|", NAMES[shape_test[i]], "| top left corner at row",
          int(top_test[i]), "column", int(left_test[i]))
    # row 0 is the class token; drop its look at itself
    print(np.round(vit.attention.numpy()[i, 0, 1:].reshape(3, 3), 2))

The model has 2,772 weights. It starts at 0.279, which is chance for four classes, and reaches 0.778.

The three grids are the interesting part. Nobody told the model where the shape is. The class token has learned to find it anyway: in the first image nearly all of its attention goes to the top middle patch, and the shape's corner is at row 0, column 5, inside that patch. In the second, the shape starts at column 2 and runs across the boundary between the bottom left and bottom middle patches, and the attention is on the bottom left one.

That is what the class token is for. It asks every patch "do you hold anything that matters?", and pulls the answer from wherever the shape happens to be.

Quick check
The transformer block in a language model masks attention so that a token can't look at later tokens. Why does a vision transformer drop that mask?
  1. An image has no order in time, so no patch needs hiding
  2. Images have too few tokens for a mask to matter
  3. The class token would be hidden by the mask
  4. Masking only works with word embeddings
Show the answer

An image has no order in time, so no patch needs hiding

A language model is trained to predict the next token, so seeing later tokens would be cheating. A classifier looking at a whole image has nothing to predict ahead of time: all the patches are given at once, and the patch in the bottom corner may be exactly what explains the one at the top.

Quick check
The model above puts a class token in front of its 9 patch tokens. What does this print: the number of tokens, and the number of attention weights its one block computes for one image?
patch_tokens = (12 // 4) ** 2
tokens = patch_tokens + 1
print(tokens, tokens * tokens)
  1. 9 81
  2. 10 20
  3. 9 90
  4. 10 100
Show the answer

10 100

Nine patches plus the class token is 10 tokens. Every token attends to every token, itself included, so the attention table is 10 by 10: 100 weights.

Drill

Read an attention map

Write most_attended(row, columns). row is the class token's attention over all tokens: the first number is its attention to itself, and the rest are its attention to the patches in reading order. The patches form a grid with columns patches across. Return a tuple (patch_row, patch_column, share): the grid position of the patch with the most attention, and that patch's share of the attention given to patches (leaving the class token's look at itself out), rounded to 3 places. If two patches tie, return the earlier one.

Uses the setup above, 68 lines.

def most_attended(row, columns):
    return (0, 0, 0.0)


print(most_attended([0.1, 0.0, 0.05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.85, 0.0], 3))
In the lesson, your code is checked as you write it.

Without positions, a bag of patches

Attention has no sense of order. It compares vectors, and if two patches swap places, their vectors are compared in exactly the same way. The only thing that tells the model where a patch sat is the position embedding added to it.

To see how much rests on that, give the model a question about place: is the shape in the lower half of the image? Then train it twice, once with position embeddings and once without.

Uses the setup above, 68 lines.

X, _, top, _ = make(1600, 0)
X_test, _, top_test, _ = make(1000, 1)
# 1 if the shape is in the lower half
lower, lower_test = (top >= 5).astype(np.int64), (top_test >= 5).astype(np.int64)

for use_position in (True, False):
    torch.manual_seed(0)
    model = train(ViT(classes=2, use_position=use_position), patches(X), lower, 300)
    print("position embeddings:", str(use_position).ljust(5),
          "| is the shape in the lower half? test accuracy:",
          accuracy(model, patches(X_test), lower_test))

With position embeddings the model answers correctly 0.941 of the time. Without them it scores 0.526, a coin flip, and no amount of training would help. A shape in the top row and the same shape in the bottom row put the same nine patches into the bag.

Language models have the same property, which is why you added position information to the GPT you built. For images it is easier to forget, since a picture looks like it has its layout built in. The layout is in the array. It is lost the moment the array becomes a set of tokens, unless you put it back.

Quick check
A trained vision transformer with no position embeddings classifies a photo. The photo's patches are then shuffled into a random order and it classifies again. What happens to its answer?
  1. Nothing: it gives the same answer for any order of the patches
  2. It fails, since the shuffled photo no longer shows an object
  3. It improves, since shuffling acts as extra training data
  4. It changes at random each time the patches are shuffled
Show the answer

Nothing: it gives the same answer for any order of the patches

Without position embeddings, nothing the model computes depends on the order of the patch tokens: attention compares the same vectors whichever slot they sit in, and the class token gathers the same total. A photo and its shuffled jigsaw look identical to it. A person finds that bizarre, and it follows directly from how attention works.

In your own words
Why does a vision transformer need position embeddings, when the image array already has rows and columns?

Write a sentence or two, then compare it with a good answer in the lesson.

In your own words
What is the class token, and why does the model still need position embeddings?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: what a CNN knows before it sees an image

A convolutional network comes with two assumptions about images built into its wiring.

  • Locality. A filter looks at a small neighborhood, since nearby pixels belong together.
  • The same pattern means the same thing anywhere. One filter is slid across the whole image, so an edge detector learned in one corner works in every corner.

Assumptions like these, fixed before any training, are called an inductive bias. A vision transformer has almost none. Its patches are cut on an arbitrary grid, any patch may attend to any other from the first layer, and what it learns about a shape in one patch tells it nothing about the same shape in another. Whatever a CNN is given for free, it has to learn from examples.

The experiment below measures the price. The same shape task is given to the vision transformer and to a very small CNN: 8 filters of 3 by 3 pixels, slid over every position, keeping each filter's strongest response. Each is trained on 100 images, then on 6,400.

Uses the setup above, 68 lines.

def windows(X):
    """Every 3 by 3 window of every image, flattened: 100 windows of 9 pixels each."""
    out = np.zeros((len(X), 10, 10, 9), dtype=np.float32)
    for dy in range(3):
        for dx in range(3):
            out[:, :, :, dy * 3 + dx] = X[:, dy:dy + 10, dx:dx + 10]
    return out.reshape(len(X), 100, 9)

class TinyCNN(nn.Module):
    def __init__(self, filters=8, classes=4):
        super().__init__()
        # a linear layer on each window is a convolution
        self.filters, self.head = nn.Linear(9, filters), nn.Linear(filters, classes)

    def forward(self, w):
        # each filter's strongest response, wherever it was
        return self.head(torch.relu(self.filters(w)).max(dim=1).values)

X_test, shape_test, _, _ = make(1000, 1)
print("weights | vision transformer:", sum(p.numel() for p in ViT().parameters()),
      "| CNN:", sum(p.numel() for p in TinyCNN().parameters()))
print("training images | vision transformer: train, test | CNN: train, test")
for n in (100, 6400):
    X, shape, _, _ = make(n, 0)
    torch.manual_seed(0)
    vit = train(ViT(), patches(X), shape, 600)
    torch.manual_seed(0)
    cnn = train(TinyCNN(), windows(X), shape, 600)
    print(str(n).rjust(15), "|", str(accuracy(vit, patches(X), shape)).ljust(5),
          str(accuracy(vit, patches(X_test), shape_test)).ljust(5), "               |",
          str(accuracy(cnn, windows(X), shape)).ljust(5),
          accuracy(cnn, windows(X_test), shape_test))

This cell trains four models and takes about ten seconds.

With 100 images, the vision transformer gets every training image right and scores 0.282 on new ones, which is chance. It has memorized 100 pictures and learned nothing about shapes. The CNN, with 116 weights to the transformer's 2,772, scores 0.968 from the same 100 images.

With 6,400 images the transformer has begun to learn the task. It reaches 0.778, and its training and test scores now agree, so it is no longer memorizing. The CNN is at 0.974, about where it was with 100.

The CNN isn't a better learner. Its assumptions fit this task exactly: a 3 by 3 filter slid over the whole image is the right tool for a small shape that can be anywhere, and one example of a flat bar teaches it flat bars in all hundred positions. The transformer has to meet each shape in each place, straddling each patch boundary, before it knows them all.

The trade runs both ways. An assumption that fits saves data, and one that doesn't fit holds a model back. A transformer given enough data can learn what the CNN assumed, and isn't limited to it.

Quick check
A team has 2,000 labeled X-ray images and wants to train a classifier from scratch, with no pretrained model. Based on this lesson, which is the safer choice, and why?
  1. Either: the architecture makes no difference at that size
  2. A CNN: its built-in assumptions do work that 2,000 images can't
  3. A vision transformer: attention copes better with small datasets
  4. A vision transformer: it has more weights, so it learns faster
Show the answer

A CNN: its built-in assumptions do work that 2,000 images can't

With little data, a model has to lean on what it assumes. A CNN assumes locality and that patterns repeat across the image, which is true of X-rays. A vision transformer would have to learn both from 2,000 examples. In practice the usual answer is a third option: start from a model pretrained on a large dataset, which is how vision transformers are normally used.

The price of small patches

One design choice remains: the size of a patch. Small patches keep fine detail, since each token covers less of the image. They also multiply the tokens, and attention compares every token with every other, so its work grows with the square of the token count.

Uses the setup above, 68 lines.

print("image side | patch | tokens with the class token | attention weights per layer")
for side, patch in ((224, 32), (224, 16), (224, 8), (384, 16), (1024, 16)):
    tokens = (side // patch) ** 2 + 1
    print(str(side).rjust(10), "|", str(patch).rjust(5), "|", str(tokens).rjust(27),
          "|", f"{tokens * tokens:,}".rjust(27))
It runs right here in your browser, with nothing to install.

At 224 pixels with 16-pixel patches, the standard setting, there are 197 tokens and 38,809 attention weights per layer. Halving the patch to 8 pixels gives four times the tokens, 785, and about sixteen times the attention, 616,225. A 1,024-pixel image at 16-pixel patches needs over 16 million weights per layer.

That is why vision transformers resize images to modest sizes, and why detection and segmentation, which need high resolution, use variants that restrict attention to local windows or shrink the grid in stages.

Go deeper How vision transformers are used in practice
  • Pretrain, then fine-tune. The paper that introduced the vision transformer found the same thing as the experiment above, at full scale: trained on a mid-sized dataset it fell short of comparable CNNs, and pretrained on hundreds of millions of images it passed them. A vision transformer is almost always started from pretrained weights.
  • Averaging instead of a class token. Many models drop the class token and average the patch tokens' final vectors. Both work.
  • Hybrids. Some designs put a few convolutional layers first, to bring back locality, and attention after.
Drill

Tokens and attention cost

Write vit_cost(side, patch, layers) for a square image side pixels on a side, cut into square patches patch pixels on a side, with one class token added. Return a tuple (tokens, attention_weights): the number of tokens including the class token, and the total number of attention weights computed, which is tokens × tokens in each of the layers layers. If the side isn't a multiple of the patch size, raise ValueError.

Uses the setup above, 68 lines.

def vit_cost(side, patch, layers):
    return (0, 0)


print(vit_cost(224, 16, 12))
In the lesson, your code is checked as you write it.
Quick check
A vision transformer's patch size is changed from 16 pixels to 4 pixels, on the same image. Roughly how does the amount of attention computation change?
  1. It grows about 256 times
  2. It stays about the same
  3. It grows about 16 times
  4. It grows about 4 times
Show the answer

It grows about 256 times

A patch a quarter as wide means 4 times as many across and 4 times as many down: 16 times the tokens. Attention compares every token with every token, so its work grows with the square of that: 16 × 16 = 256 times.

Worked example

Worked example: follow one batch through the model

The quickest way to understand any transformer is to print the shape of the data after each step. Here a batch of 5 images goes through the vision transformer's steps one at a time, with fresh, untrained weights.

Uses the setup above, 68 lines.

import math
import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
images = torch.randn(5, 12, 12)                               # a batch of 5 gray images
p, d, classes = 4, 16, 4

tokens = images.reshape(5, 3, p, 3, p).transpose(2, 3).reshape(5, 9, p * p)
print("1. cut into patches:        ", tuple(tokens.shape),
      "  5 images, 9 patches, 16 pixels each")

x = nn.Linear(p * p, d)(tokens)
print("2. patch embedding:         ", tuple(x.shape),
      "  each patch is now a vector of width 16")

cls = nn.Parameter(torch.randn(1, 1, d) * 0.02)
x = torch.cat([cls.expand(5, 1, -1), x], dim=1)
print("3. class token in front:    ", tuple(x.shape), " one more token per image")

x = x + nn.Parameter(torch.randn(1, 10, d) * 0.02)
print("4. position embeddings:     ", tuple(x.shape),
      " same shape: positions are added, not appended")

q, k, v = nn.Linear(d, d)(x), nn.Linear(d, d)(x), nn.Linear(d, d)(x)
attention = torch.softmax(q @ k.transpose(1, 2) / math.sqrt(d), dim=-1)
print("5. attention weights:       ", tuple(attention.shape),
      " every token against every token")
print("   each row sums to:        ", round(float(attention[0, 0].sum().item()), 3))

x = x + attention @ v
print("6. after the attention step:", tuple(x.shape),
      " the tokens, updated with what they gathered")

scores = nn.Linear(d, classes)(x[:, 0])
print("7. head on the class token: ", tuple(scores.shape),
      "     one score per class, per image")
In the lesson you can change this and run it.

Seven steps, and only three shapes that matter. The image becomes 9 tokens. The class token makes it 10, and from then on the shape is (5, 10, 16) all the way through the block: a transformer block changes what the tokens hold, never how many there are or how wide. Position embeddings are added to the tokens, so they change no shape either. The attention table is (5, 10, 10), one row per token saying how it spread its attention, and each row sums to 1. At the end the head ignores nine of the ten tokens and reads the four class scores from position 0. The layer norms, the MLP and the residual connections of the full block are left out here, since none of them changes a shape.

On the job: when you fine-tune an image model on a few thousand labeled photos, start from a vision transformer that was pretrained on millions of images, not from random weights. The experiment above is why: a transformer trained from scratch on little data loses to a much smaller CNN, but one that has already seen a lot learns your task from a few examples.

In your own words
You are choosing between a CNN and a vision transformer for an image task. What would you ask about the project before deciding?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

What does the class token of a vision transformer stand for at the input?

  1. No part of the image: it is a learned vector
  2. The patch in the top left corner of the image
  3. The average of all the patches in the image
  4. The label of the image, given to the model as a hint

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write