Index Ai Academy
Phase 5 · How transformers think

The feed-forward layer

Expand to four times d_model and back: shapes, parameters and what the layer stores.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers10

Why it matters

Attention let the three tokens read from each other. The other half of every transformer block does something different: it takes each token's row on its own, spreads it out to four times its width, bends it with a nonlinearity, and squeezes it back. No token sees another during this step. It looks like the plainest part of the block, and it holds about two thirds of the block's parameters.

In this lesson you run that layer on the running example, count its parameters as the hidden width changes, and meet the best current account of what all those parameters are for.

What you'll be able to do

  • Write the feed-forward equation with the shapes of W₁, b₁, W₂ and b₂, and compute it for the 3 × 8 example
  • Track how the hidden width 4 × d_model sets each matrix's shape and the parameter count
  • State what GELU does that ReLU doesn't, with its values at −1, 0 and 1
  • Describe the key-value memory view of the layer, and say what evidence it rests on
Each row is widened from 8 numbers to 32, bent by GELU and squeezed back; the boards count the parameters as the hidden width changes and compare GELU with ReLU. Transcript
Quick check
A feed-forward layer for d_model = 8 with a hidden width of 32 has two weight matrices and two bias vectors. A guess: how many parameters?
import numpy as np

W1 = np.zeros((8, 32))
W2 = np.zeros((32, 8))
print(W1.size + 32 + W2.size + 8)
  1. 1024
  2. 512
  3. 256
  4. 552
Show the answer

552

8 × 32 = 256 weights in, 32 biases, 32 × 8 = 256 weights out, 8 biases: 552. The two biases are the part people forget.

The idea: widen, bend, narrow

This rests on A layer and a small network as matrices, where a layer is a matrix, a bias and a bend. A block's second half is exactly that, twice, on every token alone.

Expand, bend, project back

After attention, each block runs a small two-layer network on every token row separately. The original transformer wrote it with ReLU:

FFN(x)=max⁡(0,  xW1+b1) W2+b2\text{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2
  • x∈Rdmodelx \in \mathbb{R}^{d_{\text{model}}} is one token's row (here 8 numbers).
  • W1∈Rdmodel×dffW_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}} and b1∈Rdffb_1 \in \mathbb{R}^{d_{\text{ff}}} expand the row to the hidden width dff=4 dmodeld_{\text{ff}} = 4\,d_{\text{model}}, 32 here.
  • max⁡(0,⋅)\max(0, \cdot), or GELU in GPT-2 and most models since, bends it: the only nonlinear step in the block apart from the softmax.
  • W2∈Rdff×dmodelW_2 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}} and b2∈Rdmodelb_2 \in \mathbb{R}^{d_{\text{model}}} project back to dmodeld_{\text{model}}, so the output can be added to the input.

For the whole prompt, X∈R3×8X \in \mathbb{R}^{3 \times 8} becomes a hidden 3×323 \times 32, then 3×83 \times 8 again. The same W1W_1 and W2W_2 serve every row, and no row touches another: the layer is position-wise. Mixing between tokens happened in attention; this step works on what each token has gathered so far. Check the shapes before any arithmetic:

import numpy as np

X = np.zeros((3, 8))                 # three token rows, d_model = 8
W1 = np.zeros((8, 32)); W2 = np.zeros((32, 8))
print((X @ W1).shape, ((X @ W1) @ W2).shape)
It runs right here in your browser, with nothing to install.

In a real block the input is XX plus the attention output, normalized. To keep one set of numbers, the example feeds the XX you already know.

Try it now

Four hidden values from two inputs. The second one comes out negative and the bend sets it to 0.

Worked example

The three shapes

X from the embedding lesson, seeded weights, GELU in the middle. Watch the width go 8 → 32 → 8.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
              [1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
              [0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
d_model, d_ff = 8, 32
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(d_model, d_ff)).round(2); b1 = np.zeros(d_ff)
W2 = rng.normal(0.0, 0.5, size=(d_ff, d_model)).round(2); b2 = np.zeros(d_model)

def gelu(x):
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x ** 3)))

H = X @ W1 + b1              # (3, 32): the expansion
out = gelu(H) @ W2 + b2      # (3, 8): back to d_model
print(X.shape, "->", H.shape, "->", out.shape)
print("hidden row for \" sat\", first 8 of 32:", H[2, :8])
print("output\n", out)
In the lesson you can change this and run it.

(3, 8) -> (3, 32) -> (3, 8). The hidden row for " sat" starts 0.17, 0.27, 0.42, -0.04, -0.80, …: 32 dot products between its 8 numbers and the 32 columns of W1W_1. After GELU and W2W_2 the output row is -1.23, -1.47, 0.79, -0.95, 0.67, 1.09, 1.84, -0.59, eight numbers ready to be added back to the token's row.

Quick check
The feed-forward layer is applied to each row on its own. If two different prompts happened to hand the layer the same row for one token, what would it output for that token?
  1. Different rows, because the layer reads the neighbors
  2. The average of the two prompts' rows
  3. The same output row: the layer never looks at the other tokens
  4. The same row only if the prompts have the same length
Show the answer

The same output row: the layer never looks at the other tokens

The layer is a function of one row at a time, with fixed weights. Context from other tokens only reaches it through what attention already mixed into that row.

GELU, the smooth gate

Without the nonlinearity the two layers would collapse: xW1W2xW_1W_2 is one 8 × 8 matrix, however wide the hidden layer. The bend is what makes the width worth having.

ReLU, max⁡(0,x)\max(0, x), zeroes every negative hidden value: about half of ours, since random weights give positive and negative dot products equally often. GELU keeps a little of the small negatives:

GELU(x)=x Φ(x)≈0.5 x(1+tanh⁡ ⁣(2/π (x+0.044715 x3)))\text{GELU}(x) = x\,\Phi(x) \approx 0.5\,x\left(1 + \tanh\!\left(\sqrt{2/\pi}\,(x + 0.044715\,x^3)\right)\right)

Φ\Phi is the standard normal's cumulative distribution, the probability that a random normal number is below xx. So GELU scales xx by how far above average it is: at −2 it gives −0.04, at −1 it gives −0.16, at 0 it gives 0, at 1 it gives 0.84, at 2 it gives 1.96. Large positives pass almost untouched, large negatives are almost removed, and the transition is smooth, which keeps gradients alive for units that sit near zero. GPT-2 uses exactly the tanh form above.

Worked example

GELU against ReLU

Five test values, then what each function does to the 96 hidden values of the example.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

def gelu(x):
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x ** 3)))

x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
print("ReLU:", np.maximum(0, x))
print("GELU:", gelu(x).round(3))

X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
              [1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
              [0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(8, 32)).round(2)
H = X @ W1
print("hidden values ReLU zeroes:", f"{(H <= 0).mean():.0%}")
print("hidden values GELU leaves near zero (|g| < 0.05):",
      f"{(np.abs(gelu(H)) < 0.05).mean():.0%}")
In the lesson you can change this and run it.

ReLU is a hard switch: 0, 0, 0, 1, 2. GELU is the soft version: -0.04, -0.16, 0, 0.84, 1.96. On the example's hidden values, ReLU would zero 51% of them outright; GELU leaves only 10% near zero, letting small negatives through at a reduced size. Both do the job the block needs, a bend between the two linear maps.

Quick check
ReLU applied to five values. What does this print?
import numpy as np

print(np.maximum(0, np.array([-2.0, -1.0, 0.0, 1.0, 2.0])))
  1. [2. 1. 0. 1. 2.]
  2. [0. 0. 0. 1. 2.]
  3. [-2. -1. 0. 1. 2.]
  4. [0. 0. 0. 0. 0.]
Show the answer

[0. 0. 0. 1. 2.]

max(0, x) keeps positives and replaces every negative with 0. The dots show these are floats.

In your own words
What would go wrong if the layer had no nonlinearity between W1 and W2?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: a module, a count and what fires

Write GELU, count the parameters, build the layer as a module, and then read which hidden units fire on the trained tiny GPT.

Drill

Write GELU

Write gelu(x) with the tanh approximation, 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x**3))), working on a NumPy array.

import numpy as np

def gelu(x):
    return np.maximum(0, x)

print(gelu(np.array([-2.0, -1.0, 0.0, 1.0, 2.0])).round(3))
In the lesson, your code is checked as you write it.

Counting parameters

The count is one formula, with dff=m⋅dmodeld_{\text{ff}} = m \cdot d_{\text{model}} for a multiplier mm:

dmodeldff+dff+dffdmodel+dmodel=2 dmodel2m+dmodel(m+1)d_{\text{model}} d_{\text{ff}} + d_{\text{ff}} + d_{\text{ff}} d_{\text{model}} + d_{\text{model}} = 2\,d_{\text{model}}^2 m + d_{\text{model}}(m + 1)

For dmodel=8d_{\text{model}} = 8: m=1m = 1 gives 144, m=2m = 2 gives 280, m=4m = 4 gives 552, m=8m = 8 gives 1,096. The count grows in step with the multiplier, because both matrices grow with it.

GPT-2's numbers show why this layer matters: dmodel=768d_{\text{model}} = 768, dff=3072d_{\text{ff}} = 3072, so one layer's feed-forward part has 768×3072+3072+3072×768+768=4,722,432768 \times 3072 + 3072 + 3072 \times 768 + 768 = 4{,}722{,}432 parameters. Its attention part has 2,362,368. Across 12 layers the feed-forward layers hold 56,669,184 parameters, about 46% of the whole 124-million model, twice what attention holds.

The example below builds the layer as a PyTorch module and counts its parameters for each multiplier. It runs on the course's teaching version of PyTorch, the same code as the real library on the CPU, slower and with different random starting values.

Quick check
For d_model = 8, which hidden-width multiplier gives a feed-forward layer of 1,096 parameters?
  1. 16, a hidden width of 128
  2. 2, a hidden width of 16
  3. 4, a hidden width of 32
  4. 8, a hidden width of 64
Show the answer

8, a hidden width of 64

2 × 8² × 8 + 8 × (8 + 1) = 1,024 + 72 = 1,096. Multiplier 4 gives 552, multiplier 2 gives 280.

Worked example

A FeedForward module with a multiplier

Change mult and both linear layers change with it. The parameter count comes from the module itself.

import torch
from torch import nn

class FeedForward(nn.Module):
    def __init__(self, d_model, mult=4):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, mult * d_model),   # expand
            nn.GELU(),                            # bend
            nn.Linear(mult * d_model, d_model),   # project back
        )

    def forward(self, x):
        return self.net(x)

torch.manual_seed(0)
x = torch.randn(3, 8)                          # three token rows of d_model = 8
for mult in (1, 2, 4, 8):
    ff = FeedForward(8, mult)
    n = sum(p.numel() for p in ff.parameters())
    print(f"mult {mult}: hidden width {8 * mult:2d}, "
          f"{n:5d} parameters, output {tuple(ff(x).shape)}")
In the lesson you can change this and run it.

The output is (3, 8) every time; only the middle changes. The counts, 144, 280, 552 and 1,096, match the formula, and each is the two Linear layers' weights and biases added up. In GPT-2 the same module with dmodel=768d_{\text{model}} = 768 and m=4m = 4 has 4,722,432.

Quick check
From the transformer block lesson: which two sublayers make up a block, in order, and what joins them?
  1. Attention, then the feed-forward layer, each wrapped in a residual add and a layer norm
  2. Two attention layers, joined by a mask
  3. The embedding layer, then attention
  4. The feed-forward layer, then attention, with a softmax between them
Show the answer

Attention, then the feed-forward layer, each wrapped in a residual add and a layer norm

A block is attention followed by the feed-forward layer, and each sublayer's output is added back to its input (the residual) with a layer norm. The feed-forward part is the second half of every block.

Drill

Count the parameters

Write ffn_params(d_model, mult) that returns the number of parameters in a feed-forward layer with hidden width mult * d_model, biases included.

def ffn_params(d_model, mult):
    return 0

print(ffn_params(8, 4), ffn_params(768, 4))
In the lesson, your code is checked as you write it.
Worked example

The tiny GPT's feed-forward layer, unit by unit

Block 0 of the trained model, on Describe the Lark headlamp . The bars show the hidden value of each of the 128 units for the token "headlamp", after GELU.

import torch
import matplotlib.pyplot as plt
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
block = model.blocks[0]
count = lambda m: sum(p.numel() for p in m.parameters())
attn, mlp = count(block.qkv) + count(block.proj), count(block.mlp)
share = mlp / (attn + mlp)
print("attention", attn, "feed-forward", mlp, f"= {share:.0%} of the block")

ids = torch.tensor([tok.encode("Describe the Lark headlamp .")])
x = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
h = block.mlp[1](block.mlp[0](block.ln2(x)))           # (1, 5, 128): after GELU
a = h[0, 3]                                              # the token "headlamp"
top = torch.topk(a.abs(), 3).indices
print("strongest units", [int(i) for i in top], [round(float(a[i]), 2) for i in top])
print(int((a.abs() > 0.5).sum()), "of", a.shape[0], "units above 0.5 in size")
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.bar(range(128), a.tolist())
ax.set_xlabel("hidden unit")
ax.set_ylabel("value after GELU")
ax.set_title("Which of the 128 hidden units fire for headlamp")
In the lesson you can change this and run it.

Attention takes 4,224 parameters and the feed-forward layer 8,352, 66% of the block, the "about two thirds" the first paragraph promised. For "headlamp", units 88, 67 and 83 fire hardest, at 3.10, 1.91 and 1.91, and 38 of the 128 units are above 0.5 in size. The picture is the memory view in action on a trained model: a minority of units carry the token's row, and the rest stay near zero.

Quick check
Block 0 has 4,224 attention parameters and 8,352 feed-forward ones. What share of the block is the feed-forward layer, to the nearest percent?
attn, ff = 4224, 8352
print(round(100 * ff / (attn + ff)))
  1. 50
  2. 80
  3. 66
  4. 33
Show the answer

66

8,352 / 12,576 is 0.664, so 66%. Layer norms add a few more parameters to the block, but the feed-forward layer is about two thirds, as in GPT-2 where it is 4.7 of 7.1 million.

What the layer stores

Why spend half a model on a step that treats each token alone? The best-supported account reads the layer as a key-value memory. Each hidden unit jj has a column of W1W_1, and the hidden value hj=x⋅W1[:,j]+b1jh_j = x \cdot W_1[:, j] + b_{1j} is a dot product, like an attention score: it measures how well the token's row matches that unit's pattern, its key. The nonlinearity decides which units fire. Each firing unit then adds its row of W2W_2, its value, scaled by how strongly it fired, into the output:

FFN(x)=∑j=1dffGELU(hj) W2[j,:]+b2\text{FFN}(x) = \sum_{j=1}^{d_{\text{ff}}} \text{GELU}(h_j)\,W_2[j,:] + b_2

The sum over units is the same matrix product as before, written one memory slot at a time:

import numpy as np

rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(8, 32)).round(2)
W2 = rng.normal(0.0, 0.5, size=(32, 8)).round(2)
g = np.tanh(np.arange(32) / 16 - 1)          # stand-in for GELU(h): 32 firing strengths
by_unit = sum(g[j] * W2[j, :] for j in range(32))
print(np.allclose(by_unit, g @ W2))

So the layer is a lookup over dffd_{\text{ff}} stored patterns: 32 in our model, 3,072 in each GPT-2 layer. The evidence comes from research, not from the design: in trained models, individual units respond to recognizable patterns (Geva and others, 2021), and editing a few rows of W2W_2 in a middle layer changes what a model states as fact while leaving the rest intact (Meng and others, 2022). That's why these layers are described as where a model's factual associations mostly live. It's the current best explanation, and research into it is ongoing.

The block then adds the output back: x+FFN(x)x + \text{FFN}(x). Every layer edits the running representation rather than replacing it, which is what lets dozens of blocks stack. GPT-2 small has 12; a 7-billion-parameter model of the Llama family has 32, the number of monoliths in the video.

Worked example

Which hidden units fire for " sat"

The three strongest hidden units for " sat", and how much of the output row their W₂ rows account for.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
              [1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
              [0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(8, 32)).round(2); b1 = np.zeros(32)
W2 = rng.normal(0.0, 0.5, size=(32, 8)).round(2); b2 = np.zeros(8)

def gelu(x):
    return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x ** 3)))

a = gelu(X[2] @ W1 + b1)                       # 32 activations for " sat"
top = np.argsort(-np.abs(a))[:3]
print("strongest units:", top, "activations:", a[top])
# the three units' W2 rows, scaled and added
partial = (a[top][:, None] * W2[top]).sum(axis=0)
full = a @ W2 + b2
print("from the top 3:", partial)
print("full output:  ", full)
print("share of the output's size:",
      f"{np.linalg.norm(partial) / np.linalg.norm(full):.0%}")
print("units above 0.5:", int((np.abs(a) > 0.5).sum()), "of 32")
In the lesson you can change this and run it.

Units 23, 12 and 21 fire hardest for " sat", at 1.06, 0.94 and 0.70, and only 4 of the 32 units exceed 0.5. Those three units' W2W_2 rows, scaled by their activations and added, give a vector that is 93% of the output's size. A few matched patterns do most of the work: that sparsity is what the memory view predicts, even here with random weights.

Quick check
In the key-value memory view, what plays the key, what decides whether a memory fires, and what plays the value?
  1. The token's ID is the key, softmax fires it, and the embedding row is the value it returns
  2. The bias b₂ is the key and W₁ the value, with the residual add as the gate
  3. W₁'s columns are keys matched by dot product, GELU gates, W₂'s rows are the values added
  4. A row of W₂ is the key and a column of W₁ is the value, matched by GELU
Show the answer

W₁'s columns are keys matched by dot product, GELU gates, W₂'s rows are the values added

h_j = x · W₁[:, j] scores the match, the nonlinearity gates it, and the fired unit adds W₂[j, :] scaled by its activation. Attention has the same three roles with different matrices.

In your own words
Where do most of a transformer's parameters live, and roughly how much more than attention?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: what the config calls it

In a Hugging Face model this layer is the mlp of each block, and its hidden width is in the config. The map shows where it sits: after attention, inside the residual.

Where this shows up in an engineer's week

On the job: the feed-forward layer is most of a model's weights, so it is where memory and speed go. Quantizing a model mostly shrinks these matrices, and mixture-of-experts models replace this one layer with many, picking a few per token. When you read a model card, intermediate_size over hidden_size tells you which design you are looking at.

Goes deeper in

In your own words
What does the residual add, x + FFN(x), do for a model with many blocks?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

For d_model = 768 and d_ff = 3072, what are the shapes of W₁ and W₂, and what shape does a prompt of 20 tokens have after W₁?

  1. W₁: 768 × 3072, W₂: 3072 × 768, hidden: 20 × 3072
  2. W₁: 3072 × 768, W₂: 768 × 3072, hidden: 20 × 768
  3. W₁: 768 × 768, W₂: 768 × 768, hidden: 20 × 768
  4. W₁: 20 × 3072, W₂: 3072 × 20, hidden: 768 × 3072

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 5 lessons

How Transformers Think: an interactive inference walkthrough

Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write