Attention let the three tokens read from each other. The other half of every transformer block does something different: it takes each token's row on its own, spreads it out to four times its width, bends it with a nonlinearity, and squeezes it back. No token sees another during this step. It looks like the plainest part of the block, and it holds about two thirds of the block's parameters.
In this lesson you run that layer on the running example, count its parameters as the hidden width changes, and meet the best current account of what all those parameters are for.
What you'll be able to do
Write the feed-forward equation with the shapes of W₁, b₁, W₂ and b₂, and compute it for the 3 × 8 example
Track how the hidden width 4 × d_model sets each matrix's shape and the parameter count
State what GELU does that ReLU doesn't, with its values at −1, 0 and 1
Describe the key-value memory view of the layer, and say what evidence it rests on
Each row is widened from 8 numbers to 32, bent by GELU and squeezed back; the boards count the parameters as the hidden width changes and compare GELU with ReLU. Transcript
Quick check
A feed-forward layer for d_model = 8 with a hidden width of 32 has two weight matrices and two bias vectors. A guess: how many parameters?
8 × 32 = 256 weights in, 32 biases, 32 × 8 = 256 weights out, 8 biases: 552. The two biases are the part people forget.
The idea: widen, bend, narrow
This rests on A layer and a small network as matrices, where a layer is a matrix, a bias and a bend. A block's second half is exactly that, twice, on every token alone.
Expand, bend, project back
After attention, each block runs a small two-layer network on every token row separately. The original transformer wrote it with ReLU:
FFN(x)=max(0,xW1+b1)W2+b2
x∈Rdmodel is one token's row (here 8 numbers).
W1∈Rdmodel×dff and b1∈Rdffexpand the row to the hidden width dff=4dmodel, 32 here.
max(0,⋅), or GELU in GPT-2 and most models since, bends it: the only nonlinear step in the block apart from the softmax.
W2∈Rdff×dmodel and b2∈Rdmodelproject back to dmodel, so the output can be added to the input.
For the whole prompt, X∈R3×8 becomes a hidden 3×32, then 3×8 again. The same W1 and W2 serve every row, and no row touches another: the layer is position-wise. Mixing between tokens happened in attention; this step works on what each token has gathered so far. Check the shapes before any arithmetic:
It runs right here in your browser, with nothing to install.
In a real block the input is X plus the attention output, normalized. To keep one set of numbers, the example feeds the X you already know.
Try it now
Four hidden values from two inputs. The second one comes out negative and the bend sets it to 0.
Worked example
The three shapes
X from the embedding lesson, seeded weights, GELU in the middle. Watch the width go 8 → 32 → 8.
import numpy as np
np.set_printoptions(precision=2, suppress=True)
X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
[1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
[0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
d_model, d_ff = 8, 32
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(d_model, d_ff)).round(2); b1 = np.zeros(d_ff)
W2 = rng.normal(0.0, 0.5, size=(d_ff, d_model)).round(2); b2 = np.zeros(d_model)
defgelu(x):
return0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x ** 3)))
H = X @ W1 + b1 # (3, 32): the expansion
out = gelu(H) @ W2 + b2 # (3, 8): back to d_modelprint(X.shape, "->", H.shape, "->", out.shape)
print("hidden row for \" sat\", first 8 of 32:", H[2, :8])
print("output\n", out)
In the lesson you can change this and run it.
(3, 8) -> (3, 32) -> (3, 8). The hidden row for " sat" starts 0.17, 0.27, 0.42, -0.04, -0.80, …: 32 dot products between its 8 numbers and the 32 columns of W1. After GELU and W2 the output row is -1.23, -1.47, 0.79, -0.95, 0.67, 1.09, 1.84, -0.59, eight numbers ready to be added back to the token's row.
Quick check
The feed-forward layer is applied to each row on its own. If two different prompts happened to hand the layer the same row for one token, what would it output for that token?
Different rows, because the layer reads the neighbors
The average of the two prompts' rows
The same output row: the layer never looks at the other tokens
The same row only if the prompts have the same length
Show the answer
The same output row: the layer never looks at the other tokens
The layer is a function of one row at a time, with fixed weights. Context from other tokens only reaches it through what attention already mixed into that row.
GELU, the smooth gate
Without the nonlinearity the two layers would collapse: xW1W2 is one 8 × 8 matrix, however wide the hidden layer. The bend is what makes the width worth having.
ReLU, max(0,x), zeroes every negative hidden value: about half of ours, since random weights give positive and negative dot products equally often. GELU keeps a little of the small negatives:
GELU(x)=xΦ(x)≈0.5x(1+tanh(2/π(x+0.044715x3)))
Φ is the standard normal's cumulative distribution, the probability that a random normal number is below x. So GELU scales x by how far above average it is: at −2 it gives −0.04, at −1 it gives −0.16, at 0 it gives 0, at 1 it gives 0.84, at 2 it gives 1.96. Large positives pass almost untouched, large negatives are almost removed, and the transition is smooth, which keeps gradients alive for units that sit near zero. GPT-2 uses exactly the tanh form above.
Worked example
GELU against ReLU
Five test values, then what each function does to the 96 hidden values of the example.
ReLU is a hard switch: 0, 0, 0, 1, 2. GELU is the soft version: -0.04, -0.16, 0, 0.84, 1.96. On the example's hidden values, ReLU would zero 51% of them outright; GELU leaves only 10% near zero, letting small negatives through at a reduced size. Both do the job the block needs, a bend between the two linear maps.
Quick check
ReLU applied to five values. What does this print?
import numpy as np
print(np.maximum(0, np.array([-2.0, -1.0, 0.0, 1.0, 2.0])))
[2. 1. 0. 1. 2.]
[0. 0. 0. 1. 2.]
[-2. -1. 0. 1. 2.]
[0. 0. 0. 0. 0.]
Show the answer
[0. 0. 0. 1. 2.]
max(0, x) keeps positives and replaces every negative with 0. The dots show these are floats.
In your own words
What would go wrong if the layer had no nonlinearity between W1 and W2?
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: a module, a count and what fires
Write GELU, count the parameters, build the layer as a module, and then read which hidden units fire on the trained tiny GPT.
Drill
Write GELU
Write gelu(x) with the tanh approximation, 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x**3))), working on a NumPy array.
For dmodel=8: m=1 gives 144, m=2 gives 280, m=4 gives 552, m=8 gives 1,096. The count grows in step with the multiplier, because both matrices grow with it.
GPT-2's numbers show why this layer matters: dmodel=768, dff=3072, so one layer's feed-forward part has 768×3072+3072+3072×768+768=4,722,432 parameters. Its attention part has 2,362,368. Across 12 layers the feed-forward layers hold 56,669,184 parameters, about 46% of the whole 124-million model, twice what attention holds.
The example below builds the layer as a PyTorch module and counts its parameters for each multiplier. It runs on the course's teaching version of PyTorch, the same code as the real library on the CPU, slower and with different random starting values.
Quick check
For d_model = 8, which hidden-width multiplier gives a feed-forward layer of 1,096 parameters?
Change mult and both linear layers change with it. The parameter count comes from the module itself.
import torch
from torch import nn
classFeedForward(nn.Module):
def__init__(self, d_model, mult=4):
super().__init__()
self.net = nn.Sequential(
nn.Linear(d_model, mult * d_model), # expand
nn.GELU(), # bend
nn.Linear(mult * d_model, d_model), # project back
)
defforward(self, x):
returnself.net(x)
torch.manual_seed(0)
x = torch.randn(3, 8) # three token rows of d_model = 8for mult in (1, 2, 4, 8):
ff = FeedForward(8, mult)
n = sum(p.numel() for p in ff.parameters())
print(f"mult {mult}: hidden width {8 * mult:2d}, "f"{n:5d} parameters, output {tuple(ff(x).shape)}")
In the lesson you can change this and run it.
The output is (3, 8) every time; only the middle changes. The counts, 144, 280, 552 and 1,096, match the formula, and each is the two Linear layers' weights and biases added up. In GPT-2 the same module with dmodel=768 and m=4 has 4,722,432.
Quick check
From the transformer block lesson: which two sublayers make up a block, in order, and what joins them?
Attention, then the feed-forward layer, each wrapped in a residual add and a layer norm
Two attention layers, joined by a mask
The embedding layer, then attention
The feed-forward layer, then attention, with a softmax between them
Show the answer
Attention, then the feed-forward layer, each wrapped in a residual add and a layer norm
A block is attention followed by the feed-forward layer, and each sublayer's output is added back to its input (the residual) with a layer norm. The feed-forward part is the second half of every block.
Drill
Count the parameters
Write ffn_params(d_model, mult) that returns the number of parameters in a feed-forward layer with hidden width mult * d_model, biases included.
In the lesson, your code is checked as you write it.
Worked example
The tiny GPT's feed-forward layer, unit by unit
Block 0 of the trained model, on Describe the Lark headlamp . The bars show the hidden value of each of the 128 units for the token "headlamp", after GELU.
import torch
import matplotlib.pyplot as plt
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
block = model.blocks[0]
count = lambda m: sum(p.numel() for p in m.parameters())
attn, mlp = count(block.qkv) + count(block.proj), count(block.mlp)
share = mlp / (attn + mlp)
print("attention", attn, "feed-forward", mlp, f"= {share:.0%} of the block")
ids = torch.tensor([tok.encode("Describe the Lark headlamp .")])
x = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
h = block.mlp[1](block.mlp[0](block.ln2(x))) # (1, 5, 128): after GELU
a = h[0, 3] # the token "headlamp"
top = torch.topk(a.abs(), 3).indices
print("strongest units", [int(i) for i in top], [round(float(a[i]), 2) for i in top])
print(int((a.abs() > 0.5).sum()), "of", a.shape[0], "units above 0.5 in size")
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.bar(range(128), a.tolist())
ax.set_xlabel("hidden unit")
ax.set_ylabel("value after GELU")
ax.set_title("Which of the 128 hidden units fire for headlamp")
In the lesson you can change this and run it.
Attention takes 4,224 parameters and the feed-forward layer 8,352, 66% of the block, the "about two thirds" the first paragraph promised. For "headlamp", units 88, 67 and 83 fire hardest, at 3.10, 1.91 and 1.91, and 38 of the 128 units are above 0.5 in size. The picture is the memory view in action on a trained model: a minority of units carry the token's row, and the rest stay near zero.
Quick check
Block 0 has 4,224 attention parameters and 8,352 feed-forward ones. What share of the block is the feed-forward layer, to the nearest percent?
8,352 / 12,576 is 0.664, so 66%. Layer norms add a few more parameters to the block, but the feed-forward layer is about two thirds, as in GPT-2 where it is 4.7 of 7.1 million.
What the layer stores
Why spend half a model on a step that treats each token alone? The best-supported account reads the layer as a key-value memory. Each hidden unit j has a column of W1, and the hidden value hj=x⋅W1[:,j]+b1j is a dot product, like an attention score: it measures how well the token's row matches that unit's pattern, its key. The nonlinearity decides which units fire. Each firing unit then adds its row of W2, its value, scaled by how strongly it fired, into the output:
FFN(x)=j=1∑dffGELU(hj)W2[j,:]+b2
The sum over units is the same matrix product as before, written one memory slot at a time:
import numpy as np
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(8, 32)).round(2)
W2 = rng.normal(0.0, 0.5, size=(32, 8)).round(2)
g = np.tanh(np.arange(32) / 16 - 1) # stand-in for GELU(h): 32 firing strengths
by_unit = sum(g[j] * W2[j, :] for j inrange(32))
print(np.allclose(by_unit, g @ W2))
So the layer is a lookup over dff stored patterns: 32 in our model, 3,072 in each GPT-2 layer. The evidence comes from research, not from the design: in trained models, individual units respond to recognizable patterns (Geva and others, 2021), and editing a few rows of W2 in a middle layer changes what a model states as fact while leaving the rest intact (Meng and others, 2022). That's why these layers are described as where a model's factual associations mostly live. It's the current best explanation, and research into it is ongoing.
The block then adds the output back: x+FFN(x). Every layer edits the running representation rather than replacing it, which is what lets dozens of blocks stack. GPT-2 small has 12; a 7-billion-parameter model of the Llama family has 32, the number of monoliths in the video.
Worked example
Which hidden units fire for " sat"
The three strongest hidden units for " sat", and how much of the output row their W₂ rows account for.
import numpy as np
np.set_printoptions(precision=2, suppress=True)
X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
[1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
[0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
rng = np.random.default_rng(11)
W1 = rng.normal(0.0, 0.5, size=(8, 32)).round(2); b1 = np.zeros(32)
W2 = rng.normal(0.0, 0.5, size=(32, 8)).round(2); b2 = np.zeros(8)
defgelu(x):
return0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x ** 3)))
a = gelu(X[2] @ W1 + b1) # 32 activations for " sat"
top = np.argsort(-np.abs(a))[:3]
print("strongest units:", top, "activations:", a[top])
# the three units' W2 rows, scaled and added
partial = (a[top][:, None] * W2[top]).sum(axis=0)
full = a @ W2 + b2
print("from the top 3:", partial)
print("full output: ", full)
print("share of the output's size:",
f"{np.linalg.norm(partial) / np.linalg.norm(full):.0%}")
print("units above 0.5:", int((np.abs(a) > 0.5).sum()), "of 32")
In the lesson you can change this and run it.
Units 23, 12 and 21 fire hardest for " sat", at 1.06, 0.94 and 0.70, and only 4 of the 32 units exceed 0.5. Those three units' W2 rows, scaled by their activations and added, give a vector that is 93% of the output's size. A few matched patterns do most of the work: that sparsity is what the memory view predicts, even here with random weights.
Quick check
In the key-value memory view, what plays the key, what decides whether a memory fires, and what plays the value?
The token's ID is the key, softmax fires it, and the embedding row is the value it returns
The bias b₂ is the key and W₁ the value, with the residual add as the gate
W₁'s columns are keys matched by dot product, GELU gates, W₂'s rows are the values added
A row of W₂ is the key and a column of W₁ is the value, matched by GELU
Show the answer
W₁'s columns are keys matched by dot product, GELU gates, W₂'s rows are the values added
h_j = x · W₁[:, j] scores the match, the nonlinearity gates it, and the fired unit adds W₂[j, :] scaled by its activation. Attention has the same three roles with different matrices.
In your own words
Where do most of a transformer's parameters live, and roughly how much more than attention?
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: what the config calls it
In a Hugging Face model this layer is the mlp of each block, and its hidden width is in the config. The map shows where it sits: after attention, inside the residual.
Where this shows up in an engineer's week
On the job: the feed-forward layer is most of a model's weights, so it is where memory and speed go. Quantizing a model mostly shrinks these matrices, and mixture-of-experts models replace this one layer with many, picking a few per token. When you read a model card, intermediate_size over hidden_size tells you which design you are looking at.
03A bigram modelThe simplest language model, built from counts.35 min
04Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
05Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
14The GPT architectureA stack of decoder blocks that predicts the next token.40 min
15Build a tiny GPTWrite and train a small GPT from scratch.45 min
16From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
17Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
27The block in today's open modelsOptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
28Long contextEfficient attention and million-token windows.35 min
29Mixture of expertsRoute each token to a few specialist sub-networks.35 min
30Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
31Jacobians: derivatives with shapesOptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
32Backprop through a linear layerOptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
33The softmax Jacobian and p − yOptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
34Backprop through attentionOptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
36Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
37Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
38Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
39Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
40Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
41Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
42Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
43Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
44Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
45Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
46Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
47The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write