This is where the whole Foundation arrives. The model behind a chat assistant is mostly one block, repeated: the same ten lines, again and again, and every one of them is a lesson you've done. Ids become vectors, the vectors take their places, attention blends them, add and norm steadies them, a small two-layer network reworks each one, add and norm again, and a last product scores every word for the next. This lesson puts the pieces together on one camp-log sentence, names the shape at every line, counts the numbers a block learns, and computes the sentence's loss, first in NumPy and then in PyTorch.
What you'll be able to do
Follow a sentence through ids, embeddings and positions, attention, add and norm, a two-layer network, add and norm, and scores for the next word, naming every shape
Count a block's parameters as a sum of products, one product per matrix
Match each line of the block to the lesson that teaches its math
Compute the loss for one sentence in NumPy, then get the same loss from PyTorch's nn.Embedding, nn.Linear and nn.LayerNorm with the same weights
Quick check
Have a guess before you start. A sentence of 5 words goes into a transformer block, each word a vector of 4 numbers. What shape comes out of the block?
5 × 4, the same as went in
1 × 4, one vector for the sentence
5 × 5, one number per pair of words
4 × 5, turned on its side
Show the answer
5 × 4, the same as went in
A block gives every word a new vector of the same width, so 5 × 4 goes in and 5 × 4 comes out. That's what lets blocks stack: one block's output is the next block's input.
The math: ten lines, one block
Purpose. Write the whole block as ten lines and know the shape at every one.
The sentence is "the tent is on the trail", in the 12-word camp-log vocabulary of lesson 65. The block reads the first five words and, at each place, scores a guess for the word that comes next. One batch of one sentence, 5 words, width 4, vocabulary 12:
ids: each word's place in the vocabulary, 5 numbers (lesson 28).
X = E[ids] + P: each id's row of the 12 × 4 table E, plus each place's row of a 5 × 4 position table P: 5 × 4 (lesson 65).
Q, K, V = XW_Q, XW_K, XW_V: three 4 × 4 layers: 5 × 4 each (lesson 56).
S = QKᵀ/√4, with the causal mask: 5 × 5 (lesson 57).
A = softmax(S) row by row, 5 × 5, then AV: 5 × 4 (lesson 113).
H = relu(XW₁ + b₁): widen each word to 16: 5 × 16 (lesson 110).
X = LN(X + HW₂ + b₂): back to 4, add and norm: 5 × 4.
scores = XEᵀ: one score per vocabulary word at each place: 5 × 12 (lesson 65).
loss: softmax each row, then the mean of −log of each right next word's chance: one number (lesson 105).
Quick check
From lesson 65: a table E is 12 × 4 and a sentence has 5 ids. What shape is E[ids]?
5 × 4
12 × 4
5 × 12
4 × 5
Show the answer
5 × 4
Each id brings back its row of 4 numbers, so 5 ids give 5 rows: 5 × 4. That's line 2's X before the positions are added, and the positions are 5 × 4 too.
The picture. Here is the block as boxes, with the shape that leaves each one written on its arrow. Press Next to walk the ten lines: each lights the box it works in and writes its shape under it. Two shapes only live inside a box, the 5 × 5 attention table and the 5 × 16 hidden layer, and between boxes it's 5 × 4 all the way.
Try it now
Every arrow between boxes carries 5 × 4: that's why blocks can be stacked.
Counting the parameters. Every number the block learns sits in one of its tables, and a table with r rows and c columns holds r × c numbers. So the count is a sum of products:
parameters=tables∑rows×columns
In plain words: size up each table, multiply its sides, add them all.
In block order: E is 12 × 4 = 48 and P is 5 × 4 = 20; W_Q, W_K and W_V are 4 × 4 = 16 each; the first layer norm's γ and β are 2 × 4 = 8; W₁ is 4 × 16 = 64 and b₁ is 1 × 16 = 16; W₂ is 16 × 4 = 64 and b₂ is 1 × 4 = 4; the second norm has 8 more. The scores reuse E, so they add nothing. Step through the total: the bars are the eleven tables, x₁ to x₁₁ in that order.
Quick check
The block's width grows from 4 to 8, with everything else the same. How many numbers does W_Q hold now?
16
64
32
8
Show the answer
64
W_Q is width × width, so 8 × 8 = 64: doubling the width multiplies it by four. That's why wide models get big fast.
In your own words
In your own words: why does every arrow between the block's boxes carry the same shape, 5 × 4?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: the whole forward pass
Purpose. Run the block on the camp-log sentence in NumPy, from ids to loss, reusing each lesson's function.
E is the table from lesson 65; P and the W's come from a seeded random generator (lesson 88), as an untrained model's would, and the biases and the norms' γ and β start at 0 and 1, so the code leaves the norms' out. The block reads ids[:5] and is judged on ids[1:], the next word at every place. probs[np.arange(5), ids[1:]] picks, in each of the 5 rows, the chance of that row's right next word.
Reading the output. The ids are 7, 8, 2, 5, 7 and 9; X comes out 5 × 4 and the chances 5 × 12. The block gives the right next words chances of 0.0225, 0.0228, 0.1711, 0.1944 and 0.0349, and the loss is 2.8665. Guessing all 12 words evenly would give each a chance of 1/12 and a loss of ln 12 = 2.4849, so this block, with random weights, does worse than guessing. Training is what pulls 2.8665 down.
Code flow. Step through the attention half of the block, lines 3 to 6, on five words already embedded (random numbers, to keep the code short). The mask goes straight into the exponential, so hidden places come out as exactly 0, and these scores are small enough to skip subtracting the largest. Watch every matrix keep 5 rows.
Worked example
Worked example: positions go in before attention
A named mistake is adding the positions after attention instead of before. Compare the last word's attention output for "the tent is on the" and "the is tent on the", both ways.
Without positions, the last word gets exactly the same output for both orders: True. Attention blends a set of vectors, and swapping "tent" and "is" doesn't change the set the last word sees. With P added first, the vectors know their places, and the two orders differ: False. Adding P after attention would be too late: attention has already blended the words without knowing their order.
Quick check
What does this print?
import numpy as np
X = np.zeros((5, 4))
W1 = np.zeros((4, 16))
E = np.zeros((12, 4))
print((X @ W1).shape, (X @ E.T).shape, (X @ X.T).shape)
(5, 16) (5, 12) (5, 5)
(5, 16) (5, 4) (4, 4)
(4, 16) (12, 5) (5, 5)
(5, 4) (5, 12) (5, 5)
Show the answer
(5, 16) (5, 12) (5, 5)
The inner sizes meet and drop out: (5 × 4)(4 × 16) is 5 × 16, (5 × 4)(4 × 12) is 5 × 12, and (5 × 4)(4 × 5) is 5 × 5. These are lines 7, 9 and 4's shapes.
Drill
Count any block
Write block_params(V, n, d, h): the parameters of this lesson's block for a vocabulary of V words, n places, width d and a hidden width h. Count E, P, W_Q, W_K, W_V, both layer norms, W₁, b₁, W₂ and b₂. The starter leaves some out.
defblock_params(V, n, d, h):
embed = V * d
attention = d * d
network = d * h + h * d
return embed + attention + network
print(block_params(12, 5, 4, 16))
In the lesson, your code is checked as you write it.
In your own words
Why must the positions be added before attention, not after?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the same block in PyTorch, and a real one
Purpose. Build the block from PyTorch's parts, get the same loss, and see a trained model run the same steps.
In PyTorch. The same block, from PyTorch's parts. nn.Embedding(12, 4) holds E and nn.Embedding(5, 4) holds P (torch.arange(5) is the places 0 to 4). Three nn.Linear(4, 4, bias=False) layers are W_Q, W_K and W_V, nn.LayerNorm(4) is each norm, and nn.Sequential stacks the network's two layers with nn.ReLU() between. The NumPy weights are loaded with copy_, each W turned around with .T, since a layer stores (out, in) (lesson 55). F.cross_entropy(scores, ids[1:]) does line 10 in one call: softmax, pick the right words, mean of −log. Last, a loop adds up numel(), the entries in each parameter.
The same loss as NumPy, 2.8665, from the same weights, and 280 parameters, the count the flow added up. The scores are 5 × 12, and scores = X @ emb.weight.T reuses the embedding table at the end, as lesson 65 called tying.
Quick check
The NumPy block multiplies X @ W_Q. Why does the PyTorch cell load W_Q.T into to_q.weight?
W_Q.T is a copy of W_Q, and copy_ only takes copies
Turning the weight around makes the layer run faster
nn.Linear stores its weight as (out, in), turned around
PyTorch reads every matrix product from right to left
Show the answer
nn.Linear stores its weight as (out, in), turned around
nn.Linear computes x @ weight.T with weight stored as (out, in). Loading W_Q.T makes weight.T equal W_Q, so to_q(X) is X @ W_Q, the NumPy line exactly.
A real one. A trained model runs these same lines, with three changes you'll meet again. It has several attention heads side by side, each with its own W_Q, W_K and W_V. It puts each norm before its part instead of after, X + attention(LN(X)), which trains more steadily in deep stacks. And its network bends with GELU, a smooth relu. Below is the academy's small trained model, a few such blocks deep, running on this page. Type a start and watch what it predicts next, then look at where its words attend.
Quick check
What does this print?
import torch
import torch.nn.functional as F
scores = torch.zeros(5, 12)
print(round(F.cross_entropy(scores, torch.tensor([8, 2, 5, 7, 9])).item(), 4))
2.4849
12.0
0.0
0.0833
Show the answer
2.4849
Equal scores give every word a chance of 1/12, so each −log is ln 12 = 2.4849, and so is their mean: the loss of guessing evenly, the line the block above failed to beat.
Drill
Does the block beat guessing?
right holds the chances the block gave the five right next words, as the NumPy cell printed them. Set loss to the sentence's loss, guess to the loss of guessing all 12 words evenly, better to whether the block beats guessing, and perplexity to e to the loss: how many words, evenly, the block is as unsure as. The starter has three slips.
import numpy as np
right = np.array([0.0225, 0.0228, 0.1711, 0.1944, 0.0349])
loss = np.mean(np.log10(right))
guess = np.log(12)
better = loss > guess
perplexity = np.exp(guess)
print(round(loss, 3), round(guess, 4), better, round(perplexity, 1))
In the lesson, your code is checked as you write it.
In your own words
You now know every line of a transformer block. What does training change, and what stays the same?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write