Attention is the step that made today's language models possible, and it has a reputation for being hard. It isn't, once its pieces are named, because every one of them is a lesson you've done. Words are vectors (lesson 47), looked up in a table by their ids (lesson 65). How well two words match is a dot product (lessons 50 and 51). The scores are scaled down by √d (lesson 10's square roots) so softmax doesn't saturate. Words aren't allowed to look ahead, which is the triangular mask of lesson 57. The scores become weights with a row-by-row softmax (lesson 104). And each word's new vector is a weighted average of the others (lessons 48 and 92). Put together, that's one line of math, softmax(QKᵀ/√d)V, and eight lines of NumPy. This lesson builds it, then shows what it's for: letting "warm" find "fire".
What you'll be able to do
Build attention from pieces you know: dot products for similarity, a 1/√d scale, a causal mask, softmax row by row, and a weighted average
Compute one row of attention weights and one output vector by hand for three words
Write attention in eight lines of NumPy for five words, with the shape at every step, and add the mask
Read what attention does for a language model: each word's vector becomes a blend of the words it attends to, and run it in PyTorch with W_Q, W_K and W_V as nn.Linear layers
Quick check
Have a guess before you start. Two word vectors point in nearly the same direction. What does that suggest about the words, and which operation measures it?
They're opposites; a sum measures it
They're the same word; a norm measures it
They're related; a dot product measures it
Nothing; vectors don't carry meaning
Show the answer
They're related; a dot product measures it
Lesson 51: vectors pointing the same way have a large dot product and cosine near 1. Attention uses exactly this to decide which words go together.
The math: score, scale, mask, softmax, blend
Purpose. Give each word a new vector that blends in the words it's most related to.
Each word has three vectors: a query q (what it's looking for), a key k (what it offers to be found by) and a value v (what it passes on). For one word's query, attention does five things:
Score every word: the dot product q · kⱼ, large when the key matches the query.
Scale each score by 1/√d, where d is the vectors' length, so long vectors don't give huge scores.
Mask the words that come later: their scores become −∞, so they get no weight.
Softmax the scores into weights that are positive and add up to 1.
Blend the values: the output is Σ weightⱼ × vⱼ, a weighted average.
Three words, "the", "fire" and "was", with keys (1, 0), (0, 1) and (1, 1) and values (1, 0), (0, 1) and (0.5, 0.5). The word "fire" asks with the query (1, 2).
Quick check
From lesson 104: what does softmax guarantee about a row of attention weights?
The largest weight is exactly 1
Every weight is the same
All positive, adding up to 1
The weights equal the scores
Show the answer
All positive, adding up to 1
So the output is a true weighted average of the value vectors: a blend, not a sum that grows with the number of words.
The picture. Step through "fire" attending to all three words, first with nothing hidden, the way a model reading a whole sentence at once would. The picture lays the whole calculation out as one table, one row per word: its key, its score, its weight and its value. The query sits on top, over the key columns, so each score is the query dotted with the key in its own row. When softmax runs, the weight column is shaded by each word's share, and the last row adds up the value columns, every value times its word's weight. Then hide a word with the mask: as the second word, "fire" may only look at "the" and itself.
Try it now
Then set the mask to 1, 1, 0 and press Use these numbers: "was" comes after "fire", so its score becomes −∞, its weight drops to 0, and the other two share all of it: 0.33 and 0.67.
Symbols and formula. Stack every word's query, key and value as the rows of Q, K and V (n × d each):
Attention(Q,K,V)=softmax(dQK⊤+M)V
In plain words: score every query against every key at once, scale, add the mask M (0 where allowed, −∞ where not), softmax each row, and blend the values with those weights.
QKᵀ is n × n: entry (i, j) is how much word i's query matches word j's key. After the softmax, row i holds word i's weights over all words, and multiplying by V (n × d) gives n × d: one new vector per word. For "fire" with nothing hidden: the scores are 1, 2 and 3, scaled by 1/√2 to 0.71, 1.41 and 2.12; softmax gives 0.140, 0.284 and 0.576; and the output is 0.140 × (1, 0) + 0.284 × (0, 1) + 0.576 × (0.5, 0.5) = (0.428, 0.572). With the mask, "was" drops out and the weights become 0.330 and 0.670.
Quick check
Queries and keys have length d = 16. By what is every score divided?
16
2
4
256
Show the answer
4
The scale is √d = √16 = 4. Without it, scores grow with d and softmax becomes nearly one-hot.
In your own words
In your own words: why is the output a weighted average of the values rather than a plain sum?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: eight lines for five words
Purpose. Compute attention for a whole sentence at once, with the causal mask.
import numpy as np
X = np.array([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0],
[0.0, 1.0, 0.0, 1.0], [1.0, 0.0, 0.0, 1.0]])
d = X.shape[1]
S = X @ X.T / np.sqrt(d)
mask = np.tril(np.ones((5, 5)))
S = np.where(mask == 1, S, -np.inf)
E = np.exp(S - S.max(axis=1, keepdims=True))
A = E / E.sum(axis=1, keepdims=True)
out = A @ X
print(S.shape, A.shape, out.shape)
print(A[3].round(3).tolist())
It runs right here in your browser, with nothing to install.
Five camp-log words, "the fire was warm tonight", each a 4-number vector, are the rows of X; here the queries, keys and values are all X itself. S is 5 × 5, one score per pair; np.where keeps a score where the mask has a 1 and puts −∞ where it has a 0; A's rows are the weights; out is 5 × 4, a new vector for every word. Row 3 is "warm": weights 0.157, 0.259, 0.157 and 0.427 on the first four words, and 0 on "tonight", which comes later.
Code flow. Step through the same lines one at a time and watch each matrix appear as a grid of boxes. Two lines are folded together here: the mask goes straight into the exponential, so the hidden places of E come out as exactly 0, e to −∞. And with scores this small, the flow skips subtracting each row's largest score first. Keep an eye on row 3 of A, "warm".
A named mistake: multiplying the scores by the mask.Lesson 57 pictured the mask as 1s and 0s laid over the scores. But a score of 0 isn't "no attention": e⁰ = 1, so a masked word would still get weight. For "fire" looking at "the fire was", multiplying by the mask leaves "was" a weight of 0.14, peeking at a word that hasn't been written yet. Masked scores must become −∞ before the softmax; then e^(−∞) = 0 and the weight is exactly 0.
Worked example
Worked example: −∞ against multiplying by the mask
"fire" attends to "the fire was" and must not see "was". Compare the right mask with the slip.
import numpy as np
defsoftmax(s):
e = np.exp(s - s.max())
return e / e.sum()
q = np.array([1.0, 2.0])
K = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
s = K @ q / np.sqrt(2)
m = np.array([1.0, 1.0, 0.0])
print(softmax(np.where(m == 1, s, -np.inf)).round(3).tolist())
print(softmax(s * m).round(3).tolist())
In the lesson you can change this and run it.
With −∞, the hidden word "was" gets exactly 0 and the visible words share everything: 0.330 and 0.670. With the slip, "was" still gets 0.140, and the visible words' weights are wrong too, since they now share only 0.86. A language model trained with the slip would learn to read the next word from the future, then fail the moment it has to write one.
Quick check
What does this print?
import numpy as np
S = np.array([[1.0, 5.0], [2.0, 2.0]])
mask = np.tril(np.ones((2, 2)))
S = np.where(mask == 1, S, -np.inf)
E = np.exp(S - S.max(axis=1, keepdims=True))
print((E / E.sum(axis=1, keepdims=True)).tolist())
[[1.0, 0.0], [1.0, 0.0]]
[[0.5, 0.5], [0.5, 0.5]]
[[1.0, 0.0], [0.5, 0.5]]
[[0.018, 0.982], [0.5, 0.5]]
Show the answer
[[1.0, 0.0], [0.5, 0.5]]
The first word may only see itself, so it takes all its weight despite the 5; the second word sees both equal scores and splits them.
Drill
Write attention
Write attention(Q, K, V, causal) that returns softmax(QKᵀ/√d)V, setting scores above the diagonal to −∞ when causal is True. Then set out_row to the masked output for word 1 ("fire") of the three-word example, rounded to 3.
import numpy as np
defattention(Q, K, V, causal):
S = Q @ K.T
E = np.exp(S)
return E @ V
Q = np.array([[2.0, 0.0], [1.0, 2.0], [1.0, 1.0]])
K = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
V = np.array([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]])
out_row = attention(Q, K, V, True)[1].round(3).tolist()
print(out_row)
In the lesson, your code is checked as you write it.
In your own words
The first word's output is always its own value vector, whatever the scores. Why?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: what attention buys a language model
Purpose. See a word's vector absorb the words it needs.
Before attention, "warm" in "the fire was warm" is the same vector in every sentence. After attention, its new vector is a blend weighted toward the words that match it, so it carries context: in this sentence, something about a fire. In the five-word example, "warm" gives itself 0.427, then "fire" 0.259, the earlier word it shares a feature with, and 0.157 each to "the" and "was". Its new vector moves toward "fire".
The weight table, the mask laid over it. Here is all of A at once, from the Python part's own scores, S = XXᵀ/√4: one row per word, each row its weights in percent, gray where the mask hides a later word. Row "warm" reads 16, 26, 16 and 43: the 0.157, 0.259, 0.157 and 0.427 the cell printed, as percentages.
Two refinements turn this into a transformer. First, the example used X itself as queries, keys and values, so each word matched itself best (a vector's dot product with itself is its squared length). Real models multiply X by three learned matrices first, Q = XW_Q, K = XW_K and V = XW_V, so a word can look for something other than itself, a verb looking for its subject, say. Second, they run several of these "heads" side by side, each free to look for something different, and join their outputs. Each is still this lesson's eight lines. Lesson 111's training loop sets all those W's.
Here is the first refinement on the camp log. Count the features from 0, as Python does: "warm" has features 1 and 3, and "fire" has 1 and 2. This W_Q keeps features 0, 1 and 2 as they are and moves feature 3 onto feature 2, so "warm" now asks for features 1 and 2: exactly what "fire" offers. W_K and W_V are np.eye(4), the identity of lesson 57, which changes nothing, so only the queries move.
import numpy as np
X = np.array([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0],
[0.0, 1.0, 0.0, 1.0], [1.0, 0.0, 0.0, 1.0]])
W_Q = np.array([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0]])
Q, K, V = X @ W_Q, X @ np.eye(4), X @ np.eye(4)
S = Q @ K.T / np.sqrt(4)
S = np.where(np.tril(np.ones((5, 5))) == 1, S, -np.inf)
E = np.exp(S - S.max(axis=1, keepdims=True))
A = E / E.sum(axis=1, keepdims=True)
print(Q[3].tolist(), A[3].round(3).tolist())
print((A @ V)[3].round(3).tolist())
"warm"'s query is now (0, 1, 1, 0), "fire"'s own vector, and its weights become 0.143, 0.387, 0.235, 0.235 and 0: "fire" gets the biggest share, more than "warm" itself. Its new vector, (0.143, 0.622, 0.622, 0.235), leans further toward "fire" than the 0.259 share could take it.
Quick check
Why do real models compute Q = XW_Q and K = XW_K instead of using X directly?
So a word can look for something other than itself
To avoid the softmax
Because X can't be multiplied by itself
To make the vectors shorter and cheaper to store
Show the answer
So a word can look for something other than itself
With raw X every word matches itself best. Learned projections let a query ask for, say, "my subject" and a key answer it, as "warm" just asked for "fire".
In PyTorch. The eight lines with tensors, then the projections as layers. torch.triu(…, diagonal=1) keeps the part of a matrix above its diagonal, so hide is True exactly where the mask has a 0, and masked_fill(hide, float("-inf")) writes −∞ there. F.softmax(S, dim=-1) runs across each row. Then three nn.Linear(4, 4, bias=False) layers hold W_Q, W_K and W_V. A layer stores its weights as (out, in) and multiplies by them turned around (lesson 55), so W_Q goes in as W_Q.T. Last, .double() turns the float32 results into float64, so they round and print as NumPy's do.
import torch
import torch.nn as nn
import torch.nn.functional as F
X = torch.tensor([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0],
[0.0, 1.0, 0.0, 1.0], [1.0, 0.0, 0.0, 1.0]])
hide = torch.triu(torch.ones(5, 5), diagonal=1) == 1
S = (X @ X.T / 4 ** 0.5).masked_fill(hide, float("-inf"))
A = F.softmax(S, dim=-1)
print(S.shape, A.shape, (A @ X).shape)
print(A[3].double().round(decimals=3).tolist())
W_Q = torch.tensor([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 1.0, 0.0]])
to_q, to_k, to_v = nn.Linear(4, 4, bias=False), nn.Linear(4, 4, bias=False), nn.Linear(4, 4, bias=False)
with torch.no_grad():
to_q.weight.copy_(W_Q.T)
to_k.weight.copy_(torch.eye(4))
to_v.weight.copy_(torch.eye(4))
Q, K, V = to_q(X), to_k(X), to_v(X)
A = F.softmax((Q @ K.T / 4 ** 0.5).masked_fill(hide, float("-inf")), dim=-1)
print(Q[3].tolist(), A[3].double().round(decimals=3).tolist())
It prints the shapes the Python part printed, 5 × 5, 5 × 5 and 5 × 4, and "warm"'s row, 0.157, 0.259, 0.157, 0.427 and 0. Then, through the three layers, the query (0, 1, 1, 0) and the weights 0.143, 0.387, 0.235, 0.235 and 0, exactly as the NumPy cell above. In a real model the three layers are wider, start random, and training sets them.
Quick check
What does this print? (Two words, queries and keys both X.)
import numpy as np
X = np.array([[1.0, 0.0], [1.0, 1.0]])
S = X @ X.T
print(S.tolist(), int(np.argmax(S[1])))
[[1.0, 1.0], [1.0, 2.0]] 1
[[2.0, 1.0], [1.0, 1.0]] 0
[[1.0, 1.0], [1.0, 2.0]] 0
[[1.0, 0.0], [1.0, 1.0]] 1
Show the answer
[[1.0, 1.0], [1.0, 2.0]] 1
Word 1's best match is itself: its dot product with itself is its squared length, 2. That's why real models learn separate W_Q and W_K.
Drill
Who does "warm" attend to?
Run masked self-attention on the five camp-log words (queries, keys and values all X). Set row to word 3 ("warm")'s weights rounded to 3, top_other to the position of the earlier word it attends to most, not counting itself, and moved to its output vector rounded to 3.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write