Index Ai Academy
Foundation · Foundation mathPutting it together

Attention as matrix math

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Attention is the step that made today's language models possible, and it has a reputation for being hard. It isn't, once its pieces are named, because every one of them is a lesson you've done. Words are vectors (lesson 47), looked up in a table by their ids (lesson 65). How well two words match is a dot product (lessons 50 and 51). The scores are scaled down by √d (lesson 10's square roots) so softmax doesn't saturate. Words aren't allowed to look ahead, which is the triangular mask of lesson 57. The scores become weights with a row-by-row softmax (lesson 104). And each word's new vector is a weighted average of the others (lessons 48 and 92). Put together, that's one line of math, softmax(QKᵀ/√d)V, and eight lines of NumPy. This lesson builds it, then shows what it's for: letting "warm" find "fire".

  • Build attention from pieces you know: dot products for similarity, a 1/√d scale, a causal mask, softmax row by row, and a weighted average
  • Compute one row of attention weights and one output vector by hand for three words
  • Write attention in eight lines of NumPy for five words, with the shape at every step, and add the mask
  • Read what attention does for a language model: each word's vector becomes a blend of the words it attends to, and run it in PyTorch with W_Q, W_K and W_V as nn.Linear layers
Quick check
Have a guess before you start. Two word vectors point in nearly the same direction. What does that suggest about the words, and which operation measures it?

The math: score, scale, mask, softmax, blend

Purpose. Give each word a new vector that blends in the words it's most related to.

Each word has three vectors: a query q (what it's looking for), a key k (what it offers to be found by) and a value v (what it passes on). For one word's query, attention does five things:

  1. Score every word: the dot product q · kⱼ, large when the key matches the query.
  2. Scale each score by 1/√d, where d is the vectors' length, so long vectors don't give huge scores.
  3. Mask the words that come later: their scores become −∞, so they get no weight.
  4. Softmax the scores into weights that are positive and add up to 1.
  5. Blend the values: the output is Σ weightⱼ × vⱼ, a weighted average.

Three words, "the", "fire" and "was", with keys (1, 0), (0, 1) and (1, 1) and values (1, 0), (0, 1) and (0.5, 0.5). The word "fire" asks with the query (1, 2).