Index Ai Academy
Foundation · Foundation mathPutting it together

A transformer block, start to finish

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

This is where the whole Foundation arrives. The model behind a chat assistant is mostly one block, repeated: the same ten lines, again and again, and every one of them is a lesson you've done. Ids become vectors, the vectors take their places, attention blends them, add and norm steadies them, a small two-layer network reworks each one, add and norm again, and a last product scores every word for the next. This lesson puts the pieces together on one camp-log sentence, names the shape at every line, counts the numbers a block learns, and computes the sentence's loss, first in NumPy and then in PyTorch.

  • Follow a sentence through ids, embeddings and positions, attention, add and norm, a two-layer network, add and norm, and scores for the next word, naming every shape
  • Count a block's parameters as a sum of products, one product per matrix
  • Match each line of the block to the lesson that teaches its math
  • Compute the loss for one sentence in NumPy, then get the same loss from PyTorch's nn.Embedding, nn.Linear and nn.LayerNorm with the same weights
Quick check
Have a guess before you start. A sentence of 5 words goes into a transformer block, each word a vector of 4 numbers. What shape comes out of the block?

The math: ten lines, one block

Purpose. Write the whole block as ten lines and know the shape at every one.

The sentence is "the tent is on the trail", in the 12-word camp-log vocabulary of lesson 65. The block reads the first five words and, at each place, scores a guess for the word that comes next. One batch of one sentence, 5 words, width 4, vocabulary 12:

  1. ids: each word's place in the vocabulary, 5 numbers (lesson 28).
  2. X = E[ids] + P: each id's row of the 12 × 4 table E, plus each place's row of a 5 × 4 position table P: 5 × 4 (lesson 65).
  3. Q, K, V = XW_Q, XW_K, XW_V: three 4 × 4 layers: 5 × 4 each (lesson 56).
  4. S = QKᵀ/√4, with the causal mask: 5 × 5 (lesson 57).
  5. A = softmax(S) row by row, 5 × 5, then AV: 5 × 4 (lesson 113).
  6. X = LN(X + AV): add and norm, 5 × 4 (lesson 114).
  7. H = relu(XW₁ + b₁): widen each word to 16: 5 × 16 (lesson 110).
  8. X = LN(X + HW₂ + b₂): back to 4, add and norm: 5 × 4.
  9. scores = XEᵀ: one score per vocabulary word at each place: 5 × 12 (lesson 65).
  10. loss: softmax each row, then the mean of −log of each right next word's chance: one number (lesson 105).