Index Ai Academy
Phase 5 · How transformers think

Inside the attention matrix

Every entry of the 3 × 3 attention matrix, computed and steered.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Up to now the three rows of XX have never met: each was built from its own token and its own position. Attention is the step where they do. Every token asks a question, every token offers an answer, and a small T×TT \times T table of weights decides who reads how much from whom. That table is the heart of the transformer, and in this lesson you compute every entry of it for "The cat sat".

The formula is one line, softmax(QK⊤/dk) V\text{softmax}(QK^\top/\sqrt{d_k})\,V. The lesson takes it apart into four matrix operations, prints each result for the running example, and then lets you push on a key and watch the weights move.

What you'll be able to do

  • Write the three projections Q = XW_Q, K = XW_K, V = XW_V with their shapes, and compute them for a 3 × 8 input
  • Compute the 3 × 3 attention matrix step by step: scores, the division by √d_k, softmax, and the blend of values
  • Explain why the scores are divided by √d_k in terms of the spread of dot products
  • Change one key and predict how the attention weights move
Queries, keys and values from the same rows, then the 3 × 3 matrix of weights: the boards show Q, K and V as 3 × 4 matrices and " sat"'s row of 0.29, 0.40 and 0.31. Transcript
Quick check
An attention score is a dot product between a query and a key. A guess: what does this print?
import numpy as np

q = np.array([-0.78, 0.10, -1.01, -0.10])    # " sat" asking
k = np.array([0.36, -0.81, 0.44, 0.14])      # " sat" offering
print(round(float(q @ k), 2))
  1. 0.82
  2. -0.82
  3. 4
  4. -0.33
Show the answer

-0.82

Multiply matching positions and add: −0.78 × 0.36 + 0.10 × (−0.81) + (−1.01) × 0.44 + (−0.10) × 0.14 = −0.82. That is " sat"'s score against its own key in this lesson's example.

The idea: three views of every token

This rests on Attention as matrix math, where you worked attention by hand on five words, and on Queries, keys and values. Here the same steps run on three real rows.

The input is the matrix X∈RT×dmodelX \in \mathbb{R}^{T \times d_{\text{model}}} from Embeddings and positional waves: 3 rows of 8 numbers for "The cat sat". Attention starts by making three different projections of it, each with its own learned weight matrix:

Q=XWQ,K=XWK,V=XWV,WQ,WK,WV∈Rdmodel×dkQ = XW_Q, \qquad K = XW_K, \qquad V = XW_V, \qquad W_Q, W_K, W_V \in \mathbb{R}^{d_{\text{model}} \times d_k}

Each product is (T×dmodel)(dmodel×dk)=T×dk(T \times d_{\text{model}})(d_{\text{model}} \times d_k) = T \times d_k. Our head uses dk=4d_k = 4, so each of QQ, KK and VV is 3 × 4. GPT-2 splits its 768 columns into 12 heads of dk=64d_k = 64; one head of GPT-2 looks like ours with longer rows.

The names describe roles, not contents. A token's query row is what it's looking for. Its key row is what it offers to be found by. Its value row is what it hands over if chosen. All three are made from the same row of XX by different weights, so training can shape them separately.

In the video: inside the first monolith, each vector stream splits into three color-coded beams, amber, magenta and cyan. Those are the token's rows of QQ, KK and VV.

The example uses the printed XX from the embedding lesson and seeded random weights, so the numbers you see here are the numbers the rest of the lesson uses.

Worked example

Project X into queries, keys and values

Three matrix multiplications, one per role. The weights are random stand-ins for trained ones.

Setup (runs before every cell) · 16 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

# "The" at position 0, " cat" at position 1, " sat" at position 2
X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
              [1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
              [0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
rng = np.random.default_rng(7)
W_Q = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
W_K = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
W_V = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
Q, K, V = X @ W_Q, X @ W_K, X @ W_V

def softmax_rows(S):
    e = np.exp(S - S.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)
# X, W_Q, W_K and W_V come from the setup above
print(X.shape, "@", W_Q.shape, "->", Q.shape)
print("Q\n", Q)
print("K\n", K)
In the lesson you can change this and run it.

(3, 8) @ (8, 4) -> (3, 4): eight numbers per token in, four out, three tokens throughout. The last row of QQ, -0.78, 0.10, -1.01, -0.10, is " sat" asking; the last row of KK, 0.36, -0.81, 0.44, 0.14, is " sat" offering. Their dot product, −0.82, was the opening question.

Quick check
X is 3 × 8 and W_Q is 8 × 4. What is Q's shape, and what would it be for a 50-token prompt with the same weights?
  1. 4 × 3, then 4 × 50
  2. 3 × 8, then 50 × 8
  3. 8 × 4 both times
  4. 3 × 4, then 50 × 4
Show the answer

3 × 4, then 50 × 4

The inner dimension, 8, is consumed by the multiplication. One row per token stays, with d_k = 4 numbers each: the weights don't change with the prompt's length.

In your own words
In one sentence each: what is a token's query, its key, and its value?

Write a sentence or two, then compare it with a good answer in the lesson.

Scores, scaling and shares

Now every query meets every key. The matrix QK⊤QK^\top is T×TT \times T: row tt, column ss is the dot product of token tt's query with token ss's key, a score for how much tt should read from ss. The scores are divided by dk\sqrt{d_k} (here 4=2\sqrt{4} = 2), and a softmax turns each row into shares that add up to 1:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
  • QK⊤∈RT×TQK^\top \in \mathbb{R}^{T \times T}: all the scores at once, 3×33 \times 3 here.
  • dk\sqrt{d_k}: the scaling, explained in the next part.
  • softmax\text{softmax} runs along each row, so each token's weights sum to 1.
  • The result times VV is T×dvT \times d_v: a blended value row per token (dv=dk=4d_v = d_k = 4 here).

For our example, " sat"'s row of scaled scores is -0.50, -0.16, -0.41: its query matches " cat"'s key best, and its own key least of the three. Softmax turns that into the weights 0.29, 0.40, 0.31.

In the video: the amber and magenta beams collide and explode into a shimmering 3 × 3 grid. That grid is this matrix, and its brighter cells are the bigger weights.

Worked example

The 3 × 3 attention matrix

Scores, scaled scores and the softmax of each row, for the Q and K computed above.

Uses the setup above, 16 lines.

scores = Q @ K.T                       # (3, 3): row t, column s = q_t . k_s
scaled = scores / np.sqrt(4)           # d_k = 4
e = np.exp(scaled - scaled.max(axis=1, keepdims=True))
A = e / e.sum(axis=1, keepdims=True)   # softmax along each row
print("scores\n", scores)
print("scaled\n", scaled)
print("weights\n", A)
print("row sums:", A.sum(axis=1))
In the lesson you can change this and run it.

Read the weights row by row. "The" (row 0) reads most from " sat" (0.49). " cat" (row 1) reads most from itself (0.47). " sat" (row 2) reads most from " cat" (0.40), then itself (0.31), then "The" (0.29). Every row sums to 1: attention is a weighted average, never more. With trained weights these patterns carry meaning; here they show the mechanics.

Quick check
Softmax of " sat"'s row of scaled scores, by hand. What does this print?
import numpy as np

z = np.array([-0.5, -0.16, -0.41])
e = np.exp(z - z.max())
print((e / e.sum()).round(2))
  1. [0.4 0.29 0.31]
  2. [0.33 0.33 0.33]
  3. [0.5 0.16 0.41]
  4. [0.29 0.4 0.31]
Show the answer

[0.29 0.4 0.31]

The largest score, −0.16 for " cat", gets the largest share, 0.40. The other two are close to each other, 0.29 and 0.31. Subtracting the max first changes nothing in the result; it only keeps the exponentials small.

Try it now

The real scaled scores from the example: pick a token to see its row of weights. The scores are this lesson's numbers, computed from seeded weights, not from a trained model.

In your own words
What does one entry of the 3 × 3 matrix mean, and why must each row add up to 1?

Write a sentence or two, then compare it with a good answer in the lesson.

Why divide by √d_k, in one line

Scaled dot-product attention showed the reason: a dot product adds dkd_k products, so its spread grows like dk\sqrt{d_k}, and dividing by dk\sqrt{d_k} keeps the softmax from saturating. Here the division is by 4=2\sqrt{4} = 2, and without it "The"'s weight on " sat" would be 0.64 instead of 0.49.

Quick check
From the scaled dot-product lesson: a head with d_k = 64 divides its scores by which number?
  1. 64, the key length itself
  2. 4,096, the square of 64
  3. 8, the square root of 64
  4. 2, the square root of 4
Show the answer

8, the square root of 64

√64 = 8. Our tiny head with d_k = 4 divides by 2; GPT-2's heads with d_k = 64 divide by 8.

Quick check
A model has d_k = 64 and someone removes the division by √d_k. What happens, and why?
  1. Nothing: softmax rescales any input to sum to 1, so the spread doesn't matter
  2. The attention matrix becomes 64 × 64 instead of T × T
  3. The weights stop summing to 1 once the scores grow large
  4. Scores spread about 8 times wider, the softmax saturates, and its gradients shrink
Show the answer

Scores spread about 8 times wider, the softmax saturates, and its gradients shrink

Softmax does keep the row summing to 1, but a wide spread of inputs pushes almost all of that sum onto the largest entry. There the function is nearly flat, so learning through it stalls.

In your own words
Why does a bigger d_k make the raw scores spread out more, and what does the division fix?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the same table from a trained model

The three rows above use seeded random weights, so the pattern means nothing. A trained head has a pattern worth reading. First the NumPy function, written once from scratch.

Drill

Scores to weights

Write attention(Q, K, V) in NumPy: scores Q @ K.T, divided by the square root of the key length, a softmax along each row (subtract the row's max first), and return (weights, weights @ V).

Uses the setup above, 16 lines.

import numpy as np

def attention(Q, K, V):
    T = Q.shape[0]
    weights = np.full((T, T), 1 / T)
    return weights, weights @ V

Q = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
K = np.array([[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]])
V = np.array([[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]])
w, out = attention(Q, K, V)
print(w.round(2))
print(out.round(2))
In the lesson, your code is checked as you write it.

Blend the values, and mask the future

The last multiplication, Z=AVZ = AV, is T×dvT \times d_v. Each row of ZZ is a weighted average of the value rows, with that token's weights:

zsat=0.29 vThe+0.40 vcat+0.31 vsatz_{\text{sat}} = 0.29\, v_{\text{The}} + 0.40\, v_{\text{cat}} + 0.31\, v_{\text{sat}}

So " sat" leaves this layer carrying mostly what " cat" offered. That is context linking, in numbers.

A decoder adds one rule: while predicting, a token may not read from tokens after it, because at generation time those don't exist yet. The causal mask sets every score above the diagonal to −∞-\infty before the softmax; e−∞=0e^{-\infty} = 0, so those weights are exactly 0 and the rest of the row renormalizes. "The" can only read from itself (row 1, 0, 0); " cat" from "The" and itself (0.28, 0.72, 0); " sat" from all three, unchanged.

Real models run many heads at once. GPT-2 runs 12 heads of dk=64d_k = 64 in each layer, joins their 12 outputs into 768 columns and multiplies by one more matrix WOW_O. Per layer that's four 768 × 768 matrices plus biases: 4×7682+4×768=2,362,3684 \times 768^2 + 4 \times 768 = 2{,}362{,}368 parameters.

In the video: the bridges of light arcing between token streams are these weights carrying value rows across, " cat"'s toward " sat" most of all.

Worked example

The output rows, with and without the mask

Z = AV for the example, then the same computation with the causal mask applied before the softmax.

Uses the setup above, 16 lines.

scaled = Q @ K.T / 2
A = softmax_rows(scaled)
print("Z = A @ V\n", A @ V)

future = np.triu(np.ones((3, 3), dtype=bool), k=1)    # True above the diagonal
A_causal = softmax_rows(np.where(future, -np.inf, scaled))
print("causal weights\n", A_causal)
In the lesson you can change this and run it.

ZZ is 3 × 4, like VV: one blended row per token. " sat"'s row, -0.83, -0.56, -0.06, -0.06, sits between the three value rows, closest to " cat"'s because that weight was largest. With the mask, "The" keeps weight 1 on itself and " cat" splits 0.28 / 0.72 between "The" and itself; " sat"'s row is the same as before, since nothing comes after it.

Quick check
After the causal mask, " cat"'s weights are 0.28 on "The", 0.72 on itself and 0 on " sat". Before the mask they were 0.18, 0.47 and 0.35. What explains the change?
  1. Nothing is renormalized; the row now sums to 0.65 instead of 1
  2. The mask changed " cat"'s query, so every score in the row moved
  3. The mask added the removed 0.35 straight onto " cat"'s own weight
  4. " sat"'s score became −∞, its weight 0, and the other two renormalized to sum to 1
Show the answer

" sat"'s score became −∞, its weight 0, and the other two renormalized to sum to 1

e^(−∞) is 0, so the masked token contributes nothing to the sum in the softmax's denominator. The two remaining exponentials are divided by a smaller total, so 0.18 and 0.47 grow to 0.28 and 0.72, keeping the ratio and the sum of 1.

Drill

Steer " sat"'s attention

Right now " sat" reads most from " cat" (0.40) and less from itself (0.31). Change only " cat"'s key, K[1], so that " sat" gives itself more weight than it gives " cat". One way: point " cat"'s key away from " sat"'s query, so their dot product falls. Leave K[0] and K[2] as they are.

Uses the setup above, 16 lines.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

X = np.array([[-0.56, 0.81, -0.38, 0.55, -0.16, 1.71, 0.92, 0.83],
              [1.07, 0.90, -0.70, 0.59, 0.63, 1.14, -0.96, 1.52],
              [0.33, -0.41, -0.36, 0.49, -0.26, 1.24, -0.16, 0.68]])
rng = np.random.default_rng(7)
W_Q = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
W_K = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
W_V = rng.normal(0.0, 0.5, size=(8, 4)).round(2)
Q, K, V = X @ W_Q, X @ W_K, X @ W_V

K[1] = K[1]            # <- change " cat"'s key here

def softmax_rows(S):
    e = np.exp(S - S.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

A = softmax_rows(Q @ K.T / 2)
print("\" sat\" reads:", A[2])
In the lesson, your code is checked as you write it.
Worked example

One head of the trained tiny GPT

Block 0 of the trained model reads "Describe the Lark headlamp .". The cell runs the layer's own steps (norm, one fused projection, split into heads, scaled scores, mask, softmax) and prints, for the last token, which token each of the four heads reads most from.

Uses the setup above, 16 lines.

import math
import torch
import matplotlib.pyplot as plt
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("Describe the Lark headlamp .")])
words = [tok.vocab[i] for i in ids[0]]
block = model.blocks[0]
x = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
B, T, d = x.shape
q, k, v = block.qkv(block.ln1(x)).split(d, dim=-1)
q, k, v = (t.view(B, T, block.h, d // block.h).transpose(1, 2) for t in (q, k, v))
scores = q @ k.transpose(-2, -1) / math.sqrt(d // block.h)
future = ~torch.tril(torch.ones(T, T)).bool()
A = torch.softmax(scores.masked_fill(future, float("-inf")), dim=-1)   # (1, 4, 5, 5)
print("weights:", tuple(A.shape), "| rows sum to 1:",
      bool(torch.allclose(A.sum(dim=-1), torch.ones(1, block.h, T))))
for h in range(block.h):
    w = A[0, h, -1]
    top = words[int(w.argmax())]
    print(f"head {h}: '.' reads most from {top!r} ({float(w.max()):.2f})")

fig, axes = plt.subplots(1, block.h, figsize=(8, 2.2))
for h, ax in enumerate(axes):
    ax.imshow(A[0, h].detach().numpy(), vmin=0, vmax=1)
    ax.set_title(f"head {h}", fontsize=8)
    ax.set_xticks(range(T), words, rotation=90, fontsize=6)
    ax.set_yticks(range(T), words, fontsize=6)
plt.tight_layout()
plt.show()
It runs right here in your browser, with nothing to install.

The four heads of one layer read the same five tokens differently. For the final ".", head 0 reads most from "the" (0.32), head 1 from "headlamp" (0.39), head 2 from the "." itself (0.26) and head 3 from "Describe" (0.41). Every row sums to 1 and the picture is lower-triangular: the zeros above the diagonal are the causal mask. This is the same softmax(QKᵀ/√d_k) as the three-row example, with 4 heads and trained weights.

Quick check
The cell's weights have the shape (1, 4, 5, 5). What do the four numbers mean, in order?
  1. 1 layer, 4 blocks, 5 tokens, 5 vocabulary entries
  2. Batch of 1, 4 tokens, 5 heads, 5 layers
  3. Batch of 1 text, 4 heads, 5 query tokens, 5 key tokens
  4. 1 head, 4 texts, 5 words, 5 letters
Show the answer

Batch of 1 text, 4 heads, 5 query tokens, 5 key tokens

There is one score table per head: T × T for T = 5 tokens. The batch holds 1 text, and the model has 4 heads.

In your own words
Read softmax(QKᵀ/√d_k)V for "The cat sat" term by term, with the shape of each piece.

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: what a library hands back

Real models run this inside one fused call. GPT-2 does it for 12 heads of 64 numbers in each of 12 layers, joins the head outputs into 768 columns and multiplies by one more matrix. You rarely write the loop. You do read its weights when you want to know what a head attends to.

On the job: attention weights are a debugging view, not an explanation. A head that reads the previous token or the first token is common and tells you the model is wiring positions, not meaning. Use the weights to check that a mask is right (zeros above the diagonal) and that rows sum to 1, then test any claim about what a head "does" by changing the input and watching the output.

In your own words
What do you check first when you read a model's attention weights?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

For a prompt of T tokens with d_model = 768 and one head of d_k = 64, what are the shapes of W_Q, Q and QKᵀ?

  1. W_Q: 768 × 64, Q: T × 64, QKᵀ: T × T
  2. W_Q: T × 64, Q: 768 × 64, QKᵀ: 64 × 64
  3. W_Q: 768 × 768, Q: T × 768, QKᵀ: T × 768
  4. W_Q: 64 × 768, Q: 64 × T, QKᵀ: T × T

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 5 lessons

How Transformers Think: an interactive inference walkthrough

Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write