Index Ai Academy
Phase 5 · How transformers think

The LM head and sampling

From the last row to a probability for every token, and the next word.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers11

Why it matters

After the last block, "The cat sat" is three rows of eight numbers, and the model has to turn that into one decision: which token comes next. Everything in this section has been leading here. One matrix turns the last row into a score for every entry in the vocabulary, a softmax turns the scores into probabilities, and a sampling rule picks one. Then the chosen token joins the prompt and the whole pipeline runs again.

This is also where the settings you meet in every API live: temperature, top-k and top-p are nothing more than small changes to the arithmetic in this lesson, and you'll compute each of them by hand.

What you'll be able to do

  • Project the last token's row to a logit for every vocabulary entry, and give the shapes involved
  • Turn logits into probabilities with softmax, and read a probability ratio off a logit gap
  • Apply a temperature to the logits and predict how the distribution sharpens or flattens
  • Pick the next token by argmax, sampling, top-k or top-p, and feed it back for the next step
The last row becomes a score for every token, softmax makes them probabilities and the chosen token joins the prompt; the boards show " on" at 0.813 and what temperature does to it. Transcript
Quick check
Two logits, 6.1 for " on" and 3.9 for " down", through a softmax. A guess: what does this print?
import numpy as np

z = np.array([6.1, 3.9])
e = np.exp(z - z.max())
print((e / e.sum()).round(2))
  1. [0.61 0.39]
  2. [0.5 0.5]
  3. [6.1 3.9]
  4. [0.9 0.1]
Show the answer

[0.9 0.1]

A gap of 2.2 in the logits is a factor of e^2.2 ≈ 9 in the probabilities, so the split is about 90/10. Softmax turns differences into ratios.

The idea: scores, chances and one choice

This rests on Softmax: scores into probabilities, which turns scores into chances, and on The feed-forward layer, which produced the last row.

One row becomes V scores

After the last block, the token that matters is the last one, " sat". Its row z∈Rdmodelz \in \mathbb{R}^{d_{\text{model}}} (normalized once more by a final layer norm) is multiplied by the unembedding or LM head matrix:

logits=z Wout+b∈RV,Wout∈Rdmodel×V\text{logits} = z\,W_{\text{out}} + b \in \mathbb{R}^{V}, \qquad W_{\text{out}} \in \mathbb{R}^{d_{\text{model}} \times V}

Eight numbers in, one raw score, a logit, for each of the V=1000V = 1000 vocabulary entries out. GPT-2's head is 768 × 50,257, and it reuses the embedding matrix transposed (weight tying), so those 38.6 million numbers do double duty: the same row that encodes " cat" on the way in scores " cat" on the way out.

Only the last row is used when generating, because the next token comes after the last one. During training every row is used at once, each predicting the token after it, which is why one pass over a text gives a training signal at every position.

The example takes " sat"'s output row from the feed-forward lesson as zz and uses a seeded random head, so its top entry is a random ID; the shapes are the point here. The rest of the lesson then uses logits with meaning.

Quick check
When a model generates text, only the last token's row goes through the LM head. Why that row?
  1. The LM head can only multiply one row at a time, so it takes the last
  2. The other rows are deleted after the last block to save memory
  3. The first row would be too small to hold a prediction over the vocabulary
  4. The last row has read the whole prompt and predicts the token after the end
Show the answer

The last row has read the whole prompt and predicts the token after the end

Each row predicts the token after its own position. At generation time the only unknown token is the one after the end, so the last row is the one that matters. During training every row is used, each predicting its successor.

Worked example

From the last row to V logits

The last token's row, a seeded 8 × 1000 head, and the resulting logits: one number per vocabulary entry.

import numpy as np
np.set_printoptions(precision=2, suppress=True)

# " sat"'s row after the last block
z = np.array([-1.23, -1.47, 0.79, -0.95, 0.67, 1.09, 1.84, -0.59])
V, d_model = 1000, 8
rng = np.random.default_rng(5)
W_out = rng.normal(0.0, 0.5, size=(d_model, V)).round(2)
b = np.zeros(V)

logits = z @ W_out + b
print(z.shape, "@", W_out.shape, "->", logits.shape)
top = np.argsort(-logits)[:3]
print("highest-scoring IDs:", top, "with logits", logits[top])
print("the logit of ID 511 (\" on\" in our vocabulary):", round(float(logits[511]), 2))
It runs right here in your browser, with nothing to install.

(8,) @ (8, 1000) -> (1000,): a thousand logits from eight numbers, one per token in the vocabulary. With random weights the top IDs mean nothing, and that's the honest picture of an untrained model: training is what moves the row for " on" to the top after "The cat sat". From here on we use logits a trained head might give.

Quick check
GPT-2 has d_model = 768 and V = 50,257. What are the shapes of its LM head and of the logits for one next-token prediction?
  1. 50,257 × 768 and (768,)
  2. 768 × 768 and (768,)
  3. T × 50,257 and (T,)
  4. 768 × 50,257 and (50,257,)
Show the answer

768 × 50,257 and (50,257,)

The head maps d_model numbers to one score per vocabulary entry, so it has V columns, and the logits are a vector of length V.

Softmax: scores to probabilities

Logits are raw numbers on no particular scale. Softmax turns the whole vector into a probability distribution over the vocabulary:

pi=ezi∑j=1Vezj,∑ipi=1p_i = \frac{e^{z_i}}{\sum_{j=1}^{V} e^{z_j}}, \qquad \sum_i p_i = 1

Every entry becomes positive, the largest logit gets the largest share, and only differences matter: adding the same constant to every logit changes nothing, which is why the code subtracts the maximum first. The ratio between two probabilities is the exponential of their logit gap:

pipj=e zi−zj\frac{p_i}{p_j} = e^{\,z_i - z_j}

Here are logits a trained head might give for eight candidates after "The cat sat". They're illustrative, chosen to show the shape of a real distribution: a clear favorite, a few plausible runners-up, and a long tail (the other 992 tokens, left out here, would share a fraction of a percent).

Token" on"" down"" there"" quietly"" by"" under"" in"" up"
Logit6.13.93.22.42.01.71.10.6

The gap between " on" and " down" is 2.2, so " on" is e2.2≈9e^{2.2} \approx 9 times as likely.

Worked example

Eight candidates after "The cat sat"

Softmax over the eight logits, the probability ratio from the top gap, and the check that the shares sum to 1.

import numpy as np

words = [" on", " down", " there", " quietly", " by", " under", " in", " up"]
logits = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])

e = np.exp(logits - logits.max())
probs = e / e.sum()
for w, p in zip(words, probs):
    print(f"{w:10s} {p:.3f}")
print("sum:", round(float(probs.sum()), 3))
print("p(on) / p(down):", round(float(probs[0] / probs[1]), 2), " e^2.2:",
      round(float(np.exp(2.2)), 2))
In the lesson you can change this and run it.

" on" takes 0.813, " down" 0.090, " there" 0.045, and the rest share the remaining 5%. The ratio 0.813 / 0.090 is 9.03, exactly e2.2e^{2.2}. The shares sum to 1 because the softmax divides by the total.

Quick check
The top two logits are 6.1 and 3.9. How many times as likely is the top token? What does this print?
import numpy as np

print(round(float(np.exp(6.1 - 3.9)), 2))
  1. 0.9
  2. 2.2
  3. 1.56
  4. 9.03
Show the answer

9.03

The ratio of two softmax probabilities is e to the gap between their logits: e^2.2 = 9.03.

Try it now

The eight candidates. Drag a logit and watch every share move, since they always sum to 1; then move the temperature.

Temperature

Temperature divides every logit by a number TT before the softmax:

pi(T)=ezi/T∑jezj/Tp_i(T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

Dividing by TT scales every gap by 1/T1/T. With T<1T < 1 the gaps grow and the favorite takes more; as T→0T \to 0 the distribution becomes a single spike on the largest logit, the same as always picking the top token. With T>1T > 1 the gaps shrink and the probability spreads toward the tail. For our eight candidates, " on" gets 1.000 at T=0.1T = 0.1, 0.984 at 0.5, 0.813 at 1, 0.610 at 1.5 and 0.474 at 2. The order never changes; only the margins do.

One way to put a number on the spread is entropy, in bits: 0.0 at T=0.1T = 0.1, 0.14 at 0.5, 1.09 at 1, 1.92 at 1.5, 2.37 at 2. Higher means the draw is less predictable.

The bar in the video reads 98%: that's this distribution at a temperature of 0.5. Temperature is set by the caller, not learned, which is why the same model can sound careful at 0.3 and inventive at 1.2.

Worked example

The same logits at five temperatures

The top candidate's share and the entropy of the whole distribution, as T rises.

import numpy as np

words = [" on", " down", " there", " quietly", " by", " under", " in", " up"]
logits = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])

def with_temperature(logits, T):
    e = np.exp(logits / T - (logits / T).max())
    return e / e.sum()

for T in (0.1, 0.5, 1.0, 1.5, 2.0):
    p = with_temperature(logits, T)
    nz = p[p > 0]
    entropy = float(-(nz * np.log2(nz)).sum())
    print(f"T = {T:3.1f}: on {p[0]:.3f}  down {p[1]:.3f}  "
          f"there {p[2]:.3f}  up {p[7]:.3f}  entropy {entropy:.2f} bits")
In the lesson you can change this and run it.

At T=0.1T = 0.1 everything piles onto " on". At T=2T = 2 " on" keeps less than half, " down" rises to 0.158 and even " up" reaches 0.030. The entropy climbs from 0 to 2.37 bits as the draw gets harder to predict.

Quick check
At T = 0.1 the top probability prints as 1.000. Why?
  1. The logits themselves are changed by training at low temperature, raising the top one
  2. The softmax stops dividing by the sum at low temperatures, so the top stays at 1
  3. Temperature 0.1 removes all but the top token from the vocabulary before the softmax
  4. Dividing by 0.1 turns the 2.2 gap into 22, a ratio of e^22, so the runner-up rounds to 0
Show the answer

Dividing by 0.1 turns the 2.2 gap into 22, a ratio of e^22, so the runner-up rounds to 0

Temperature only rescales the logits before the softmax. A tenfold gap in logits is a ratio of e^22 ≈ 3.6 billion in probability, far below three decimals.

Choosing the token, and the loop

A probability distribution isn't a token yet. Four rules are in common use:

  • Greedy (argmax): take the largest probability, " on". Deterministic, and prone to repeating itself in long outputs.
  • Sampling: draw a token at random with the given probabilities. " on" about 81 times in 100, " down" about 9.
  • Top-k: keep only the kk largest, renormalize, then sample. With k=3k = 3: " on" 0.857, " down" 0.095, " there" 0.047.
  • Top-p (nucleus): keep the smallest set of tokens whose probabilities add up to at least pp, then sample. With p=0.9p = 0.9: " on" + " down" = 0.903 ≥ 0.9, so those two survive, renormalized to 0.900 and 0.100.
kept = {" on": 0.813, " down": 0.090}            # the smallest set reaching p = 0.9
total = sum(kept.values())
print({tok: round(p / total, 3) for tok, p in kept.items()})

Whichever rule picks it, the chosen token's ID (511 for " on") is appended to the sequence. "The cat sat" becomes "The cat sat on", TT goes from 3 to 4, and the whole pipeline runs again from the tokenizer's output: new embeddings, a 4 × 4 attention matrix, four rows through every block, and the LM head on the new last row. That repeat is the autoregressive loop, one token per pass, until a stop token or a length limit. A real server avoids redoing the old rows' keys and values each time; that's the KV cache, in the inference lesson.

In your own words
What does temperature change about the distribution, and what does it leave alone?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the head on a trained model

The tiny GPT has the same last step as GPT-2: a final norm, a linear head, and a softmax over the last row. Here is the loop, written out, on a model whose answers mean something.

Drill

Logits to probabilities

Write softmax_probs(logits) that returns the probabilities for a NumPy array of logits. Subtract the maximum before exponentiating, so very large logits can't overflow.

import numpy as np

def softmax_probs(logits):
    return logits / logits.sum()

print(softmax_probs(np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])).round(3))
In the lesson, your code is checked as you write it.
Drill

Write the temperature softmax

Write with_temperature(logits, T) that divides the logits by T and returns the softmax probabilities.

import numpy as np

def with_temperature(logits, T):
    e = np.exp(logits - logits.max())
    return e / e.sum()

z = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])
print(with_temperature(z, 2.0).round(3))
In the lesson, your code is checked as you write it.
Worked example

The last row, the chances and a five-token loop

model(ids) gives logits for every position. Only the last row is used. The loop appends the likeliest token five times.

import torch
import torch.nn.functional as F
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("The Lark headlamp is")])
logits = model(ids)
last = logits[:, -1, :]
print(tuple(logits.shape), "->", tuple(last.shape))
p = F.softmax(last, dim=-1)[0]
top = torch.topk(p, 3)
pairs = zip(top.values, top.indices)
print([(tok.vocab[int(i)], round(float(v), 3)) for v, i in pairs])

out = ids
for step in range(5):
    nxt = model(out[:, -model.context:])[:, -1, :].argmax(dim=-1, keepdim=True)
    out = torch.cat([out, nxt], dim=1)
    print(step + 1, tok.vocab[int(nxt)], tuple(out.shape))
print(tok.decode(out[0].tolist()))
same = model.generate(ids, 5, temperature=0)
print("generate agrees:", torch.equal(same, out))
In the lesson you can change this and run it.

The prompt is four tokens, so the logits are (1, 4, 195): a score for each of the 195 tokens at each of the four positions, and the last row is (1, 195). The softmax gives " made" 0.986 and almost nothing to the rest: the model has seen this phrase in its catalog. Each loop pass appends one token and the shape grows from (1, 5) to (1, 9): made, from, a, light, polymer. generate with temperature=0 runs the same loop.

Worked example

Temperature on the model's own logits

The same last row, divided by three temperatures before the softmax. The top chance falls as the temperature rises.

import torch
import torch.nn.functional as F
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("The Lark headlamp is")])
last = model(ids)[:, -1, :]
for T in (0.5, 1.0, 2.0):
    q = F.softmax(last / T, dim=-1)[0]
    print(T, "top chance", round(float(q.max()), 3), tok.vocab[int(q.argmax())])
In the lesson you can change this and run it.

At temperature 0.5 the favorite " made" has a chance of 1.0 (rounded), at 1 it has 0.986 and at 2 it falls to 0.591. The ranking never changes, only the margins. For real outputs this is the knob: below 1 the model repeats its best guess, above 1 it wanders.

Quick check
From the sampling lesson: with p = 0.9 on these probabilities, 0.813, 0.090, 0.045, 0.020, …, which tokens does top-p keep?
  1. " on" alone, since 0.813 is by far the largest single share
  2. The top three, because 0.9 rounds up to three tokens in a list of eight
  3. " on" and " down": 0.813 + 0.090 = 0.903 reaches 0.9
  4. Every token with a probability above 0.04: " on", " down" and " there"
Show the answer

" on" and " down": 0.813 + 0.090 = 0.903 reaches 0.9

Top-p adds tokens from the largest down until the running total reaches p. 0.813 falls short of 0.9; adding 0.090 passes it, so two tokens survive.

Worked example

Greedy, top-k and top-p on the eight candidates

The four rules on the same probabilities, with a seeded draw so the sampled token is repeatable.

import numpy as np

words = [" on", " down", " there", " quietly", " by", " under", " in", " up"]
logits = np.array([6.1, 3.9, 3.2, 2.4, 2.0, 1.7, 1.1, 0.6])
e = np.exp(logits - logits.max()); probs = e / e.sum()

print("greedy:", words[int(np.argmax(probs))])

rng = np.random.default_rng(0)
print("sampled:", rng.choice(words, p=probs))

k = 3
keep = np.argsort(-probs)[:k]
top_k = np.zeros_like(probs); top_k[keep] = probs[keep]; top_k /= top_k.sum()
print("top-k = 3:", [(words[i], round(float(top_k[i]), 3)) for i in keep])

order = np.argsort(-probs)
cumulative = np.cumsum(probs[order])
nucleus = order[: int(np.searchsorted(cumulative, 0.9)) + 1]
top_p = np.zeros_like(probs); top_p[nucleus] = probs[nucleus]; top_p /= top_p.sum()
print("top-p = 0.9:", [(words[i], round(float(top_p[i]), 3)) for i in nucleus])
In the lesson you can change this and run it.

Greedy always says " on". The seeded draw picks one token by the probabilities; run it again with another seed and the pick can change. Top-k keeps three tokens and spreads the removed 5% over them. Top-p keeps the smallest set reaching 0.9: two tokens, because " on" alone is 0.813 and " on" plus " down" is 0.903.

In your own words
Why does only the last row go through the LM head when generating, but every row during training?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: logits and generate in practice

In Hugging Face the model returns logits for every position, and generate wraps this lesson's loop with the sampling settings you saw. The map shows the end of the pipeline.

Where this shows up in an engineer's week

On the job: most output bugs here are settings. A temperature left at 1 on an extraction job makes answers differ between runs. A temperature of 0 on a creative job makes every answer the same. Log the sampling settings with every response, and for models that fix them for you, record which ones the provider ignores.

Two more habits help. Check that a stop token is set, since a loop with no stop runs to the length limit. And read the top few chances, not only the chosen token: a flat spread is the model telling you it is unsure.

Goes deeper in

In your own words
A product returns a different answer each time for the same extraction prompt. Which setting do you check first, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What does the LM head do, and what shape is its output for one next-token prediction?

  1. It multiplies the last row by a d_model × V matrix: one logit per vocabulary entry, length V
  2. It multiplies the whole T × d_model matrix by a V × V matrix, giving a V × V table of scores
  3. It looks the last token up in the embedding table, giving a row of d_model numbers back
  4. It runs a softmax over the T rows, giving T probabilities, one per position

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 5 lessons

How Transformers Think: an interactive inference walkthrough

Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write