Index Ai Academy
Phase 5 · Math under the GPT

The softmax Jacobian and p − y

Every output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.

About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers11

Why it matters

Softmax sits at two places in a GPT: the attention weights in every head and the output distribution over the vocabulary. Its derivative is unlike the elementwise functions: every output depends on every input, so its Jacobian is a full matrix with a simple formula, pi(δij−pj)p_i(\delta_{ij} - p_j). Put cross-entropy on top and it collapses to the cleanest gradient in deep learning, p−yp - y: the probabilities minus the one-hot truth.

This lesson derives both, checks them against the numbers and against autograd, and reads the result off the tiny GPT itself: on "Our Kestrel tent is made from", the gradient at the right token is −0.475 and at the leading wrong one +0.463, and the 195 entries sum to zero. You will also see what temperature does to the gradient, and why it is a sampling setting, not a training one.

What you'll be able to do

  • Derive the softmax Jacobian, ∂p_i/∂z_j = p_i(δ_ij − p_j), and check it numerically
  • Compute the softmax vector-Jacobian product without forming the matrix
  • Show that cross-entropy on top of softmax has gradient p − y, and read it off the tiny GPT's logits
  • Explain what temperature does to the gradient and why training keeps T = 1
Quick check
p = softmax(z). Raise one logit z_1 a little and leave the others alone. What happens to the other probabilities p_j?
  1. They stay the same, because each p_j depends only on its own z_j
  2. They all rise, because the denominator of softmax gets bigger
  3. Only the neighbors p_0 and p_2 change; the rest are unaffected
  4. They all fall, because the probabilities must still sum to 1
Show the answer

They all fall, because the probabilities must still sum to 1

The shared denominator grows, so every other probability shrinks. That coupling is why softmax's Jacobian is full rather than diagonal.

The idea: every output depends on every input

This rests on Softmax: scores into probabilities, the function itself, and on Jacobians: derivatives with shapes, the table of slopes.

A full Jacobian with a simple formula

Softmax turns logits zz into probabilities pi=ezi/∑kezkp_i = e^{z_i} / \sum_k e^{z_k}. The denominator is shared, so every pip_i depends on every zjz_j: raise one logit and its probability rises while all the others fall. The Jacobian is a full matrix, not a diagonal one, but its entries have one formula:

∂pi∂zj=pi(δij−pj),\frac{\partial p_i}{\partial z_j} = p_i(\delta_{ij} - p_j),

where δij\delta_{ij} is 1 when i=ji = j and 0 otherwise. On the diagonal that is pi(1−pi)p_i(1 - p_i), the slope of a probability with respect to its own logit; off the diagonal it is −pipj-p_i p_j, what pip_i loses when zjz_j gains. In matrix form, J=diag(p)−pp⊤J = \mathrm{diag}(p) - pp^{\top}.

The derivation is the quotient rule. For i=ji = j, the numerator's derivative is ezie^{z_i} and so is the denominator's, giving pi−pi2p_i - p_i^2. For i≠ji \ne j, the numerator does not move and the denominator moves by ezje^{z_j}, giving −pipj-p_i p_j. Every row sums to zero, because adding the same constant to every logit leaves the probabilities unchanged. The next cell builds JJ from the formula for z=[2,0,1]z = [2, 0, 1] and compares it with the numerical estimate.

Worked example

Formula against numerical estimate

diag(p) − ppᵀ for z = [2, 0, 1], the numerical Jacobian beside it, and the row sums.

import torch

z = torch.tensor([2.0, 0.0, 1.0])
p = torch.softmax(z, dim=0)
print("p:", p)
J = torch.eye(3) * p - p.view(3, 1) @ p.view(1, 3)       # diag(p) - p p^T
print("formula J:", J)

def jacobian(f, x, h=1e-3):
    y = f(x)
    Jn = torch.zeros(y.shape[0], x.shape[0])
    for j in range(x.shape[0]):
        e = torch.zeros_like(x); e[j] = h
        Jn[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return Jn

print("numerical:", jacobian(lambda v: torch.softmax(v, dim=0), z))
print("row sums:", J.sum(1))
It runs right here in your browser, with nothing to install.

p = [0.665, 0.090, 0.245]. The diagonal entry J[0][0] = 0.665 × (1 − 0.665) = 0.2227; the off-diagonal J[0][1] = −0.665 × 0.090 = −0.0599. The matrix is symmetric, because −p_i p_j does not care about order. The numerical estimate agrees to four decimals, and each row sums to zero within float32 rounding (the entries of order 1e-08 are zeros): push all three logits up together and nothing happens to p.

Quick check
A row of the softmax Jacobian, summed. What does this print?
import torch
p = torch.softmax(torch.tensor([2.0, 0.0, 1.0]), dim=0)
J = torch.eye(3) * p - p.view(3, 1) @ p.view(1, 3)
print(round(J.sum(1)[0].item(), 3))
  1. 0.223
  2. 0.0
  3. 0.665
  4. 1.0
Show the answer

0.0

Row i sums to p_i(1 − p_i) − p_i Σ_{j≠i} p_j = p_i − p_i Σ_j p_j = p_i − p_i = 0. Shifting every logit by the same amount changes nothing.

The vector-Jacobian product

Backprop does not need JJ, only J⊤gJ^{\top} g for the gradient g=∂L/∂pg = \partial L/\partial p arriving from above. Because JJ is symmetric, J⊤g=JgJ^{\top} g = Jg, and the formula collapses:

∂L∂z=p⊙(g−g⋅p),\frac{\partial L}{\partial z} = p \odot (g - g \cdot p),

the upstream gradient minus its pp-weighted average, times pp. Three vector operations on nn numbers and no n×nn \times n matrix. For the attention weights in a head, where nn is the sequence length, that is the difference between a backward pass that fits in memory and one that does not.

The next cell checks the shortcut three ways: autograd, the formula, and the explicit J⊤gJ^{\top} g from the matrix built above.

Worked example

Three ways to the same gradient

Autograd, the vector formula p ⊙ (g − g·p), and the explicit Jᵀg, for an upstream gradient g = [1, 0, −1].

import torch

z = torch.tensor([2.0, 0.0, 1.0], requires_grad=True)
p = torch.softmax(z, dim=0)
g = torch.tensor([1.0, 0.0, -1.0])            # the gradient arriving at p
(p * g).sum().backward()
print("autograd:     ", z.grad)
pd = p.detach()
print("p * (g - g.p):", pd * (g - (g * pd).sum()))
J = torch.eye(3) * pd - pd.view(3, 1) @ pd.view(1, 3)
print("J^T g:        ", J.T @ g)
In the lesson you can change this and run it.

g·p = 0.665 − 0.245 = 0.420, so the centered gradient is [0.580, −0.420, −1.420], and multiplying by p gives [0.3855, −0.0379, −0.3476]. Autograd and the explicit matrix product agree. Notice the middle entry: its upstream gradient was zero, yet its logit gets a negative gradient, because raising logit 1 would steal probability from the other two, and the loss cares about those. Full Jacobian, full coupling.

Cross-entropy collapses it to p − y

The loss on one token is L=−log⁡pyL = -\log p_y, where yy is the correct index. Its gradient with respect to pp is −1/py-1/p_y at position yy and zero elsewhere. Feed that gg into the softmax Jacobian:

∂L∂zj=∑igi pi(δij−pj)=−1py py(δyj−pj)=pj−δyj.\frac{\partial L}{\partial z_j} = \sum_i g_i\, p_i(\delta_{ij} - p_j) = -\frac{1}{p_y}\, p_y(\delta_{yj} - p_j) = p_j - \delta_{yj}.

So ∂L/∂z=p−y\partial L/\partial z = p - y, with yy the one-hot vector of the correct answer. The same result comes faster from log⁡py=zy−log⁡∑kezk\log p_y = z_y - \log \sum_k e^{z_k}: the derivative is δyj−pj\delta_{yj} - p_j, and the minus sign of the loss flips it.

Read it as instructions to the logits: raise the correct one by 1−py1 - p_y, the probability still missing, and lower every other one by exactly its own probability. The entries sum to zero, so the gradient never changes the overall level of the logits, only their differences. The next two cells check it on the three-logit example and then on the tiny GPT's real logits for "Our Kestrel tent is made from", where the answer is silnylon.

Worked example

Cross-entropy gives p − y

Loss −log p_2 on z = [2, 0, 1]. Autograd's gradient against p minus the one-hot target, and the sum of the entries.

import torch

z = torch.tensor([2.0, 0.0, 1.0], requires_grad=True)
y = 2                                         # the correct index
loss = -torch.log_softmax(z, dim=0)[y]
loss.backward()
onehot = torch.zeros(3); onehot[y] = 1.0
p = torch.softmax(z.detach(), dim=0)
print("loss:", round(loss.item(), 4))
print("autograd:", z.grad)
print("p - y:   ", p - onehot)
print("sum of the gradient:", z.grad.sum().item())
In the lesson you can change this and run it.

The correct logit, index 2, had probability 0.2447, so its gradient is 0.2447 − 1 = −0.7553: a descent step raises it. The other two get +0.6652 and +0.0900, their own probabilities: they fall, the confident wrong one fastest. The loss is −ln 0.2447 = 1.4076. The sum of the three entries is −1.2e-07, zero in float32.

Quick check
Two equal logits, the first one correct. What does this print?
import torch
p = torch.softmax(torch.tensor([0.0, 0.0]), dim=0)
print((p - torch.tensor([1.0, 0.0])).tolist())
  1. [0.5, -0.5]
  2. [0.0, 0.0]
  3. [-0.5, 0.5]
  4. [-1.0, 1.0]
Show the answer

[-0.5, 0.5]

p = [0.5, 0.5]; subtracting the one-hot [1, 0] gives [−0.5, 0.5]: raise the correct logit by the half that is missing, lower the other by its half.

Try it now

Cross-entropy for the three leading tokens (illustrative logits). The loss is −log of the correct bar; each logit's gradient is its bar's height, minus 1 for the correct one.

In your own words
Why is p − y a sensible training signal in plain words, and what would go wrong if the gradient at the wrong tokens were the same size for all of them?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the gradient on a trained model

Temperature, and why it belongs to sampling

Divide the logits by a temperature TT before the softmax and the chain rule adds one factor: ∂L/∂z=(pT−y)/T\partial L/\partial z = (p_T - y)/T, where pTp_T is the softmax at that temperature. A low temperature sharpens pTp_T and divides by a small number: at T=0.5T = 0.5 the gradient at the correct logit in the three-logit example is −1.7654, more than twice the −0.7553 at T=1T = 1. A high temperature flattens and shrinks it: −0.3464 at T=2T = 2.

That is why temperature is a sampling knob and not a training one. Training at T=1T = 1 gives the clean p−yp - y; changing TT during training would rescale every gradient, a disguised learning-rate change. At generation time there is no gradient, and temperature reshapes the distribution exactly as the sampling lesson described. The next cell prints the gradient at three temperatures with the formula beside it.

Worked example

Temperature scales the gradient

Loss −log softmax(z/T)[2] at T = 0.5, 1 and 2: autograd against (p_T − y)/T.

import torch

z = torch.tensor([2.0, 0.0, 1.0])
onehot = torch.tensor([0.0, 0.0, 1.0])
for T in [0.5, 1.0, 2.0]:
    zg = z.clone().requires_grad_(True)
    loss = -torch.log_softmax(zg / T, dim=0)[2]
    loss.backward()
    pT = torch.softmax(z / T, dim=0)
    print("T", T, " autograd", zg.grad, " (p_T - y) / T", (pT - onehot) / T)
In the lesson you can change this and run it.

At T = 0.5 the distribution sharpens to p_T = [0.867, 0.016, 0.117] and the division by 0.5 doubles everything: the correct logit's gradient is −1.7654. At T = 2 the distribution flattens and the division halves: −0.3464. Same logits, same answer, gradients that differ by a factor of five between the extremes. Hold T at 1 for training.

Quick check
From the cross-entropy lesson: with one correct answer, the cross-entropy loss equals:
  1. 1 minus the probability given to the correct answer
  2. The sum of the probabilities given to the wrong answers
  3. −log of the probability the model gave the correct answer
  4. The number of bits in the one-hot target vector
Show the answer

−log of the probability the model gave the correct answer

With a one-hot target only one term of −Σ y_i log p_i survives: −log p_y. Its gradient with respect to the logits is p − y.

Quick check
The entries of p − y always sum to zero. What does that mean for an update that follows the gradient?
  1. It raises every logit by the same amount so the probabilities stay normalized
  2. It changes the logits' differences, never their common level, which softmax ignores anyway
  3. It leaves the logits unchanged, because the pushes cancel out
  4. It changes only the correct logit, since the others sum to its opposite
Show the answer

It changes the logits' differences, never their common level, which softmax ignores anyway

A zero-sum gradient has no component along the all-ones direction, the direction softmax cannot see. The update spends itself entirely on differences between logits.

Drill

The softmax vector-Jacobian product

Write softmax_vjp(p, g): p is a 1-D tensor of probabilities (a softmax output) and g the gradient arriving at p. Return the gradient with respect to the logits, p * (g - (g * p).sum()), without building the Jacobian.

import torch

def softmax_vjp(p, g):
    return g

p = torch.softmax(torch.tensor([2.0, 0.0, 1.0]), dim=0)
print(softmax_vjp(p, torch.tensor([1.0, 0.0, -1.0])))
In the lesson, your code is checked as you write it.
Drill

p − y as a function

Write cross_entropy_backward(z, y): z is a 1-D tensor of logits and y the correct index. Return the gradient of −log softmax(z)[y] with respect to z, which is softmax(z) minus the one-hot vector of y.

import torch

def cross_entropy_backward(z, y):
    return z

print(cross_entropy_backward(torch.tensor([2.0, 0.0, 1.0]), 2))
In the lesson, your code is checked as you write it.
Worked example

The softmax Jacobian as a picture

diag(p) minus p p-transpose for six logits, as a heat map. The diagonal is positive, the rest negative, and each row adds to zero.

import torch
import matplotlib.pyplot as plt

z = torch.tensor([2.0, 0.0, 1.0, -1.0, 0.5, 1.5])
p = torch.softmax(z, dim=0)
J = torch.diag(p) - p.view(-1, 1) @ p.view(1, -1)
fig, ax = plt.subplots(figsize=(4.5, 3.8))
im = ax.imshow(J.tolist(), cmap="RdBu", vmin=-0.25, vmax=0.25)
ax.set_xlabel("input logit")
ax.set_ylabel("output probability")
ax.set_title("Softmax Jacobian for six logits")
fig.colorbar(im, ax=ax)
print("largest diagonal entry:", round(float(J.diag().max()), 3))
print("row sums:", [round(float(r.sum()), 3) for r in J])
In the lesson you can change this and run it.

The strongest diagonal entry is 0.244, for the token with the largest probability, 0.42: 0.42 x 0.58. Every row sums to 0 (the printed -0.0 is a rounding of a tiny negative), so the picture is balanced: red on the diagonal, blue off it.

Worked example

On the tiny GPT's logits

The last position's 195 logits for "Our Kestrel tent is made from", the loss against silnylon, and the gradient autograd leaves at the logits.

import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
ids = torch.tensor([tok.encode("Our Kestrel tent is made from")])
target = tok.token_id("silnylon")
logits = model(ids)[0, -1]                   # the last position's 195 logits
logits.retain_grad()                         # keep the gradient at this non-leaf tensor
loss = -torch.log_softmax(logits, dim=-1)[target]
loss.backward()
p = torch.softmax(logits.detach(), dim=-1)
print("loss", round(loss.item(), 3), " p(silnylon)", round(p[target].item(), 3))
top = torch.topk(p, 3)
for prob, i in zip(top.values, top.indices):
    print(f"{tok.vocab[int(i)]:10s} p {prob.item():.3f}  "
          f" gradient {logits.grad[int(i)].item():+.3f}")
onehot = torch.zeros(len(tok.vocab)); onehot[target] = 1.0
print("max |gradient - (p - y)|:", (logits.grad - (p - onehot)).abs().max().item())
print("sum over the vocabulary:", round(logits.grad.sum().item(), 6))
In the lesson you can change this and run it.

The model gives silnylon 0.525 and recycled 0.463: nearly a coin flip between two materials, with the right one ahead. The gradient at silnylon is 0.525 − 1 = −0.475, at recycled +0.463, at everything else its own small probability (waxed: 0.004). One training step on this example pushes the two apart. Autograd's 195 numbers match p − y to 6e-08, and they sum to zero. This vector, computed at every position of every training sequence, is the entire signal that trains a language model.

Quick check
After one descent step along this gradient, what happens to the three leading logits?
  1. silnylon rises, recycled falls by almost as much, waxed barely moves
  2. silnylon and recycled both rise, since both have high probability
  3. waxed rises most, because it has the most room to grow
  4. All three fall, because the loss is positive
Show the answer

silnylon rises, recycled falls by almost as much, waxed barely moves

Descent moves each logit against its gradient: −0.475 means silnylon rises; +0.463 means recycled falls; waxed's +0.004 is a nudge.

In your own words
A colleague proposes training with T = 0.5 so that the model 'learns to be more decisive'. Using the gradient formula, what would actually happen?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: why the loss takes logits

In a training loop the softmax and the loss are one call, and the gradient at the logits is p minus y at every position.

Where this shows up in an engineer's week

On the job: apply the softmax and then the log yourself and you will hit a NaN the first time a probability underflows to 0. Pass logits to the fused loss. And when a loss will not go down, look at p minus y on a batch: if the correct token's entry is already near 0, the model is confident and right, and the problem is elsewhere.

Goes deeper in

In your own words
Why does F.cross_entropy take logits and not probabilities?

Write a sentence or two, then compare it with a good answer in the lesson.

You can now

  • Derive the softmax Jacobian, p_i(δ_ij − p_j), and check it numerically.
  • Compute the softmax vector-Jacobian product, p ⊙ (g − g·p), without the matrix.
  • Show that cross-entropy on softmax has gradient p − y, and read it off the tiny GPT's logits.
  • Explain what temperature does to the gradient and why training keeps T = 1.

Next: backprop through attention, where every row of the attention matrix is a softmax.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 10 review cards for spaced repetition

Try a quiz question

The entry ∂p_i/∂z_j of the softmax Jacobian for i ≠ j is:

  1. −p_i p_j: raising z_j takes probability away from p_i
  2. p_i(1 − p_j): the slope of a probability with respect to a different logit
  3. 0: each probability depends only on its own logit
  4. p_i + p_j: the two probabilities combined

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

The math behind this

The Phase 2 lessons this one rests on, from the course map.

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write