Index Ai Academy
Phase 5 · Math under the GPT

Jacobians: derivatives with shapes

The derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.

About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers11

Why it matters

Phase 2 taught the chain rule for numbers: multiply the slopes. A GPT is made of vectors and matrices, and when a function takes a vector in and gives a vector out, its derivative is a table of slopes, the Jacobian. The chain rule still holds, but "multiply" now means matrix multiplication and the shapes have to fit.

Every gradient formula in the other "Math under the GPT" lessons is a Jacobian argument with the shapes worked out: the linear layer's X⊤dYX^{\top} dY in Backprop through a linear layer, the softmax's p−yp - y in The softmax Jacobian and p − y, and Backprop through attention. This lesson gives you the tool: what a Jacobian is, how to measure one, why backprop multiplies by it without ever building it, and why that makes a derivative with fourteen million entries cost a few hundred thousand operations.

What you'll be able to do

  • Write the Jacobian of a vector function, with its shape, and estimate it numerically
  • Apply the chain rule to vector functions as a product of Jacobians in the right order
  • Explain why backprop computes vector-Jacobian products and never builds a Jacobian
  • Recognize diagonal Jacobians in elementwise functions and the full one in softmax
Quick check
A function takes 3 numbers in and gives 2 numbers out. How many partial derivatives does it have, and how are they arranged?
  1. Two, one per output: each output has one slope that applies to all three inputs
  2. Five, the inputs and outputs added together, one slope for each of them
  3. Three, one per input: each input has one slope that applies to both outputs
  4. Six, as a 2 × 3 matrix: one row per output, one column per input
Show the answer

Six, as a 2 × 3 matrix: one row per output, one column per input

Every output can move when any input moves: 2 outputs × 3 inputs = 6 slopes. Arranged by output row and input column, they are the Jacobian.

The idea: a table of slopes

This rests on Jacobians and Hessians, where the table of slopes was built by hand, and on The chain rule with vectors, the rule that multiplies them.

From one slope to a table of slopes

A function from a number to a number has one derivative. A function from a vector of nn numbers to a vector of mm numbers has m×nm \times n of them: how much each output moves when each input moves. Arranged with one row per output and one column per input, they form the Jacobian JJ, with Jij=∂yi/∂xjJ_{ij} = \partial y_i / \partial x_j.

Two cases you already know are Jacobians in disguise. A scalar function of a vector, like a loss, has a Jacobian with one row: the gradient from the gradients lesson, written across. A linear map y=Wxy = Wx has Jacobian WW itself: output ii is ∑jWijxj\sum_j W_{ij} x_j, so its slope with respect to xjx_j is WijW_{ij}, at every xx.

You can estimate any Jacobian numerically the way the gradients lesson checked a gradient: nudge one input up and down by a small hh, take the change in every output divided by 2h2h, and you have one column. Do it once per input. The next cell does exactly that for y=Wxy = Wx with a 2 × 3 matrix, so the answer is known before the code runs.

Worked example

A numerical Jacobian

One column per input, by central differences, for a linear map whose Jacobian we know.

Setup (runs before every cell) · 9 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import torch

def jacobian(f, x, h=1e-3):
    y = f(x)
    J = torch.zeros(y.shape[0], x.shape[0])
    for j in range(x.shape[0]):
        e = torch.zeros_like(x); e[j] = h
        J[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return J
import torch

W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
x = torch.tensor([1.0, 1.0, 1.0])

def jacobian(f, x, h=1e-3):
    y = f(x)
    J = torch.zeros(y.shape[0], x.shape[0])  # one row per output, one column per input
    for j in range(x.shape[0]):
        e = torch.zeros_like(x); e[j] = h
        # column j: every output's slope along input j
        J[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return J

print(jacobian(lambda v: W @ v, x))
print(W)
It runs right here in your browser, with nothing to install.

Two rows, three columns, one entry per (output, input) pair, and the estimate matches W to three decimals. The 0.9999 and 3.0001 in place of 1 and 3 are float32 rounding in the differences, not a property of the function. Read it by rows: output 0 depends on inputs 0 and 1 with slopes 1 and 2 and ignores input 2; output 1 ignores input 0. The Jacobian of a linear map is the matrix, at every point.

Quick check
The Jacobian of a map from 3 inputs to 2 outputs. What does this print?
import torch
W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
def jacobian(f, x, h=1e-3):
    y = f(x)
    J = torch.zeros(y.shape[0], x.shape[0])
    for j in range(x.shape[0]):
        e = torch.zeros_like(x); e[j] = h
        J[:, j] = (f(x + e) - f(x - e)) / (2 * h)
    return J
print(jacobian(lambda v: W @ v, torch.tensor([1.0, 1.0, 1.0])).shape)
  1. torch.Size([2, 3])
  2. torch.Size([3, 2])
  3. torch.Size([3, 3])
  4. torch.Size([2])
Show the answer

torch.Size([2, 3])

Rows follow the outputs (2), columns follow the inputs (3). The Jacobian has the shape of W, outputs × inputs.

Try it now

Each Step nudges one input and writes one column. The result is W itself, because the map is linear.

In your own words
A loss is a scalar function of a vector of parameters. In Jacobian terms, what shape is its derivative, and what did Phase 2 call it?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: Jacobians you can run

The chain rule becomes matrix multiplication

With numbers, the chain rule for y=f(g(x))y = f(g(x)) multiplies two slopes. With vectors, each slope is a Jacobian and the multiplication is a matrix product:

Jy/x=Jf(h) Jg(x),h=g(x).J_{y/x} = J_f(h)\, J_g(x), \qquad h = g(x).

The order is fixed: the outer function's Jacobian on the left. The shapes enforce it. If gg maps nn inputs to kk values and ff maps those kk to mm outputs, then JfJ_f is m×km \times k, JgJ_g is k×nk \times n, and their product is the m×nm \times n Jacobian of the whole function. Any other arrangement does not multiply.

Elementwise functions, the activations, have the simplest Jacobians of all. If yi=tanh⁡(hi)y_i = \tanh(h_i), output ii depends only on input ii, so the Jacobian is diagonal: 1−tanh⁡2(hi)1 - \tanh^2(h_i) on the diagonal and zero elsewhere. The next cell takes tanh⁡\tanh of the linear map from before, builds that diagonal matrix DD, and checks that DWDW equals the Jacobian measured numerically.

Worked example

The chain rule as a matrix product

tanh after the linear map: the diagonal Jacobian of tanh times W, against the numerical estimate.

Uses the setup above, 9 lines.

import torch

W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
x = torch.tensor([1.0, 1.0, 1.0])

h = W @ x                                      # the inner function's output: [3, 2]
D = torch.eye(2) * (1 - torch.tanh(h) ** 2)    # the diagonal Jacobian of tanh at h
print("numerical:", jacobian(lambda v: torch.tanh(W @ v), x))
print("D @ W:    ", D @ W)
print("D:", D)
In the lesson you can change this and run it.

The inner map sends x to h = [3, 2], where tanh is nearly flat: 1 − tanh²(3) = 0.0099 and 1 − tanh²(2) = 0.0707, the two diagonal entries of D. Multiplying D into W scales each row of W by its output's slope: row 0 becomes 0.0099 × [1, 2, 0] = [0.0099, 0.0197, 0], row 1 becomes 0.0707 × [0, −1, 3] = [0, −0.0707, 0.2120]. The numerical Jacobian agrees to four decimals. Chain rule, shapes and all: (2 × 2)(2 × 3) = 2 × 3.

Quick check
g maps 5 inputs to 3 values and f maps those 3 values to 2 outputs. The Jacobian of f(g(x)) is:
  1. 2 × 3 and 3 × 5 kept apart, since they cannot be multiplied
  2. 2 × 5, the product of a 2 × 3 and a 3 × 5 matrix
  3. 5 × 2, the product of a 5 × 3 and a 3 × 2 matrix
  4. 3 × 3, since the inner values are where the two meet
Show the answer

2 × 5, the product of a 2 × 3 and a 3 × 5 matrix

Outer Jacobian (2 × 3) on the left, inner (3 × 5) on the right; the shared 3 disappears and 2 × 5 remains: outputs × inputs of the whole function.

Drill

Write the numerical Jacobian

Write jacobian(f, x, h=1e-3): f maps a 1-D tensor to a 1-D tensor. Return a tensor of shape (outputs, inputs) whose column j is the central difference (f(x + h·e_j) − f(x − h·e_j)) / (2h).

Uses the setup above, 9 lines.

import torch

def jacobian(f, x, h=1e-3):
    return torch.zeros(0, 0)

W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
print(jacobian(lambda v: W @ v, torch.tensor([1.0, 1.0, 1.0])))
In the lesson, your code is checked as you write it.

Backprop never builds the Jacobian

Training needs the gradient of one number, the loss, with respect to everything. By the chain rule that is ∂L/∂x=J⊤ ∂L/∂y\partial L/\partial x = J^{\top}\, \partial L/\partial y: the Jacobian's transpose times the upstream gradient, a vector-Jacobian product. The result has the shape of xx, as a gradient always has the shape of the thing it is for.

Backprop computes that product without ever forming JJ. The autograd engine from the autograd lesson does it one node at a time: the line that adds the other input's value times the output's gradient into a factor's gradient is a vector-Jacobian product with a 1 × 1 Jacobian, the local slope, times the gradient arriving from above. Every backward function in that class is the same move.

The saving is not cosmetic. The tiny GPT's output layer turns 48 hidden vectors of 32 numbers into 48 rows of 195 logits. The Jacobian of logits with respect to hidden states has 9,360 × 1,536 entries, over fourteen million. The vector-Jacobian product needs one multiply-add per weight per position, 299,520 of them, 48 times fewer, and the ratio grows with the sequence length. The next two cells show the product agreeing with autograd on the tanh example, then the count.

Worked example

The vector-Jacobian product is what autograd computes

A loss whose gradient at y is a chosen vector g. Autograd's x.grad against Jᵀg from the numerical Jacobian.

Uses the setup above, 9 lines.

import torch

W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
x = torch.tensor([1.0, 1.0, 1.0])

g = torch.tensor([1.0, -2.0])  # the gradient arriving from above, one number per output
xg = torch.tensor([1.0, 1.0, 1.0], requires_grad=True)
y = torch.tanh(W @ xg)
(y * g).sum().backward()       # a loss whose gradient with respect to y is exactly g
print("autograd x.grad:", xg.grad)
print("J^T g:          ", jacobian(lambda v: torch.tanh(W @ v), x).T @ g)
In the lesson you can change this and run it.

Multiplying y by a fixed g and summing makes a loss whose gradient with respect to y is g, so autograd's x.grad is the vector-Jacobian product. It matches Jᵀg entry for entry: 0.0099 = 1 × 0.0099, 0.1610 = 1 × 0.0197 − 2 × (−0.0707), −0.4239 = −2 × 0.2120, with the numerical 0.1611 a rounding away. Autograd never computed the 2 × 3 matrix: it pushed the two upstream numbers back through tanh (multiply by the slopes) and then through W (multiply by Wᵀ).

Worked example

Jacobian sizes in the tiny GPT

The output layer: 48 positions, 32 hidden numbers each, 195 logits each. The full Jacobian against the product backprop computes.

Uses the setup above, 9 lines.

T, d, V = 48, 32, 195                 # tokens, hidden width, vocabulary of the tiny GPT
logits, hidden = T * V, T * d
print("logits:", logits, " hidden:", hidden)
print("full Jacobian entries:", logits * hidden)
print("vector-Jacobian multiply-adds:", logits * d)
print(f"ratio: {logits * hidden / (logits * d):.0f}")
In the lesson you can change this and run it.

Every logit depends on the hidden numbers at its own position only, so most of the 14,376,960 Jacobian entries would be zeros, and the useful ones are the 6,240 weights of the output layer. The product needs each weight once per position: 9,360 × 32 = 299,520 multiply-adds. The ratio is the sequence length, 48, because the full Jacobian pretends every position could affect every other. A production model's vocabulary and context are each hundreds of times larger, which makes the full matrix unthinkable and the product routine.

Quick check
Wᵀ times an upstream gradient of ones. What does this print?
import torch
W = torch.tensor([[1.0, 2.0, 0.0], [0.0, -1.0, 3.0]])
g = torch.tensor([1.0, 1.0])
print((W.T @ g).tolist())
  1. [1.0, 2.0, 0.0]
  2. [1.0, 1.0, 3.0]
  3. [3.0, 2.0]
  4. [0.0, -1.0, 3.0]
Show the answer

[1.0, 1.0, 3.0]

Wᵀ is 3 × 2; times a vector of two it gives three numbers, the column sums of W: 1 + 0, 2 − 1, 0 + 3. A gradient has the shape of its variable, here the three inputs.

Quick check
From the chain-rule lesson: for y = f(g(x)) with ordinary numbers, dy/dx equals:
  1. g′(f(x)) · f′(x): the inner slope at the outer value, times the outer slope
  2. f′(g(x)) · g′(x): the outer slope at the inner value, times the inner slope
  3. f′(x) · g′(x): the two slopes at the same input, multiplied
  4. f′(g(x)) + g′(x): the outer slope at the inner value, plus the inner slope
Show the answer

f′(g(x)) · g′(x): the outer slope at the inner value, times the inner slope

The outer slope is taken where the outer function actually runs, at g(x). With vectors, the same rule reads J_f(g(x)) · J_g(x).

Elementwise, diagonal, cheap

Put the two ideas together for the activations. An elementwise function has a diagonal Jacobian, so its vector-Jacobian product is an elementwise multiply: ∂L/∂h=f′(h)⊙∂L/∂y\partial L/\partial h = f'(h) \odot \partial L/\partial y. No matrix is built and nothing is summed; the backward pass through ReLU, GELU or tanh costs the same as the forward pass, one operation per number. The next cell measures ReLU's Jacobian at [1,−2,3][1, -2, 3]: a diagonal of 1, 0, 1, because ReLU passes positive inputs with slope 1 and blocks negative ones with slope 0.

Softmax is the exception, and the reason those other lessons exist. Every output of softmax depends on every input through the shared denominator, so its Jacobian is a full matrix, and so is the Jacobian of attention, which contains a softmax. Those two take a lesson each: The softmax Jacobian and p − y and Backprop through attention. The linear layer, whose Jacobian is the weight matrix, has its own lesson.

Worked example

An elementwise function's Jacobian is diagonal

ReLU at [1, −2, 3], measured numerically.

Uses the setup above, 9 lines.

import torch

print(jacobian(torch.relu, torch.tensor([1.0, -2.0, 3.0])))
In the lesson you can change this and run it.

Diagonal, as every elementwise function's Jacobian is: output i moves only with input i. The slopes are 1 for the positive inputs and 0 for −2, where ReLU is flat; the 0.9999 is the finite-difference estimate of 1, as before. Backprop through this layer multiplies the upstream gradient by [1, 0, 1], which is why a unit whose input is negative passes no gradient: its diagonal entry is 0.

Quick check
The upstream gradient at a tanh layer is [2, 4] and tanh′ there is [0.5, 0.1]. The gradient passed back to the layer's input is:
  1. [1.0, 0.4]: each entry times its own slope
  2. [0.5, 0.1]: the slopes themselves, since the gradient is 1
  3. [1.4, 1.4]: the dot product, repeated
  4. [2.4, 0.6]: the Jacobian summed and multiplied
Show the answer

[1.0, 0.4]: each entry times its own slope

A diagonal Jacobian makes the vector-Jacobian product an elementwise multiply: 2 × 0.5 and 4 × 0.1.

Drill

Multiply a chain of Jacobians

Write chain_jacobians(Js): Js lists the Jacobians of a chain of functions from the innermost to the outermost. Return the Jacobian of the whole chain (the outermost on the left). Raise ValueError if two neighbors cannot be multiplied.

Uses the setup above, 9 lines.

import torch

def chain_jacobians(Js):
    return Js[0]

J_g = torch.ones(3, 5)        # inner: 5 inputs to 3 values
J_f = torch.ones(2, 3)        # outer: 3 values to 2 outputs
print(chain_jacobians([J_g, J_f]).shape)
In the lesson, your code is checked as you write it.
Worked example

Two Jacobians as pictures: diagonal and full

The numerical Jacobian of tanh (elementwise) and of softmax (every output depends on every input), drawn as heat maps.

Uses the setup above, 9 lines.

import matplotlib.pyplot as plt

x = torch.tensor([0.5, -1.0, 2.0, 0.0])
J_tanh = jacobian(torch.tanh, x)
J_soft = jacobian(lambda v: torch.softmax(v, dim=0), x)
fig, axes = plt.subplots(1, 2, figsize=(6.5, 3))
for ax, J, name in zip(axes, (J_tanh, J_soft), ("tanh", "softmax")):
    im = ax.imshow(J.tolist(), cmap="RdBu", vmin=-1, vmax=1)
    ax.set_title(name)
    ax.set_xlabel("input")
    ax.set_ylabel("output")
off = J_tanh - torch.diag(torch.diag(J_tanh))
print("tanh off-diagonal is zero:", bool((off.abs() < 1e-3).all()))
print("softmax rows sum to about 0:", [round(float(r.sum()), 3) for r in J_soft])
In the lesson you can change this and run it.

The tanh picture is a diagonal: output i moves only with input i. The softmax picture is full, with positive entries on the diagonal and negative ones off it, because raising one logit takes probability from all the others. Each row of the softmax Jacobian sums to about 0, since the probabilities always add to 1.

In your own words
In Jacobian terms, why does a ReLU unit whose input is negative stop learning, and why does that not happen to tanh?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: why autograd works this way

An engineer meets Jacobians as shapes and memory, not as tables. The map shows where the gradient starts.

Where this shows up in an engineer's week

On the job: the Jacobian view explains three everyday facts. A gradient has the shape of the tensor it belongs to, since it is a vector-Jacobian product with one number per input. A model's backward pass costs about twice its forward pass, since each layer does two such products. And an out-of-memory error in training is nearly always stored activations, not Jacobians, which are never built.

Goes deeper in

In your own words
Why does backprop multiply a vector by a Jacobian instead of building the Jacobian?

Write a sentence or two, then compare it with a good answer in the lesson.

You can now

  • Write the Jacobian of a vector function, with its shape, and estimate it numerically.
  • Apply the chain rule to vector functions as a product of Jacobians, outer on the left.
  • Explain why backprop computes vector-Jacobian products and never builds a Jacobian, with the tiny GPT's numbers.
  • Recognize diagonal Jacobians in elementwise functions and the full one in softmax.

Next: the three formulas for a linear layer, derived by shape and by entry.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 10 review cards for spaced repetition

Try a quiz question

A function maps 4 inputs to 6 outputs. Its Jacobian has shape:

  1. 6 × 4: one row per output, one column per input
  2. 4 × 6: one row per input, one column per output
  3. 4 × 4: one entry per pair of inputs
  4. 6: one slope per output

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

The math behind this

The Phase 2 lessons this one rests on, from the course map.

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write