The last lesson's model paid 1.75 bits of surprise per outcome on average, where the truth itself needs only 1.5. The extra quarter of a bit has a name: the KL divergence, how far the model's chances sit from the truth's. Cross-entropy is always the truth's own entropy plus this gap, and the data fixes the entropy, so when training lowers a model's cross-entropy, what really shrinks is the KL. It's a loss in its own right too. A small model learns to copy a large one by shrinking it, variational and diffusion models are trained on bounds built from it, and when a chat model is tuned on people's preferences, a KL term keeps it from drifting far from where it started.
What you'll be able to do
Compute KL(p ‖ q) by hand: each outcome's gap, log(p ÷ q), weighted by the truth's chance p and added
Show that cross-entropy is entropy plus KL, so lowering one lowers the other by the same amount
See that KL is 0 only when the two distributions match, and that KL(p ‖ q) and KL(q ‖ p) differ
Compute a soft-target loss, the KL a small model learns from a large one, in NumPy and with F.kl_div
Quick check
Have a guess before you start. A model's cross-entropy on some data is 1.75 bits, and the data's own entropy is 1.5 bits. Can more training bring the model's cross-entropy down to 1.2 bits?
No: 1.5 bits is the floor
Yes, but only with a bigger model
Yes, given enough training steps
Yes, if the data is shuffled first
Show the answer
No: 1.5 bits is the floor
The entropy is the smallest average surprise anyone can have about this data, even knowing the true chances exactly. Training can only close the 0.25 bits above it. That gap is the KL divergence, and this lesson takes it apart.
The math: the gap between two distributions
Purpose. Measure how far a model's chances q are from the truth's chances p, outcome by outcome, then in one number.
Take lesson 105's example again. The truth gives fire, tent and trail the chances p = (0.5, 0.25, 0.25), and a model guesses q = (0.25, 0.25, 0.5). For each outcome, set two surprises side by side: the model's, log₂(1/q), and the truth's own, log₂(1/p). Their difference is the outcome's gap: how much more the model is surprised than it needs to be. A log law turns the difference of two logs into one:
log2qi1−log2pi1=log2qipi
In plain words: an outcome's gap is the log of how many times more chance the truth gives it than the model does.
Quick check
From lesson 25: log₂ 8 − log₂ 2 is the log₂ of what?
6, which is 8 − 2
3, which is log₂ 8
16, which is 8 × 2
4, which is 8 ÷ 2
Show the answer
4, which is 8 ÷ 2
Subtracting logs divides inside them: log₂ 8 − log₂ 2 = log₂(8 ÷ 2) = log₂ 4 = 2. The same law turns the model's surprise minus the truth's into log₂(p ÷ q).
Fire's gap: the model gives it 0.25 where the truth gives 0.5, so log₂(0.5 ÷ 0.25) = log₂ 2 = 1 bit. Giving an outcome half the chance it deserves costs one extra bit each time it happens. Tent's gap is log₂ 1 = 0: the model has it right. Trail's is log₂(0.25 ÷ 0.5) = −1. The model gives trail double its chance, so trail surprises the model less than it surprises the truth. Weight each gap by how often its outcome really happens, p, and add: that total is the KL divergence.
The picture. Step through it. The cells write each outcome's two surprises and the gap between them. The picture starts with p and q side by side, then draws each gap as a bar: above the line where the model gave too little chance, below it where it gave too much. The last step shrinks each bar by its chance p and adds them up.
Try it now
Then set the model's chances to the truth's, 0.5, 0.25, 0.25, and press Use these numbers: every gap is 0, and so is the KL.
Symbols and formula. For a truth p and a model q over the same outcomes:
KL(p∥q)=i∑pilog2qipi=H(p,q)−H(p)≥0
In plain words: KL is each outcome's gap weighted by how often it really happens. It is also the cross-entropy minus the entropy, the extra surprise of using q when the truth is p, and it is never negative.
The flow's last step showed both forms: the gaps weighted by p give 0.25, and the same weights give the model's average surprise, 1.75, and the truth's, 1.5. So cross-entropy = entropy + KL. The data fixes the entropy; only the KL depends on the model. Lowering a model's cross-entropy and lowering its KL are the same job, step for step.
Zero only for a match. If q = p, every ratio is 1, every gap log₂ 1 = 0, and the KL is 0. Any other q gives a positive KL, though some of its gaps are negative, like trail's. Why don't they win?
Quick check
Trail's gap was −1 bit, yet KL is never negative. Why don't the negative gaps win?
The log of a chance is always a positive number
KL adds up the size of each gap, ignoring the sign
Negative gaps are dropped before the weighted sum
Negative gaps sit where p is below q, so p weighs them less
Show the answer
Negative gaps sit where p is below q, so p weighs them less
The model's chances add to 1, like the truth's, so any extra it gives one outcome it takes from another. A negative gap sits where q is above p, so its weight p is the smaller of the two chances; a positive gap sits where p is above q, so its weight is the bigger. Weighted by p, the positive gaps always come out ahead, or tie at 0 when q matches p.
Not symmetric. KL compares in one direction. The first distribution, p, supplies the weights, and it sits on top of every ratio. Swap the two and both change. The camp log shows rain on 1 night in 10, p = (0.9, 0.1) for dry and rain, and a lazy forecaster always says 50%, q = (0.5, 0.5). Then KL(p ‖ q) is 0.531 bits: what the 50% forecast costs on the camp's real nights. KL(q ‖ p) is 0.737: what the true chances would cost if nights really went 50/50. Different questions, different answers. That's why KL is called a divergence, not a distance.
Quick check
A colleague computes KL(q ‖ p) where the task asked for KL(p ‖ q). When does that give the same number?
Only in special cases, like q = p; usually the two differ
Never: they can't be equal
Only when both are measured in nats
Always: KL is a distance, the same both ways
Show the answer
Only in special cases, like q = p; usually the two differ
Swapping changes the weights and flips every ratio, so the two usually differ: 0.531 against 0.737 bits for the camp-log pair. They agree when q = p, both 0, and in a few lucky cases. The fire, tent and trail pair is one: swapped, it gives 0.25 × (−1) + 0.25 × 0 + 0.5 × 1 = 0.25, the same as before.
In your own words
In your own words: why is the KL divergence 0 when the model's chances match the truth's, and positive otherwise?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: KL in one line
Purpose. Compute KL with NumPy in one line, and check that it equals cross-entropy minus entropy.
The formula's Σ pᵢ × log₂(pᵢ ÷ qᵢ) is a dot product (lesson 50): p dotted with the log of the ratios, p / q taken entry by entry.
import numpy as np
p = np.array([0.5, 0.25, 0.25])
q = np.array([0.25, 0.25, 0.5])
kl = p @ np.log2(p / q)
H = p @ np.log2(1 / p)
CE = p @ np.log2(1 / q)
print(kl, CE - H, H + kl)
The one line gives 0.25 bits, the flow's answer. Cross-entropy minus entropy gives 0.25 too, and entropy plus KL gives back the cross-entropy, 1.75: the identity, in numbers.
Code flow. Here is that one line taken apart. Step through it and watch the boxes: the ratios, then the gaps, then each gap weighted by its chance, then the total. They are the flow's numbers.
Worked example
Worked example: the identity on three pairs
Check cross-entropy − entropy = KL on three pairs: the fire, tent and trail pair, and the camp log's rain forecast both ways round.
cross_entropy(p, p) is the entropy: a model that matches the truth. On every pair the two columns agree: 0.25, then 0.531, then 0.737. The last two pairs are the same forecast, swapped, so they also show the asymmetry: 0.531 bits one way and 0.737 the other. Neither number is wrong; they answer different questions. KL(p ‖ q) asks how much q costs when the truth is p.
The first is KL(q ‖ p): q gives the weights and sits on top of the ratios, 0.737. The second is KL(p ‖ q), 0.531. Same two distributions, two answers.
A named mistake: a zero in q where p isn't zero. Watch the KL as the model's chance for trail shrinks toward 0, while trail still really happens a quarter of the time:
import numpy as np
p = np.array([0.5, 0.25, 0.25])
for tiny in [0.1, 0.01, 0.0001, 0.00000001]:
q = np.array([0.75 - tiny, 0.25, tiny])
print(tiny, round(float(p @ np.log2(p / q)), 3))
The KL climbs from 0.141 to 0.878, 2.53 and 5.851 bits, and it has no ceiling. At exactly 0 there is no answer at all: dividing by 0 inside the log is an error.
import math
print(0.25 * math.log2(0.25 / 0.0))
A model that says "impossible" about something that happens is infinitely wrong. That's one reason losses are built from log-chances, which lesson 104's F.log_softmax keeps finite, and why code that takes the log of chances often clips them first, with np.clip(q, 1e-12, 1). A zero in p is different: an outcome that never happens adds nothing, so skip it, as the last lesson's entropy did with p[p > 0].
Drill
KL in both directions
Fix kl(p, q) so it returns KL(p ‖ q) in bits and skips outcomes where p is 0. Then set forward and backward to KL(p ‖ q) and KL(q ‖ p) for the camp-log pair, rounded to 3 places, and hot to the KL of the one-hot truth from the model below, rounded to 4. The starter has two slips.
In the lesson, your code is checked as you write it.
In your own words
For the camp-log forecast, KL(p ‖ q) is 0.531 bits and KL(q ‖ p) is 0.737. Judging the 50% forecaster against the camp's real weather, which would you report, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: learning from a teacher's chances
Purpose. Use KL as a loss: a small model learns to copy a large model's softened chances.
A teacher and a student. The trail club has a large sign reader that is accurate but slow, and wants a small one that runs on a phone. Distillation trains the small model, the student, to copy the large one's chances, not just its answers. Take one photo of an arrow sign. The teacher scores it (4.0, 1.5, 0.5) for arrow, tent and trail; the student, so far, (2.0, 1.0, 1.0). Both are softened with a temperature of 2 from lesson 104, and the KL is taken in nats, with the natural log, as libraries do:
import numpy as np
defsoftmax(z, t):
z = z / t
e = np.exp(z - z.max())
return e / e.sum()
teacher = np.array([4.0, 1.5, 0.5])
student = np.array([2.0, 1.0, 1.0])
print(softmax(teacher, 1.0).round(3).tolist())
p = softmax(teacher, 2.0)
q = softmax(student, 2.0)
print(p.round(3).tolist(), q.round(3).tolist())
print(round(float(p @ np.log(p / q)), 4))
At temperature 1 the teacher is 0.899 sure of arrow, and its other two chances are too small to say much. At 2 the same scores give 0.685, 0.196 and 0.119: an arrow, but one that looks far more like a tent than a trail sign. The student's softened chances, 0.452, 0.274 and 0.274, tie tent and trail. Its KL from the teacher is 0.1198 nats, and training pushes that toward 0: arrow up, and tent above trail.
Quick check
Why soften the teacher's chances with a temperature above 1 before the student copies them?
It lets the student stop learning the arrow class
It shows which wrong answers the teacher finds close
Softmax needs a temperature above 1 to add up to 1
It makes the teacher's top answer more accurate
Show the answer
It shows which wrong answers the teacher finds close
At temperature 1 the teacher's 0.899 for arrow drowns out the rest. Softened, tent's 0.196 against trail's 0.119 says what the teacher has learned: an arrow looks more like a tent than a trail sign. Copying that teaches the student more than the label alone.
In PyTorch.F.kl_div computes the KL, with two things to watch. Its first argument is the student's log-chances, log q, which F.log_softmax gives directly, and its second is the target chances, p: the reverse of the order KL(p ‖ q) is written in. reduction="sum" adds up the terms. The cell also checks the identity once more: F.cross_entropy takes target chances as well as right answers, and with the teacher's chances as targets, its cross-entropy minus the teacher's entropy is the KL.
F.kl_div gives 0.1198, the NumPy cell's KL. The soft-target cross-entropy is 0.952 nats, the teacher's entropy 0.8321, and their difference 0.1198 again. The student can't change the teacher's entropy, so training on either loss moves it the same way. Real distillation also multiplies the softened loss by the temperature squared, 4 here: softening shrinks the slopes by about that much, and the factor puts them back on the scale of the ordinary loss that is usually trained alongside it.
Quick check
What does this print?
import torch
import torch.nn.functional as F
p = torch.tensor([0.5, 0.25, 0.25])
q = torch.tensor([0.25, 0.25, 0.5])
print(round(F.kl_div(torch.log(q), p, reduction="sum").item(), 4), round(F.kl_div(q, p, reduction="sum").item(), 4))
0.1733 -1.3522
0.25 0.25
0.25 -1.3522
0.1733 0.1733
Show the answer
0.1733 -1.3522
The math part's KL was 0.25 bits, and in nats that's 0.25 × 0.6931 = 0.1733. Given q itself instead of log q, F.kl_div still runs, but it returns a number that isn't a KL at all: a negative one, which a real KL never is.
Drill
A soft-target loss
Fix soft_loss(teacher, student, t) so it softens both sets of scores with the temperature t and returns KL(teacher's chances ‖ student's) in nats with F.kl_div. Then set cool and warm to the loss at t = 1 and t = 2, rounded to 4 places. The starter has two slips.
In the lesson, your code is checked as you write it.
In your own words
A student model reaches the same answers as its teacher on every photo, but its KL from the teacher's softened chances is still well above 0. What hasn't it learned yet?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Distillation: training a small model to imitate a large one.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write