KL divergence at work
Forward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.
What it covers6
Why it matters
Cross-entropy trains a model to match the data. KL divergence measures how far one distribution is from another, and it shows up wherever a model is asked to stay close to something: a small model learning from a large one (distillation), a tuned model told not to drift from its base (the penalty in reinforcement learning from human feedback), a diffusion model matching noise. It is not symmetric, and the order of its two arguments, and of the two arguments of PyTorch's function that computes it, decides what the training does. Getting the order wrong runs without an error.
What you'll be able to do
- Compute KL(p‖q) by hand for small distributions, and say why KL(p‖q) and KL(q‖p) differ
- Predict which way a fit leans: forward KL covers every peak, reverse KL commits to one
- Use F.kl_div with log-probabilities first and probabilities second, and shape the input so batchmean divides by the batch
- Write a distillation loss with temperature T and its T² factor, and a KL penalty against a reference model
Show the answer
Not in general: KL divergence is not symmetric
KL(p‖q) and KL(q‖p) weight the gaps differently, so they usually give different numbers. Which one you minimize changes what the model learns.
The math: the same gap, weighed two ways
Purpose. Compute KL by hand, and see why swapping its arguments changes what it rewards.
Surprise, entropy and cross-entropy (Foundation lesson 105) defined cross-entropy and showed it equals the data's own entropy plus a gap. That gap is the KL divergence of the data's distribution p from the model's q:
In plain words: for each outcome, how many extra bits (or nats) you pay for using q's code when p is the truth, averaged with p's own weights. It is 0 only when q equals p, and never negative.
Take bits (log base 2). With p = (0.5, 0.5, 0) and q = (0.25, 0.25, 0.5), the zero in p drops out: KL(p‖q) = 0.5 log₂(2) + 0.5 log₂(2) = 1 bit. Swap them and q puts 0.5 where p has 0: the term 0.5 log₂(0.5 / 0) is infinite, so KL(q‖p) = ∞. The same two distributions, 1 bit one way and infinite the other.
Show the answer
Cross-entropy is entropy plus a positive KL gap
H(p, q) = H(p) + KL(p‖q). KL is zero only when q = p, so any other q costs extra.
Which fit is closer to p depends on which KL you ask. The next part measures both.
Write a sentence or two, then compare it with a good answer in the lesson.
In NumPy and PyTorch: both directions, and F.kl_div
Purpose. Compute both directions by code, then use PyTorch's functions and their argument order.
The sum is one line in NumPy. Here p is the two-peak truth from the picture, q₁ spreads its chance evenly, and q₂ commits to peak A.
Worked example: forward and reverse KL, in nats
Compute KL(p‖q) and KL(q‖p) for the two fits, and see which fit each direction prefers.
import numpy as np p = np.array([0.45, 0.05, 0.05, 0.45]) q_cover = np.array([0.25, 0.25, 0.25, 0.25]) q_peak = np.array([0.85, 0.05, 0.05, 0.05]) def kl(a, b): return float((a * np.log(a / b)).sum()) for name, q in [("cover", q_cover), ("peak", q_peak)]: print(name, round(kl(p, q), 4), round(kl(q, p), 4))
The even fit has KL(p‖q) = 0.3681 and KL(q‖p) = 0.5108. The peak-committed fit has KL(p‖q) = 0.7026 and KL(q‖p) = 0.4307. Forward KL (p first) prefers the even fit, 0.368 against 0.703: it can't forgive a model that ignores peak D. Reverse KL (q first) prefers the committed fit, 0.431 against 0.511: it punishes a model that puts mass where the truth has little, and is happy to ignore a peak. Training with forward KL, as cross-entropy does, gives models that cover; training with reverse KL gives models that pick a mode.
Show the answer
KL(p‖q): forward, cross-entropy up to a constant
Forward KL averages over the data's p, so an outcome with real mass and tiny q costs a lot. That is why cross-entropy training produces models that cover everything the data does.
Why this matters for training. Everything above is a choice of which mistake to fear more. Fitting a model with cross-entropy, as every language model is trained, is forward KL: the model is punished most for ignoring something real, so it learns to give at least a little chance to every word that does follow. Methods that minimize the reverse direction, used in some reinforcement learning and in variational methods, instead produce models that commit confidently to a few good answers and ignore the rest. Neither is wrong. They suit different jobs, and a good engineer can say which one a loss is using by looking at which distribution sits first.
import numpy as np p = np.array([0.5, 0.5]) print(float((p * np.log(p / p)).sum()))
Show the answer
0.0
p / p is 1 for every outcome and log 1 = 0, so a distribution has no divergence from itself.
PyTorch's F.kl_div takes its arguments the other way around from the formula. The first argument is the model's log-probabilities, log q; the second is the target probabilities p; and reduction="batchmean" divides the total by the first dimension of the input. So a single distribution must be shaped as a batch of one, (1, 4), or the sum is divided by 4 instead of 1. The cell below checks F.kl_div against the NumPy sum, and torch.distributions.kl_divergence for two normals against their closed form. The page runs a teaching subset of PyTorch; the same lines run on a real install.
import numpy as np import torch import torch.nn.functional as F from torch.distributions import Normal, kl_divergence p = np.array([0.45, 0.05, 0.05, 0.45]) q = np.full(4, 0.25) hand = float((p * np.log(p / q)).sum()) log_q = torch.log(torch.tensor(q)).unsqueeze(0) target = torch.tensor(p).unsqueeze(0) print(round(hand, 4), round(F.kl_div(log_q, target, reduction="batchmean").item(), 4)) print(round(F.kl_div(log_q[0], target[0], reduction="batchmean").item(), 4)) a = Normal(torch.tensor(0.0), torch.tensor(1.0)) b = Normal(torch.tensor(1.0), torch.tensor(2.0)) closed = np.log(2 / 1) + (1 + 1) / (2 * 4) - 0.5 print(round(kl_divergence(a, b).item(), 4), round(float(closed), 4))
The first line is 0.3681 twice: the hand sum and F.kl_div agree. The second shows the trap: the same call on a flat vector gives 0.092, a quarter, because batchmean divided by the vector's length, 4. The last line is the closed form for two normals, 0.4431, matched by the library.
Write the KL function
Write kl(p, q) for two probability arrays: the sum of p log(p / q), skipping outcomes where p is 0. The starter computes the other direction.
import numpy as np def kl(p, q): m = q > 0 return float((q[m] * np.log(q[m] / p[m])).sum()) p = np.array([0.7, 0.3]) q = np.array([0.5, 0.5]) print(round(kl(p, q), 4))
Write a sentence or two, then compare it with a good answer in the lesson.
In training code: distillation, a drift penalty, and swapped arguments
Purpose. See the two jobs KL does in modern training, and the bug that hides in its argument order.
Distillation. A small student learns from a large teacher's soft answers. Dividing both models' scores by a temperature T > 1 makes the chances less peaked, so the teacher's second and third choices show. That scaling shrinks the gradient by about T², so the loss is multiplied by T² to keep it the same size: loss = T² × KL(teacher ‖ student). With scores (2, 1, 0) for the teacher and (1, 1, 1) for the student, T = 1 gives a KL of 0.2662, and T = 2 gives 0.0784, which times T² = 4 is 0.3137.
import torch import torch.nn.functional as F zt = torch.tensor([[2.0, 1.0, 0.0]]) zs = torch.tensor([[1.0, 1.0, 1.0]]) for T in (1, 2): teacher = F.softmax(zt / T, dim=-1) log_student = F.log_softmax(zs / T, dim=-1) kl = F.kl_div(log_student, teacher, reduction="batchmean") print(T, round(kl.item(), 4), round((kl * T * T).item(), 4))
A drift penalty. In reinforcement learning from human feedback, a tuned model is rewarded for what people like, minus β times how far its token choices have moved from the base model's: reward − β × (log π − log π_ref), summed over tokens. If a response gets reward 1.0 and its tokens' log-chances are (−1.0, −2.0, −0.5) under the tuned model and (−1.2, −2.5, −0.4) under the base, the log ratios are 0.2, 0.5 and −0.1: a sum of 0.6, a penalty of 0.06 at β = 0.1, and a net reward of 0.94.
F.kl_div(F.softmax(zt, -1), F.log_softmax(zs, -1), reduction='batchmean'). What is wrong?Show the answer
The arguments are swapped: log-probabilities must come first
kl_div expects log-probabilities as input and probabilities as target. Swapped, it takes the log of a log-probability, which can be nan, and runs without a shape error.
import numpy as np p = np.array([0.7, 0.3]) q = np.array([0.5, 0.5]) forward = float((p * np.log(p / q)).sum()) reverse = float((q * np.log(q / p)).sum()) print(round(forward, 4), round(reverse, 4))
Show the answer
0.0823 0.0872
KL(p‖q) weighs the log ratio by p and gives 0.0823; KL(q‖p) weighs it by q and gives 0.0872. Close, but not equal: KL is not symmetric.
A distillation loss
Write distill(zt, zs, T) for logits of shape (batch, classes): T² times F.kl_div of the student's log-softmax at temperature T against the teacher's softmax at T, with batchmean. Use NumPy: compute the KL yourself, averaged over the batch.
import numpy as np def softmax(z): e = np.exp(z - z.max(axis=-1, keepdims=True)) return e / e.sum(axis=-1, keepdims=True) def distill(zt, zs, T): p = softmax(zt / T) q = softmax(zs / T) return float((q * np.log(q / p)).sum(axis=-1).mean()) zt = np.array([[2.0, 1.0, 0.0]]) zs = np.array([[1.0, 1.0, 1.0]]) print(round(distill(zt, zs, 2), 4))
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Distillation: training a student on a teacher's soft answers.
- Reinforcement learning from human feedback: the reward, and the KL penalty that keeps a tuned model near its base.
- Cross-entropy: the loss whose gap above entropy is this KL.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
Which statement is true of KL(p‖q)?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 2 · about 23 hours of study
Math for ML engineers
The Foundation's formulas at working depth: batches, shapes, autograd, stable arithmetic and error bars
30 lessons 1 free
- Reading a paper's equations, with shapes FreeΣ, subscripts, hats and bars, translated into Python you can run and test.35 min
- Floating point and log spaceGrowth, decay, and why logs turn products into sums.35 min
- Shapes at transformer scaleArrays with more axes: batches, sequences, heads.36 min
- The linear layer as algebraMany dot products at once: what a neural network layer computes.40 min
- Solving and least squaresThe matrix operations you'll see in every formula.36 min
- Projections, orthogonality and rank-1 updatesA matrix stretches, rotates and projects space.34 min
- Eigenvalues of symmetric matricesThe directions a transformation leaves pointing the same way.36 min
- SVD at workCompress data to its most important directions.40 min
- Strange geometry in high dimensionsNear-perpendicular random directions, bunched-up distances, and the curse of dimensionality.38 min
- Derivatives of the functions models usePower, sum and product rules, so you rarely compute from scratch.35 min
- The chain rule with vectorsDerivatives of functions inside functions: the heart of backprop.40 min
- Backprop through a linear layer: gradients with shapesThe slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.40 min
- Gradients in matrix form, and how to check themThe vector of all partial derivatives: which way is uphill.35 min
- Why training is fast or slowSecond derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method.40 min
- The chain rule of probabilityUpdate beliefs when new evidence arrives.40 min
- Distributions as objectsRandom variables and the shapes their outcomes take.35 min
- Expectation and variance of estimatorsExpected value, variance and standard deviation.35 min
- Gaussians at workThe bell curve, and why it shows up everywhere.35 min
- Covariance matrices and the multivariate normalCovariance and correlation, curves, outliers and common causes, rank correlation, skew and kurtosis.40 min
- Losses are negative log-likelihoodsChoose the parameters that make the data most probable.35 min
- Samples and confidence: error bars for an evalEstimate from a sample and say how sure you are.40 min
- Priors, MAP and regularizationPriors as pseudo-counts, Beta posteriors, shrinkage for small samples, and which of two rates is really higher.40 min
- Markov chainsStates, a table of what comes next, the chance of a path, and where a chain settles.38 min
- Softmax at working depthTurn any list of scores into probabilities.35 min
- Cross-entropy from logitsThe loss function language models are trained on.40 min
- KL divergence at workForward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.36 min
- A/B tests and significance, for prompts and modelsTell a real lift from noise.40 min
- Comparing two models on the same testPaired results, disagreements, a quick z and a bootstrap for the gap between two models.35 min
- Ranking from head-to-head resultsElo updates, Bradley-Terry strengths, and how model leaderboards turn votes into ratings.37 min
- Capstone: one problem through five tracksA three-class softmax classifier written from its formula: notation to NumPy, matrix products, cross-entropy, the gradient, and an honest report.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write