Index Ai Academy
Phase 2 · Information

KL divergence at work

Forward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.

About 36 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Cross-entropy trains a model to match the data. KL divergence measures how far one distribution is from another, and it shows up wherever a model is asked to stay close to something: a small model learning from a large one (distillation), a tuned model told not to drift from its base (the penalty in reinforcement learning from human feedback), a diffusion model matching noise. It is not symmetric, and the order of its two arguments, and of the two arguments of PyTorch's function that computes it, decides what the training does. Getting the order wrong runs without an error.

What you'll be able to do

  • Compute KL(p‖q) by hand for small distributions, and say why KL(p‖q) and KL(q‖p) differ
  • Predict which way a fit leans: forward KL covers every peak, reverse KL commits to one
  • Use F.kl_div with log-probabilities first and probabilities second, and shape the input so batchmean divides by the batch
  • Write a distillation loss with temperature T and its T² factor, and a KL penalty against a reference model
Quick check
Have a guess before you start. A model says the next word is 'cat' with chance 0.25 and 'dog' with 0.75. The truth is 0.5 and 0.5. Does 'how far the model is from the truth' equal 'how far the truth is from the model'?
  1. Always: distance is distance
  2. Not in general: KL divergence is not symmetric
  3. Only when the chances are all positive
  4. Only for two outcomes
Show the answer

Not in general: KL divergence is not symmetric

KL(p‖q) and KL(q‖p) weight the gaps differently, so they usually give different numbers. Which one you minimize changes what the model learns.

The math: the same gap, weighed two ways

Purpose. Compute KL by hand, and see why swapping its arguments changes what it rewards.

Surprise, entropy and cross-entropy (Foundation lesson 105) defined cross-entropy and showed it equals the data's own entropy plus a gap. That gap is the KL divergence of the data's distribution p from the model's q:

KL(p ∥ q)=∑ipilog⁡piqi=H(p,q)−H(p)\mathrm{KL}(p \,\|\, q) = \sum_i p_i \log\frac{p_i}{q_i} = H(p, q) - H(p)

In plain words: for each outcome, how many extra bits (or nats) you pay for using q's code when p is the truth, averaged with p's own weights. It is 0 only when q equals p, and never negative.

Take bits (log base 2). With p = (0.5, 0.5, 0) and q = (0.25, 0.25, 0.5), the zero in p drops out: KL(p‖q) = 0.5 log₂(2) + 0.5 log₂(2) = 1 bit. Swap them and q puts 0.5 where p has 0: the term 0.5 log₂(0.5 / 0) is infinite, so KL(q‖p) = ∞. The same two distributions, 1 bit one way and infinite the other.

Quick check
From the Foundation: the cross-entropy of p = (0.5, 0.5) under q = (0.25, 0.75) is larger than p's own entropy. Why?
  1. Entropy is always the largest of the three
  2. Cross-entropy is entropy plus a positive KL gap
  3. Cross-entropy ignores q and uses only p
  4. q has more outcomes than p does
Show the answer

Cross-entropy is entropy plus a positive KL gap

H(p, q) = H(p) + KL(p‖q). KL is zero only when q = p, so any other q costs extra.

Try it now

Which fit is closer to p depends on which KL you ask. The next part measures both.

In your own words
In your own words: why is KL(p‖q) not equal to KL(q‖p)?

Write a sentence or two, then compare it with a good answer in the lesson.

In NumPy and PyTorch: both directions, and F.kl_div

Purpose. Compute both directions by code, then use PyTorch's functions and their argument order.

The sum is one line in NumPy. Here p is the two-peak truth from the picture, q₁ spreads its chance evenly, and q₂ commits to peak A.

Worked example

Worked example: forward and reverse KL, in nats

Compute KL(p‖q) and KL(q‖p) for the two fits, and see which fit each direction prefers.

import numpy as np

p = np.array([0.45, 0.05, 0.05, 0.45])
q_cover = np.array([0.25, 0.25, 0.25, 0.25])
q_peak = np.array([0.85, 0.05, 0.05, 0.05])


def kl(a, b):
    return float((a * np.log(a / b)).sum())


for name, q in [("cover", q_cover), ("peak", q_peak)]:
    print(name, round(kl(p, q), 4), round(kl(q, p), 4))
It runs right here in your browser, with nothing to install.

The even fit has KL(p‖q) = 0.3681 and KL(q‖p) = 0.5108. The peak-committed fit has KL(p‖q) = 0.7026 and KL(q‖p) = 0.4307. Forward KL (p first) prefers the even fit, 0.368 against 0.703: it can't forgive a model that ignores peak D. Reverse KL (q first) prefers the committed fit, 0.431 against 0.511: it punishes a model that puts mass where the truth has little, and is happy to ignore a peak. Training with forward KL, as cross-entropy does, gives models that cover; training with reverse KL gives models that pick a mode.

Quick check
A model must never assign near-zero chance to anything that really happens. Which direction should its training loss use?
  1. KL(q‖p): reverse, the other direction
  2. Either one; the two directions are equal
  3. KL(p‖q): forward, cross-entropy up to a constant
  4. Neither one; use plain accuracy instead
Show the answer

KL(p‖q): forward, cross-entropy up to a constant

Forward KL averages over the data's p, so an outcome with real mass and tiny q costs a lot. That is why cross-entropy training produces models that cover everything the data does.

Why this matters for training. Everything above is a choice of which mistake to fear more. Fitting a model with cross-entropy, as every language model is trained, is forward KL: the model is punished most for ignoring something real, so it learns to give at least a little chance to every word that does follow. Methods that minimize the reverse direction, used in some reinforcement learning and in variational methods, instead produce models that commit confidently to a few good answers and ignore the rest. Neither is wrong. They suit different jobs, and a good engineer can say which one a loss is using by looking at which distribution sits first.

Quick check
What does this print?
import numpy as np

p = np.array([0.5, 0.5])
print(float((p * np.log(p / p)).sum()))
  1. 0.693
  2. 0.0
  3. -0.693
  4. 1.0
Show the answer

0.0

p / p is 1 for every outcome and log 1 = 0, so a distribution has no divergence from itself.

PyTorch's F.kl_div takes its arguments the other way around from the formula. The first argument is the model's log-probabilities, log q; the second is the target probabilities p; and reduction="batchmean" divides the total by the first dimension of the input. So a single distribution must be shaped as a batch of one, (1, 4), or the sum is divided by 4 instead of 1. The cell below checks F.kl_div against the NumPy sum, and torch.distributions.kl_divergence for two normals against their closed form. The page runs a teaching subset of PyTorch; the same lines run on a real install.

import numpy as np
import torch
import torch.nn.functional as F
from torch.distributions import Normal, kl_divergence

p = np.array([0.45, 0.05, 0.05, 0.45])
q = np.full(4, 0.25)
hand = float((p * np.log(p / q)).sum())
log_q = torch.log(torch.tensor(q)).unsqueeze(0)
target = torch.tensor(p).unsqueeze(0)
print(round(hand, 4), round(F.kl_div(log_q, target, reduction="batchmean").item(), 4))
print(round(F.kl_div(log_q[0], target[0], reduction="batchmean").item(), 4))

a = Normal(torch.tensor(0.0), torch.tensor(1.0))
b = Normal(torch.tensor(1.0), torch.tensor(2.0))
closed = np.log(2 / 1) + (1 + 1) / (2 * 4) - 0.5
print(round(kl_divergence(a, b).item(), 4), round(float(closed), 4))

The first line is 0.3681 twice: the hand sum and F.kl_div agree. The second shows the trap: the same call on a flat vector gives 0.092, a quarter, because batchmean divided by the vector's length, 4. The last line is the closed form for two normals, 0.4431, matched by the library.

Drill

Write the KL function

Write kl(p, q) for two probability arrays: the sum of p log(p / q), skipping outcomes where p is 0. The starter computes the other direction.

import numpy as np


def kl(p, q):
    m = q > 0
    return float((q[m] * np.log(q[m] / p[m])).sum())


p = np.array([0.7, 0.3])
q = np.array([0.5, 0.5])
print(round(kl(p, q), 4))
In the lesson, your code is checked as you write it.
In your own words
In your own words: in F.kl_div(a, b), which argument is the model's and what form must it be in?

Write a sentence or two, then compare it with a good answer in the lesson.

In training code: distillation, a drift penalty, and swapped arguments

Purpose. See the two jobs KL does in modern training, and the bug that hides in its argument order.

Distillation. A small student learns from a large teacher's soft answers. Dividing both models' scores by a temperature T > 1 makes the chances less peaked, so the teacher's second and third choices show. That scaling shrinks the gradient by about T², so the loss is multiplied by T² to keep it the same size: loss = T² × KL(teacher ‖ student). With scores (2, 1, 0) for the teacher and (1, 1, 1) for the student, T = 1 gives a KL of 0.2662, and T = 2 gives 0.0784, which times T² = 4 is 0.3137.

import torch
import torch.nn.functional as F

zt = torch.tensor([[2.0, 1.0, 0.0]])
zs = torch.tensor([[1.0, 1.0, 1.0]])
for T in (1, 2):
    teacher = F.softmax(zt / T, dim=-1)
    log_student = F.log_softmax(zs / T, dim=-1)
    kl = F.kl_div(log_student, teacher, reduction="batchmean")
    print(T, round(kl.item(), 4), round((kl * T * T).item(), 4))

A drift penalty. In reinforcement learning from human feedback, a tuned model is rewarded for what people like, minus β times how far its token choices have moved from the base model's: reward − β × (log π − log π_ref), summed over tokens. If a response gets reward 1.0 and its tokens' log-chances are (−1.0, −2.0, −0.5) under the tuned model and (−1.2, −2.5, −0.4) under the base, the log ratios are 0.2, 0.5 and −0.1: a sum of 0.6, a penalty of 0.06 at β = 0.1, and a net reward of 0.94.

Quick check
A distillation loop calls F.kl_div(F.softmax(zt, -1), F.log_softmax(zs, -1), reduction='batchmean'). What is wrong?
  1. softmax should be log_softmax in both places
  2. The arguments are swapped: log-probabilities must come first
  3. Nothing is wrong: the argument order doesn't matter
  4. batchmean should be sum
Show the answer

The arguments are swapped: log-probabilities must come first

kl_div expects log-probabilities as input and probabilities as target. Swapped, it takes the log of a log-probability, which can be nan, and runs without a shape error.

Quick check
What does this print? The same two distributions, in both directions.
import numpy as np

p = np.array([0.7, 0.3])
q = np.array([0.5, 0.5])
forward = float((p * np.log(p / q)).sum())
reverse = float((q * np.log(q / p)).sum())
print(round(forward, 4), round(reverse, 4))
  1. 0.0872 0.0823
  2. 0.2 0.2
  3. 0.0823 0.0872
  4. 0.0823 0.0823
Show the answer

0.0823 0.0872

KL(p‖q) weighs the log ratio by p and gives 0.0823; KL(q‖p) weighs it by q and gives 0.0872. Close, but not equal: KL is not symmetric.

Drill

A distillation loss

Write distill(zt, zs, T) for logits of shape (batch, classes): T² times F.kl_div of the student's log-softmax at temperature T against the teacher's softmax at T, with batchmean. Use NumPy: compute the KL yourself, averaged over the batch.

import numpy as np


def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)


def distill(zt, zs, T):
    p = softmax(zt / T)
    q = softmax(zs / T)
    return float((q * np.log(q / p)).sum(axis=-1).mean())


zt = np.array([[2.0, 1.0, 0.0]])
zs = np.array([[1.0, 1.0, 1.0]])
print(round(distill(zt, zs, 2), 4))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what is the KL term doing in a drift penalty, and what does the weight β control?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Which statement is true of KL(p‖q)?

  1. It is 0 only when q equals p, and never negative
  2. It always equals KL(q‖p), whatever p and q are
  3. It can be negative
  4. It measures the angle between p and q

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 2 · about 23 hours of study

Math for ML engineers

The Foundation's formulas at working depth: batches, shapes, autograd, stable arithmetic and error bars 30 lessons 1 free
  1. Reading a paper's equations, with shapes FreeΣ, subscripts, hats and bars, translated into Python you can run and test.35 min
  2. Floating point and log spaceGrowth, decay, and why logs turn products into sums.35 min
  3. Shapes at transformer scaleArrays with more axes: batches, sequences, heads.36 min
  4. The linear layer as algebraMany dot products at once: what a neural network layer computes.40 min
  5. Solving and least squaresThe matrix operations you'll see in every formula.36 min
  6. Projections, orthogonality and rank-1 updatesA matrix stretches, rotates and projects space.34 min
  7. Eigenvalues of symmetric matricesThe directions a transformation leaves pointing the same way.36 min
  8. SVD at workCompress data to its most important directions.40 min
  9. Strange geometry in high dimensionsNear-perpendicular random directions, bunched-up distances, and the curse of dimensionality.38 min
  10. Derivatives of the functions models usePower, sum and product rules, so you rarely compute from scratch.35 min
  11. The chain rule with vectorsDerivatives of functions inside functions: the heart of backprop.40 min
  12. Backprop through a linear layer: gradients with shapesThe slope of the loss through Y = XW + b: dX, dW and db derived from one entry, found again from shapes, and checked against autograd.40 min
  13. Gradients in matrix form, and how to check themThe vector of all partial derivatives: which way is uphill.35 min
  14. Why training is fast or slowSecond derivatives, minimum or maximum, why curvature caps the learning rate, convex losses and Newton's method.40 min
  15. The chain rule of probabilityUpdate beliefs when new evidence arrives.40 min
  16. Distributions as objectsRandom variables and the shapes their outcomes take.35 min
  17. Expectation and variance of estimatorsExpected value, variance and standard deviation.35 min
  18. Gaussians at workThe bell curve, and why it shows up everywhere.35 min
  19. Covariance matrices and the multivariate normalCovariance and correlation, curves, outliers and common causes, rank correlation, skew and kurtosis.40 min
  20. Losses are negative log-likelihoodsChoose the parameters that make the data most probable.35 min
  21. Samples and confidence: error bars for an evalEstimate from a sample and say how sure you are.40 min
  22. Priors, MAP and regularizationPriors as pseudo-counts, Beta posteriors, shrinkage for small samples, and which of two rates is really higher.40 min
  23. Markov chainsStates, a table of what comes next, the chance of a path, and where a chain settles.38 min
  24. Softmax at working depthTurn any list of scores into probabilities.35 min
  25. Cross-entropy from logitsThe loss function language models are trained on.40 min
  26. KL divergence at workForward against reverse KL, KL between normals, F.kl_div's argument order, a distillation loss with temperature, and a KL penalty against a base model.36 min
  27. A/B tests and significance, for prompts and modelsTell a real lift from noise.40 min
  28. Comparing two models on the same testPaired results, disagreements, a quick z and a bootstrap for the gap between two models.35 min
  29. Ranking from head-to-head resultsElo updates, Bradley-Terry strengths, and how model leaderboards turn votes into ratings.37 min
  30. Capstone: one problem through five tracksA three-class softmax classifier written from its formula: notation to NumPy, matrix products, cross-entropy, the gradient, and an honest report.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write