KL divergence: how far one distribution is from another
Why this matters
The last lesson's model paid 1.75 bits of surprise per outcome on average, where the truth itself needs only 1.5. The extra quarter of a bit has a name: the KL divergence, how far the model's chances sit from the truth's. Cross-entropy is always the truth's own entropy plus this gap, and the data fixes the entropy, so when training lowers a model's cross-entropy, what really shrinks is the KL. It's a loss in its own right too. A small model learns to copy a large one by shrinking it, variational and diffusion models are trained on bounds built from it, and when a chat model is tuned on people's preferences, a KL term keeps it from drifting far from where it started.
- Compute KL(p ‖ q) by hand: each outcome's gap, log(p ÷ q), weighted by the truth's chance p and added
- Show that cross-entropy is entropy plus KL, so lowering one lowers the other by the same amount
- See that KL is 0 only when the two distributions match, and that KL(p ‖ q) and KL(q ‖ p) differ
- Compute a soft-target loss, the KL a small model learns from a large one, in NumPy and with F.kl_div
The math: the gap between two distributions
Purpose. Measure how far a model's chances q are from the truth's chances p, outcome by outcome, then in one number.
Take lesson 105's example again. The truth gives fire, tent and trail the chances p = (0.5, 0.25, 0.25), and a model guesses q = (0.25, 0.25, 0.5). For each outcome, set two surprises side by side: the model's, log₂(1/q), and the truth's own, log₂(1/p). Their difference is the outcome's gap: how much more the model is surprised than it needs to be. A log law turns the difference of two logs into one:
In plain words: an outcome's gap is the log of how many times more chance the truth gives it than the model does.