Index Ai Academy
Foundation · Foundation mathStatistics and information

Surprise, entropy and cross-entropy

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Nearly every classifier and every language model is trained on the same number: cross-entropy. Lesson 100 met it as a negative log-likelihood and lesson 102 as the log loss, without asking what it measures. This lesson gives it a meaning. An unlikely event is surprising, and surprise can be measured in bits: a coin toss carries one bit, a 1-in-8 event three. The average surprise of a source is its entropy, how unpredictable it is at best. The average surprise of the truth under a model's predictions is the cross-entropy, and it can never be less than the entropy: the gap is exactly what the model gets wrong. So a falling training loss means a model is being surprised less by real text. Perplexity, the number language models are compared by, is the same quantity read as "how many equally likely choices the model is torn between".

  • Measure surprise as −log₂ p: rare events surprise more, and a 1-in-2 event is one bit
  • Compute entropy, the average surprise of a distribution, and cross-entropy, the average surprise of the truth under a model
  • Compute both as dot products with a log, and see that with one right answer cross-entropy is −log of its probability
  • Recognize cross-entropy as a language model's training loss, report it as perplexity, and check its value and its slope p − y with PyTorch's F.cross_entropy
Quick check
Have a guess before you start. Which surprises you more: a fair coin landing heads, or a fair die landing on 6?

The math: surprise, averaged

Purpose. Measure how wrong a predicted distribution is, in one number.

Surprise. An event with chance p carries a surprise of −log₂ p, also written log₂(1/p), measured in bits. A 1-in-2 event is 1 bit, 1-in-4 is 2 bits, 1-in-8 is 3: each halving of the chance adds one bit. A certain event, p = 1, is 0 bits: no surprise. An event with p = 0.9 is 0.152 bits; one with p = 0.01 is 6.6 bits. The log makes surprises of independent events add, just as their chances multiply (lesson 89).

Entropy. The entropy of a distribution is its average surprise, each outcome's surprise weighted by its chance: H(p) = Σ pᵢ × (−log₂ pᵢ). A fair coin: ½ × 1 + ½ × 1 = 1 bit. A coin with chances 0.3 and 0.7: 0.88 bits, less, since it's more predictable. A two-headed coin: 0 bits. A fair die: log₂ 6 = 2.585 bits. Entropy is the smallest average surprise anyone can have about that source.