Entropy and surprise

Entropy: the average surprise

Surprise is about one event. Entropy is about a whole situation: every outcome that could happen, with its probability. A list like that is a distribution.

Picture two towns. In a desert town, it's sunny 9 days in 10 and rains 1 day in 10. In a coin-flip town, sun and rain are equally likely.

In the desert town, most days bring almost no news. A sunny day has a surprise of $\log_2 \frac{1}{0.9}$, about 0.15 bits. A rainy day is a shock: $\log_2 \frac{1}{0.1}$, about 3.32 bits. But rain is rare. So on average, how much news does the weather bring?

Weight each surprise by how often it happens, then add:

• sunny: 0.9 × 0.152 = 0.137 bits;
• rainy: 0.1 × 3.322 = 0.332 bits;
• average: 0.137 + 0.332 = 0.469 bits per day.

The coin-flip town gets a full 1 bit every day. Its weather is harder to predict, so its entropy is higher.

That weighted average is the entropy. As a formula:

The letter $H$ stands for entropy. $p_i$ is the probability of outcome number $i$. The big $\Sigma$ (sigma, a Greek capital S) means "add this up over every outcome". In Python, it's a for loop or a sum.

In plain words: for each outcome, multiply its probability by its surprise. Add up the results. The total is how many bits of news you get on average. That's how uncertain the situation is.

One small rule: an outcome with probability 0 never happens, so it adds nothing. Books say $0 \log 0$ is treated as 0. The code below drops those outcomes with p[p > 0]. Then it multiplies each probability by its surprise, np.log2(1 / p), and adds the results.

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return np.sum(p * np.log2(1 / p))     # same as -Σ p log₂ p

print("fair coin:        ", entropy([0.5, 0.5]))
print("biased coin 90/10:", round(entropy([0.9, 0.1]), 3))
print("certain:          ", entropy([1.0, 0.0]))
print("fair 8-sided die: ", entropy([1 / 8] * 8))

The 90/10 coin gives 0.469 bits, the same as the desert town. The 8-sided die gives 3 bits, the same as the 8-card game. Three patterns show up:

• Entropy is zero when one outcome is certain. There's nothing to learn.
• It's largest when all outcomes are equally likely. Call the number of outcomes $K$. Then the largest possible entropy is $\log_2 K$ bits. The die has 8 sides, and $\log_2 8 = 3$.
• In between, the more lopsided the distribution, the lower the entropy.

How sure is the model?

A language model predicts the next token (a word or a piece of a word). It gives a probability for every token it knows. That list is a distribution, so it has an entropy too.

After "The capital of France is", nearly all the probability sits on " Paris". The entropy is close to 0: the model is confident. After "My favorite color is", the probability spreads across many colors. The entropy is higher: the model is less sure, and rightly so.

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return np.sum(p * np.log2(1 / p))     # same as -Σ p log₂ p

capital = {"Paris": 0.96, "a": 0.02, "located": 0.01, "the": 0.01}
color = {"blue": 0.28, "green": 0.2, "red": 0.17, "purple": 0.13, "black": 0.12, "yellow": 0.1}
print("after 'The capital of France is':", round(entropy(list(capital.values())), 2), "bits")
print("after 'My favorite color is':   ", round(entropy(list(color.values())), 2), "bits")

The capital prompt scores about 0.3 bits. The color prompt scores about 2.5 bits. Six equally likely colors would give the maximum, $\log_2 6$, about 2.58 bits. So the model's color guess is close to a shrug.

This is useful in practice. High entropy on a factual question can signal that the model is unsure. That's a hint it might hallucinate: give a fluent answer that's made up. Some systems use entropy to decide when to fetch more context or ask a clarifying question.

Watch out: low entropy means confident, not correct. A model can put 96% on a wrong answer.

import math

sam = [0.25, 0.25, 0.25, 0.25]   # four spots, equally often
alex = [0.7, 0.1, 0.1, 0.1]      # a favorite cafe, plus three others

for name, probs in [("Sam", sam), ("Alex", alex)]:
    h = sum(p * math.log2(1 / p) for p in probs)
    print(name, "entropy:", round(h, 2), "bits")

print("Alex at the cafe:   ", round(math.log2(1 / 0.7), 2), "bits of surprise")
print("Alex anywhere else: ", round(math.log2(1 / 0.1), 2), "bits of surprise")

Entropy and compression

Think about how you text. You type "lol" and "brb" for things you say all the time. You spell out rare words in full. Short codes for common things, longer codes for rare ones. Morse code works the same way: E, the most common letter in English, is a single dot.

Entropy sets the limit on this trick. On average, you can't record outcomes in fewer bits than the entropy of their distribution. Predictable data has low entropy, so it shrinks a lot when you zip it. Random data has high entropy, so it barely shrinks at all.

This links straight back to language models. A model that predicts text well can be used to compress text well. In a precise sense, a good predictor of text is a good compressor of text. A model's training loss is measured in the same units as entropy. You'll meet that loss, cross-entropy, in the next lesson.

Note: with natural logs (np.log), entropy comes out in nats instead of bits. 1 nat is about 1.44 bits. Training code usually uses natural logs. The ideas are identical.