Glossary · Phase 2 · Math, when you need it
Information content
How surprising one outcome is, in bits: rare events carry more information than common ones.
The rarer an event, the more you learn when it happens. Information theory turns this into a number called surprise. It's also called information content. It's measured in bits.
Try it now
Entropy of a next-word distribution. Push one score far above the others and entropy falls toward 0. Make the scores equal and it reaches the maximum, log₂ 4 = 2 bits.
Learn it properly
Entropy and surprise · Measure uncertainty in bits. About 25 minutes.
Other terms from the same lesson
- bit: The unit of information: the answer to one fair yes-or-no question, like one coin flip.
- entropy: How unpredictable something is. A fair coin has more entropy than a coin that nearly always lands heads.
- nat: A unit of information that uses natural logs instead of base 2. One nat is about 1.44 bits.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.