Glossary · Phase 5 · Transformers
Add-one smoothing
Adding 1 to every count before turning counts into chances, so nothing the model never saw gets a chance of exactly 0.
The unigram model adds 1 to every count first. This is called add-one smoothing. Without it, a character that never shows up in training would get a chance of 0. Its surprise, minus the log of 0, would be infinite, and one such character would wreck the whole score.
Try it now
A language model is this table, for every possible context. Press Sample and it writes the next word by drawing from these made-up odds.
Learn it properly
Language modeling · Predict the next token, and score it with cross-entropy. About 30 minutes.
Other terms from the same lesson
- corpus: The collection of text a language model learns from, such as product descriptions or a slice of the web.
- language model: A model that predicts the next token from the ones before it.
- unigram: The simplest language model: it predicts each character or word from how common it is overall, ignoring what came before.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.