Glossary · Phase 5 · Transformers
Corpus
The collection of text a language model learns from, such as product descriptions or a slice of the web.
Try it now
A language model is this table, for every possible context. Press Sample and it writes the next word by drawing from these made-up odds.
Learn it properly
Language modeling · Predict the next token, and score it with cross-entropy. About 30 minutes.
Other terms from the same lesson
- add-one smoothing: Adding 1 to every count before turning counts into chances, so nothing the model never saw gets a chance of exactly 0.
- language model: A model that predicts the next token from the ones before it.
- unigram: The simplest language model: it predicts each character or word from how common it is overall, ignoring what came before.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.