Unigram
The simplest language model: it predicts each character or word from how common it is overall, ignoring what came before.
What it means
A unigram is a single item of text, usually one word or one character. It's the smallest member of the n-gram family: a bigram is two items in a row, and a trigram is three. In "the cat sat", the unigrams are the, cat and sat, and the bigrams are the cat and cat sat.
A unigram model is the simplest language model there is. It gives each item a probability from how common it is overall, and it ignores everything that came before.
An example in Python
from collections import Counter words = "the cat sat on the mat and the dog sat on the log".split() counts = Counter(words) print("unigrams:", len(words), "words,", len(counts), "different ones") print("bigrams: ", list(zip(words, words[1:]))[:3]) for word, n in counts.most_common(3): print(f"P({word}) = {n}/{len(words)} = {n / len(words):.2f}")
unigrams: 13 words, 8 different ones
bigrams: [('the', 'cat'), ('cat', 'sat'), ('sat', 'on')]
P(the) = 4/13 = 0.31
P(sat) = 2/13 = 0.15
P(on) = 2/13 = 0.15That's the entire model: each word's count divided by the total. It predicts "the" with the same 31% chance whatever came before, so if you sampled from it you'd get word soup such as "the the on sat". Knowing how common words are is useful, but knowing the order is what makes language.
Where you'll meet unigrams
They're a baseline: a language model has to beat a unigram model to show it has learned anything about order. They're also the basis of bag-of-words features in search and text classification. A word that never appeared in training would get a probability of zero, so counts usually start at one, which is called add-one smoothing.
Try it now
A language model is this table, for every possible context. Press Sample and it writes the next word by drawing from these made-up odds.
Learn it properly
Language modeling · Predict the next token, and score it with cross-entropy. About 30 minutes.
Other terms from the same lesson
- add-one smoothing: Adding 1 to every count before turning counts into chances, so nothing the model never saw gets a chance of exactly 0.
- corpus: The collection of text a language model learns from, such as product descriptions or a slice of the web.
- language model: A model that predicts the next token from the ones before it.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.