Bigram
A pair of neighboring tokens. A bigram model predicts the next token from only the one before.
What it means
A bigram is a pair of neighboring items in text, usually two words or two characters. In "the cat sat", the bigrams are the cat and cat sat. A bigram model predicts the next item from only the one before it, using counts: of all the times it saw "the", what came next?
An example in Python
from collections import Counter, defaultdict words = "the cat sat on the mat and the cat ran to the mat".split() following = defaultdict(Counter) for current, next_word in zip(words, words[1:]): following[current][next_word] += 1 print("after 'the':", dict(following["the"])) total = sum(following["the"].values()) for word, n in following["the"].most_common(): print(f"P({word} | the) = {n}/{total} = {n / total:.2f}")
after 'the': {'cat': 2, 'mat': 2}
P(cat | the) = 2/4 = 0.50
P(mat | the) = 2/4 = 0.50That's the whole model. It already beats a unigram model, which ignores order altogether, because it knows that "cat" follows "the" and that "the" rarely follows "the".
Where it stops working
One word of context isn't much. A bigram model can't tell that "the cat that the dog chased" is still about a cat. Counting longer runs doesn't rescue it either: most long sequences never appear in the training text, and the table grows enormously. Neural language models solve that by learning to generalize, and bigram models remain a useful baseline. The bigram lesson builds one from counts.
Try it now
A bigram model has one row like this for each letter. Generating text is just sampling from the current row, again and again.
Learn it properly
A bigram model · The simplest language model, built from counts. About 35 minutes.
Other terms from the same lesson
- n-gram: A model that predicts the next item from the previous n − 1 items. A bigram uses one, a trigram two.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.