Language modeling
Why this matters
Every large language model, from GPT-2 to Claude, is trained on one simple task. It reads some text and guesses what comes next. Grammar, facts, style and even reasoning all get learned because they make that guess better. This phase builds a language model step by step, from counting letters to a working transformer. It starts here, by pinning down two things: what "guess the next piece of text" means, and how you score a model's guesses.
- Define a language model as a probability distribution over the next token
- Break a text's probability into next-token predictions with the chain rule
- Score a model with average cross-entropy (loss) and perplexity
- Build and evaluate the simplest baselines: uniform and unigram
The game: guess what comes next
Your phone's keyboard plays a small version of this game. Type "Thanks for your" and it suggests "help", "order" or "time". It has learned which words tend to follow which.
A language model does the same thing more carefully. It reads the text so far. Then it gives a chance, a probability, to every piece of text that could come next. The pieces are called tokens. In this lesson a token is a single character, such as "a", "!" or a space. Later lessons use word pieces instead.
The full list of tokens the model knows is its vocabulary. So the model's answer is a list of chances, one for each token in the vocabulary, adding up to 1.
To write text, the model repeats four steps:
- Look at everything written so far.
- Give every possible next token a chance.
- Pick one token and add it to the end.
- Go back to step 1.
That's all generation is. The sampling lesson later in this phase covers how to pick in step 3.