What an LLM actually does
Why this matters
You can already send a request to a model and read the reply. To build something reliable on that call, you need a picture of what happens in between. Most prompting advice in Phase 1 follows from one fact, and so do most of the ways LLM apps fail. The model writes its reply by predicting the next token, over and over. Once that clicks, you'll know when to trust a model, when to check it, and when to hand the job to ordinary code.
- Describe how a model writes a reply one token at a time
- Explain where a model's knowledge comes from, and why it can sound sure and still be wrong
- Spot tasks that need checking or plain code instead: counting, recent facts, exact math
- Simulate the generation loop in Python with a toy probability table
Autocomplete, run in a loop
Your phone's keyboard suggests the next word as you type. A large language model (LLM) does the same job at an enormous scale. You give it some text. It works out how likely each possible next piece of text is.
Those pieces are called tokens. As in your first LLM call, a token is a word or a piece of a word. The model reads tokens, which are chunks of characters, so individual letters aren't directly visible to it: that's why it can miscount the letters in a word. The full list of tokens a model knows is its vocabulary. The model gives every token in its vocabulary a probability. That's a number from 0 to 1 saying how likely the token is to come next. A probability of 0.31 means "about 31 times in 100".
Take the text Our spring sale ends. A model might score the candidates like this (numbers invented for illustration):
| Next token | Probability |
|---|---|
Sunday | 0.31 |
soon | 0.22 |
tomorrow | 0.18 |
March | 0.09 |
| thousands of others | the remaining 0.20 |
Notice the space before Sunday in the table. Tokens often include the space in front of a word. The probabilities across the whole vocabulary add up to 1, the same way the slices of a pie add up to the whole pie.
In plain words: the model looks at your text and asks "what usually comes next?" Its answer is a list of options, each with a score for how likely it is.