Language modeling

The game: guess what comes next

Your phone's keyboard plays a small version of this game. Type "Thanks for your" and it suggests "help", "order" or "time". It has learned which words tend to follow which.

A language model does the same thing more carefully. It reads the text so far. Then it gives a chance, a probability, to every piece of text that could come next. The pieces are called tokens. In this lesson a token is a single character, such as "a", "!" or a space. Later lessons use word pieces instead.

The full list of tokens the model knows is its vocabulary. So the model's answer is a list of chances, one for each token in the vocabulary, adding up to 1.

To write text, the model repeats four steps:

1. Look at everything written so far.
2. Give every possible next token a chance.
3. Pick one token and add it to the end.
4. Go back to step 1.

That's all generation is. The sampling lesson later in this phase covers how to pick in step 3.

The chance of a whole text

A model can also score a whole text: how likely is it? You get that from the same next-token guesses. Multiply them together.

Take the word "cat". Say the model gives these chances:

• "c" as the first letter: 0.2
• "a" next, given "c": 0.5
• "t" next, given "ca": 0.9

The chance of "cat" is 0.2 × 0.5 × 0.9 = 0.09. This is the chain rule of probability from Phase 2. Written out for any text, it looks like this:

Here $x_1$ is the first token and $x_n$ is the last one. The letter $n$ is how many tokens the text has. $P$ means "the probability of". The bar $\mid$ is read "given".

In plain words: the chance of a whole text is the chance of its first token, times the chance of the second given the first, and so on to the end.

In Python, the long product is a loop:

probs = [0.2, 0.5, 0.9]   # P("c"), P("a" | "c"), P("t" | "ca")
p_text = 1
for p in probs:
    p_text *= p
print(round(p_text, 3))

It prints 0.09. So a model only ever needs to answer one question: given everything so far, how likely is each possible next token? Answer that well, and both writing and scoring follow.

Scoring a model: the loss

To tell a good model from a bad one, test it on text it has never seen. This is called held-out text. It's like testing a student on new questions, not the ones they practiced.

Walk through the held-out text one position at a time. At each one, look up the chance the model gave to the token that really came next. A high chance is good. A low chance is bad.

Turn each chance into a surprise score, as in the cross-entropy lesson. The surprise is minus the log of the chance. Here "log" is the natural log: ln on a calculator, math.log in Python. A chance of 1 gives surprise 0. A tiny chance gives a big surprise.

The loss is the average surprise over the whole text:

The $\sum$ sign (sigma) means "add up". The letter $i$ counts the positions from 1 to $n$. And $x_{<i}$ means "all the tokens before position $i$".

In plain words: for each position, take the chance the model gave the real next token. Take minus its log. Then average those numbers. Lower is better.

Say a model gave the real next tokens chances of 0.5, 0.4 and 0.1. The surprises are about 0.693, 0.916 and 2.303. Their average, the loss, is about 1.304.

import math

probs = [0.5, 0.4, 0.1]   # chance given to each token that really came next
losses = [-math.log(p) for p in probs]
loss = sum(losses) / len(losses)
print([round(x, 3) for x in losses])
print(round(loss, 3), round(math.exp(loss), 2))

The second line also prints 3.68. That's the perplexity, coming up next.

This is the same cross-entropy you used for classifiers. Picking the next token is a classification problem: the model chooses one answer from a fixed list, the vocabulary. A loss measured with the natural log comes in units called nats.

Perplexity: the loss as a number of choices

A loss of 1.3 nats is hard to picture. Perplexity turns it into a number of equally likely choices. To get it, undo the log. Raise $e$, a fixed number about 2.718, to the power of the loss. In Python that's math.exp.

In plain words: perplexity is how many options the model is "effectively choosing among" at each step.

Think of a fair six-sided die. Someone who guesses the roll gives the right face a chance of 1/6. Their loss is −ln(1/6) ≈ 1.792 nats. Raise $e$ to that power and you get 6 back: six choices.

import math

V = 6                      # a fair die: six equally likely faces
loss = -math.log(1 / V)    # every guess gives the right face 1/6
print(round(loss, 3), round(math.exp(loss), 3))

It prints 1.792 6.0. The same holds for any vocabulary. Write $V$ for the number of tokens in it. A model that guesses evenly among them has loss $\ln V$ and perplexity $V$. A perfect model gives every real token a chance of 1. Its loss is 0 and its perplexity is 1.

The example above, with loss 1.304, had perplexity 3.68. That model is about as unsure as someone picking from three or four equal options.

You can also measure loss with log base 2. Then it's in bits, and perplexity is 2 to the power of the loss. The perplexity comes out the same either way.

import math
loss = 0.0
print(math.exp(loss))

The course text, and two simple baselines

Every model needs something to beat. A baseline is the simplest model you can think of. A weather forecaster's baseline might be "tomorrow will be like today". If a fancy model can't beat that, something is wrong.

This phase uses a corpus, a collection of text to learn from. It holds 300 short product descriptions for a made-up outdoor brand, served by the practice API. For now you'll model it character by character. Tokens made of word pieces come in the next lessons.

The code keeps the first 270 descriptions for training. It holds out the last 30 to measure the loss honestly. Then it tries two baselines:

• Uniform: every character gets the same chance. With $V$ characters, each gets $1/V$.
• Unigram: each character gets its share of the training text. If 1 character in 6 is a space, a space gets a chance of about 1/6.

The unigram model adds 1 to every count first. This is called add-one smoothing. Without it, a character that never shows up in training would get a chance of 0. Its surprise, minus the log of 0, would be infinite, and one such character would wreck the whole score.

Here's add-one smoothing on a tiny example. The training text is "banana", and the vocabulary is "a", "b", "n" and "x". The counts are 3, 1, 2 and 0. Add 1 to each to get 4, 2, 3 and 1. Then divide by the new total, 6 + 4 = 10.

from collections import Counter

train = "banana"
vocab = ["a", "b", "n", "x"]
counts = Counter(train)
probs = {c: (counts[c] + 1) / (len(train) + len(vocab)) for c in vocab}
print(probs)

It prints chances of 0.4, 0.2, 0.3 and 0.1. The "x" never appeared, but it still gets 0.1. Now the real corpus:

import math
from collections import Counter
import numpy as np
import requests

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
docs = text.split("\n")
train, val = "\n".join(docs[:270]), "\n".join(docs[270:])
print(docs[0])
print(f"\n{len(text):,} characters, {len(set(text))} distinct characters")

vocab = sorted(set(text))
V = len(vocab)
print(f"\nuniform guessing: loss {math.log(V):.3f}, perplexity {V}")

counts = Counter(train)
probs = {c: (counts[c] + 1) / (len(train) + V) for c in vocab}     # +1 so unseen characters aren't impossible
loss = -np.mean([math.log(probs[c]) for c in val])
print(f"unigram (letter frequencies): loss {loss:.3f}, perplexity {math.exp(loss):.1f}")
print("most common characters:", [c for c, _ in counts.most_common(8)])

The corpus uses 56 different characters. Uniform guessing has loss ln 56 ≈ 4.025 and perplexity

1. The unigram model's loss is about 3.1, a perplexity of about 22. Knowing which characters are

common (spaces, e, a, t...) cuts the effective choices from 56 to about 22.

What a unigram model can't see

The unigram model ignores order. It scores each character on its own, so shuffling the letters changes nothing. Here it scores a sentence and the same letters scrambled:

import math
import random
from collections import Counter
import requests

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
counts, V = Counter(text), len(set(text))
logp = lambda s: sum(math.log((counts[c] + 1) / (len(text) + V)) for c in s)

sentence = "It is light and packable."
scrambled = "".join(random.Random(1).sample(sentence, len(sentence)))
for s in [sentence, scrambled]:
    print(f"{s!r:<30} log probability {logp(s):.2f}")

The code adds up the log of each character's chance. That sum is the log of the whole text's chance. Both lines print exactly the same number. To the unigram model, nonsense is as likely as real English.

A useful model has to use context, the text that came before. After "packa", the next letter is almost surely "b". The next lessons add context step by step:

1. one previous character (a bigram model);
2. several previous characters, fed through a neural network;
3. finally, attention over the whole text.

The same game at scale

A large language model (LLM) plays exactly this game, with three differences:

• It predicts tokens that are word pieces, not single characters. Its vocabulary has roughly 50,000 to 200,000 of them.
• Its context is the whole conversation, not one or two characters.
• The model doing the guessing is a transformer. It has billions of settings (parameters) and learns from trillions of tokens.

Training losses for LLMs are reported per token, and lower is better. But you can only compare two losses fairly when the models split text into tokens the same way and test on the same data. A model with bigger tokens has fewer, harder guesses to make, so its loss per token isn't comparable.

This simple task still produces very capable models. To guess the next token of real text well, a model has to know a lot. It needs spelling, grammar, the topic and the writer's style. Finishing "The capital of Japan is" needs a fact. Training rewards anything that lowers the loss, so the model picks all of these up along the way.

import math

# chances the model gave each real next character of "tent"
probs = [0.1, 0.5, 0.4, 0.8]
p_text = math.prod(probs)
loss = sum(-math.log(p) for p in probs) / len(probs)
print(round(p_text, 4))
print(round(loss, 3), round(math.exp(loss), 2))