Glossary · Phase 2 · Math, when you need it
KL divergence
A measure of how different one set of probabilities is from another.
Try it now
Pick which word actually came next. Raise its score and the loss falls toward 0; give probability to the wrong words and it climbs. Training nudges the logits exactly this way.
Learn it properly
Cross-entropy and KL divergence · The loss function language models are trained on. About 30 minutes.
Other terms from the same lesson
- cross-entropy: A score for how surprised a model is by the right answer. Lower is better; it's the usual training loss for classifiers and language models.
- perplexity: How many words a language model is effectively choosing between at each step. Lower is better; it's e to the cross-entropy.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.