Glossary · Phase 6 · Modern LLM engineering
Distillation
Training a small model to imitate a big one, keeping much of its skill at lower cost.
Try it now
At temperature 1, the teacher looks certain. Raise the temperature and its hidden knowledge appears: a cat is much more like a dog than like a car.
Learn it properly
Distillation · Train a small model to imitate a large one. About 35 minutes.
Other terms from the same lesson
- soft labels: A teacher's full spread of probabilities over the answers, which carries more information than just the single right answer (a hard label).
- student model: In distillation, the smaller model trained to copy a larger teacher model.
- teacher model: In distillation, the large model whose answers or probabilities a smaller student model learns to copy.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.