Student model
In distillation, the smaller model trained to copy a larger teacher model.
What it means
In knowledge distillation, the student model is the smaller model you train to copy a larger one, the teacher. It learns from the teacher's outputs, often its full spread of probabilities, called soft labels, rather than only from the right answers. The aim is to keep most of the quality at a fraction of the cost.
Why it works
A teacher's answers say more than a label does. When it gives a tent 81% and a tarp 18%, the student learns that those two are similar, which a plain "tent" wouldn't tell it. So a student can often learn more from a teacher than it would from the original data alone.
DistilBERT is a well-known result: its authors reported 40% fewer parameters and 60% faster running than BERT, while keeping over 95% of its performance.
Worth knowing
A student rarely beats its teacher, and it inherits the teacher's mistakes, so evaluate it separately. And check the license first: many AI providers' terms forbid using their outputs to train a competing model. The distillation lesson goes through how it's done.
Try it now
At temperature 1, the teacher looks certain. Raise the temperature and its hidden knowledge appears: a cat is much more like a dog than like a car.
Learn it properly
Distillation · Train a small model to imitate a large one. About 35 minutes.
Other terms from the same lesson
- distillation: Training a small model to imitate a big one, keeping much of its skill at lower cost.
- soft labels: A teacher's full spread of probabilities over the answers, which carries more information than just the single right answer (a hard label).
- teacher model: In distillation, the large model whose answers or probabilities a smaller student model learns to copy.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.