Soft labels
A teacher's full spread of probabilities over the answers, which carries more information than just the single right answer (a hard label).
What it means
A hard label gives one right answer: this is a tent. A soft label gives a spread of probabilities over all the answers: probably a tent, possibly a tarp, almost certainly not a mug. That spread carries more information, because it says how the classes relate to each other.
An example with NumPy
import numpy as np def softmax(logits, temperature=1.0): scaled = np.array(logits) / temperature return np.exp(scaled) / np.exp(scaled).sum() classes = ["tent", "tarp", "mug"] logits = [4.0, 2.5, -1.0] # a teacher model's raw scores print("hard label: ", [1, 0, 0]) print("soft label: ", softmax(logits).round(3)) print("softer, temperature 3:", softmax(logits, 3).round(3)) # Label smoothing is a simpler kind of soft label, with no teacher involved. print("label smoothing 0.1: ", np.round(np.array([1, 0, 0]) * 0.9 + 0.1 / 3, 3))
hard label: [1, 0, 0] soft label: [0.813 0.181 0.005] softer, temperature 3: [0.557 0.338 0.105] label smoothing 0.1: [0.933 0.033 0.033]
The hard label says nothing about the tarp. The soft label shows the teacher sees a tent as far more like a tarp than a mug. Raising the temperature flattens the spread, which brings out those smaller differences.
Where they're used
Knowledge distillation trains a smaller student model on a bigger model's soft labels. Geoffrey Hinton and colleagues showed in 2015 that a student learns better this way. Label smoothing is a simpler kind, with no teacher: it takes a little from the right answer and shares it among the others, as the last line shows, so the model doesn't grow overconfident. Soft labels also fit data where people disagree about the right answer.
Try it now
At temperature 1, the teacher looks certain. Raise the temperature and its hidden knowledge appears: a cat is much more like a dog than like a car.
Learn it properly
Distillation · Train a small model to imitate a large one. About 35 minutes.
Other terms from the same lesson
- distillation: Training a small model to imitate a big one, keeping much of its skill at lower cost.
- student model: In distillation, the smaller model trained to copy a larger teacher model.
- teacher model: In distillation, the large model whose answers or probabilities a smaller student model learns to copy.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.