Double descent
When a model is made far bigger than the data seems to justify, its test error can fall again after first rising. It's one reason huge neural networks still generalize.
The U-shaped picture holds well for classical models. As you add complexity, validation error first falls, then rises again. Very large neural networks often break this pattern. Past the point where a model can fit its training data exactly, test error can fall again as the model keeps growing. This is called double descent. It's part of why LLMs with billions of weights, trained on enormous datasets, generalize well.
Try it now
Twelve refits on fresh samples. At degree 1 they agree and all miss. At degree 12 they disagree wildly.
Learn it properly
Bias and variance · The two ways a model can be wrong. About 30 minutes.
Other terms from the same lesson
- bias-variance tradeoff: Simple models miss in a steady way (bias); flexible models swing with every change in the data (variance). The best model balances the two.
- learning curve: A chart of training and validation scores as the training set grows. The gap between the lines shows whether more data would help.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.