Overfitting, visualized

Raise a model's complexity and watch training error keep falling while error on new data turns back up. Then see bias, variance and regularization at work.

A model can score perfectly on its training data and still be bad, because it has memorized the data rather than learned the pattern. The only way to see that is to measure it on data it hasn't seen. These curves are fitted live, on this page.

Raise the degree. Training error keeps dropping, and error on new data turns back up.

Bias and variance

A model can be wrong in two ways. A model that's too simple misses in the same way every time, which is bias. A model that's too flexible changes wildly with each new sample, which is variance. Refit the same model on fresh samples and you can see both.

Twelve refits on fresh samples. At degree 1 they agree and all miss. At degree 12 they disagree wildly.

One way to fight it

Regularization adds a penalty for complexity to the loss, so the model only gets complicated where the data earns it. Too little does nothing, and too much flattens the model.

Slide the penalty and watch the error on new points fall, then rise again when the penalty is too strong.

Learn how this works, properly

A free account opens all 244 lessons, with Python that runs in your browser, saved progress and certificates.

Create a free account