Bias and variance

Splitting the error into three parts

There's a neat fact about these two kinds of miss. Suppose you measure error by squaring it, as mean squared error does. Then the total error splits exactly into three pieces:

Take it one symbol at a time. The letter $y$ is the real outcome, noise and all. The prediction is $\hat{y}$, said "y-hat". The $\mathbb{E}$ in front means "expected value", the long-run average you met in Phase 2. Here it averages over every training set you might have drawn, and over the noise too. So the left side is the average squared error. On the right, $\sigma^2$ (said "sigma squared") measures the size of the noise.

In plain words: your average squared error is made of three parts. Bias squared is how far off the average prediction is, squared. Variance is how much the predictions spread out around their own average. Noise is the randomness no model can predict.

Bias shows up squared because the error itself was squared. So a steady miss too high and a steady miss too low both count as positive. The noise part is the same for every model. Two customers who look identical on paper can still behave differently, and nothing can predict that. So you can only fight the other two parts. They usually trade against each other: pushing one down tends to push the other up.

A small example

A delivery app predicts how long one order will take. You retrain its model on four different months of data, so you get four models. For the same order they predict 22, 26, 24 and 28 minutes. Say the true typical time for this order is 20 minutes. Leave noise out for now.

• The average prediction is (22 + 26 + 24 + 28) / 4 = 25. That's 5 minutes too high, so bias² = 5² = 25.
• Each prediction's distance from the average of 25 is −3, 1, −1 and 3. Squared, those are 9, 1, 1 and 9. Their average is 20 / 4 = 5. That's the variance.
• The errors against the truth are 2, 6, 4 and 8. Squared, they're 4, 36, 16 and 64. Their average is 120 / 4 = 30.

And 25 + 5 = 30. The split is exact.

preds = [22, 26, 24, 28]   # four retrained models, one order
truth = 20                 # the true typical delivery time
avg = sum(preds) / len(preds)
bias_sq = (avg - truth) ** 2
variance = sum((p - avg) ** 2 for p in preds) / len(preds)
mse = sum((p - truth) ** 2 for p in preds) / len(preds)
print(avg, bias_sq, variance, mse)

It prints 25.0 25.0 5.0 30.0. Those are the average prediction, bias², variance, and the mean squared error.

preds = [9, 11]     # two retrained models, one customer
truth = 7
avg = sum(preds) / len(preds)
print((avg - truth) ** 2, sum((p - avg) ** 2 for p in preds) / len(preds))

Seeing it: refit two hundred times

Now watch the trade-off happen. In the code below, the true pattern is a smooth wave called a sine curve. It's the dashed gray line. Each training set is 25 points taken from the wave, with random noise added.

The code fits a polynomial to each training set. A polynomial is a curve whose degree sets how many bends it can make. Degree 1 is a straight line. Degree 3 can bend twice. Degree 7 can bend up to six times.

For each degree, the code fits 200 different noisy samples and draws 20 of the fits. Then it measures bias² and variance across all 200. It averages them over 50 points along the wave.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
f = lambda x: np.sin(2 * np.pi * x)
grid = np.linspace(0.05, 0.95, 50)

def fits(degree, n_sets=200, n=25, noise=0.3):
    preds = []
    for _ in range(n_sets):
        x = rng.uniform(0, 1, n)
        y = f(x) + rng.normal(0, noise, n)
        preds.append(np.polyval(np.polyfit(x, y, degree), grid))
    return np.array(preds)

fig, axes = plt.subplots(1, 3, figsize=(10, 3.2), sharey=True)
for ax, d in zip(axes, [1, 3, 7]):
    P = fits(d)
    for p in P[:20]:
        ax.plot(grid, p, alpha=0.35, lw=1)
    ax.plot(grid, f(grid), "--", color="gray", lw=2)
    ax.set_title(f"degree {d}")
    ax.set_ylim(-2, 2)
    bias2 = np.mean((P.mean(axis=0) - f(grid)) ** 2)
    var = np.mean(P.var(axis=0))
    print(f"degree {d}:  bias² {bias2:.3f}   variance {var:.3f}   sum {bias2 + var:.3f}")

Read the panels from left to right.

• Degree 1: the lines agree with each other, and they're all wrong in the same way. That's high bias and low variance. Its bias² is about 0.15.
• Degree 7: the curves are right on average, but each one wiggles differently. That's low bias and high variance. Its variance is about 0.72.
• Degree 3: both parts are small, and it has the lowest sum, about 0.02. Its bends can follow the wave, without many spare bends left over to chase noise.

Learning curves: diagnosing a real model

In real work you can't collect a hundred fresh datasets. You have one. A learning curve gets the same diagnosis from it. You train on bigger and bigger slices of the data. At each size, you score the model twice: once on the data it trained on, and once on validation data it didn't see. Then you plot both scores.

The code below does this for the customer data you've used before. Each row says whether a customer churned, which means they stopped buying. It compares two models. The first is logistic regression, which draws one simple, straight boundary between leavers and stayers. The second is an unpruned decision tree, one that's allowed to grow as deep as it likes.

The score here is AUC. It measures how well a model ranks customers who leave above customers who stay. A score of 0.5 is no better than a coin flip, and 1.0 is a perfect ranking. A later lesson covers it in full. The cv=5 setting repeats each fit on five different splits of the data and averages the results, so the curves are less jumpy.

import pandas as pd
import requests
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X, y = df[features], df["churned"]

models = {"logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
          "unpruned tree": DecisionTreeClassifier(random_state=0)}
fig, axes = plt.subplots(1, 2, figsize=(10, 3.2), sharey=True)
for ax, (name, model) in zip(axes, models.items()):
    sizes, tr, va = learning_curve(model, X, y, train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0],
                                   cv=5, scoring="roc_auc")
    ax.plot(sizes, tr.mean(axis=1), marker="o", label="training")
    ax.plot(sizes, va.mean(axis=1), marker="o", label="validation")
    ax.set_title(name)
    ax.set_xlabel("training examples")
    print(f"{name}: final train AUC {tr.mean(axis=1)[-1]:.2f}; validation AUC as data grows "
          + " → ".join(f"{v:.2f}" for v in va.mean(axis=1)))
axes[0].set_ylabel("AUC")
axes[0].legend()

Here's how to read the two charts.

• Logistic regression: the training and validation curves sit close together. More data won't move them much. If the score isn't good enough, bias is the limit. The model needs better features or a more flexible design.
• Unpruned tree: training AUC is a perfect 1.0 at every size, and validation is far below. That big gap is the classic sign of high variance. The validation score is noisy, but it ends highest with the most data. The fixes are more data, pruning (limiting how deep the tree can grow), or averaging many trees, which is what a random forest does.

The table below sums up the two patterns. Two of its fixes may be new. Regularization adds a penalty that pushes a model toward simpler answers. Bagging trains many models on different random samples of the data and averages them.

The knobs you already know, by what they trade

Most settings you tune move a model along the same seesaw. Turn a knob one way, and bias falls while variance rises. Turn it the other way, and the reverse happens.

• Regularization strength λ (said "lambda"): how hard the penalty pushes the model toward simpler answers. Turning it up means less variance and more bias.
• Tree depth, polynomial degree, network size: turning these up means less bias and more variance.
• k in nearest neighbors: a nearest-neighbors model predicts from the k most similar past examples. A small k is flexible, so it has high variance. A large k averages more neighbors, so it has higher bias.
• Bagging and random forests average many high-variance models. Their scattered misses partly cancel out, which cuts variance.
• Boosting adds models one after another. Each new one works on the errors the earlier ones left behind. That cuts bias.

A caveat for deep learning

The U-shaped picture holds well for classical models. As you add complexity, validation error first falls, then rises again. Very large neural networks often break this pattern. Past the point where a model can fit its training data exactly, test error can fall again as the model keeps growing. This is called double descent. It's part of why LLMs with billions of weights, trained on enormous datasets, generalize well.

The diagnosis tools still apply. Say a fine-tune's training loss keeps falling while its validation loss climbs. That model is showing high variance.

For marketers: a forecast that's always 10% low has a bias problem. One that swings wildly whenever you add a month of data has a variance problem. The fixes are different, so find out which one you have before you change anything.

truth = 20
models = {"simple": [14, 15, 14, 15], "flexible": [12, 29, 16, 23]}
for name, preds in models.items():
    avg = sum(preds) / len(preds)
    bias_sq = (avg - truth) ** 2
    variance = sum((p - avg) ** 2 for p in preds) / len(preds)
    print(f"{name:<9} average {avg:5.1f}  bias² {bias_sq:5.2f}  variance {variance:5.2f}  total {bias_sq + variance:5.2f}")