Overfitting and underfitting

The complexity curve

How flexible a model is gets called its complexity. A simple model can only draw broad patterns. A complex one can bend to fit tiny details, including noise.

A decision tree makes this easy to see. A tree learns a flowchart of yes/no questions. Its depth is the longest chain of questions it may ask before it gives an answer. A tree of depth 1 asks one question. A tree of depth 15 can ask 15 in a row, so it can split the customers into very small groups.

So here's the experiment. Train trees of depth 1, 2, 3 and so on, up to 15. For each one, record the training score and the validation score. Then plot both against depth. That plot is called a complexity curve.

The code below does this on the churn data from earlier lessons. (A customer churned if they stopped buying.) It keeps 30% of customers back for validation. It draws both curves and puts a dashed gray line at the depth with the best validation score. Then it prints that depth and its score.

import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X_tr, X_val, y_tr, y_val = train_test_split(df[features], df["churned"], test_size=0.3, random_state=0, stratify=df["churned"])

depths = range(1, 16)
train_acc, val_acc = [], []
for d in depths:
    tree = DecisionTreeClassifier(max_depth=d, random_state=0).fit(X_tr, y_tr)
    train_acc.append(tree.score(X_tr, y_tr))
    val_acc.append(tree.score(X_val, y_val))

best = depths[int(np.argmax(val_acc))]
plt.plot(depths, train_acc, marker="o", label="training")
plt.plot(depths, val_acc, marker="o", label="validation")
plt.axvline(best, linestyle="--", color="gray")
plt.xlabel("max tree depth (complexity)")
plt.ylabel("accuracy")
plt.legend()
print("best depth by validation:", best, " validation accuracy:", round(max(val_acc), 3))

Look at the two lines. Training accuracy climbs steadily toward 1.0, which means perfect. The deeper the tree, the better it fits the customers it learned from.

Validation accuracy tells a different story. It rises a little, peaks at a shallow depth, and then falls. Past the peak, each extra level lets the tree memorize more quirks of the training customers. Those quirks don't carry over to new customers, so the validation score drops.

The best model sits where validation peaks, not where training does. That's the depth you should pick.

train_acc = [0.81, 0.84, 0.88, 0.93, 0.99]
val_acc = [0.80, 0.83, 0.82, 0.78, 0.74]
depths = [1, 2, 3, 4, 5]
print(depths[val_acc.index(max(val_acc))])

What to do about overfitting

There are five standard fixes. Each one either gives the model more to learn from or makes it harder to memorize.

1. More data. This is the single most reliable fix. Noise averages out across many examples. The real pattern doesn't, so it stands out more clearly.
2. A simpler model. Use shallower trees or fewer features. In nearest neighbors, which predicts from the most similar past examples, ask more neighbors (a larger k). One odd neighbor then matters less.
3. Regularization. Give the model a penalty for being complicated, so it only adds detail that pays off. That's the next lesson.
4. Ensembles. Train many models and average their answers. Each overfit tree makes its own random mistakes, and averaging cancels many of them out. A random forest works this way. The lesson on random forests and boosting covers it.
5. Early stopping. Some models learn step by step, in passes over the training data. Watch the validation score as they train. Stop when it stops improving.

Underfitting needs the opposite medicine. Try a more flexible model, better features, less regularization, or longer training.

train = [(5, 0), (12, 0), (20, 0), (33, 1), (45, 0), (70, 1), (85, 1), (90, 0)]
val = [(8, 0), (25, 0), (50, 1), (65, 1), (80, 1), (95, 1)]

memory = dict(train)                    # learn every training customer by heart

def memorizer(days):
    return memory.get(days, 0)          # a customer it hasn't seen: guess "stays" (0)

def rule(days):
    return 1 if days > 60 else 0        # churn if no order for more than 60 days

def accuracy(model, rows):
    return sum(model(days) == label for days, label in rows) / len(rows)

for name, model in [("memorizer", memorizer), ("rule", rule)]:
    tr, va = accuracy(model, train), accuracy(model, val)
    print(f"{name:<10} train {tr:.2f}  val {va:.2f}  gap {tr - va:.2f}")

The same problem with prompts and LLMs

Overfitting isn't only about trees or numbers. It shows up whenever you tune something on a fixed set of examples.

• Prompts. Say you edit a prompt until all 30 of your eval examples pass. Each fix targets one failing case. In the end the prompt handles those 30 well and may act oddly on the 31st. That's overfitting to those 30 examples. Keep a held-out set that you only check at the end (Phase 1's evals lesson).
• Fine-tuning. Fine-tuning means training an existing model further on your own small dataset. Train for too many passes and the model starts repeating its training examples. It also loses some of its general skill. A rising validation loss, the error on held-back examples, is the warning sign (Phase 6).

For marketers: say you "optimize" a campaign on last month's data, down to the exact hour, audience and phrasing. It may be overfit to last month. Patterns that carry over are usually simpler than the ones that fit history perfectly.