Train, validation and test sets

Three sets, three jobs

Think of a school course. You learn from the textbook and homework. You check your progress on practice tests, and you change how you study based on them. At the end there's one final exam that you've never seen.

Data gets split the same way, into three parts:

• The training set is the homework. The model learns its parameters from it. Parameters are the numbers inside the model that training adjusts.
• The validation set is the practice tests. You use it to compare choices you make yourself. Examples are how deep the tree may grow or which columns to use. These choices are called settings (or hyperparameters).
• The test set is the final exam. You use it once, at the very end, for the score you report.

Why you need a validation set

Why not compare settings on the test set? Say you try some settings, score them on a set of data, and keep the winner. Each time you do that, you fit a little to that data. Part of the winner's score is luck. It happened to suit those particular rows.

Do that fifty times on the test set and its score is no longer honest. It has become a practice test. So you tune on validation and touch the test set once.

When you have little data, holding back a fixed validation set wastes too much of it. Cross-validation takes its place. It gets its own lesson later.

How to split

There are four common ways to cut your data. Which one you use depends on what the data looks like.

• Random, with a seed. Shuffle the rows and cut. A seed is a starting number for the shuffle. The same seed always gives the same shuffle. In scikit-learn you set it with random_state, so the split is repeatable.
• Stratified. Here about 19% of customers churn and 81% stay. A random cut can leave a test set with too few churners. A stratified split keeps the share of each label the same in every part.
• By time. Say you'll use the model to predict the future. Then train on the past and test on the most recent period. A random split lets the model peek at future patterns. It can't do that once it's in real use.
• By group. Sometimes one customer has many rows, one per order. Keep all of a customer's rows in the same part. Otherwise the model can recognize the customer instead of learning the pattern.

The code below makes small test sets, 10% of the customers each time, five times over. It prints the churn rate in each test set. The first line uses plain random splits. The second uses stratified splits.

import numpy as np
import pandas as pd
import requests
from sklearn.model_selection import train_test_split

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
y = df["churned"]
for strat in [None, y]:
    rates = []
    for seed in range(5):
        _, _, _, y_test = train_test_split(df, y, test_size=0.1, random_state=seed, stratify=strat)
        rates.append(round(float(y_test.mean()), 3))
    print("stratified" if strat is not None else "random    ", rates)

With small test sets, random splits make the churn rate swing around from seed to seed. Stratified splits hold it steady.

rows = list(range(10))
split = int(len(rows) * 0.8)
train, test = rows[:split], rows[split:]
print(len(train), len(test))

import numpy as np

labels = np.array([1] * 4 + [0] * 16)   # 20 customers: 4 churned (1), 16 stayed (0)

counts = []
for seed in range(10):
    order = np.random.default_rng(seed).permutation(20)  # shuffle the positions 0 to 19
    test = order[:5]                                     # the first 5: a 25% test set
    counts.append(int(labels[test].sum()))               # churners that landed in test
print("random split, churners in test:", counts)

# Stratified: take 25% of each group separately
print("stratified: churners", round(4 * 0.25), "+ stayers", round(16 * 0.25))

How test data leaks into training

Data leakage means information from the test data sneaks into training. The model then looks better than it really is on new data. Leaks are easy to miss, because nothing crashes. The score looks good, and that's all you see.

Watch for these:

• Tuning on the test set. Every look that changes a choice leaks a little.
• A random split on time-ordered data. The model trains on days that come after some of the days it's tested on.
• The same customer in both parts. The model recognizes the person instead of the pattern.
• Preparing data before you split. Say you fill missing values with the average of the whole table. That average includes the test rows, so a little of the test set has reached training. Split first. Then work out averages and other fill-in values from the training rows only.

A good habit is to split first, before you explore or clean anything. Then put the test set away until the end.