Stratified split
Splitting data so each part keeps the same mix of classes as the whole, for example the same share of churners in train and test.
What it means
A stratified split divides data into parts, usually training and test sets, so that each part keeps the same mix of classes as the whole. If 20% of your customers churn, then 20% of the training set and 20% of the test set churn too.
A plain random split only gets that right on average. On any one split, chance can leave the test set with too few of a rare class, and the score you get from it will swing with the luck of the draw.
An example with scikit-learn
from sklearn.model_selection import train_test_split labels = [1] * 20 + [0] * 80 # 20 of 100 customers churn rows = list(range(100)) for seed in range(4): _, _, _, plain = train_test_split(rows, labels, test_size=0.2, random_state=seed) _, _, _, strat = train_test_split(rows, labels, test_size=0.2, random_state=seed, stratify=labels) print(f"seed {seed}: random split {sum(plain)} churners in test, stratified {sum(strat)}")
seed 0: random split 5 churners in test, stratified 4 seed 1: random split 2 churners in test, stratified 4 seed 2: random split 6 churners in test, stratified 4 seed 3: random split 3 churners in test, stratified 4
A test set of 20 should hold 4 churners. The random splits hold 5, 2, 6 and 3, so their scores can't be fairly compared. The stratified splits hold 4 every time. In scikit-learn you pass the labels as stratify, and StratifiedKFold does the same for cross-validation.
When to use one, and when not to
Use it for classification, above all when one class is rare or the dataset is small. Don't use it when the order of time matters: to predict next month from past months, split by date, or the model gets to learn from the future. And if the same customer appears on several rows, keep all their rows on one side, or the test set isn't really unseen.
Try it now
Each set answers one question. Mixing their jobs is how inflated scores happen.
Learn it properly
Train, validation and test sets · Hold data back so you can trust your results. About 25 minutes.
Other terms from the same lesson
- data leakage: When information from the test data sneaks into training, so the model looks better than it really is on new data.
- test set: Examples kept aside until the very end to give an honest score on unseen data.
- training set: The examples a model learns from.
- validation set: Held-back examples used to compare and tune models while you work.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.