Cross-validation

You end up with $k$ scores, one per fold. Call the score on fold 1 $s_1$. The score on fold 2 is $s_2$, and so on. The cross-validated score is their average:

In plain words: add up the score from every fold, then divide by the number of folds.

You also report the spread: how far the fold scores typically sit from that average. This is the standard deviation from Phase 2. People write the pair as "0.81 ± 0.02". Read it as "0.81 on average, give or take about 0.02". A big spread warns you that a single split could have misled you.

For classification, scikit-learn uses stratified folds by default. A stratified fold keeps the churn rate the same in every fold, so no fold ends up with too few churners.

scores = [0.80, 0.84, 0.78, 0.82, 0.81]   # AUC on each of the 5 folds
k = len(scores)

mean = sum(scores) / k
spread = (sum((s - mean) ** 2 for s in scores) / k) ** 0.5
print(f"CV score: {mean:.2f} ± {spread:.2f}")
print(f"lowest fold {min(scores):.2f}, highest fold {max(scores):.2f}")

scores = [0.70, 0.74, 0.72, 0.76, 0.78]
print(round(sum(scores) / len(scores), 2))

Comparing models with cross-validation

scikit-learn does the rotating for you. cross_val_score takes a model, the data and the number of folds. It returns one score per fold.

The code below compares three models you've met in this phase:

• logistic regression, which turns a weighted sum of the features into a probability;
• a decision tree, depth 4, a flowchart that asks each customer at most four yes/no questions;
• a random forest, 200 trees that vote.

import pandas as pd
import requests
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X, y = df[features], df["churned"]

candidates = {
    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
    "tree, depth 4": DecisionTreeClassifier(max_depth=4, random_state=0),
    "random forest": RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=0),
}
for name, model in candidates.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    print(f"{name:<20} AUC {scores.mean():.3f} ± {scores.std():.3f}   folds: {' '.join(f'{s:.2f}' for s in scores)}")

Logistic regression comes out ahead. The fold scores also show how much a single split could have misled you. The depth-4 tree alone ranges from 0.64 to 0.79.

Keep the preparation inside each fold

Look at the logistic regression line. The scaler sits inside the pipeline. A scaler puts every feature on a similar scale, and it learns each feature's average and spread to do it. A pipeline chains the preparation steps and the model into one object.

cross_val_score refits the whole pipeline in every fold. So the scaler only ever learns from that fold's training part. Suppose you scaled all the data first instead. Then the averages would include the validation fold's customers. A little information about them would leak into training, and the score would come out a bit too rosy.

Tuning with cross-validation

Models have settings you choose yourself, such as how deep a tree may grow. GridSearchCV tries every combination of settings you list. It cross-validates each one. Then it refits the best one on all the data you gave it.

Keep a test set outside the search. The code below first sets aside 20% of the customers as the test set, 120 people. The search only sees the other 80%.

The grid lists 6 depths and 3 values of min_samples_leaf, the smallest number of customers allowed at the end of a branch. That's 18 combinations. With 5 folds each, the search trains 90 models, plus one final refit.

import pandas as pd
import requests
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X_dev, X_test, y_dev, y_test = train_test_split(df[features], df["churned"], test_size=0.2,
                                                random_state=0, stratify=df["churned"])

search = GridSearchCV(DecisionTreeClassifier(random_state=0),
                      {"max_depth": [2, 3, 4, 5, 6, 8], "min_samples_leaf": [1, 5, 20]},
                      cv=5, scoring="roc_auc").fit(X_dev, y_dev)
print("best settings:", search.best_params_)
print(f"best cross-validated AUC: {search.best_score_:.3f}")
print(f"held-out test AUC:        {roc_auc_score(y_test, search.predict_proba(X_test)[:, 1]):.3f}")

Here the test score happens to come out higher than the cross-validated one. With only 120 test customers, either number can be a few points off. The test set's value is that nothing was tuned on it. So it's the honest one to report.

Choosing the splitter

Random folds assume every row is independent of the others. Two common cases break that.

• Grouped data: several rows per customer, store or user. Say one customer's rows land in both training and validation. Then the model is graded partly on people it has already seen. Use GroupKFold, which keeps each group on one side.
• Time-ordered data: for example, forecasting next month from the past. Random folds let the model train on the future. TimeSeriesSplit always validates on data that comes after the training data, as it will in real use.

The cost

k-fold trains the model k times. That's nothing for the models in this phase. For large neural networks it's far too slow. That's why deep learning usually relies on one fixed validation set with lots of data.

For LLM evals, the same idea shows up as reporting uncertainty. Score the prompt on several samples. Or bootstrap the eval set: build many new eval sets by drawing examples at random from the original, with repeats allowed, and see how much the score moves. Then be skeptical of any difference smaller than that spread.