The scikit-learn workflow

A pipeline for messy data

The practice shop's customer table is messy in three ways:

• Some number columns have gaps. email_open_rate is blank for people who never subscribed.
• Some text columns have blanks. region is sometimes an empty string.
• Some columns hold words, like "west" or "referral". A model can only do sums with numbers.

You met a fix for each of these in Preparing data. Here you wire the fixes together. Two tools do the wiring:

• A ColumnTransformer works like a mail sorter. It sends each group of columns down its own chute. Number columns get one set of steps, and word columns get another.
• A Pipeline works like a recipe card. It lists every step in order, ending with the model. The whole card then acts as one tool, with its own fit and predict.

import pandas as pd
import requests
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]

prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), numeric),
    ("cat", make_pipeline(SimpleImputer(missing_values="", strategy="constant", fill_value="unknown"),
                          OneHotEncoder(handle_unknown="ignore")), categorical),
])
pipe = Pipeline([("prep", prep), ("model", LogisticRegression(max_iter=1000))])

X, y = df[numeric + categorical], df["churned"]
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
scores = cross_val_score(pipe, X_dev, y_dev, cv=5, scoring="roc_auc")
print(f"cross-validated AUC: {scores.mean():.3f} ± {scores.std():.3f}")

pipe.fit(X_dev, y_dev)
print("columns the model sees:", len(pipe.named_steps["prep"].get_feature_names_out()))

Here's what the code does, in order:

1. It loads the customers. It lists the number columns (numeric) and the word columns (categorical).
2. prep is the mail sorter. make_pipeline builds a short chain of steps for each chute.
3. pipe is the recipe card: preparation first, then a logistic regression model.
4. train_test_split locks away 20% of the customers as a test set. The other 80%, called X_dev and y_dev here, is your working data. stratify=y keeps the share of customers who left about the same on both sides.
5. cross_val_score runs 5-fold cross-validation on the working data. It prints the average AUC. After the ± sign comes the spread of the five fold scores. AUC measures how well the model ranks leavers above stayers: 1.0 is perfect and 0.5 is a coin flip.
6. Last, it fits the pipeline on all the working data and counts the columns the model receives.

That count is bigger than the 11 columns you started with. The steps below explain why.

What each chute does, learned from training rows only:

• Numbers: fill gaps with the median, add a 0/1 "was missing" column, then standardize. Standardizing puts every column on the same scale. Then order values in dollars and shares between 0 and 1 can sit side by side. The flag is there because a gap can itself be a signal. Never subscribing to email might predict that a customer leaves.
• Words: treat blanks as "unknown", then one-hot encode. One-hot encoding gives each category its own 0/1 column. handle_unknown="ignore" deals with surprises. Say a new sales channel appears next quarter, one the model never saw in training. Its row gets all zeros in the channel columns instead of an error.

Why bother making it one object? Because cross_val_score now refits every step inside every fold. The medians, the scaling and the list of categories are all relearned from that fold's training rows. Nothing from the validation rows leaks in. Leakage is when information from the rows you test on sneaks into training, so your score looks better than it really is. Later, when you predict, there's no step to forget. The pipeline runs them all.

Reaching a setting inside a pipeline

A pipeline holds several steps, and each step has its own settings. To reach a setting inside a step, join the step's name and the setting's name with two underscores: step__setting. So model__C means "the C of the step called model". It works like a folder path, such as model/C, with __ in place of the slash.

As a reminder from Regularization, C sets the penalty in logistic regression. A smaller C means a stronger penalty and a simpler model.

Tune, compare, test once, save

GridSearchCV tries every setting you list. It cross-validates each one and keeps the best. You can even search over which model fills the "model" step. The code below compares three kinds of model:

• logistic regression, with three values of C;
• a random forest, with two values of min_samples_leaf, the fewest customers allowed in each leaf of a tree;
• boosted trees (HistGradientBoostingClassifier), with two values of max_depth, how many questions deep each tree can go.

Then it scores the winner once on the locked test set. Last, it saves the winner to a file, loads it back and scores two new customers.

import joblib
import pandas as pd
import requests
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), numeric),
    ("cat", make_pipeline(SimpleImputer(missing_values="", strategy="constant", fill_value="unknown"),
                          OneHotEncoder(handle_unknown="ignore")), categorical),
])
pipe = Pipeline([("prep", prep), ("model", LogisticRegression(max_iter=1000))])
X_dev, X_test, y_dev, y_test = train_test_split(df[numeric + categorical], df["churned"], test_size=0.2,
                                                random_state=0, stratify=df["churned"])

grid = [
    {"model": [LogisticRegression(max_iter=1000)], "model__C": [0.01, 0.1, 1]},
    {"model": [RandomForestClassifier(n_estimators=100, random_state=0)], "model__min_samples_leaf": [5, 20]},
    {"model": [HistGradientBoostingClassifier(random_state=0)], "model__max_depth": [2, 3]},
]
search = GridSearchCV(pipe, grid, cv=5, scoring="roc_auc").fit(X_dev, y_dev)
for params, score in zip(search.cv_results_["params"], search.cv_results_["mean_test_score"]):
    setting = {k: v for k, v in params.items() if k != "model"}
    print(f"{type(params['model']).__name__:<32} {setting}  AUC {score:.3f}")
print(f"\nwinner's test AUC (scored once): {roc_auc_score(y_test, search.predict_proba(X_test)[:, 1]):.3f}")

joblib.dump(search.best_estimator_, "churn_model.joblib")
model = joblib.load("churn_model.joblib")
new_customers = pd.DataFrame([
    {"tenure_months": 3, "orders_last_year": 1, "avg_order_value": 55.0, "days_since_last_order": 140,
     "discount_share": 0.4, "email_open_rate": None, "support_tickets": 2, "trail_club": 0,
     "region": "", "channel": "podcast", "top_category": "home"},
    {"tenure_months": 40, "orders_last_year": 7, "avg_order_value": 130.0, "days_since_last_order": 12,
     "discount_share": 0.05, "email_open_rate": 0.4, "support_tickets": 0, "trail_club": 1,
     "region": "west", "channel": "referral", "top_category": "apparel"},
])
print("churn probability for two new customers:", model.predict_proba(new_customers)[:, 1].round(2))

Each of the first lines is one setting and its cross-validated AUC. Next comes the winner's AUC on the test set. That's the number you report, because the model never saw those rows while you were choosing. In predict_proba(...)[:, 1], the [:, 1] picks the second column of probabilities: the chance of leaving. The first column is the chance of staying.

Logistic regression with a strong penalty wins here. The dataset is small, and each feature mostly pushes churn risk up or down on its own. The more flexible tree models find nothing extra here: they score a few points lower, so they don't pay off. Some larger datasets have features that work together: the effect of one depends on another. On data like that, boosted trees often come first. That's why it's worth searching across models, not only across settings.

The first new customer is a stress test. Their open rate is missing and their region is blank. Their channel, "podcast", never appeared in training. The pipeline handles all three with no extra code.

Saving a model safely

joblib.dump writes the whole fitted pipeline to a file, and joblib.load reads it back. Two cautions:

• Only load files you trust. Loading a joblib or pickle file can run code hidden inside it. Treat a model file from a stranger like a program from a stranger.
• Pin the library versions. A model saved with one version of scikit-learn may not load under another. Or it may load and behave differently. Write down the exact versions next to the file, and use those same versions wherever you load it.

The workflow, start to finish

1. Split off a test set first, and don't look at it.
2. Put every preparation step and the model in one pipeline.
3. Cross-validate on the rest to compare models and tune settings.
4. Score the winner once on the test set, and report that number.
5. If you like, refit on all the labeled data. Then save it with its versions, and score new rows through the same pipeline.
6. Monitor it. Customers change over time, and this slow change is called drift. A churn model trained last year slowly goes stale.