Capstone: churn prediction

Step 2: from scores to a campaign

The model gives every customer a churn probability. The team needs decisions, not probabilities. Three ideas turn one into the other. You'll try them on ten made-up customers first, then on the real test set.

Here are the ten, riskiest first. The letter $p$ stands for the model's churn probability for one customer. The last row shows who really churned (1) and who stayed (0).

Lift: is the top of the list rich in churners?

Lift compares the churn rate in your top group with the churn rate of everyone.

In plain words: lift says how many times more churners your top group holds than a random group of the same size. A lift of 1 means the model does no better than chance.

Worked example: two of the ten customers churned, so the overall rate is 2 ÷ 10 = 20%. The top five (A to E) hold both churners, so their rate is 2 ÷ 5 = 40%. The lift is 40% ÷ 20% = 2.0. Your top five are twice as rich in churners as the customer base.

Who to contact: expected value

Phase 2 introduced expected value: each outcome's worth times its chance, added up. Saving a churner is worth $80. So an offer to a customer with churn probability $p$ is worth $p \times 80$ dollars on average. It costs $15 whatever happens. Contact the customer when the first number beats the second:

In plain words: send the offer when the customer's chance of leaving, times what keeping them is worth, comes to more than the offer costs.

Worked example: customer C has $p = 0.30$. Then 0.30 × 80 = $24, which beats $15. So you contact C and expect to gain $9 on average. Customer E has $p = 0.15$. Then 0.15 × 80 = $12, which is less than $15. You'd expect to lose $3, so you skip E.

The break-even point is 15 ÷ 80 = 0.1875. Anyone above about a 19% risk gets the offer. Here that's A, B, C and D.

print(0.25 * 80 - 15)

Net value: what would the plan have earned?

Once you know who really churned, you can add up the result. Every churner you reached earns $80. Every offer costs $15, whoever gets it:

• net value = 80 × churners reached − 15 × offers sent.

Worked example: the plan contacts A to D and reaches two churners, A and C. That's 2 × 80 − 4 × 15 = 160 − 60 = $100. Contacting all ten reaches the same two churners but sends ten offers: 2 × 80 − 10 × 15 = 160 − 150 = $10. Most of those extra offers went to people who were staying anyway.

Watch out: this counts every reached churner as a full $80. That's the team's assumption, not a measured fact. Step 4 shows how to measure it.

probs   = [0.70, 0.45, 0.30, 0.22, 0.15, 0.12, 0.08, 0.05, 0.04, 0.02]   # riskiest first
churned = [1,    0,    1,    0,    0,    0,    0,    0,    0,    0]      # 1 = really left
OFFER, VALUE = 15, 80

base_rate = sum(churned) / len(churned)
top5_rate = sum(churned[:5]) / 5
print(f"base rate {base_rate:.0%}, top 5 rate {top5_rate:.0%}, lift {top5_rate / base_rate:.1f}x")

contact = [p * VALUE > OFFER for p in probs]
print("customers to contact:", contact.count(True))

def net_value(chosen):
    reached = sum(c for c, pick in zip(churned, chosen) if pick)
    return VALUE * reached - OFFER * sum(chosen)

print("model plan:", net_value(contact))
print("contact everyone:", net_value([True] * 10))

p = 0.18
print(p * 80 > 15)

Step 2, for real: one look at the test set

Now fit the chosen pipeline on the whole development set and score the test set, once. Why only once? If you kept checking the test set and adjusting, it would stop being a fair exam. Its score would look better than the model will do on new customers.

The code then answers the business questions directly:

• Lift: order = np.argsort(-p) sorts the test customers riskiest first. The loop prints the churn rate and lift for the top 15, 30 and 50.
• Who to contact: everyone whose expected value is positive, p × $80 > $15.
• Value: realized is the net value from above. It compares the model's plan with contacting everyone and with today's 90-day rule.

import numpy as np
import pandas as pd
import requests
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]
X_dev, X_test, y_dev, y_test = train_test_split(df[numeric + categorical], df["churned"], test_size=0.25,
                                                random_state=42, stratify=df["churned"])
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), numeric),
    ("cat", make_pipeline(SimpleImputer(missing_values="", strategy="constant", fill_value="unknown"),
                          OneHotEncoder(handle_unknown="ignore")), categorical),
])
model = Pipeline([("prep", prep), ("model", LogisticRegression(C=0.1, max_iter=1000))]).fit(X_dev, y_dev)
p = model.predict_proba(X_test)[:, 1]
y = y_test.to_numpy()
print(f"test AUC: {roc_auc_score(y, p):.3f}\n")

order = np.argsort(-p)
for k in [15, 30, 50]:
    rate = y[order[:k]].mean()
    print(f"top {k:>2} by risk: churn rate {rate:.0%} (lift {rate / y.mean():.1f}x), {int(y[order[:k]].sum())} of {int(y.sum())} churners")

OFFER, VALUE = 15, 80
def realized(contacted):
    return VALUE * int(y[contacted].sum()) - OFFER * int(contacted.sum())
plans = {
    "contact everyone": np.ones(len(y), dtype=bool),
    "today's 90-day rule": X_test["days_since_last_order"].to_numpy() > 90,
    "model: p x $80 > $15": p * VALUE > OFFER,
}
print()
for name, contacted in plans.items():
    print(f"{name:<22} contacts {int(contacted.sum()):>3}   net value on the test set: ${realized(contacted):>5}")

On the test customers, the model's top-ranked groups churn at well over twice the base rate. The expected-value plan earns far more than either alternative. Contacting everyone barely breaks even, because most offers go to customers who were staying anyway. It's the ten-customer story again, at full size.

Step 3: what drives churn, and can the probabilities be trusted?

Which columns does the model lean on?

To find out which ingredient matters in a recipe, you could swap one for something random and taste the result. Permutation importance does that with columns. It shuffles one column at a time in the test set, so that column's values land on the wrong customers. Then it measures how much the AUC drops. If the model leans on a column, scrambling it hurts. If the model ignores a column, the score barely moves. The code shuffles each column 10 times and averages the drops, to smooth out luck.

Does a 30% risk mean 30%?

The contact rule multiplies probabilities by dollars. So the probabilities have to mean what they say. Among customers given about a 30% risk, about 30% should churn. A model like that is called calibrated. A weather forecast works the same way: on days it says "30% chance of rain", it should rain about 30% of the time.

Ranking well and being calibrated are different things. A model could put customers in exactly the right order, with a high AUC, while every probability is twice too high.

The calibration table groups the test customers into bands by predicted risk: up to 10%, 10% to 20%, 20% to 40%, and above 40%. For each band it shows the number of customers, the average prediction, and the share who really churned.

import pandas as pd
import requests
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]
X_dev, X_test, y_dev, y_test = train_test_split(df[numeric + categorical], df["churned"], test_size=0.25,
                                                random_state=42, stratify=df["churned"])
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), numeric),
    ("cat", make_pipeline(SimpleImputer(missing_values="", strategy="constant", fill_value="unknown"),
                          OneHotEncoder(handle_unknown="ignore")), categorical),
])
model = Pipeline([("prep", prep), ("model", LogisticRegression(C=0.1, max_iter=1000))]).fit(X_dev, y_dev)

imp = permutation_importance(model, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0)
ranked = pd.Series(imp.importances_mean, index=X_test.columns).sort_values(ascending=False)
print("AUC lost when each column is shuffled:")
print(ranked.head(5).round(3).to_string())

p = model.predict_proba(X_test)[:, 1]
table = pd.DataFrame({"predicted": p, "churned": y_test.to_numpy()})
table["band"] = pd.cut(table["predicted"], [0, 0.1, 0.2, 0.4, 1.0])
print("\ncalibration on the test set:")
print(table.groupby("band", observed=True).agg(customers=("churned", "size"),
      predicted=("predicted", "mean"), actual=("churned", "mean")).round(2).to_string())

Recency and order frequency carry most of the signal, followed by loyalty-club membership. Customers who are drifting away show it in their buying first.

Calibration looks reasonable at the low and high ends. The middle band churned more than predicted. But that band holds only 26 customers, so the gap could be noise. Re-check it on more data before you use the probabilities for revenue forecasts.

Step 4: what the model can't tell you

• Who will leave isn't who an offer will save. Think of the café again. Some regulars who stop coming have moved away, and no voucher brings them back. Others would have come back anyway. The voucher only matters for the people in between. Finding the people an offer really changes is called uplift modeling, and it needs data from a past campaign.
• So test the campaign. Take a random slice of the customers you'd contact, and don't send them the offer. This group is the holdout. Compare churn between the two groups. The difference measures what the campaign really did, and that number replaces the assumed $80.
• Models change the world they predict. Once offers go out, next year's churn labels reflect the campaign. Retrain with that in mind, and keep the holdout group.
• Watch for drift. Track the spread of risk scores and the real churn rate every month. Retrain when they move.
• Use the data responsibly. Stick to features customers would expect a retailer to use. Check that the contact list isn't skewed against a group in ways you can't justify.

The one-paragraph summary

Here's what you'd send the marketing director:

We built a churn model on 600 customers, and it ranks risk better than our 90-day rule. On customers it had never seen, its highest-risk group churned at more than twice the average rate. Contacting customers whose expected value is positive would have earned about 70% more than the 90-day rule. It would also have earned far more than contacting everyone. Those figures assume a $15 offer and an $80 value for each would-be churner reached. Recency and order frequency drive the predictions. Next step: run the campaign with a 20% holdout group to measure the real effect.