ROC curves and AUC

What AUC means

AUC is short for "area under the curve": the area under the ROC curve. It runs from 0.5 for random guessing to 1.0 for a perfect model. (A model below 0.5 ranks things backwards.)

AUC also has a direct meaning. Pick one churner at random and one customer who stays. Ask the model which of the two is riskier. AUC is the chance it picks the churner. A tie counts as half right.

Think of a tournament where every churner plays every happy customer once. The model is the referee, and the higher score wins. AUC is the churners' win rate.

Here's a small example. Two churners scored 0.9 and 0.6. Three happy customers scored 0.8, 0.4 and 0.2. That makes 2 × 3 = 6 pairs.

• The churner at 0.9 beats all three happy customers.
• The churner at 0.6 beats 0.4 and 0.2, but loses to 0.8.

That's 5 wins out of 6 pairs, so AUC = 5 / 6 ≈ 0.833.

churners = [0.9, 0.6]
stayers = [0.8, 0.4, 0.2]
wins = sum(c > s for c in churners for s in stayers)
pairs = len(churners) * len(stayers)
print(wins, "of", pairs, "pairs ->", round(wins / pairs, 3))

It prints 5 of 6 pairs -> 0.833.

So AUC measures ranking: does the model put the right people near the front of the line? That's exactly what you need when the plan is "contact the 50 riskiest customers".

The cell below checks this on the real churn test set. It lines up every churner's score against every happy customer's score in one grid. (pos[:, None] > neg[None, :] builds that grid: one row per churner, one column per happy customer.) It takes the share of pairs ranked correctly, adds half the ties, and prints it next to scikit-learn's roc_auc_score. The two numbers match.

import numpy as np
import pandas as pd
import requests
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(0)
features = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
            "discount_share", "email_open_rate", "support_tickets", "trail_club"]
X_tr, X_te, y_tr, y_te = train_test_split(df[features], df["churned"], test_size=0.3,
                                          random_state=0, stratify=df["churned"])
scores = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]

pos, neg = scores[y_te.values == 1], scores[y_te.values == 0]
ranked_right = (pos[:, None] > neg[None, :]).mean() + 0.5 * (pos[:, None] == neg[None, :]).mean()
print(f"{len(pos) * len(neg)} (churner, stayer) pairs; ranked correctly: {ranked_right:.3f}")
print(f"roc_auc_score: {roc_auc_score(y_te, scores):.3f}")

churners = [0.7, 0.3]
stayers = [0.5, 0.3]
pairs = [(c, s) for c in churners for s in stayers]
wins = sum(1 if c > s else 0.5 if c == s else 0 for c, s in pairs)
print(wins / len(pairs))

scores    = [0.95, 0.85, 0.7, 0.5, 0.4, 0.1]   # model's chance each lead buys
converted = [1,    1,    0,   1,   0,   0]     # 1 = really bought
P, N = converted.count(1), converted.count(0)

points = [(0.0, 0.0)]
for t in scores:                               # already sorted, highest first
    tp = sum(1 for s, y in zip(scores, converted) if s >= t and y == 1)
    fp = sum(1 for s, y in zip(scores, converted) if s >= t and y == 0)
    points.append((fp / N, tp / P))
print([(round(x, 2), round(y, 2)) for x, y in points])

area = sum((x2 - x1) * (y1 + y2) / 2 for (x1, y1), (x2, y2) in zip(points, points[1:]))
pos = [s for s, y in zip(scores, converted) if y == 1]
neg = [s for s, y in zip(scores, converted) if y == 0]
wins = sum(a > b for a in pos for b in neg)
print(round(area, 3), round(wins / (P * N), 3))

When positives are rare: precision-recall curves

The false positive rate divides by all the negatives. When negatives vastly outnumber positives, that can hide a lot of false alarms.

Take card fraud, where 1 transaction in 1,000 is fraud. Out of a million transactions, 1,000 are fraud and 999,000 are honest. Say a model catches 900 frauds. It also flags 5,000 honest ones.

• TPR = 900 / 1,000 = 0.9. That looks great.
• FPR = 5,000 / 999,000 ≈ 0.005. That looks tiny.
• Precision = 900 / (900 + 5,000) ≈ 0.15. Fewer than 1 in 6 alerts is real fraud.

frauds, honest = 1000, 999000            # 1 fraud in every 1,000
caught, false_alarms = 900, 5000         # what the model flags
print("TPR", caught / frauds)
print("FPR", round(false_alarms / honest, 4))
print("precision", round(caught / (caught + false_alarms), 2))

It prints TPR 0.9, FPR 0.005 and precision 0.15. The ROC curve looks great while the fraud team drowns in false alarms.

A precision-recall curve shows this problem. It plots precision against recall for every threshold. Its summary number is average precision (average_precision_score).

Watch the baseline. Random guessing scores the positive rate, not 0.5. For the fraud example that's 0.001. Always compare against that number.

Rule of thumb:

• Use ROC AUC to compare rankers on reasonably balanced data.
• Use precision-recall when positives are rare and false alarms are what hurts.

What AUC doesn't tell you

• The threshold. AUC averages over every threshold. You still pick one from what mistakes cost, as in the previous lesson.
• Calibration. A model is calibrated when its probabilities match reality. Think of a weather forecast: on days it says "30% chance of rain", it should rain about 30% of the time. A model can rank perfectly while its probabilities are off. For example, everyone it calls "0.9" might churn only half the time. Sometimes you use the probabilities themselves, for example to work out expected lost revenue. Then check calibration: among customers given about 0.3, about 30% should churn. scikit-learn's CalibratedClassifierCV can fix a model that ranks well but is miscalibrated.
• Whether it ranks well where it matters. Two models with the same AUC can differ at the very top of the list. That's where your 50 calls go.

Beyond churn

AUC works any time a system gives scores and you have the true labels. A few places you'll meet it:

• a lead-scoring model: are buyers scored above non-buyers?
• a search or retrieval system: are relevant documents scored above irrelevant ones?
• an LLM judge's 1–10 ratings, checked against human pass/fail labels.