Index Ai Academy
Phase 3 · Evaluation

Calibration: can the probability be trusted?

Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.

About 30 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Every cost rule in this phase multiplies a probability by a number of dollars: contact a customer when the chance of churn times the value saved beats the cost of the call. That arithmetic is only right if a 0.3 means 30%. AUC cannot tell you, because it only looks at the order of the scores. A model can rank perfectly and still say 0.9 for people who stay half the time. This lesson measures how far a probability can be trusted, shows which kinds of model lean which way, and repairs them. The same question matters when an LLM says it is 90% sure.

What you'll be able to do

  • Say what it means for a probability to be calibrated, and check it by comparing bands of predicted risk with what happened
  • Compute a Brier score by hand and read it beside a no-skill baseline
  • Explain why a naive Bayes model is overconfident and a forest is cautious, and see both on a calibration plot
  • Fix a miscalibrated model with CalibratedClassifierCV on rows the model did not fit, and check that the ranking did not change
Quick check
A guess first. A model gives 200 customers a churn probability of 0.3. If the probabilities can be trusted, about how many of those 200 should churn?
  1. About 60
  2. About 140
  3. About 100
  4. About 30
Show the answer

About 60

A probability of 0.3 is a claim: of customers given 0.3, about three in ten churn. 30% of 200 is 60. This lesson is about checking that claim against what happened.

The idea: a probability is a claim, so check it

Purpose. Measure whether the numbers a model gives are probabilities in the everyday sense, and not only scores that sort people.

A model that predicts churn gives each customer a number between 0 and 1. If you only contact the top 50, only the order matters, and AUC measures that. But the moment you multiply the number by a cost, "a 0.3 chance of losing $80 is worth $24", the number itself has to be right. A model is calibrated when its probabilities match what happens: among customers it gives 0.3, about 30% churn.

You check it by sorting predictions into bands and comparing, in each band, the average prediction with the share who really churned. The Brier score is a single number for the whole set: the mean of the squared gaps between each probability and the outcome (1 for churned, 0 for stayed).

Brier=1n∑i(pi−yi)2\text{Brier} = \frac{1}{n}\sum_i (p_i - y_i)^2

In plain words: for each customer, take how far the probability was from what happened, square it, and average. 0 is perfect, and lower is better.

Here is a toy: ten customers. The model said 0.8 for five of them and 0.2 for the other five.

Setup (runs before every cell) · 31 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import pandas as pd
import requests
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value",
           "days_since_last_order", "discount_share", "email_open_rate",
           "support_tickets", "trail_club"]
X, y = df[numeric].astype(float), df["churned"]
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.3,
                                                  stratify=y, random_state=0)
X_fit, X_cal, y_fit, y_cal = train_test_split(X_rest, y_rest, test_size=0.4,
                                              stratify=y_rest, random_state=0)
impute = SimpleImputer(strategy="median")
models = {
    "logistic regression": make_pipeline(impute, StandardScaler(),
                                         LogisticRegression(max_iter=1000)),
    "random forest": make_pipeline(
        impute, RandomForestClassifier(n_estimators=60, min_samples_leaf=5,
                                       random_state=0)),
    "naive Bayes": make_pipeline(impute, GaussianNB()),
}
for model in models.values():
    model.fit(X_fit, y_fit)
import numpy as np

p = np.array([0.8] * 5 + [0.2] * 5)
y = np.array([1, 1, 1, 0, 0] + [1, 0, 0, 0, 0])
for band in (0.8, 0.2):
    rows = p == band
    print("said", band, "| churned", y[rows].mean(), "| customers", rows.sum())
print("Brier score:", round(float(((p - y) ** 2).mean()), 3))
It runs right here in your browser, with nothing to install.

The 0.2 band is calibrated: it said 0.2 and one in five churned. The 0.8 band is over-confident: it said 0.8 and three in five churned.

Quick check
From the lesson on ROC curves and AUC: a model gives churners higher scores than stayers every time, so its AUC is 1.0, but all its scores are between 0.6 and 0.9. Is it calibrated?
  1. No: AUC of 1.0 means the model is overfit
  2. Yes: a perfect ranking means the probabilities are right
  3. Not necessarily: AUC checks the order, not whether 0.9 means 90%
  4. Yes: scores between 0.6 and 0.9 are always probabilities
Show the answer

Not necessarily: AUC checks the order, not whether 0.9 means 90%

AUC only asks whether churners are scored above stayers. A model whose scores are all between 0.6 and 0.9 can rank perfectly while saying 0.9 for people who churn 50% of the time. Calibration is a separate property.

The picture. The two bands of the toy model, said against what happened. The 0.2 band says 0.2 and gets 0.2. The 0.8 band says 0.8 and gets 0.6. The gap between the two bars in a band is how wrong the probability was.

Try it now

Calibration is the match between the two bars, in every band.

In your own words
In your own words: what does it mean for a probability to be calibrated, and why can't AUC tell you?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: three models, one plot

Purpose. Draw the calibration curve for three common models, and read why each one bends the way it does.

The setup above fits three models on 252 customers: a logistic regression, a random forest and a Gaussian naive Bayes. It keeps 168 customers for a later calibration step and 180 for testing. First the band check as a loop, on a toy.

Worked example

Worked example: three models on the held-back customers

For each model, print its AUC, its Brier score beside a no-skill baseline, and the total of its predicted probabilities beside the number who really churned.

Uses the setup above, 31 lines.

from sklearn.metrics import brier_score_loss, roc_auc_score

baseline = brier_score_loss(y_test, np.full(len(y_test), y_fit.mean()))
print("customers", len(y_test), "| churned", int(y_test.sum()),
      "| Brier of 'everyone is average':", round(baseline, 3))
for name, model in models.items():
    p = model.predict_proba(X_test)[:, 1]
    print(f"{name:20} AUC {roc_auc_score(y_test, p):.3f}  "
          f"Brier {brier_score_loss(y_test, p):.3f}  "
          f"predicted churners {p.sum():.1f}")
In the lesson you can change this and run it.

The three models rank about equally well, with AUCs close to 0.77. Their probabilities are not equally good. The total of a calibrated model's probabilities should be close to the number who really churned, and the Brier score should sit well under the baseline. Naive Bayes stands out: it predicts more churners than there were and has the worst Brier score of the three, though its AUC is as high as the others. A ranking metric would have called the three models a tie.

The curves show where each model goes wrong. calibration_curve cuts the predictions into bands with equal numbers of customers (strategy="quantile") and returns, for each band, the share who churned and the mean prediction:

Uses the setup above, 31 lines.

import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

plt.plot([0, 1], [0, 1], "--", color="gray", label="calibrated")
for name, model in models.items():
    p = model.predict_proba(X_test)[:, 1]
    actual, said = calibration_curve(y_test, p, n_bins=5, strategy="quantile")
    plt.plot(said, actual, marker="o", label=name)
    print(f"{name:20} said {said.round(2).tolist()}")
    print(f"{'':20} churned {actual.round(2).tolist()}")
plt.xlabel("mean predicted probability, in each band")
plt.ylabel("share who churned")
plt.legend()
plt.show()

A point on the dashed diagonal is calibrated. A point below it is over-confident: the model said more than happened. Naive Bayes sits far below the diagonal at the top: it says about 0.84 for a band where about half churn. Its estimates multiply the evidence of many features as if each were independent, and when the features overlap, the product is pushed toward 0 and 1. The random forest's average of many trees is pulled toward the middle, so its extremes are too cautious. Logistic regression is fit to produce probabilities, and it lands closest to the diagonal.

Quick check
A model's predicted probabilities for 100 customers add up to 46, and 35 of them churned. What does that suggest?
  1. The model is over-confident: it predicts more churners than there were
  2. The model is under-confident: it predicts fewer churners than there were
  3. The model has an AUC of 0.46 on these customers
  4. The model is calibrated: the total matches the count of churners
Show the answer

The model is over-confident: it predicts more churners than there were

A calibrated model's probabilities add up to about the number of positives. 46 against 35 means it says too much risk overall. It is one quick check, and the band plot says where.

Drill

Said against happened, in bands

Write band_check(probs, labels, edges). edges is a list of band edges, such as [0, 0.5, 1.0]: band k holds the probabilities from edges[k] up to but not including edges[k + 1] (the last band also includes its upper edge). Return a list with one (mean_said, share_churned) tuple per band, each rounded to 3 places, skipping bands with no customers.

Uses the setup above, 31 lines.

def band_check(probs, labels, edges):
    ...
In the lesson, your code is checked as you write it.
Quick check
Five customers get these probabilities, and the labels say who churned. What does this print: the total of the probabilities, and the number who churned?
p = [0.9, 0.7, 0.2, 0.1, 0.1]
y = [1, 1, 0, 0, 1]
print(round(sum(p), 1), sum(y))
  1. 0.4 3
  2. 2.0 3
  3. 2.0 2
  4. 3.0 2
Show the answer

2.0 3

The probabilities add up to 0.9 + 0.7 + 0.2 + 0.1 + 0.1 = 2.0, so the model expects 2 churners. Three really churned, so on these five customers it under-predicts.

Quick check
A model is calibrated, and it is used to find the top 50 customers to call. Does the calibration change who is in the top 50?
  1. Yes: calibrated models always pick different customers
  2. No: a calibrated model has no top 50
  3. Yes: it makes the AUC rise
  4. No: calibration changes the numbers but not their order
Show the answer

No: calibration changes the numbers but not their order

Calibration maps each score to a better probability in a way that keeps the order (for sigmoid and isotonic methods, at least in the common case). The top 50 and the AUC stay about the same, and the dollar arithmetic improves.

Quick check
A model says 0.5 for every customer, and 2 of 4 churn. What does this print: its Brier score?
p = [0.5, 0.5, 0.5, 0.5]
y = [1, 0, 1, 0]
print(sum((a - b) ** 2 for a, b in zip(p, y)) / len(p))
  1. 1.0
  2. 0.25
  3. 0.5
  4. 0.0
Show the answer

0.25

Every customer misses by exactly 0.5, and 0.5 squared is 0.25, whoever churned. A model that cannot beat 0.25 has learned nothing about who will churn, though it is perfectly calibrated at 0.5.

Drill

Find the bug: a Brier score that isn't

brier(probs, labels) should be the mean squared gap between each probability and its label. This version scores the model too kindly for large misses. Fix it.

Uses the setup above, 31 lines.

def brier(probs, labels):
    return sum(abs(p - y) for p, y in zip(probs, labels)) / len(probs)
In the lesson, your code is checked as you write it.
In your own words
Naive Bayes ranks as well as the other models but has the worst Brier score. Why can those two things both be true?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: repair, then check

Purpose. Repair a miscalibrated model on rows it did not learn from, and see what the repair does to the dollar arithmetic.

CalibratedClassifierCV learns a small mapping from a model's scores to probabilities, using customers the model did not fit. method="sigmoid" fits an S-shaped curve with two numbers, which suits small sets. method="isotonic" fits a free staircase, which is more flexible and needs more rows. FrozenEstimator tells scikit-learn to keep the already-fitted model as it is and fit only the mapping on the calibration rows:

Uses the setup above, 31 lines.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from sklearn.metrics import brier_score_loss, roc_auc_score

for name in ("naive Bayes", "random forest"):
    raw = models[name].predict_proba(X_test)[:, 1]
    print(f"{name:14} raw       Brier {brier_score_loss(y_test, raw):.3f}  "
          f"predicted churners {raw.sum():.1f}")
    for method in ("sigmoid", "isotonic"):
        cal = CalibratedClassifierCV(FrozenEstimator(models[name]), method=method)
        p = cal.fit(X_cal, y_cal).predict_proba(X_test)[:, 1]
        print(f"{'':14} {method:9} Brier {brier_score_loss(y_test, p):.3f}  "
              f"predicted churners {p.sum():.1f}  AUC {roc_auc_score(y_test, p):.3f}")
print("churned:", int(y_test.sum()))

Both repairs bring the total of the probabilities down to roughly the real count, and the Brier score falls most for naive Bayes. The AUC barely moves, because the mapping keeps the order. Two cautions: fit the repair on customers the model did not train on, otherwise it learns the model's over-confidence on its own training rows and fixes nothing; and keep enough rows in the calibration set, because isotonic regression on a few dozen customers overfits.

Two more places calibration matters. A model trained on rebalanced data, with oversampling or class weights, reports probabilities for a world with far more positives than the real one, so its numbers are too high by design; the fix is a correction or a calibration step, as the lesson on imbalanced classes explains. And a language model's stated confidence, "I'm 90% sure", is a probability like any other, and is checked the same way: group the answers by stated confidence and compare with how often they were right. The lesson on why models sound confident shows why the stated number is rarely trustworthy by default.

Where this turns up in an engineer's week. Any decision rule that multiplies a probability by a cost, a threshold chosen to hit a target rate of alerts, a dashboard that shows a risk percentage to a person, and a model whose scores feed another model. Check the band plot and the total of the probabilities before the numbers are used for anything but ranking.

On the job: keep a calibration plot of the live model on recent outcomes. Calibration drifts when customers change, even when the ranking holds, and it is the first thing a cost rule notices.

Goes deeper in

In your own words
Your model has a good AUC, and the finance team wants to use its probabilities to forecast how many customers will churn next quarter. What do you check first?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 8 review cards for spaced repetition

Try a quiz question

A model's calibration curve lies below the diagonal at the high end. What does that mean?

  1. It is over-confident: for the riskiest customers it says more than happens
  2. It is under-confident: for the riskiest customers it says less than happens
  3. Its AUC is below 0.5, so it ranks customers backwards
  4. It has no skill at all and is no better than a coin flip

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write