Calibration: can the probability be trusted?
Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.
What it covers6
Why it matters
Every cost rule in this phase multiplies a probability by a number of dollars: contact a customer when the chance of churn times the value saved beats the cost of the call. That arithmetic is only right if a 0.3 means 30%. AUC cannot tell you, because it only looks at the order of the scores. A model can rank perfectly and still say 0.9 for people who stay half the time. This lesson measures how far a probability can be trusted, shows which kinds of model lean which way, and repairs them. The same question matters when an LLM says it is 90% sure.
What you'll be able to do
- Say what it means for a probability to be calibrated, and check it by comparing bands of predicted risk with what happened
- Compute a Brier score by hand and read it beside a no-skill baseline
- Explain why a naive Bayes model is overconfident and a forest is cautious, and see both on a calibration plot
- Fix a miscalibrated model with CalibratedClassifierCV on rows the model did not fit, and check that the ranking did not change
Show the answer
About 60
A probability of 0.3 is a claim: of customers given 0.3, about three in ten churn. 30% of 200 is 60. This lesson is about checking that claim against what happened.
The idea: a probability is a claim, so check it
Purpose. Measure whether the numbers a model gives are probabilities in the everyday sense, and not only scores that sort people.
A model that predicts churn gives each customer a number between 0 and 1. If you only contact the top 50, only the order matters, and AUC measures that. But the moment you multiply the number by a cost, "a 0.3 chance of losing $80 is worth $24", the number itself has to be right. A model is calibrated when its probabilities match what happens: among customers it gives 0.3, about 30% churn.
You check it by sorting predictions into bands and comparing, in each band, the average prediction with the share who really churned. The Brier score is a single number for the whole set: the mean of the squared gaps between each probability and the outcome (1 for churned, 0 for stayed).
In plain words: for each customer, take how far the probability was from what happened, square it, and average. 0 is perfect, and lower is better.
Here is a toy: ten customers. The model said 0.8 for five of them and 0.2 for the other five.
Setup (runs before every cell) · 31 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import pandas as pd import requests from sklearn.ensemble import RandomForestClassifier from sklearn.impute import SimpleImputer from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.naive_bayes import GaussianNB from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"]) numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order", "discount_share", "email_open_rate", "support_tickets", "trail_club"] X, y = df[numeric].astype(float), df["churned"] X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0) X_fit, X_cal, y_fit, y_cal = train_test_split(X_rest, y_rest, test_size=0.4, stratify=y_rest, random_state=0) impute = SimpleImputer(strategy="median") models = { "logistic regression": make_pipeline(impute, StandardScaler(), LogisticRegression(max_iter=1000)), "random forest": make_pipeline( impute, RandomForestClassifier(n_estimators=60, min_samples_leaf=5, random_state=0)), "naive Bayes": make_pipeline(impute, GaussianNB()), } for model in models.values(): model.fit(X_fit, y_fit)
import numpy as np p = np.array([0.8] * 5 + [0.2] * 5) y = np.array([1, 1, 1, 0, 0] + [1, 0, 0, 0, 0]) for band in (0.8, 0.2): rows = p == band print("said", band, "| churned", y[rows].mean(), "| customers", rows.sum()) print("Brier score:", round(float(((p - y) ** 2).mean()), 3))
The 0.2 band is calibrated: it said 0.2 and one in five churned. The 0.8 band is over-confident: it said 0.8 and three in five churned.
Show the answer
Not necessarily: AUC checks the order, not whether 0.9 means 90%
AUC only asks whether churners are scored above stayers. A model whose scores are all between 0.6 and 0.9 can rank perfectly while saying 0.9 for people who churn 50% of the time. Calibration is a separate property.
The picture. The two bands of the toy model, said against what happened. The 0.2 band says 0.2 and gets 0.2. The 0.8 band says 0.8 and gets 0.6. The gap between the two bars in a band is how wrong the probability was.
Calibration is the match between the two bars, in every band.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: three models, one plot
Purpose. Draw the calibration curve for three common models, and read why each one bends the way it does.
The setup above fits three models on 252 customers: a logistic regression, a random forest and a Gaussian naive Bayes. It keeps 168 customers for a later calibration step and 180 for testing. First the band check as a loop, on a toy.
Worked example: three models on the held-back customers
For each model, print its AUC, its Brier score beside a no-skill baseline, and the total of its predicted probabilities beside the number who really churned.
Uses the setup above, 31 lines.
from sklearn.metrics import brier_score_loss, roc_auc_score baseline = brier_score_loss(y_test, np.full(len(y_test), y_fit.mean())) print("customers", len(y_test), "| churned", int(y_test.sum()), "| Brier of 'everyone is average':", round(baseline, 3)) for name, model in models.items(): p = model.predict_proba(X_test)[:, 1] print(f"{name:20} AUC {roc_auc_score(y_test, p):.3f} " f"Brier {brier_score_loss(y_test, p):.3f} " f"predicted churners {p.sum():.1f}")
The three models rank about equally well, with AUCs close to 0.77. Their probabilities are not equally good. The total of a calibrated model's probabilities should be close to the number who really churned, and the Brier score should sit well under the baseline. Naive Bayes stands out: it predicts more churners than there were and has the worst Brier score of the three, though its AUC is as high as the others. A ranking metric would have called the three models a tie.
The curves show where each model goes wrong. calibration_curve cuts the predictions into bands with equal numbers of customers (strategy="quantile") and returns, for each band, the share who churned and the mean prediction:
Uses the setup above, 31 lines.
import matplotlib.pyplot as plt from sklearn.calibration import calibration_curve plt.plot([0, 1], [0, 1], "--", color="gray", label="calibrated") for name, model in models.items(): p = model.predict_proba(X_test)[:, 1] actual, said = calibration_curve(y_test, p, n_bins=5, strategy="quantile") plt.plot(said, actual, marker="o", label=name) print(f"{name:20} said {said.round(2).tolist()}") print(f"{'':20} churned {actual.round(2).tolist()}") plt.xlabel("mean predicted probability, in each band") plt.ylabel("share who churned") plt.legend() plt.show()
A point on the dashed diagonal is calibrated. A point below it is over-confident: the model said more than happened. Naive Bayes sits far below the diagonal at the top: it says about 0.84 for a band where about half churn. Its estimates multiply the evidence of many features as if each were independent, and when the features overlap, the product is pushed toward 0 and 1. The random forest's average of many trees is pulled toward the middle, so its extremes are too cautious. Logistic regression is fit to produce probabilities, and it lands closest to the diagonal.
Show the answer
The model is over-confident: it predicts more churners than there were
A calibrated model's probabilities add up to about the number of positives. 46 against 35 means it says too much risk overall. It is one quick check, and the band plot says where.
Said against happened, in bands
Write band_check(probs, labels, edges). edges is a list of band edges, such as [0, 0.5, 1.0]: band k holds the probabilities from edges[k] up to but not including edges[k + 1] (the last band also includes its upper edge). Return a list with one (mean_said, share_churned) tuple per band, each rounded to 3 places, skipping bands with no customers.
Uses the setup above, 31 lines.
def band_check(probs, labels, edges): ...
p = [0.9, 0.7, 0.2, 0.1, 0.1] y = [1, 1, 0, 0, 1] print(round(sum(p), 1), sum(y))
Show the answer
2.0 3
The probabilities add up to 0.9 + 0.7 + 0.2 + 0.1 + 0.1 = 2.0, so the model expects 2 churners. Three really churned, so on these five customers it under-predicts.
Show the answer
No: calibration changes the numbers but not their order
Calibration maps each score to a better probability in a way that keeps the order (for sigmoid and isotonic methods, at least in the common case). The top 50 and the AUC stay about the same, and the dollar arithmetic improves.
p = [0.5, 0.5, 0.5, 0.5] y = [1, 0, 1, 0] print(sum((a - b) ** 2 for a, b in zip(p, y)) / len(p))
Show the answer
0.25
Every customer misses by exactly 0.5, and 0.5 squared is 0.25, whoever churned. A model that cannot beat 0.25 has learned nothing about who will churn, though it is perfectly calibrated at 0.5.
Find the bug: a Brier score that isn't
brier(probs, labels) should be the mean squared gap between each probability and its label. This version scores the model too kindly for large misses. Fix it.
Uses the setup above, 31 lines.
def brier(probs, labels): return sum(abs(p - y) for p, y in zip(probs, labels)) / len(probs)
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: repair, then check
Purpose. Repair a miscalibrated model on rows it did not learn from, and see what the repair does to the dollar arithmetic.
CalibratedClassifierCV learns a small mapping from a model's scores to probabilities, using customers the model did not fit. method="sigmoid" fits an S-shaped curve with two numbers, which suits small sets. method="isotonic" fits a free staircase, which is more flexible and needs more rows. FrozenEstimator tells scikit-learn to keep the already-fitted model as it is and fit only the mapping on the calibration rows:
Uses the setup above, 31 lines.
from sklearn.calibration import CalibratedClassifierCV from sklearn.frozen import FrozenEstimator from sklearn.metrics import brier_score_loss, roc_auc_score for name in ("naive Bayes", "random forest"): raw = models[name].predict_proba(X_test)[:, 1] print(f"{name:14} raw Brier {brier_score_loss(y_test, raw):.3f} " f"predicted churners {raw.sum():.1f}") for method in ("sigmoid", "isotonic"): cal = CalibratedClassifierCV(FrozenEstimator(models[name]), method=method) p = cal.fit(X_cal, y_cal).predict_proba(X_test)[:, 1] print(f"{'':14} {method:9} Brier {brier_score_loss(y_test, p):.3f} " f"predicted churners {p.sum():.1f} AUC {roc_auc_score(y_test, p):.3f}") print("churned:", int(y_test.sum()))
Both repairs bring the total of the probabilities down to roughly the real count, and the Brier score falls most for naive Bayes. The AUC barely moves, because the mapping keeps the order. Two cautions: fit the repair on customers the model did not train on, otherwise it learns the model's over-confidence on its own training rows and fixes nothing; and keep enough rows in the calibration set, because isotonic regression on a few dozen customers overfits.
Two more places calibration matters. A model trained on rebalanced data, with oversampling or class weights, reports probabilities for a world with far more positives than the real one, so its numbers are too high by design; the fix is a correction or a calibration step, as the lesson on imbalanced classes explains. And a language model's stated confidence, "I'm 90% sure", is a probability like any other, and is checked the same way: group the answers by stated confidence and compare with how often they were right. The lesson on why models sound confident shows why the stated number is rarely trustworthy by default.
Where this turns up in an engineer's week. Any decision rule that multiplies a probability by a cost, a threshold chosen to hit a target rate of alerts, a dashboard that shows a risk percentage to a person, and a model whose scores feed another model. Check the band plot and the total of the probabilities before the numbers are used for anything but ranking.
On the job: keep a calibration plot of the live model on recent outcomes. Calibration drifts when customers change, even when the ranking holds, and it is the first thing a cost rule notices.
Goes deeper in
- Precision, recall and F1: the cost-based threshold that needs these probabilities.
- Naive Bayes: why its independence assumption pushes probabilities to the extremes.
- Random forests and boosting: why averaging many trees pulls probabilities to the middle.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 8 review cards for spaced repetition
Try a quiz question
A model's calibration curve lies below the diagonal at the high end. What does that mean?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write