Explaining a model: what it uses, and why this prediction
Permutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.
What it covers6
Why it matters
A churn model that scores 0.80 AUC raises two questions the score cannot answer. What is it using? And for this one customer, why is the risk so high? The first decides whether you trust the model and what you change in the business. The second decides what the account manager says on the phone. Both have cheap, honest answers for a tabular model, and both have well-known ways to go wrong. This lesson builds the three standard tools by hand and in scikit-learn, and ends with what they cannot tell you.
What you'll be able to do
- Measure what a model relies on by shuffling one column of held-back rows and watching the score drop
- Explain why impurity importances mislead on a random column, and why permutation importance does not
- Read one prediction from a linear model as bias plus weight times value, feature by feature
- State what an importance score cannot say: cause, and how credit is split between correlated columns
Show the answer
The model does not rely on that column
If scrambling a column leaves the score where it was, the model was not using it. This idea, shuffle and measure the drop, is the whole of permutation importance.
The idea: what a model uses, and why this prediction
Purpose. Answer two questions about a model you did not hand-build: which columns does it lean on, and what drove this one score?
There are two different questions, and two kinds of answer.
Across all customers: how much does the model rely on a column? Take rows the model did not train on and score them. Shuffle one column, so each customer gets another customer's value, and score again. A column the model leaned on now feeds it nonsense, and the score falls. The size of the fall is the column's permutation importance. It uses only predictions, so it works for any model.
For one customer: what pushed the score up or down? For a model that is a weighted sum, this has an exact answer. The score is a bias plus a weight times a value for each feature, so each term is that feature's contribution:
In plain words: a customer's score is a starting point plus one push per feature, and each push is that feature's weight times how far this customer is from average on it.
A third tool, partial dependence, answers a different question again: how does the average prediction change as one feature moves, with every customer's other values left as they are? Here is a toy customer with three standardized values and three weights:
Setup (runs before every cell) · 25 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import pandas as pd import requests from sklearn.ensemble import RandomForestClassifier from sklearn.impute import SimpleImputer from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"]) numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order", "discount_share", "email_open_rate", "support_tickets", "trail_club"] df["random_id"] = np.random.default_rng(0).integers(0, 10000, len(df)) X = df[numeric + ["random_id"]].astype(float) y = df["churned"] X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0) forest = make_pipeline( SimpleImputer(strategy="median"), RandomForestClassifier(n_estimators=60, min_samples_leaf=5, random_state=0), ).fit(X_dev, y_dev) logit = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_dev[numeric], y_dev)
import math weights = {"days since last order": 0.8, "tenure": -0.5, "tickets": 0.3} values = {"days since last order": 1.5, "tenure": -1.0, "tickets": 2.0} bias = -1.0 parts = {name: round(weights[name] * values[name], 2) for name in weights} score = round(bias + sum(parts.values()), 2) print(parts) print("score", score, "chance", round(1 / (1 + math.exp(-score)), 3))
import math z = 0.0 print(round(1 / (1 + math.exp(-z)), 2), round(1 / (1 + math.exp(-2.0)), 2))
Show the answer
0.5 0.88
A score of 0 means even odds, 0.5. A score of 2 is clearly on the churn side: 1 / (1 + e^-2) is 0.88. The contributions add into z, and z becomes the chance.
The picture. The toy customer's score, one push per feature. The bars are the contributions from the cell above: 0.8 × 1.5 = 1.2, −0.5 × −1.0 = 0.5 and 0.3 × 2.0 = 0.6. The bias, −1.0, pulls the total down from there.
A prediction explained by a model that is a sum: every term can be read.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: contributions, permutation and partial dependence
Purpose. Run each tool on the shop's customers, and see how the usual shortcut, the forest's own importance, goes wrong.
The setup above loads the 600 customers, adds a column of random whole numbers called random_id that cannot predict anything, holds back 25% of the customers for testing, and fits two models: a logistic regression and a random forest.
One customer, from the logistic model
The model standardizes the columns, then adds weight times value. The next cell takes the held-out customer the model rates as the riskiest and lists the three biggest pushes.
Worked example: why is this customer flagged?
Break the riskiest held-out customer's score into one push per feature, and check that the pushes rebuild the model's own chance.
Uses the setup above, 25 lines.
import numpy as np risk = logit.predict_proba(X_test[numeric])[:, 1] i = int(risk.argmax()) row = logit[:-1].transform(X_test[numeric])[i] parts = logit[-1].coef_[0] * row for j in np.argsort(-np.abs(parts))[:3]: print(f"{numeric[j]:24} {parts[j]:+.2f}") score = parts.sum() + logit[-1].intercept_[0] chance = 1 / (1 + np.exp(-score)) print("score", round(float(score), 2), "chance", round(float(chance), 3)) print("model says", round(float(risk[i]), 3))
The last two numbers agree: the explanation is not an approximation, it is the model's own arithmetic. The biggest pushes are a long gap since the last order and few orders in the past year. An account manager can say exactly that: "this customer hasn't ordered in a long time, and ordered rarely before that". Each push is a weight times a standardized value, so a push says how far this customer is from the average one on that feature, scaled by how much the model cares.
Code flow. The sum above is a loop over features. Step through it with the toy numbers: the score starts at the bias, and each feature adds its weight times its value.
import math z = -2 + 1.5 + 0.5 - 0.25 print(z, round(1 / (1 + math.exp(-z)), 3))
Show the answer
-0.25 0.438
The contributions and the bias add to −0.25, a score slightly on the stay side. The sigmoid of −0.25 is 0.438. A feature with a negative contribution pulled the score toward staying.
Across all customers: permutation against impurity
A random forest keeps its own importance numbers, from how much each column reduced impurity when the trees split on it. They are free, and they have a known fault: a column with many different values gives the trees many chances to split, even when it means nothing. The column random_id was made for this test. Compare the two kinds of importance on rows the forest never saw:
Uses the setup above, 25 lines.
import matplotlib.pyplot as plt from sklearn.inspection import permutation_importance impurity = forest[-1].feature_importances_ perm = permutation_importance(forest, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0) for name, a, b in zip(X.columns, impurity, perm.importances_mean): print(f"{name:24} impurity {a:.3f} permutation {b:.3f}") plt.barh(list(X.columns), perm.importances_mean) plt.xlabel("drop in test AUC when the column is shuffled") plt.show()
The impurity column ranks the random ID above several real features: it gave the trees hundreds of places to split, and some splits looked like a gain by chance. The permutation column puts it at zero, because shuffling it changes no prediction on held-back rows. Two real columns, days since the last order and orders last year, carry nearly all of the drop. Permutation importance is measured on rows the model did not learn from, which is why it cannot reward memorizing a column.
random_id above tenure_months, while permutation importance on held-back rows puts it near zero. What is the best reading?Show the answer
The impurity score rewards columns with many distinct values, even useless ones
A column with many distinct values offers the trees many split points, and some look good by chance on training rows. Shuffling the column on rows the forest never saw shows it was contributing nothing.
Permutation importance of one column
Write drop(model, X, y, column, seed=0): the model's score on X and y, minus its score after the values in column are shuffled. Shuffle a copy of X, so the caller's table is not changed.
Uses the setup above, 25 lines.
import numpy as np def drop(model, X, y, column, seed=0): ...
Sweeping one feature: partial dependence
Importance says how much a column matters. It does not say which way. Partial dependence answers that: for each value in a grid, set the column to that value for every customer, predict, and average. The result is a curve:
Uses the setup above, 25 lines.
import matplotlib.pyplot as plt from sklearn.inspection import partial_dependence pdp = partial_dependence(forest, X_dev, ["days_since_last_order"], grid_resolution=6) grid, average = pdp["grid_values"][0], pdp["average"][0] print(grid.round(0).tolist()) print(average.round(3).tolist()) plt.plot(grid, average, marker="o") plt.xlabel("days since last order") plt.ylabel("average predicted churn") plt.show()
The average predicted chance of churn climbs from 0.15 at 11 days to 0.39 at 294 days, with small dips on the way: a forest's curve is a staircase of many splits, so it is bumpy. A curve like this is the picture to show a non-technical reader. It also has a caveat: it forces every customer to the same value, including combinations that never occur, such as a customer with 180 days since their last order and 20 orders in the past year. When columns move together, read it with care.
days_since_last_order rises from 0.2 at 10 days to 0.6 at 180 days. What does it show?Show the answer
The average predicted churn rises as the gap grows, with other columns left as they are
Each point is the model's average prediction when every customer is set to that gap. It describes the model's behavior, not the world, and not each customer: some customers' chances move far more or less than the average.
Find the bug: shuffling that spreads
drop_all(model, X, y) should return each column's drop in score when only that column is shuffled. But it shuffles the table in place and never puts the column back, so every later column is measured on a table that is already scrambled. Fix it.
Uses the setup above, 25 lines.
import numpy as np def drop_all(model, X, y): base = model.score(X, y) rng = np.random.default_rng(0) out = {} for column in X.columns: X[column] = rng.permutation(X[column].values) out[column] = base - model.score(X, y) return out
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: what an importance does and doesn't tell you
Purpose. Know the limits of an explanation before it goes into a decision or a model card.
What an importance does not say.
- It is not cause. A column the model leans on is a good predictor, not a lever. Days since the last order predicts churn, but sending an email that resets the count would not make a customer stay.
- Correlated columns share credit. If two columns carry the same information, shuffling one leaves the other to fill in, so each looks less important than the pair is. Check how columns move together before reading a ranking, and consider shuffling a group of related columns together.
- It depends on the score. A column can matter for AUC and little for accuracy. Say which score the importance is measured on.
- Contributions explain the model, not the world. They add up exactly for a linear model. For a forest or a boosted model there is no such sum, and a method such as SHAP values gives a fair split of one prediction among the features, at a cost in compute. SHAP is a separate package, so it is only named here.
Show the answer
Importance shows what the model leans on, not what would change the outcome
A column the model leans on is a good predictor. Whether changing it would change churn needs an experiment, because the gap may be a symptom of a customer already leaving.
Where this turns up in an engineer's week. A review asks "why did it flag this account?" and you open the contributions. A new feature lands and you check its permutation importance before keeping it. A model's score drops in production and you compare which columns changed. A model card lists what the model leans on, and a reader uses it to decide where to look when something goes wrong.
On the job: compute importances on held-back rows, add a random column as a control, and report the control's importance. Any real column that does not beat it has not earned a place on the list.
Goes deeper in
- The scikit-learn workflow: the pipelines these explanations were run on.
- Ensembles: why a forest's own importance has the fault shown above.
- Capstone: churn prediction: where a model card for the final model goes.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
A forest's impurity importance ranks a random ID column above several real features. What is the usual cause?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write