Index Ai Academy
Phase 3 · The workflow

Explaining a model: what it uses, and why this prediction

Permutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A churn model that scores 0.80 AUC raises two questions the score cannot answer. What is it using? And for this one customer, why is the risk so high? The first decides whether you trust the model and what you change in the business. The second decides what the account manager says on the phone. Both have cheap, honest answers for a tabular model, and both have well-known ways to go wrong. This lesson builds the three standard tools by hand and in scikit-learn, and ends with what they cannot tell you.

What you'll be able to do

  • Measure what a model relies on by shuffling one column of held-back rows and watching the score drop
  • Explain why impurity importances mislead on a random column, and why permutation importance does not
  • Read one prediction from a linear model as bias plus weight times value, feature by feature
  • State what an importance score cannot say: cause, and how credit is split between correlated columns
Quick check
A guess first. You shuffle one column of the held-back customers, so its values no longer match their rows, and the model's score does not change. What does that tell you?
  1. The model is broken
  2. The column is important and the model is robust
  3. The column should be removed from the data
  4. The model does not rely on that column
Show the answer

The model does not rely on that column

If scrambling a column leaves the score where it was, the model was not using it. This idea, shuffle and measure the drop, is the whole of permutation importance.

The idea: what a model uses, and why this prediction

Purpose. Answer two questions about a model you did not hand-build: which columns does it lean on, and what drove this one score?

There are two different questions, and two kinds of answer.

Across all customers: how much does the model rely on a column? Take rows the model did not train on and score them. Shuffle one column, so each customer gets another customer's value, and score again. A column the model leaned on now feeds it nonsense, and the score falls. The size of the fall is the column's permutation importance. It uses only predictions, so it works for any model.

For one customer: what pushed the score up or down? For a model that is a weighted sum, this has an exact answer. The score is a bias plus a weight times a value for each feature, so each term is that feature's contribution:

z=b+∑iwixicontribution of feature i=wixiz = b + \sum_i w_i x_i \qquad \text{contribution of feature } i = w_i x_i

In plain words: a customer's score is a starting point plus one push per feature, and each push is that feature's weight times how far this customer is from average on it.

A third tool, partial dependence, answers a different question again: how does the average prediction change as one feature moves, with every customer's other values left as they are? Here is a toy customer with three standardized values and three weights:

Setup (runs before every cell) · 25 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import pandas as pd
import requests
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value",
           "days_since_last_order", "discount_share", "email_open_rate",
           "support_tickets", "trail_club"]
df["random_id"] = np.random.default_rng(0).integers(0, 10000, len(df))
X = df[numeric + ["random_id"]].astype(float)
y = df["churned"]
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.25,
                                                stratify=y, random_state=0)
forest = make_pipeline(
    SimpleImputer(strategy="median"),
    RandomForestClassifier(n_estimators=60, min_samples_leaf=5, random_state=0),
).fit(X_dev, y_dev)
logit = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
                      LogisticRegression(max_iter=1000)).fit(X_dev[numeric], y_dev)
import math

weights = {"days since last order": 0.8, "tenure": -0.5, "tickets": 0.3}
values = {"days since last order": 1.5, "tenure": -1.0, "tickets": 2.0}
bias = -1.0
parts = {name: round(weights[name] * values[name], 2) for name in weights}
score = round(bias + sum(parts.values()), 2)
print(parts)
print("score", score, "chance", round(1 / (1 + math.exp(-score)), 3))
It runs right here in your browser, with nothing to install.
Quick check
From the logistic regression lesson: the score z is turned into a chance by the sigmoid. What does this print?
import math

z = 0.0
print(round(1 / (1 + math.exp(-z)), 2), round(1 / (1 + math.exp(-2.0)), 2))
  1. 1.0 0.12
  2. 0.0 0.88
  3. 0.5 2.0
  4. 0.5 0.88
Show the answer

0.5 0.88

A score of 0 means even odds, 0.5. A score of 2 is clearly on the churn side: 1 / (1 + e^-2) is 0.88. The contributions add into z, and z becomes the chance.

The picture. The toy customer's score, one push per feature. The bars are the contributions from the cell above: 0.8 × 1.5 = 1.2, −0.5 × −1.0 = 0.5 and 0.3 × 2.0 = 0.6. The bias, −1.0, pulls the total down from there.

Try it now

A prediction explained by a model that is a sum: every term can be read.

In your own words
In your own words: how is the question about all customers different from the question about one customer?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: contributions, permutation and partial dependence

Purpose. Run each tool on the shop's customers, and see how the usual shortcut, the forest's own importance, goes wrong.

The setup above loads the 600 customers, adds a column of random whole numbers called random_id that cannot predict anything, holds back 25% of the customers for testing, and fits two models: a logistic regression and a random forest.

One customer, from the logistic model

The model standardizes the columns, then adds weight times value. The next cell takes the held-out customer the model rates as the riskiest and lists the three biggest pushes.

Worked example

Worked example: why is this customer flagged?

Break the riskiest held-out customer's score into one push per feature, and check that the pushes rebuild the model's own chance.

Uses the setup above, 25 lines.

import numpy as np

risk = logit.predict_proba(X_test[numeric])[:, 1]
i = int(risk.argmax())
row = logit[:-1].transform(X_test[numeric])[i]
parts = logit[-1].coef_[0] * row
for j in np.argsort(-np.abs(parts))[:3]:
    print(f"{numeric[j]:24} {parts[j]:+.2f}")
score = parts.sum() + logit[-1].intercept_[0]
chance = 1 / (1 + np.exp(-score))
print("score", round(float(score), 2), "chance", round(float(chance), 3))
print("model says", round(float(risk[i]), 3))
In the lesson you can change this and run it.

The last two numbers agree: the explanation is not an approximation, it is the model's own arithmetic. The biggest pushes are a long gap since the last order and few orders in the past year. An account manager can say exactly that: "this customer hasn't ordered in a long time, and ordered rarely before that". Each push is a weight times a standardized value, so a push says how far this customer is from the average one on that feature, scaled by how much the model cares.

Code flow. The sum above is a loop over features. Step through it with the toy numbers: the score starts at the bias, and each feature adds its weight times its value.

Quick check
A linear model has bias −2, and a customer's three contributions are +1.5, +0.5 and −0.25. What does this print?
import math

z = -2 + 1.5 + 0.5 - 0.25
print(z, round(1 / (1 + math.exp(-z)), 3))
  1. -0.25 -0.438
  2. -0.25 0.438
  3. 0.25 0.562
  4. 2.25 0.905
Show the answer

-0.25 0.438

The contributions and the bias add to −0.25, a score slightly on the stay side. The sigmoid of −0.25 is 0.438. A feature with a negative contribution pulled the score toward staying.

Across all customers: permutation against impurity

A random forest keeps its own importance numbers, from how much each column reduced impurity when the trees split on it. They are free, and they have a known fault: a column with many different values gives the trees many chances to split, even when it means nothing. The column random_id was made for this test. Compare the two kinds of importance on rows the forest never saw:

Uses the setup above, 25 lines.

import matplotlib.pyplot as plt
from sklearn.inspection import permutation_importance

impurity = forest[-1].feature_importances_
perm = permutation_importance(forest, X_test, y_test, scoring="roc_auc",
                              n_repeats=10, random_state=0)
for name, a, b in zip(X.columns, impurity, perm.importances_mean):
    print(f"{name:24} impurity {a:.3f}  permutation {b:.3f}")
plt.barh(list(X.columns), perm.importances_mean)
plt.xlabel("drop in test AUC when the column is shuffled")
plt.show()

The impurity column ranks the random ID above several real features: it gave the trees hundreds of places to split, and some splits looked like a gain by chance. The permutation column puts it at zero, because shuffling it changes no prediction on held-back rows. Two real columns, days since the last order and orders last year, carry nearly all of the drop. Permutation importance is measured on rows the model did not learn from, which is why it cannot reward memorizing a column.

Quick check
A forest's impurity importance puts random_id above tenure_months, while permutation importance on held-back rows puts it near zero. What is the best reading?
  1. Permutation importance is broken and should not be trusted
  2. The forest was trained on the wrong rows of the table
  3. random_id is a strong predictor of churn that permutation misses
  4. The impurity score rewards columns with many distinct values, even useless ones
Show the answer

The impurity score rewards columns with many distinct values, even useless ones

A column with many distinct values offers the trees many split points, and some look good by chance on training rows. Shuffling the column on rows the forest never saw shows it was contributing nothing.

Drill

Permutation importance of one column

Write drop(model, X, y, column, seed=0): the model's score on X and y, minus its score after the values in column are shuffled. Shuffle a copy of X, so the caller's table is not changed.

Uses the setup above, 25 lines.

import numpy as np


def drop(model, X, y, column, seed=0):
    ...
In the lesson, your code is checked as you write it.

Sweeping one feature: partial dependence

Importance says how much a column matters. It does not say which way. Partial dependence answers that: for each value in a grid, set the column to that value for every customer, predict, and average. The result is a curve:

Uses the setup above, 25 lines.

import matplotlib.pyplot as plt
from sklearn.inspection import partial_dependence

pdp = partial_dependence(forest, X_dev, ["days_since_last_order"], grid_resolution=6)
grid, average = pdp["grid_values"][0], pdp["average"][0]
print(grid.round(0).tolist())
print(average.round(3).tolist())
plt.plot(grid, average, marker="o")
plt.xlabel("days since last order")
plt.ylabel("average predicted churn")
plt.show()

The average predicted chance of churn climbs from 0.15 at 11 days to 0.39 at 294 days, with small dips on the way: a forest's curve is a staircase of many splits, so it is bumpy. A curve like this is the picture to show a non-technical reader. It also has a caveat: it forces every customer to the same value, including combinations that never occur, such as a customer with 180 days since their last order and 20 orders in the past year. When columns move together, read it with care.

Quick check
A partial dependence curve for days_since_last_order rises from 0.2 at 10 days to 0.6 at 180 days. What does it show?
  1. Churn is caused by long gaps, so a shorter gap would lower churn directly
  2. The average predicted churn rises as the gap grows, with other columns left as they are
  3. Every customer's chance of churn rises from 0.2 at 10 days to 0.6 at 180 days
  4. Customers with long gaps make up 0.6 of the customer base
Show the answer

The average predicted churn rises as the gap grows, with other columns left as they are

Each point is the model's average prediction when every customer is set to that gap. It describes the model's behavior, not the world, and not each customer: some customers' chances move far more or less than the average.

Drill

Find the bug: shuffling that spreads

drop_all(model, X, y) should return each column's drop in score when only that column is shuffled. But it shuffles the table in place and never puts the column back, so every later column is measured on a table that is already scrambled. Fix it.

Uses the setup above, 25 lines.

import numpy as np


def drop_all(model, X, y):
    base = model.score(X, y)
    rng = np.random.default_rng(0)
    out = {}
    for column in X.columns:
        X[column] = rng.permutation(X[column].values)
        out[column] = base - model.score(X, y)
    return out
In the lesson, your code is checked as you write it.
In your own words
Why are permutation importances computed on held-back rows, and not on the rows the model trained on?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: what an importance does and doesn't tell you

Purpose. Know the limits of an explanation before it goes into a decision or a model card.

What an importance does not say.

  • It is not cause. A column the model leans on is a good predictor, not a lever. Days since the last order predicts churn, but sending an email that resets the count would not make a customer stay.
  • Correlated columns share credit. If two columns carry the same information, shuffling one leaves the other to fill in, so each looks less important than the pair is. Check how columns move together before reading a ranking, and consider shuffling a group of related columns together.
  • It depends on the score. A column can matter for AUC and little for accuracy. Say which score the importance is measured on.
  • Contributions explain the model, not the world. They add up exactly for a linear model. For a forest or a boosted model there is no such sum, and a method such as SHAP values gives a fair split of one prediction among the features, at a cost in compute. SHAP is a separate package, so it is only named here.
Quick check
A model card says: "days since last order has the highest importance, so keeping that number low will reduce churn." What is wrong with it?
  1. Importance shows what the model leans on, not what would change the outcome
  2. The card should list the lowest importances instead of the highest
  3. Importance can only be computed for linear models like logistic regression
  4. Nothing: a high importance shows the column can be changed to cut churn
Show the answer

Importance shows what the model leans on, not what would change the outcome

A column the model leans on is a good predictor. Whether changing it would change churn needs an experiment, because the gap may be a symptom of a customer already leaving.

Where this turns up in an engineer's week. A review asks "why did it flag this account?" and you open the contributions. A new feature lands and you check its permutation importance before keeping it. A model's score drops in production and you compare which columns changed. A model card lists what the model leans on, and a reader uses it to decide where to look when something goes wrong.

On the job: compute importances on held-back rows, add a random column as a control, and report the control's importance. Any real column that does not beat it has not earned a place on the list.

Goes deeper in

In your own words
A column has high permutation importance. What can you conclude, and what can't you?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A forest's impurity importance ranks a random ID column above several real features. What is the usual cause?

  1. A column with many distinct values gives the trees many chances to split
  2. The ID column carries real information about who will churn
  3. Impurity importance is always the most accurate measure of a column
  4. The random forest used too few trees to measure importance

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write