Imbalanced classes in depth
Rare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.
What it covers6
Why it matters
Fraud, faults, churn, disease, abuse: the thing you most want to find is usually the rarest thing in the data. The lesson on precision and recall gave you the first aid: don't trust accuracy, and move the threshold. This lesson goes into what teams argue about next. Should you rebalance the data? Why did the probabilities go strange when you did? Which score should go on the dashboard? How many fraud cases are enough? Each question gets an experiment, and several of the answers are not what folklore says.
What you'll be able to do
- Show that class weights and resampling mostly move the threshold, and leave the ranking alone
- Correct probabilities from a rebalanced model back to the real mix of classes
- Use average precision and precision at a review budget when the positive class is rare
- Explain why the number of rare cases matters more than the ratio, in training and in testing
- Set a threshold from what a review and a miss each cost
Show the answer
About 2%
In the training sample, "0.5" meant "as likely as an average payment to be fraud", since half of that sample was fraud. In real traffic an average payment is fraud 2% of the time. The model's opinion of this payment is "average", so the honest number is about 2%. Rebalancing changed what the model's numbers mean, and part three of this lesson puts them back.
The idea: a model that looks broken, and isn't
This rests on Logistic regression (Foundation lesson 102): the model's output is a probability, and the threshold is a separate choice.
The data is made-up card payments. Each has six numbers about it, and under 2% are fraud. The model is a plain logistic regression, trained on the data exactly as it comes.
Setup (runs before every cell) · 46 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np def sigmoid(z): return 1 / (1 + np.exp(-z)) def transactions(n, seed): """Made-up card payments: six numbers about each one (four matter, two are noise), and whether it was fraud. Under 2% are.""" rng = np.random.default_rng(seed) X = rng.normal(0, 1, (n, 6)) chance = sigmoid( -5.4 + 1.2 * X[:, 0] + 0.9 * X[:, 1] + 0.7 * X[:, 2] + 0.5 * X[:, 3]) return X, (rng.random(n) < chance).astype(float) def fit(X, y, weights=None): """Logistic regression, fit by Newton's method. `weights` makes some rows count for more.""" A = np.column_stack([np.ones(len(X)), X]) w, sw = np.zeros(A.shape[1]), np.ones(len(y)) if weights is None else weights for step in range(25): p = sigmoid(A @ w) H = (A * (sw * p * (1 - p))[:, None]).T @ A + 1e-8 * np.eye(A.shape[1]) w = w + np.linalg.solve(H, A.T @ (sw * (y - p))) return w def predict(w, X): return sigmoid(w[0] + X @ w[1:]) X, y = transactions(20000, 0) # training data X_new, y_new = transactions(50000, 1) # new data, with the real mix of fraud and not def auc(p, y): """The chance that a random fraud is scored above a random non-fraud.""" order = np.argsort(p) ranks = np.empty(len(p)) ranks[order] = np.arange(1, len(p) + 1) pos = y == 1 n_pos, n_neg = pos.sum(), (~pos).sum() return float((ranks[pos].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)) def average_precision(p, y): """Go down the list from the highest score. At each fraud, note the precision so far. Average those.""" hits = y[np.argsort(-p)] precision = np.cumsum(hits) / np.arange(1, len(y) + 1) return float(precision[hits == 1].mean())
p = predict(fit(X, y), X_new) print("fraud in the training data:", int(y.sum()), "of", len(y), "| in the new data:", int(y_new.sum()), "of", len(y_new), "=", f"{100 * y_new.mean():.2f}%") print("accuracy at a threshold of 0.5:", f"{100 * ((p >= 0.5) == y_new).mean():.2f}%", "| accuracy of never flagging anything:", f"{100 * (y_new == 0).mean():.2f}%") print("average predicted chance of fraud:", f"{100 * p.mean():.2f}%") print() print( "threshold | flagged | of those, fraud (precision) | of all fraud, caught (recall)") for t in (0.5, 0.2, 0.1, 0.05, 0.02): flagged = p >= t print(str(t).rjust(9), "|", str(int(flagged.sum())).rjust(7), "|", f"{y_new[flagged].mean():27.3f}", "|", f"{flagged[y_new == 1].mean():.3f}")
At the usual threshold of 0.5 the model is 98.32% accurate. Never flagging anything scores 98.31%. It flags 46 payments of 50,000 and catches 3% of the fraud. It looks useless.
Read down the table before judging it. At a threshold of 0.1 it catches 37.9% of the fraud, and 1 flag in 5 is right. At 0.02 it catches 73.9%. The model knows a good deal. What failed was the 0.5.
A threshold of 0.5 asks, "is this payment more likely fraud than not?" When fraud is under 2% of payments, almost nothing is, and a model that says so is being accurate. Look at the third line: its average prediction is 1.61%, close to the real 1.69%. Its probabilities are honest. A payment it scores at 0.2 is more than ten times riskier than average, and well worth a look, though still probably fine.
So the first thing to do about imbalance costs nothing: stop using 0.5. Whether to do more is the next question.
The picture. Rare cases and a threshold. The dots are 200 payments placed by a model's score: fraud above, ordinary below. Slide the threshold and then slide the share of fraud down.
Move the threshold left of 0.5 and watch recall rise. Then lower the share of fraud and watch accuracy stop telling you anything.
Show the answer
The chance that a random positive is scored above a random negative
AUC is about ranking: pick one positive and one negative at random, and it is the chance the model scores the positive higher. It doesn't depend on any threshold, and it doesn't check whether the probabilities are honest. Both facts matter in this lesson: they are why AUC stays the same when a model is rebalanced, and why it can hide how hard a rare problem is.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: four remedies, one ranking
Purpose. Train the remedies, see that they move the threshold and not the ranking, and put the probabilities back.
Four remedies, one ranking
The usual advice is to rebalance, in one of three ways:
- Class weights: every fraud counts 59 times in the loss, the ratio of ordinary rows to fraud.
- Undersampling: train on all the fraud and an equal number of ordinary rows, and discard the rest.
- Oversampling: repeat fraud rows until there are as many as ordinary ones.
The cell trains all three, and the untouched model, and scores them on the same new data.
Uses the setup above, 46 lines.
rng = np.random.default_rng(5) fraud, fine = np.flatnonzero(y == 1), np.flatnonzero(y == 0) # about 59 ordinary payments for each fraud ratio = len(fine) / len(fraud) # every fraud, and as many ordinary rows under = np.concatenate([fraud, rng.choice(fine, len(fraud), replace=False)]) # every ordinary row, and frauds repeated to match over = np.concatenate([fine, rng.choice(fraud, len(fine), replace=True)]) models = { "nothing done": fit(X, y), "class weights": fit(X, y, np.where(y == 1, ratio, 1.0)), "undersampling": fit(X[under], y[under]), "oversampling": fit(X[over], y[over]), } # a team that can review 5% of payments budget = int(0.05 * len(y_new)) print( " | ranking | at a threshold of 0.5 | " "reviewing the top 5%") print( "training | AUC | avg precision | flagged | recall | precision | recall | " "precision | average predicted chance") for name, w in models.items(): p = predict(w, X_new) flagged, top = p >= 0.5, np.argsort(-p)[:budget] print(name.ljust(15), "|", f"{auc(p, y_new):.3f}", "|", f"{average_precision(p, y_new):13.3f}", "|", str(int(flagged.sum())).rjust(7), "|", f"{flagged[y_new == 1].mean():6.3f}", "|", f"{y_new[flagged].mean():9.3f}", "|", f"{y_new[top].sum() / y_new.sum():6.3f}", "|", f"{y_new[top].mean():9.3f}", "|", f"{p.mean():.3f}")
Start in the middle, at a threshold of 0.5. Untouched, the model flags 46 payments and catches 3% of the fraud. Each remedy flags over 10,000 and catches about 78%. That looks like a cure.
Now the two columns on the left, which judge the ranking and involve no threshold. AUC is 0.877, 0.877, 0.875 and 0.877. Average precision, which the next part explains, is 0.195 untouched and slightly lower for all three remedies. And give all four the same job, reviewing the top-scored 5% of payments: they catch 46.6%, 45.9%, 46.0% and 46.1% of the fraud.
The remedies did not teach the model anything about fraud. They put the same payments in almost the same order, and shifted every score upward until 0.5 landed in a more useful place. Rebalancing is, mostly, an expensive way to move the threshold. Undersampling paid for it by throwing away 97% of the data, and came last.
One caveat. This is a logistic regression. For trees and neural networks, rebalancing can change the ranking a little more, in either direction. The test is the same: compare rankings on data with the real mix.
Show the answer
They rank payments alike, and differ in how high their scores run
Flagging the top 5% depends only on the order of the scores, and the four orders are nearly the same. A fixed threshold depends on the size of the scores, and rebalancing inflates those. So the models differ at 0.5 and agree at a fixed budget. To compare models, use a measure of ranking, or give each the same budget.
Putting the probabilities back
Rebalancing has a cost that the last table showed in its final column. The untouched model's average prediction was 0.016. The others averaged 0.281: they believe that more than a quarter of payments are fraud. Any number they report, and any threshold chosen from costs, is wrong.
It can be repaired exactly, by working in odds: the chance of fraud divided by the chance of not. Undersampling kept every fraud and a share of the ordinary rows, here 0.017 of them. For any kind of payment, the frauds in the sample are all there and the ordinary ones are 59 times scarcer, so the model learned odds that are 59 times too high. Multiply the odds by the share kept, and they are right again:
corrected chance = kept × p / (kept × p + 1 − p)
Uses the setup above, 46 lines.
rng = np.random.default_rng(5) fraud, fine = np.flatnonzero(y == 1), np.flatnonzero(y == 0) # about 59 ordinary payments for each fraud ratio = len(fine) / len(fraud) # every fraud, and as many ordinary rows under = np.concatenate([fraud, rng.choice(fine, len(fraud), replace=False)]) # every ordinary row, and frauds repeated to match over = np.concatenate([fine, rng.choice(fraud, len(fine), replace=True)]) models = { "nothing done": fit(X, y), "class weights": fit(X, y, np.where(y == 1, ratio, 1.0)), "undersampling": fit(X[under], y[under]), "oversampling": fit(X[over], y[over]), } p_under = predict(models["undersampling"], X_new) # the share of ordinary rows that undersampling kept kept = len(fraud) / len(fine) # scale the odds back down by that share p_fixed = kept * p_under / (kept * p_under + 1 - p_under) print("share of ordinary rows kept:", round(kept, 4)) print("average predicted chance: as trained", round(float(p_under.mean()), 3), "| corrected", round(float(p_fixed.mean()), 4), "| real share of fraud", round(float(y_new.mean()), 4)) print() print( "predicted chance | as trained: payments, how many were fraud | corrected: " "payments, how many were fraud") for low, high in ((0, 0.01), (0.01, 0.05), (0.05, 0.2), (0.2, 0.5), (0.5, 1.01)): a, b = (p_under >= low) & (p_under < high), (p_fixed >= low) & (p_fixed < high) label = f"{low:.2f} to {min(high, 1):.2f}" print(label.ljust(16), "|", str(int(a.sum())).rjust(20), f"{y_new[a].mean():19.3f}", "|", str(int(b.sum())).rjust(19), f"{y_new[b].mean():19.3f}")
After the correction the average prediction is 0.0152, against a real rate of 0.0169.
The table checks it band by band. As trained, the model put 10,322 payments above 0.5, and 6.4% of them were fraud. Corrected, the payments scored between 0.20 and 0.50 turn out to be fraud 29.9% of the time, and those between 0.05 and 0.20 turn out to be fraud 10.2% of the time. The numbers mean what they say again.
The same correction works for class weights and for oversampling, with the weight ratio in place of the share kept. The order of the payments doesn't change, since the correction never swaps two scores.
kept = 0.02 print([round(kept * p / (kept * p + 1 - p), 3) for p in (0.5, 0.9)])
Show the answer
[0.02, 0.153]
At 0.5 the model's odds are 1 to 1. Multiplied by 0.02 they are 0.02 to 1, a chance of 0.02 / 1.02 ≈ 0.02. At 0.9 the odds are 9 to 1, which become 0.18 to 1, a chance of 0.18 / 1.18 ≈ 0.153. Even the payments the rebalanced model is most sure about are, in real traffic, probably fine.
Correct a chance for a change of mix
Write correct(p, train_rate, real_rate). A model trained where the positive class had the share train_rate gives the chance p. Return the chance where the share is real_rate: multiply the odds p / (1 - p) by the odds of real_rate and divide by the odds of train_rate, then turn the odds back into a chance. Both rates are strictly between 0 and 1. It must also work when p is exactly 0 or 1.
Uses the setup above, 46 lines.
def correct(p, train_rate, real_rate): return p print(round(correct(0.5, 0.5, 0.02), 4), round(correct(0.9, 0.5, 0.02), 4))
The score that feels the rarity
Which single number should describe a model for a rare class? AUC is the habit, and it has a blind spot. The cell scores one model on two versions of the new data: as it is, and with 9 frauds in 10 removed, so that fraud is ten times rarer.
Uses the setup above, 46 lines.
p = predict(fit(X, y), X_new) rng = np.random.default_rng(7) print( "share of fraud | AUC | average precision | precision at 50% recall | false " "alarms for each fraud caught") # all the fraud, then only 1 fraud in 10: a rarer problem, same model for keep in (1.0, 0.1): rows = (y_new == 0) | (rng.random(len(y_new)) < keep) scores, labels = p[rows], y_new[rows] hits = labels[np.argsort(-scores)] recall = np.cumsum(hits) / hits.sum() precision = np.cumsum(hits) / np.arange(1, len(hits) + 1) # how far down the list until half the fraud is found at = int(np.argmax(recall >= 0.5)) print(f"{100 * labels.mean():13.2f}%", "|", f"{auc(scores, labels):.3f}", "|", f"{average_precision(scores, labels):17.3f}", "|", f"{precision[at]:23.3f}", "|", f"{(1 - precision[at]) / precision[at]:.1f}")
AUC is 0.877 in one case and 0.883 in the other. It compares a random fraud with a random ordinary payment, so it has no idea how many of each there are. By AUC, nothing happened.
For the team reviewing flags, a great deal happened. To catch half the fraud at 1.69%, 14% of their flags are right: about 6 false alarms for each catch. At 0.14%, 1.9% are right: about 52 false alarms for each catch. Same model, same AUC, more than eight times the wasted work.
Average precision registers it: 0.195 falls to 0.034. It is computed by walking down the list from the highest score, and at each fraud noting the precision so far, then averaging those. A model that puts every fraud at the top scores 1. A model that ranks at random scores about the share of fraud, so 0.195 at a 1.69% rate is far better than chance, and far from solved.
Use AUC to ask "does the model separate the classes?" Use average precision, or precision at the number of flags you can afford, to ask "what will it be like to use?"
hits = [1, 0, 1, 0, 0] found, total = 0, 0.0 for rank, hit in enumerate(hits, start=1): if hit: found += 1 total += found / rank print(round(total / found, 3))
Show the answer
0.833
The first fraud is at rank 1, where the precision so far is 1 of 1. The second is at rank 3, where it is 2 of 3. Their average is (1 + 0.667) / 2 = 0.833. Had the two frauds been ranked first and second, it would be 1.0. Each ordinary payment ranked above a fraud pulls it down.
Average precision from scores
Write average_precision(scores, labels). Order the items from the highest score down, keeping the given order among equal scores. At each item whose label is 1, take the precision so far: the number of 1s up to and including it, divided by its rank. Return the mean of those, or 0.0 if no label is 1.
Uses the setup above, 46 lines.
def average_precision(scores, labels): return 0.0 print(round(average_precision([0.9, 0.2, 0.8, 0.4, 0.1], [1, 0, 0, 1, 0]), 3))
Show the answer
Its precision at the number of parts the team can inspect
At 1 in 5,000, even a model with an AUC of 0.95 will flag mostly good parts: there are so many of them that a small share of mistakes swamps the real faults. Precision at the team's budget says how many inspections find something. AUC can't, since it ignores how rare the faults are.
See it. Precision against recall for the same model on the new data as it is, and with fraud ten times rarer. The curve for the rarer problem sits lower even though the model is the same.
Uses the setup above, 46 lines.
import matplotlib.pyplot as plt p = predict(fit(X, y), X_new) rng = np.random.default_rng(7) for keep, label in [(1.0, "fraud as it is"), (0.1, "fraud 10x rarer")]: rows = (y_new == 0) | (rng.random(len(y_new)) < keep) hits = y_new[rows][np.argsort(-p[rows])] recall = np.cumsum(hits) / hits.sum() precision = np.cumsum(hits) / np.arange(1, len(hits) + 1) plt.plot(recall, precision, label=label) plt.xlabel("recall") plt.ylabel("precision") plt.legend() print(round(average_precision(p, y_new), 3))
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: count the rare cases, and set the threshold from costs
Purpose. What to measure, how many rare cases you need, and the rule that turns two costs into one threshold.
Count the rare cases, not the ratio
"Our data is 99 to 1" sounds like the problem. The cell argues that the ratio matters much less than a plainer number: how many rare cases you have. First for testing, then for training.
Uses the setup above, 46 lines.
p = predict(fit(X, y), X_new) rng = np.random.default_rng(3) print( "One model, one threshold (0.1), measured on 200 different test sets of each size") print("test rows | frauds in a typical one | recall: 5th to 95th percentile") for size in (1000, 5000, 50000): recalls = [] for trial in range(200): rows = rng.choice(len(y_new), size, replace=False) if y_new[rows].sum() > 0: recalls.append(float((p[rows][y_new[rows] == 1] >= 0.1).mean())) print(str(size).rjust(9), "|", str(round(size * float(y_new.mean()))).rjust(23), "|", f"{np.percentile(recalls, 5):.2f} to {np.percentile(recalls, 95):.2f}") X_pool, y_pool = transactions(300000, 9) fraud, fine = np.flatnonzero(y_pool == 1), np.flatnonzero(y_pool == 0) print() print( "training rows: fraud | ordinary | average precision on the new data (mean of 8 " "training sets)") for n_fraud, n_fine in ((20, 1000), (20, 20000), (200, 1000), (200, 20000), (2000, 20000)): scores = [] for seed in range(8): pick = np.random.default_rng(seed) rows = np.concatenate( [pick.choice(fraud, n_fraud, replace=False), pick.choice(fine, n_fine, replace=False)]) scores.append( average_precision(predict(fit(X_pool[rows], y_pool[rows]), X_new), y_new)) print(str(n_fraud).rjust(20), "|", str(n_fine).rjust(8), "|", f"{np.mean(scores):.3f}")
The first table measures one model, at one threshold, on 200 different test sets of each size. Its recall on all 50,000 rows is 0.38. On a test set of 1,000 rows, which holds about 17 frauds, the measured recall runs from 0.20 to 0.58. One team would report a model that catches a fifth of the fraud, and another a model that catches most of it. With 85 frauds the range narrows to 0.29 to 0.47.
A thousand rows sounds like a decent test set. For a rare class it is a test on 17 examples. So split with stratification, which keeps the share of each class equal in every part. Report how many positives a number rests on, and distrust any comparison between models that rests on a few dozen.
The second table is about training. With 20 frauds and 1,000 ordinary rows, average precision is 0.167. Twenty times more ordinary rows lifts it to 0.173. Ten times more frauds lifts it to 0.190. At 2,000 frauds it reaches 0.196.
So when a rare-class model is weak, the useful questions are how to get more positive examples, by labeling, by waiting, or by widening the definition, and whether the features can tell the classes apart at all. Juggling the ratio answers neither.
Show the answer
Very little: the gap is 2 cases out of 12
0.75 of 12 is 9 cases and 0.58 is 7. The whole comparison rests on two examples, and a different draw of 12 could easily reverse it, as the first table showed for 17. Gather more positives for the test set, or use cross-validation with stratified folds so that every positive is tested once, before choosing.
Worked example: the threshold that costs least
A threshold should come from what mistakes cost. Say a review costs $5, and a fraud that gets through costs $200. Flagging a payment always costs $5. Leaving it costs $200 times its chance of being fraud. So flag when chance × $200 is more than $5: when the chance is above 5 / 200 = 0.025. That rule needs honest probabilities, and the last lines show what happens without them.
Uses the setup above, 46 lines.
rng = np.random.default_rng(5) fraud, fine = np.flatnonzero(y == 1), np.flatnonzero(y == 0) # about 59 ordinary payments for each fraud ratio = len(fine) / len(fraud) # every fraud, and as many ordinary rows under = np.concatenate([fraud, rng.choice(fine, len(fraud), replace=False)]) # every ordinary row, and frauds repeated to match over = np.concatenate([fine, rng.choice(fraud, len(fine), replace=True)]) models = { "nothing done": fit(X, y), "class weights": fit(X, y, np.where(y == 1, ratio, 1.0)), "undersampling": fit(X[under], y[under]), "oversampling": fit(X[over], y[over]), } # a review costs $5. A fraud that gets through costs $200 review, missed = 5.0, 200.0 def cost(p, threshold): flagged = p >= threshold return review * flagged.sum() + missed * ((y_new == 1) & ~flagged).sum() p = predict(models["nothing done"], X_new) print("flag nothing: $" + format(int(missed * y_new.sum()), ","), "| review everything: $" + format(int(review * len(y_new)), ",")) print() print("threshold | flagged | total cost") for t in (0.5, 0.1, review / missed, 0.01): print(f"{t:9.3f}", "|", str(int((p >= t).sum())).rjust(7), "|", "$" + format(int(cost(p, t)), ",")) tried = np.linspace(0.002, 0.3, 150) best = min(tried, key=lambda t: cost(p, t)) print("lowest cost of 150 thresholds tried:", round(float(best), 3), "at $" + format(int(cost(p, best)), ",")) p_under = predict(models["undersampling"], X_new) kept = len(fraud) / len(fine) p_fixed = kept * p_under / (kept * p_under + 1 - p_under) print() print("the undersampled model at the same threshold,", review / missed) print(" as trained:", int((p_under >= review / missed).sum()), "flagged, $" + format(int(cost(p_under, review / missed)), ",")) print(" corrected: ", int((p_fixed >= review / missed).sum()), "flagged, $" + format(int(cost(p_fixed, review / missed)), ","))
Flagging nothing costs $169,200 in fraud, and reviewing everything costs $250,000. The untouched model at 0.5 saves almost nothing: $164,430. At the threshold the costs imply, 0.025, it flags 7,331 payments and the total falls to $88,455. A search over 150 thresholds finds its best at 0.024 and $88,230: the rule from the costs landed within $225 of it, with no search. Then the undersampled model, used at the same 0.025 without correction. Its scores are inflated, so it flags 45,517 of 50,000 payments and costs $227,785, worse than doing nothing. With the correction from part three it flags 7,101 and costs $88,705. That is the practical reason to keep probabilities honest: a calibrated model can be given a threshold by arithmetic, and a rebalanced one, uncorrected, turns that arithmetic into a very expensive mistake.
review, missed = 5, 200 print(review / missed, 0.03 * missed > review, 0.02 * missed > review)
Show the answer
0.025 True False
The threshold is 5 / 200 = 0.025. A payment at 3% has an expected loss of $6 if left alone, more than the $5 review, so flag it. At 2% the expected loss is $4, less than the review, so let it through. When a miss costs 40 times a review, a 3% chance is already worth acting on.
Goes deeper in
- Precision, recall and F1: the measures this lesson leans on.
- ROC curves and AUC: the ranking score and its blind spot.
- Anomaly detection: when there are almost no labeled rare cases at all.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
A logistic regression is retrained with class weights on a dataset where 2% of rows are positive. What mainly changes?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write