Naive Bayes: classifying with counts
A classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.
What it covers6
Why it matters
Naive Bayes is the classifier you can train by counting. It has no gradient descent and no learning rate: it reads the training data once, keeps a table, and it is done. For decades it was how spam filters worked, and it is still a strong first model for text when labeled examples are few. It also rests on an assumption that is plainly false, and seeing exactly what that costs, and what it doesn't, is one of the better lessons in how machine learning models get away with being wrong.
What you'll be able to do
- Turn word counts into the chance that a message is spam, using Bayes' rule
- State the naive assumption, and explain why multiplying one chance per word needs logarithms
- Fix the zero-count problem with smoothing, and say when it matters most
- Show why naive Bayes is overconfident when its evidence overlaps
- Decide when naive Bayes is the right first model, and read why it made a decision
Show the answer
About 83%: 19 of the 23 messages with that word
The question is about messages that contain "winner", so look only at those: 19 of 23 were spam, about 83%. The 35% was your belief before seeing the word. The word is evidence, and it moved the estimate a long way. That move, from a starting belief to an updated one, is all Bayes' rule is.
The idea: from one word to a verdict
This rests on Conditional probability and Bayes' rule (Foundation lesson 90): how a word changes the chance of a class.
The data for this lesson is made up, so that it can be as large as needed: short messages, each a list of words and a label. Spam and ordinary mail ("ham") use the same words at different rates. "winner" turns up in both, far more often in spam.
Setup (runs before every cell) · 67 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math, random # How often each word turns up in the two kinds of message. Every word can appear in # both: only the rates differ. SPAM_WORDS = {"free": 5, "winner": 2, "prize": 2, "click": 4, "offer": 4, "money": 4, "urgent": 3, "now": 4, "account": 4, "deal": 3, "meeting": 2, "report": 2, "lunch": 1, "team": 2, "thanks": 3, "tomorrow": 2, "project": 2, "invoice": 3, "call": 3, "today": 4, "raffle": 0.3, "refund": 0.5} HAM_WORDS = {"free": 2, "winner": 0.3, "prize": 0.3, "click": 2, "offer": 2, "money": 2, "urgent": 2, "now": 4, "account": 4, "deal": 1, "meeting": 5, "report": 5, "lunch": 4, "team": 5, "thanks": 5, "tomorrow": 4, "project": 5, "invoice": 3, "call": 4, "today": 4, "raffle": 0.5, "refund": 1} def messages(n, seed): """n made-up messages, each a list of words and whether it is spam. About 4 in 10 are spam.""" rng, out = random.Random(seed), [] for i in range(n): spam = rng.random() < 0.4 weights = SPAM_WORDS if spam else HAM_WORDS out.append( (rng.choices(list(weights), list(weights.values()), k=rng.randint(5, 12)), spam)) return out def fit(data, smooth=1): """Training is counting: messages of each kind, and each word's uses in each kind.""" model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0}, "messages": {True: 0, False: 0}, "smooth": smooth, "vocabulary": sorted({w for words, spam in data for w in words})} for words, spam in data: model["messages"][spam] += 1 for w in words: model["words"][spam][w] = model["words"][spam].get(w, 0) + 1 model["total"][spam] += 1 return model def word_chance(model, word, spam): """The chance that a word drawn from this kind of message is this word, with `smooth` added to every count.""" count = model["words"][spam].get(word, 0) + model["smooth"] return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"])) def log_score(model, words, spam): """log of: the chance of this kind of message, times the chance of each of its words.""" score = math.log(model["messages"][spam] / sum(model["messages"].values())) for w in words: p = word_chance(model, w, spam) if p == 0: return float("-inf") score += math.log(p) return score def chance_of_spam(model, words): a, b = log_score(model, words, True), log_score(model, words, False) if a == b: return 0.5 top = max(a, b) return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top)) def accuracy(model, data): return sum((chance_of_spam(model, words) >= 0.5) == spam for words, spam in data) / len(data)
train = messages(150, 0) spam_count = sum(spam for words, spam in train) print("messages:", len(train), "| spam:", spam_count, "| so before reading a word, the chance of spam is", round(spam_count / len(train), 3)) print() print("word | messages with it | of those, spam | chance of spam, given the word") for word in ("winner", "free", "account", "meeting"): having = [spam for words, spam in train if word in words] print(word.ljust(8), "|", str(len(having)).rjust(16), "|", str(sum(having)).rjust(14), "|", round(sum(having) / len(having), 3))
Before reading a word, the chance of spam is 0.353: 53 of 150 messages. That starting belief is called the prior.
Each word then moves it. "winner" moves it to 0.826, and "meeting" down to 0.242. "account" leaves it at 0.351, almost exactly where it started: a word used equally by both kinds of message is no evidence at all.
Counting messages that contain a word works for one word. A real message has many, and you will never find 23 earlier messages containing exactly the same eight words. So the calculation is turned round, with Bayes' rule:
chance of spam, given the words ∝ chance of spam × chance of those words in spam
The symbol ∝ means "in proportion to". Work the right-hand side out for spam, and again for ham, and compare the two. The first factor is the prior. The second asks how likely this message would be if it were spam, and the next part is about how to get that from counts.
Set the column to keep to 1: given no winner, the shares are 34/128 and 93/128.
Show the answer
The condition's rarity matters too: most positives can be false alarms
Bayes' rule combines the evidence with the prior. When a condition is rare, the few true positives are outnumbered by false alarms from the many healthy people, so a positive result may still mean a small chance. Naive Bayes does the same with messages: the prior, how common spam is, multiplies the evidence from the words.
Many words, and one bold assumption
How likely is a whole message, if it is spam? Naive Bayes makes an assumption that turns a hard question into a product:
Given the kind of message, each word appears independently of the others.
Then the chance of the message is just the chance of its first word in spam, times the chance of its second, and so on, each read from a table of counts. Training is filling in that table.
spam = {"free": 0.20, "now": 0.10}
ham = {"free": 0.05, "now": 0.10}
as_spam = 0.5 * spam["free"] * spam["now"]
as_ham = 0.5 * ham["free"] * ham["now"]
print(round(as_spam / (as_spam + as_ham), 2))Show the answer
0.8
As spam the message scores 0.5 × 0.20 × 0.10 = 0.01. As ham it scores 0.5 × 0.05 × 0.10 = 0.0025. The chance of spam is 0.01 out of the two together: 0.01 / 0.0125 = 0.8. "now" is equally likely in both and cancels out. All the evidence came from "free", which is four times as common in spam.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: counts, logs and smoothing
Purpose. Train by counting, score with added logs, and fix the zero-count trap.
Uses the setup above, 67 lines.
def fit(data, smooth=1): """Training is counting: messages of each kind, and each word's uses in each kind.""" model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0}, "messages": {True: 0, False: 0}, "smooth": smooth, "vocabulary": sorted({w for words, spam in data for w in words})} for words, spam in data: model["messages"][spam] += 1 for w in words: model["words"][spam][w] = model["words"][spam].get(w, 0) + 1 model["total"][spam] += 1 return model def word_chance(model, word, spam): """The chance that a word drawn from this kind of message is this word, with `smooth` added to every count.""" count = model["words"][spam].get(word, 0) + model["smooth"] return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"])) def log_score(model, words, spam): """log of: the chance of this kind of message, times the chance of each of its words.""" score = math.log(model["messages"][spam] / sum(model["messages"].values())) for w in words: p = word_chance(model, w, spam) if p == 0: return float("-inf") score += math.log(p) return score def chance_of_spam(model, words): a, b = log_score(model, words, True), log_score(model, words, False) if a == b: return 0.5 top = max(a, b) return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top)) def accuracy(model, data): return sum((chance_of_spam(model, words) >= 0.5) == spam for words, spam in data) / len(data) train, test = messages(150, 0), messages(1000, 1) model = fit(train) for words in (["winner", "prize", "click", "now"], ["meeting", "tomorrow", "team", "lunch"], ["free", "lunch", "tomorrow", "offer"]): print(" ".join(words).ljust(28), "| log score as spam", f"{log_score(model, words, True):7.2f}", "| as ham", f"{log_score(model, words, False):7.2f}", "| chance of spam", f"{chance_of_spam(model, words):.3f}") print() print("right on", round(100 * accuracy(model, test), 1), "% of 1,000 new messages. Always guessing 'not spam' would be right on", round(100 * sum(not spam for words, spam in test) / len(test), 1), "%") product = 1.0 for i in range(400): product *= word_chance(model, "free", True) print("a 400-word message, chances multiplied:", product, "| logs added:", round(400 * math.log(word_chance(model, "free", True)), 1))
The first message is given a 0.989 chance of spam and the second 0.039. The third mixes both kinds of word and lands at 0.415: unsure, as it should be.
On 1,000 new messages the model is right 83.9% of the time. Always answering "not spam" would be right 58.1% of the time, so the table of counts has learned something real, with no training loop.
The assumption is false, and obviously so. Words in a message are not independent: "prize" makes "winner" more likely. That is what naive means in the name. The model works anyway, since to pick the right class it only needs the two scores to come out in the right order, and it usually does that even when both are badly off.
The last line is a practical point. The scores are added logarithms. Multiplying 400 small chances gives exactly 0.0 on a computer, for both classes, and nothing can be compared. Their logs add up to an ordinary number, −1,041.9.
The chance of spam, without underflow
Write chance_of_spam(prior, spam_chances, ham_chances). prior is the chance a message is spam before reading it. The two lists hold the chance of each of the message's words under spam and under ham. Work in logarithms: add up log(prior) and the logs of the spam chances, and log(1 - prior) and the logs of the ham chances. Then turn the two totals into a chance of spam. Subtract the larger total from both before using math.exp, so that long messages don't underflow.
Uses the setup above, 67 lines.
import math def chance_of_spam(prior, spam_chances, ham_chances): return 0.5 print(round(chance_of_spam(0.5, [0.20, 0.10], [0.05, 0.10]), 3))
The word it has never seen
A table of counts has a trap in it. Suppose the word "lunch" has never appeared in a spam message in the training data. Its chance in spam is then 0, and one zero in a product makes the whole product zero. A single unseen word rules a class out completely, however strong the rest of the evidence.
Uses the setup above, 67 lines.
test = messages(1000, 1) small = messages(30, 200) plain, smoothed = fit(small, smooth=0), fit(small, smooth=1) never = [w for w in plain["vocabulary"] if w not in plain["words"][True] or w not in plain["words"][False]] print("after 30 messages, words never yet seen in one of the two kinds:", never) words = ["winner", "team", "lunch", "thanks", "tomorrow"] print(" ".join(words), "| without smoothing: as spam", round(log_score(plain, words, True), 2), "as ham", log_score(plain, words, False), "| with smoothing:", round(log_score(smoothed, words, True), 2), round(log_score(smoothed, words, False), 2)) print() print( "training messages | accuracy, raw counts | accuracy, 1 added to every count | " "test messages with a kind ruled out") for n in (30, 60, 150): raw = added = ruled_out = 0 # 20 different training sets of each size for seed in range(20): data = messages(n, 200 + seed) a, b = fit(data, smooth=0), fit(data, smooth=1) raw += accuracy(a, test) / 20 added += accuracy(b, test) / 20 pairs = [(log_score(a, w, True), log_score(a, w, False)) for w, spam in test] ruled_out += sum(float("-inf") in pair for pair in pairs) / 20 print(str(n).rjust(17), "|", f"{raw:.3f}".rjust(20), "|", f"{added:.3f}".rjust(32), "|", round(ruled_out), "of 1,000")
After 30 messages, four words have never been seen in one kind or the other. The message in the second line contains two of them, one missing from each side: its score is minus infinity as spam and as ham. The model has proved that the message can't exist.
The repair is smoothing: add 1 to every count before dividing, as if each word had been seen once more in each class than it was. No chance is ever exactly zero, and a word never seen becomes merely unlikely. With it, the same message gets two ordinary scores and a sensible answer.
The table shows when this matters. With 30 training messages, 352 of 1,000 test messages have a class ruled out, and smoothing lifts accuracy from 0.747 to 0.787. With 150 messages only 8 are affected, and the two columns nearly agree. Zeros are a disease of small data and big vocabularies. Real text has vocabularies of tens of thousands of words, so some smoothing is always on.
count, total, vocabulary = 0, 200, 50 print(count / total, (count + 1) / (total + vocabulary))
Show the answer
0.0 0.004
Raw, the chance is 0 / 200 = 0. Smoothed, the count becomes 1, and the total grows by one for each of the 50 words in the vocabulary, since every word had 1 added: 1 / 250 = 0.004. Adding the vocabulary size to the total keeps the chances of all the words adding up to 1.
A chance that is never zero
Write smoothed(count, total, vocabulary_size, smooth): the chance of a word in a class, as (count + smooth) divided by (total + smooth * vocabulary_size). If that bottom number is 0, there is nothing to go on: return 0.0.
Uses the setup above, 67 lines.
def smoothed(count, total, vocabulary_size, smooth): return 0.0 print(smoothed(0, 200, 50, 1), smoothed(30, 200, 50, 1))
Sure of itself, and quick to learn
The naive assumption has a price, and it is paid in confidence. The cell trains the same model three times. In the second and third runs every word in every message is typed two or four times. That adds no information at all, and a model that understood the words were copies would not change.
Uses the setup above, 67 lines.
train, test = messages(150, 0), messages(1000, 1) print( "each word counted | accuracy | messages it is 99% sure about | of those, how many " "it got right") for copies in (1, 2, 4): # every word typed `copies` times: no new information model = fit([(words * copies, spam) for words, spam in train]) chances = [(chance_of_spam(model, words * copies), spam) for words, spam in test] sure = [(p, spam) for p, spam in chances if p >= 0.99 or p <= 0.01] right = sum((p >= 0.5) == spam for p, spam in sure) print((str(copies) + " time" + ("s" if copies > 1 else "")).ljust(17), "|", f"{sum((p >= 0.5) == spam for p, spam in chances) / len(test):.3f}".rjust(8), "|", str(len(sure)).rjust(29), "|", f"{100 * right / len(sure):.1f}%")
Accuracy doesn't move: 0.839, 0.840, 0.842. The confidence does. With each word counted once, the model is 99% sure about 89 messages, and right about all of them. With each word counted four times it is 99% sure about 697, and right about 92.7% of those. It is claiming 99% and delivering 93%.
Naive Bayes treats every word as a fresh, independent piece of evidence. When words travel together, as "prize" and "winner" do, it counts the same evidence twice and overstates its case. So its labels can be trusted more than its chances. If you need honest probabilities, they have to be recalibrated afterward.
Now the reason it has survived.
Uses the setup above, 67 lines.
import math, random # How often each word turns up in the two kinds of message. Every word can appear in # both: only the rates differ. SPAM_WORDS = {"free": 5, "winner": 2, "prize": 2, "click": 4, "offer": 4, "money": 4, "urgent": 3, "now": 4, "account": 4, "deal": 3, "meeting": 2, "report": 2, "lunch": 1, "team": 2, "thanks": 3, "tomorrow": 2, "project": 2, "invoice": 3, "call": 3, "today": 4, "raffle": 0.3, "refund": 0.5} HAM_WORDS = {"free": 2, "winner": 0.3, "prize": 0.3, "click": 2, "offer": 2, "money": 2, "urgent": 2, "now": 4, "account": 4, "deal": 1, "meeting": 5, "report": 5, "lunch": 4, "team": 5, "thanks": 5, "tomorrow": 4, "project": 5, "invoice": 3, "call": 4, "today": 4, "raffle": 0.5, "refund": 1} def messages(n, seed): """n made-up messages, each a list of words and whether it is spam. About 4 in 10 are spam.""" rng, out = random.Random(seed), [] for i in range(n): spam = rng.random() < 0.4 weights = SPAM_WORDS if spam else HAM_WORDS out.append( (rng.choices(list(weights), list(weights.values()), k=rng.randint(5, 12)), spam)) return out def fit(data, smooth=1): """Training is counting: messages of each kind, and each word's uses in each kind.""" model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0}, "messages": {True: 0, False: 0}, "smooth": smooth, "vocabulary": sorted({w for words, spam in data for w in words})} for words, spam in data: model["messages"][spam] += 1 for w in words: model["words"][spam][w] = model["words"][spam].get(w, 0) + 1 model["total"][spam] += 1 return model def word_chance(model, word, spam): """The chance that a word drawn from this kind of message is this word, with `smooth` added to every count.""" count = model["words"][spam].get(word, 0) + model["smooth"] return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"])) def log_score(model, words, spam): """log of: the chance of this kind of message, times the chance of each of its words.""" score = math.log(model["messages"][spam] / sum(model["messages"].values())) for w in words: p = word_chance(model, w, spam) if p == 0: return float("-inf") score += math.log(p) return score def chance_of_spam(model, words): a, b = log_score(model, words, True), log_score(model, words, False) if a == b: return 0.5 top = max(a, b) return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top)) def accuracy(model, data): return sum((chance_of_spam(model, words) >= 0.5) == spam for words, spam in data) / len(data) import numpy as np test = messages(1000, 1) vocabulary = sorted(SPAM_WORDS) column = {w: i for i, w in enumerate(vocabulary)} def as_counts(data): X = np.zeros((len(data), len(vocabulary))) for row, (words, spam) in enumerate(data): for w in words: X[row, column[w]] += 1 return X, np.array([spam for words, spam in data], dtype=float) def logistic_regression(X, y, steps=500, rate=0.3, penalty=0.01): w, b = np.zeros(X.shape[1]), 0.0 for step in range(steps): p = 1 / (1 + np.exp(-(X @ w + b))) w -= rate * (X.T @ (p - y) / len(y) + penalty * w) b -= rate * (p - y).mean() return w, b X_test, y_test = as_counts(test) print("training messages | naive Bayes | logistic regression") for n in (10, 20, 50, 200, 1000): bayes, logistic = [], [] # 10 training sets of each size for seed in range(10): data = messages(n, 100 + seed) bayes.append(accuracy(fit(data), test)) w, b = logistic_regression(*as_counts(data)) logistic.append(float((((X_test @ w + b) > 0) == y_test).mean())) print(str(n).rjust(17), "|", f"{sum(bayes) / 10:.3f}".rjust(11), "|", f"{sum(logistic) / 10:.3f}")
With 10 training messages naive Bayes is right 70.4% of the time, and logistic regression 65.3%. It stays ahead through 50 and 200. By 1,000 the two are level, at 0.861 and 0.857, and with more data still logistic regression usually pulls in front.
The strong assumption is the reason for both halves. A model that assumes a lot needs little data to fill in the rest, and is held back by its assumptions once data is plentiful. Use naive Bayes for a first model on text, a baseline to beat, or when labels are scarce. It trains in one pass, updates by adding to its counts, and can explain itself.
Show the answer
Trust its labels, and don't treat its chances as probabilities
This is the normal behavior of naive Bayes. Counting overlapping evidence as independent pushes its chances toward 0 and 1, so "99%" means something less. The ranking and the labels remain useful. If a later step needs real probabilities, such as an expected cost, the scores should be recalibrated on held-out data first.
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: a first model for text, and why it decided
Purpose. Know when counting is the right first model, check the library against your own code, and explain one decision.
Worked example: why was this message flagged?
Since the scores are sums of logs, the difference between a message's spam score and its ham score is a sum too: one number per word, saying how far that word pushes toward spam (positive) or ham (negative). That makes a naive Bayes decision easy to explain.
Uses the setup above, 67 lines.
model = fit(messages(150, 0)) def evidence(model, word): """How far one use of this word moves the score toward spam (positive) or toward ham (negative).""" return (math.log(word_chance(model, word, True)) - math.log(word_chance(model, word, False))) ranked = sorted(model["vocabulary"], key=lambda w: evidence(model, w)) print("strongest signs of ham :", [(w, round(evidence(model, w), 2)) for w in ranked[:4]]) print("strongest signs of spam:", [(w, round(evidence(model, w), 2)) for w in ranked[-4:][::-1]]) message = ["urgent", "invoice", "call", "offer", "prize", "today", "team"] start = math.log(model["messages"][True] / model["messages"][False]) print() print("message:", " ".join(message)) print(" before any word".ljust(22), f"{start:+.2f}") total = start for w in message: total += evidence(model, w) print(" ", w.ljust(18), f"{evidence(model, w):+.2f}", "-> running total", f"{total:+.2f}") print("chance of spam:", round(1 / (1 + math.exp(-total)), 3), "| the function gives", round(chance_of_spam(model, message), 3))
The strongest signs of spam are "prize" and "winner", each worth about +2.07, and the strongest signs of ham are "lunch" and "report". Then one message is taken apart. Before any word, the total is −0.60: spam is the less common kind, so the starting point leans to ham. "urgent", "invoice" and "call" barely move it. "offer" adds +1.01 and "prize" +2.07, and the total is at +2.74. "today" and "team" pull back a little, and the message ends at +1.85, which is a chance of 0.864, exactly what the model's own function reports. Two things are worth taking from this. The decision came almost entirely from two words, which you can tell a user. And the model is a weighted sum of word counts passed through the same curve as logistic regression. The two models have the same form. They differ in how the weights are found: one by counting, the other by gradient descent.
See why. The push each word gives, as a bar: positive toward spam, negative toward ham.
Uses the setup above, 67 lines.
import matplotlib.pyplot as plt model = fit(messages(150, 0)) message = ["winner", "prize", "meeting", "free", "team"] push = [math.log(word_chance(model, w, True)) - math.log(word_chance(model, w, False)) for w in message] plt.bar(message, push) plt.axhline(0, color="gray") plt.ylabel("push toward spam (log ratio)") print([round(p, 2) for p in push])
Add the bars and you have the difference between the two scores, which is what makes a naive Bayes decision easy to explain.
import math in_spam = {"prize": 0.08, "now": 0.05, "lunch": 0.01} in_ham = {"prize": 0.01, "now": 0.05, "lunch": 0.04} print([round(math.log(in_spam[w] / in_ham[w]), 2) for w in in_spam])
Show the answer
[2.08, 0.0, -1.39]
"prize" is 8 times as likely in spam, and log(8) is 2.08: strong evidence for spam. "now" is equally likely in both, a ratio of 1, and log(1) is 0: no evidence. "lunch" is a quarter as likely in spam, and log(0.25) is −1.39: evidence for ham. Taking logs turns ratios into amounts that can be added, with 0 meaning "says nothing".
The library, checked equal. CountVectorizer turns each message into word counts, and MultinomialNB(alpha=1) is the same counting with 1 added to every count. On the same messages it should match the from-scratch accuracy:
Uses the setup above, 67 lines.
from sklearn.feature_extraction.text import CountVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import make_pipeline train, test = messages(150, 0), messages(1000, 1) text = lambda data: [" ".join(words) for words, spam in data] label = lambda data: [spam for words, spam in data] pipe = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1)) pipe.fit(text(train), label(train)) print("scikit-learn:", round(pipe.score(text(test), label(test)), 3)) print("from scratch:", round(accuracy(fit(train), test), 3))
Goes deeper in
- Text mining: TF-IDF weights, search and labels.
- Logistic regression: the discriminative model that learns the weights instead of counting.
- Imbalanced classes: what changes when the prior is far from 50%.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 8 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
What is the "naive" assumption in naive Bayes?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write