Index Ai Academy
Phase 3 · Models

Naive Bayes: classifying with counts

A classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Naive Bayes is the classifier you can train by counting. It has no gradient descent and no learning rate: it reads the training data once, keeps a table, and it is done. For decades it was how spam filters worked, and it is still a strong first model for text when labeled examples are few. It also rests on an assumption that is plainly false, and seeing exactly what that costs, and what it doesn't, is one of the better lessons in how machine learning models get away with being wrong.

What you'll be able to do

  • Turn word counts into the chance that a message is spam, using Bayes' rule
  • State the naive assumption, and explain why multiplying one chance per word needs logarithms
  • Fix the zero-count problem with smoothing, and say when it matters most
  • Show why naive Bayes is overconfident when its evidence overlaps
  • Decide when naive Bayes is the right first model, and read why it made a decision
Quick check
A guess before you start. In your inbox, 35% of messages are spam. The word "winner" appears in 23 messages, and 19 of those are spam. A new message contains "winner". Before reading anything else, what is a fair estimate of the chance it is spam?
  1. About 35%: the share of all messages that are spam
  2. About 83%: 19 of the 23 messages with that word
  3. 100%: only spam ever uses the word "winner"
  4. About 54%: the average of 35% and 83%, to be safe
Show the answer

About 83%: 19 of the 23 messages with that word

The question is about messages that contain "winner", so look only at those: 19 of 23 were spam, about 83%. The 35% was your belief before seeing the word. The word is evidence, and it moved the estimate a long way. That move, from a starting belief to an updated one, is all Bayes' rule is.

The idea: from one word to a verdict

This rests on Conditional probability and Bayes' rule (Foundation lesson 90): how a word changes the chance of a class.

The data for this lesson is made up, so that it can be as large as needed: short messages, each a list of words and a label. Spam and ordinary mail ("ham") use the same words at different rates. "winner" turns up in both, far more often in spam.

Setup (runs before every cell) · 67 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math, random

# How often each word turns up in the two kinds of message. Every word can appear in
# both: only the rates differ.
SPAM_WORDS = {"free": 5, "winner": 2, "prize": 2, "click": 4, "offer": 4, "money": 4,
              "urgent": 3, "now": 4, "account": 4, "deal": 3, "meeting": 2,
              "report": 2, "lunch": 1, "team": 2, "thanks": 3, "tomorrow": 2,
              "project": 2, "invoice": 3, "call": 3, "today": 4, "raffle": 0.3,
              "refund": 0.5}
HAM_WORDS = {"free": 2, "winner": 0.3, "prize": 0.3, "click": 2, "offer": 2, "money": 2,
             "urgent": 2, "now": 4, "account": 4, "deal": 1, "meeting": 5,
             "report": 5, "lunch": 4, "team": 5, "thanks": 5, "tomorrow": 4,
             "project": 5, "invoice": 3, "call": 4, "today": 4, "raffle": 0.5,
             "refund": 1}

def messages(n, seed):
    """n made-up messages, each a list of words and whether it is spam. About 4 in 10
    are spam."""
    rng, out = random.Random(seed), []
    for i in range(n):
        spam = rng.random() < 0.4
        weights = SPAM_WORDS if spam else HAM_WORDS
        out.append(
            (rng.choices(list(weights), list(weights.values()), k=rng.randint(5, 12)),
             spam))
    return out

def fit(data, smooth=1):
    """Training is counting: messages of each kind, and each word's uses in each
    kind."""
    model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0},
             "messages": {True: 0, False: 0}, "smooth": smooth,
             "vocabulary": sorted({w for words, spam in data for w in words})}
    for words, spam in data:
        model["messages"][spam] += 1
        for w in words:
            model["words"][spam][w] = model["words"][spam].get(w, 0) + 1
            model["total"][spam] += 1
    return model

def word_chance(model, word, spam):
    """The chance that a word drawn from this kind of message is this word, with
    `smooth` added to every count."""
    count = model["words"][spam].get(word, 0) + model["smooth"]
    return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"]))

def log_score(model, words, spam):
    """log of: the chance of this kind of message, times the chance of each of its
    words."""
    score = math.log(model["messages"][spam] / sum(model["messages"].values()))
    for w in words:
        p = word_chance(model, w, spam)
        if p == 0:
            return float("-inf")
        score += math.log(p)
    return score

def chance_of_spam(model, words):
    a, b = log_score(model, words, True), log_score(model, words, False)
    if a == b:
        return 0.5
    top = max(a, b)
    return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top))

def accuracy(model, data):
    return sum((chance_of_spam(model, words) >= 0.5) == spam
               for words, spam in data) / len(data)
train = messages(150, 0)
spam_count = sum(spam for words, spam in train)
print("messages:", len(train), "| spam:", spam_count,
      "| so before reading a word, the chance of spam is",
      round(spam_count / len(train), 3))
print()
print("word     | messages with it | of those, spam | chance of spam, given the word")
for word in ("winner", "free", "account", "meeting"):
    having = [spam for words, spam in train if word in words]
    print(word.ljust(8), "|", str(len(having)).rjust(16), "|",
          str(sum(having)).rjust(14), "|", round(sum(having) / len(having), 3))

Before reading a word, the chance of spam is 0.353: 53 of 150 messages. That starting belief is called the prior.

Each word then moves it. "winner" moves it to 0.826, and "meeting" down to 0.242. "account" leaves it at 0.351, almost exactly where it started: a word used equally by both kinds of message is no evidence at all.

Counting messages that contain a word works for one word. A real message has many, and you will never find 23 earlier messages containing exactly the same eight words. So the calculation is turned round, with Bayes' rule:

chance of spam, given the words ∝ chance of spam × chance of those words in spam

The symbol ∝ means "in proportion to". Work the right-hand side out for spam, and again for ham, and compare the two. The first factor is the prior. The second asks how likely this message would be if it were spam, and the next part is about how to get that from counts.

Try it now

Set the column to keep to 1: given no winner, the shares are 34/128 and 93/128.

Quick check
From the lesson on conditional probability and Bayes: a test for a rare condition comes back positive. Why isn't the chance of having the condition simply the test's accuracy?
  1. Rare conditions can't be detected by any test
  2. The accuracy applies only to people who are tested twice
  3. The condition's rarity matters too: most positives can be false alarms
  4. A positive test is always certain, whatever its accuracy
Show the answer

The condition's rarity matters too: most positives can be false alarms

Bayes' rule combines the evidence with the prior. When a condition is rare, the few true positives are outnumbered by false alarms from the many healthy people, so a positive result may still mean a small chance. Naive Bayes does the same with messages: the prior, how common spam is, multiplies the evidence from the words.

Many words, and one bold assumption

How likely is a whole message, if it is spam? Naive Bayes makes an assumption that turns a hard question into a product:

Given the kind of message, each word appears independently of the others.

Then the chance of the message is just the chance of its first word in spam, times the chance of its second, and so on, each read from a table of counts. Training is filling in that table.

Quick check
A tiny model has these chances for two words. With a prior of 0.5 for each class, what does this print: the chance that the message "free now" is spam?
spam = {"free": 0.20, "now": 0.10}
ham = {"free": 0.05, "now": 0.10}
as_spam = 0.5 * spam["free"] * spam["now"]
as_ham = 0.5 * ham["free"] * ham["now"]
print(round(as_spam / (as_spam + as_ham), 2))
  1. 0.2
  2. 0.01
  3. 0.8
  4. 0.5
Show the answer

0.8

As spam the message scores 0.5 × 0.20 × 0.10 = 0.01. As ham it scores 0.5 × 0.05 × 0.10 = 0.0025. The chance of spam is 0.01 out of the two together: 0.01 / 0.0125 = 0.8. "now" is equally likely in both and cancels out. All the evidence came from "free", which is four times as common in spam.

In your own words
The word "account" left the chance of spam almost unchanged. What makes a word good evidence?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: counts, logs and smoothing

Purpose. Train by counting, score with added logs, and fix the zero-count trap.

Uses the setup above, 67 lines.

def fit(data, smooth=1):
    """Training is counting: messages of each kind, and each word's uses in each
    kind."""
    model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0},
             "messages": {True: 0, False: 0}, "smooth": smooth,
             "vocabulary": sorted({w for words, spam in data for w in words})}
    for words, spam in data:
        model["messages"][spam] += 1
        for w in words:
            model["words"][spam][w] = model["words"][spam].get(w, 0) + 1
            model["total"][spam] += 1
    return model

def word_chance(model, word, spam):
    """The chance that a word drawn from this kind of message is this word, with
    `smooth` added to every count."""
    count = model["words"][spam].get(word, 0) + model["smooth"]
    return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"]))

def log_score(model, words, spam):
    """log of: the chance of this kind of message, times the chance of each of its
    words."""
    score = math.log(model["messages"][spam] / sum(model["messages"].values()))
    for w in words:
        p = word_chance(model, w, spam)
        if p == 0:
            return float("-inf")
        score += math.log(p)
    return score

def chance_of_spam(model, words):
    a, b = log_score(model, words, True), log_score(model, words, False)
    if a == b:
        return 0.5
    top = max(a, b)
    return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top))

def accuracy(model, data):
    return sum((chance_of_spam(model, words) >= 0.5) == spam
               for words, spam in data) / len(data)

train, test = messages(150, 0), messages(1000, 1)
model = fit(train)

for words in (["winner", "prize", "click", "now"],
              ["meeting", "tomorrow", "team", "lunch"],
              ["free", "lunch", "tomorrow", "offer"]):
    print(" ".join(words).ljust(28), "| log score as spam",
          f"{log_score(model, words, True):7.2f}", "| as ham",
          f"{log_score(model, words, False):7.2f}", "| chance of spam",
          f"{chance_of_spam(model, words):.3f}")

print()
print("right on", round(100 * accuracy(model, test), 1),
      "% of 1,000 new messages. Always guessing 'not spam' would be right on",
      round(100 * sum(not spam for words, spam in test) / len(test), 1), "%")

product = 1.0
for i in range(400):
    product *= word_chance(model, "free", True)
print("a 400-word message, chances multiplied:", product, "| logs added:",
      round(400 * math.log(word_chance(model, "free", True)), 1))

The first message is given a 0.989 chance of spam and the second 0.039. The third mixes both kinds of word and lands at 0.415: unsure, as it should be.

On 1,000 new messages the model is right 83.9% of the time. Always answering "not spam" would be right 58.1% of the time, so the table of counts has learned something real, with no training loop.

The assumption is false, and obviously so. Words in a message are not independent: "prize" makes "winner" more likely. That is what naive means in the name. The model works anyway, since to pick the right class it only needs the two scores to come out in the right order, and it usually does that even when both are badly off.

The last line is a practical point. The scores are added logarithms. Multiplying 400 small chances gives exactly 0.0 on a computer, for both classes, and nothing can be compared. Their logs add up to an ordinary number, −1,041.9.

Drill

The chance of spam, without underflow

Write chance_of_spam(prior, spam_chances, ham_chances). prior is the chance a message is spam before reading it. The two lists hold the chance of each of the message's words under spam and under ham. Work in logarithms: add up log(prior) and the logs of the spam chances, and log(1 - prior) and the logs of the ham chances. Then turn the two totals into a chance of spam. Subtract the larger total from both before using math.exp, so that long messages don't underflow.

Uses the setup above, 67 lines.

import math


def chance_of_spam(prior, spam_chances, ham_chances):
    return 0.5


print(round(chance_of_spam(0.5, [0.20, 0.10], [0.05, 0.10]), 3))
In the lesson, your code is checked as you write it.

The word it has never seen

A table of counts has a trap in it. Suppose the word "lunch" has never appeared in a spam message in the training data. Its chance in spam is then 0, and one zero in a product makes the whole product zero. A single unseen word rules a class out completely, however strong the rest of the evidence.

Uses the setup above, 67 lines.

test = messages(1000, 1)
small = messages(30, 200)
plain, smoothed = fit(small, smooth=0), fit(small, smooth=1)
never = [w for w in plain["vocabulary"]
         if w not in plain["words"][True] or w not in plain["words"][False]]
print("after 30 messages, words never yet seen in one of the two kinds:", never)
words = ["winner", "team", "lunch", "thanks", "tomorrow"]
print(" ".join(words), "| without smoothing: as spam",
      round(log_score(plain, words, True), 2), "as ham", log_score(plain, words, False),
      "| with smoothing:", round(log_score(smoothed, words, True), 2),
      round(log_score(smoothed, words, False), 2))

print()
print(
    "training messages | accuracy, raw counts | accuracy, 1 added to every count | "
    "test messages with a kind ruled out")
for n in (30, 60, 150):
    raw = added = ruled_out = 0
    # 20 different training sets of each size
    for seed in range(20):
        data = messages(n, 200 + seed)
        a, b = fit(data, smooth=0), fit(data, smooth=1)
        raw += accuracy(a, test) / 20
        added += accuracy(b, test) / 20
        pairs = [(log_score(a, w, True), log_score(a, w, False)) for w, spam in test]
        ruled_out += sum(float("-inf") in pair for pair in pairs) / 20
    print(str(n).rjust(17), "|", f"{raw:.3f}".rjust(20), "|", f"{added:.3f}".rjust(32),
          "|", round(ruled_out), "of 1,000")

After 30 messages, four words have never been seen in one kind or the other. The message in the second line contains two of them, one missing from each side: its score is minus infinity as spam and as ham. The model has proved that the message can't exist.

The repair is smoothing: add 1 to every count before dividing, as if each word had been seen once more in each class than it was. No chance is ever exactly zero, and a word never seen becomes merely unlikely. With it, the same message gets two ordinary scores and a sensible answer.

The table shows when this matters. With 30 training messages, 352 of 1,000 test messages have a class ruled out, and smoothing lifts accuracy from 0.747 to 0.787. With 150 messages only 8 are affected, and the two columns nearly agree. Zeros are a disease of small data and big vocabularies. Real text has vocabularies of tens of thousands of words, so some smoothing is always on.

Quick check
A word appears 0 times among the 200 spam words in the training data. The vocabulary has 50 words. What does this print: the word's chance in spam with raw counts, and with 1 added to every count?
count, total, vocabulary = 0, 200, 50
print(count / total, (count + 1) / (total + vocabulary))
  1. 0.0 0.02
  2. 0.0 0.004
  3. 0.0 0.005
  4. 0.004 0.004
Show the answer

0.0 0.004

Raw, the chance is 0 / 200 = 0. Smoothed, the count becomes 1, and the total grows by one for each of the 50 words in the vocabulary, since every word had 1 added: 1 / 250 = 0.004. Adding the vocabulary size to the total keeps the chances of all the words adding up to 1.

Drill

A chance that is never zero

Write smoothed(count, total, vocabulary_size, smooth): the chance of a word in a class, as (count + smooth) divided by (total + smooth * vocabulary_size). If that bottom number is 0, there is nothing to go on: return 0.0.

Uses the setup above, 67 lines.

def smoothed(count, total, vocabulary_size, smooth):
    return 0.0


print(smoothed(0, 200, 50, 1), smoothed(30, 200, 50, 1))
In the lesson, your code is checked as you write it.

Sure of itself, and quick to learn

The naive assumption has a price, and it is paid in confidence. The cell trains the same model three times. In the second and third runs every word in every message is typed two or four times. That adds no information at all, and a model that understood the words were copies would not change.

Uses the setup above, 67 lines.

train, test = messages(150, 0), messages(1000, 1)
print(
    "each word counted | accuracy | messages it is 99% sure about | of those, how many "
    "it got right")
for copies in (1, 2, 4):
    # every word typed `copies` times: no new information
    model = fit([(words * copies, spam) for words, spam in train])
    chances = [(chance_of_spam(model, words * copies), spam) for words, spam in test]
    sure = [(p, spam) for p, spam in chances if p >= 0.99 or p <= 0.01]
    right = sum((p >= 0.5) == spam for p, spam in sure)
    print((str(copies) + " time" + ("s" if copies > 1 else "")).ljust(17), "|",
          f"{sum((p >= 0.5) == spam for p, spam in chances) / len(test):.3f}".rjust(8),
          "|", str(len(sure)).rjust(29), "|", f"{100 * right / len(sure):.1f}%")

Accuracy doesn't move: 0.839, 0.840, 0.842. The confidence does. With each word counted once, the model is 99% sure about 89 messages, and right about all of them. With each word counted four times it is 99% sure about 697, and right about 92.7% of those. It is claiming 99% and delivering 93%.

Naive Bayes treats every word as a fresh, independent piece of evidence. When words travel together, as "prize" and "winner" do, it counts the same evidence twice and overstates its case. So its labels can be trusted more than its chances. If you need honest probabilities, they have to be recalibrated afterward.

Now the reason it has survived.

Uses the setup above, 67 lines.

import math, random

# How often each word turns up in the two kinds of message. Every word can appear in
# both: only the rates differ.
SPAM_WORDS = {"free": 5, "winner": 2, "prize": 2, "click": 4, "offer": 4, "money": 4,
              "urgent": 3, "now": 4, "account": 4, "deal": 3, "meeting": 2,
              "report": 2, "lunch": 1, "team": 2, "thanks": 3, "tomorrow": 2,
              "project": 2, "invoice": 3, "call": 3, "today": 4, "raffle": 0.3,
              "refund": 0.5}
HAM_WORDS = {"free": 2, "winner": 0.3, "prize": 0.3, "click": 2, "offer": 2, "money": 2,
             "urgent": 2, "now": 4, "account": 4, "deal": 1, "meeting": 5,
             "report": 5, "lunch": 4, "team": 5, "thanks": 5, "tomorrow": 4,
             "project": 5, "invoice": 3, "call": 4, "today": 4, "raffle": 0.5,
             "refund": 1}

def messages(n, seed):
    """n made-up messages, each a list of words and whether it is spam. About 4 in 10
    are spam."""
    rng, out = random.Random(seed), []
    for i in range(n):
        spam = rng.random() < 0.4
        weights = SPAM_WORDS if spam else HAM_WORDS
        out.append(
            (rng.choices(list(weights), list(weights.values()), k=rng.randint(5, 12)),
             spam))
    return out

def fit(data, smooth=1):
    """Training is counting: messages of each kind, and each word's uses in each
    kind."""
    model = {"words": {True: {}, False: {}}, "total": {True: 0, False: 0},
             "messages": {True: 0, False: 0}, "smooth": smooth,
             "vocabulary": sorted({w for words, spam in data for w in words})}
    for words, spam in data:
        model["messages"][spam] += 1
        for w in words:
            model["words"][spam][w] = model["words"][spam].get(w, 0) + 1
            model["total"][spam] += 1
    return model

def word_chance(model, word, spam):
    """The chance that a word drawn from this kind of message is this word, with
    `smooth` added to every count."""
    count = model["words"][spam].get(word, 0) + model["smooth"]
    return count / (model["total"][spam] + model["smooth"] * len(model["vocabulary"]))

def log_score(model, words, spam):
    """log of: the chance of this kind of message, times the chance of each of its
    words."""
    score = math.log(model["messages"][spam] / sum(model["messages"].values()))
    for w in words:
        p = word_chance(model, w, spam)
        if p == 0:
            return float("-inf")
        score += math.log(p)
    return score

def chance_of_spam(model, words):
    a, b = log_score(model, words, True), log_score(model, words, False)
    if a == b:
        return 0.5
    top = max(a, b)
    return math.exp(a - top) / (math.exp(a - top) + math.exp(b - top))

def accuracy(model, data):
    return sum((chance_of_spam(model, words) >= 0.5) == spam
               for words, spam in data) / len(data)

import numpy as np

test = messages(1000, 1)
vocabulary = sorted(SPAM_WORDS)
column = {w: i for i, w in enumerate(vocabulary)}

def as_counts(data):
    X = np.zeros((len(data), len(vocabulary)))
    for row, (words, spam) in enumerate(data):
        for w in words:
            X[row, column[w]] += 1
    return X, np.array([spam for words, spam in data], dtype=float)

def logistic_regression(X, y, steps=500, rate=0.3, penalty=0.01):
    w, b = np.zeros(X.shape[1]), 0.0
    for step in range(steps):
        p = 1 / (1 + np.exp(-(X @ w + b)))
        w -= rate * (X.T @ (p - y) / len(y) + penalty * w)
        b -= rate * (p - y).mean()
    return w, b

X_test, y_test = as_counts(test)
print("training messages | naive Bayes | logistic regression")
for n in (10, 20, 50, 200, 1000):
    bayes, logistic = [], []
    # 10 training sets of each size
    for seed in range(10):
        data = messages(n, 100 + seed)
        bayes.append(accuracy(fit(data), test))
        w, b = logistic_regression(*as_counts(data))
        logistic.append(float((((X_test @ w + b) > 0) == y_test).mean()))
    print(str(n).rjust(17), "|", f"{sum(bayes) / 10:.3f}".rjust(11), "|",
          f"{sum(logistic) / 10:.3f}")
It runs right here in your browser, with nothing to install.

With 10 training messages naive Bayes is right 70.4% of the time, and logistic regression 65.3%. It stays ahead through 50 and 200. By 1,000 the two are level, at 0.861 and 0.857, and with more data still logistic regression usually pulls in front.

The strong assumption is the reason for both halves. A model that assumes a lot needs little data to fill in the rest, and is held back by its assumptions once data is plentiful. Use naive Bayes for a first model on text, a baseline to beat, or when labels are scarce. It trains in one pass, updates by adding to its counts, and can explain itself.

Quick check
A naive Bayes spam filter is about 85% accurate. Of the messages it calls spam with 99% confidence, about 93% really are. What should a team take from that?
  1. Its accuracy would be 99% if the threshold were raised
  2. The model is broken and should be retrained from scratch
  3. Its confidence is exact, and the test data must be wrong
  4. Trust its labels, and don't treat its chances as probabilities
Show the answer

Trust its labels, and don't treat its chances as probabilities

This is the normal behavior of naive Bayes. Counting overlapping evidence as independent pushes its chances toward 0 and 1, so "99%" means something less. The ranking and the labels remain useful. If a later step needs real probabilities, such as an expected cost, the scores should be recalibrated on held-out data first.

In your own words
Why does one unseen word do so much damage to a naive Bayes model without smoothing?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: a first model for text, and why it decided

Purpose. Know when counting is the right first model, check the library against your own code, and explain one decision.

Worked example

Worked example: why was this message flagged?

Since the scores are sums of logs, the difference between a message's spam score and its ham score is a sum too: one number per word, saying how far that word pushes toward spam (positive) or ham (negative). That makes a naive Bayes decision easy to explain.

Uses the setup above, 67 lines.

model = fit(messages(150, 0))

def evidence(model, word):
    """How far one use of this word moves the score toward spam (positive) or toward
    ham (negative)."""
    return (math.log(word_chance(model, word, True))
            - math.log(word_chance(model, word, False)))

ranked = sorted(model["vocabulary"], key=lambda w: evidence(model, w))
print("strongest signs of ham :",
      [(w, round(evidence(model, w), 2)) for w in ranked[:4]])
print("strongest signs of spam:",
      [(w, round(evidence(model, w), 2)) for w in ranked[-4:][::-1]])

message = ["urgent", "invoice", "call", "offer", "prize", "today", "team"]
start = math.log(model["messages"][True] / model["messages"][False])
print()
print("message:", " ".join(message))
print("  before any word".ljust(22), f"{start:+.2f}")
total = start
for w in message:
    total += evidence(model, w)
    print("  ", w.ljust(18), f"{evidence(model, w):+.2f}", "-> running total",
          f"{total:+.2f}")
print("chance of spam:", round(1 / (1 + math.exp(-total)), 3), "| the function gives",
      round(chance_of_spam(model, message), 3))
In the lesson you can change this and run it.

The strongest signs of spam are "prize" and "winner", each worth about +2.07, and the strongest signs of ham are "lunch" and "report". Then one message is taken apart. Before any word, the total is −0.60: spam is the less common kind, so the starting point leans to ham. "urgent", "invoice" and "call" barely move it. "offer" adds +1.01 and "prize" +2.07, and the total is at +2.74. "today" and "team" pull back a little, and the message ends at +1.85, which is a chance of 0.864, exactly what the model's own function reports. Two things are worth taking from this. The decision came almost entirely from two words, which you can tell a user. And the model is a weighted sum of word counts passed through the same curve as logistic regression. The two models have the same form. They differ in how the weights are found: one by counting, the other by gradient descent.

See why. The push each word gives, as a bar: positive toward spam, negative toward ham.

Uses the setup above, 67 lines.

import matplotlib.pyplot as plt

model = fit(messages(150, 0))
message = ["winner", "prize", "meeting", "free", "team"]
push = [math.log(word_chance(model, w, True)) - math.log(word_chance(model, w, False))
        for w in message]
plt.bar(message, push)
plt.axhline(0, color="gray")
plt.ylabel("push toward spam (log ratio)")
print([round(p, 2) for p in push])

Add the bars and you have the difference between the two scores, which is what makes a naive Bayes decision easy to explain.

Quick check
Each word's evidence is the log of its chance in spam divided by its chance in ham. What does this print for the three words?
import math

in_spam = {"prize": 0.08, "now": 0.05, "lunch": 0.01}
in_ham = {"prize": 0.01, "now": 0.05, "lunch": 0.04}
print([round(math.log(in_spam[w] / in_ham[w]), 2) for w in in_spam])
  1. [0.07, 0.0, -0.03]
  2. [2.08, 0.0, -1.39]
  3. [8.0, 1.0, 0.25]
  4. [2.08, 1.0, -1.39]
Show the answer

[2.08, 0.0, -1.39]

"prize" is 8 times as likely in spam, and log(8) is 2.08: strong evidence for spam. "now" is equally likely in both, a ratio of 1, and log(1) is 0: no evidence. "lunch" is a quarter as likely in spam, and log(0.25) is −1.39: evidence for ham. Taking logs turns ratios into amounts that can be added, with 0 meaning "says nothing".

The library, checked equal. CountVectorizer turns each message into word counts, and MultinomialNB(alpha=1) is the same counting with 1 added to every count. On the same messages it should match the from-scratch accuracy:

Uses the setup above, 67 lines.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

train, test = messages(150, 0), messages(1000, 1)
text = lambda data: [" ".join(words) for words, spam in data]
label = lambda data: [spam for words, spam in data]
pipe = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1))
pipe.fit(text(train), label(train))
print("scikit-learn:", round(pipe.score(text(test), label(test)), 3))
print("from scratch:", round(accuracy(fit(train), test), 3))

Goes deeper in

In your own words
You have 60 labeled support messages and need a classifier by tomorrow. Make the case for naive Bayes, and say what you would warn the team about.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

What is the "naive" assumption in naive Bayes?

  1. Given the class, each feature is independent of the others
  2. Every class is equally likely before any evidence
  3. Every word is equally likely in every class
  4. The training data contains no mistakes

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write