Conditional probability and Bayes

Language models are conditional probabilities

A language model reads some text and gives a chance for every possible next token. (A token is a word or a piece of a word.) Each of those chances depends on the text so far. After "Happy", the token "birthday" is likely. After "The", it isn't.

So the model estimates $P(\text{next token} \mid \text{all previous tokens})$.

To get the chance of a whole sentence, you chain these together one step at a time. Call the first word $w_1$ (w for word). The second word is $w_2$, and the third is $w_3$.

In plain words: the chance of the whole sentence is the chance of the first word, times the chance of the second word given the first, times the chance of the third word given the first two. The small gaps between the pieces mean "times".

A worked example, with made-up numbers to keep the sums easy. Say a model gives "the" a chance of 0.05 to start a sentence. Given "the", it gives "cat" a chance of 0.01. Given "the cat", it gives "sat" a chance of 0.2. Then "the cat sat" has a chance of 0.05 × 0.01 × 0.2 = 0.0001.

p_first = 0.05    # P("the")
p_second = 0.01   # P("cat" | "the")
p_third = 0.2     # P("sat" | "the cat")
print(round(p_first * p_second * p_third, 6))

It prints 0.0001. A long text multiplies hundreds of small numbers like these, and the result gets tiny fast. That's why the log rules matter: logs turn multiplying into adding. In practice these products become sums of log-probabilities.

Bayes' rule: flipping the condition

Often you know a chance the other way around from the one you want. The maker of a fraud filter tells you how often it flags fraud: $P(\text{flag} \mid \text{fraud})$. But you have a flagged order in front of you. You want $P(\text{fraud} \mid \text{flag})$. Bayes' rule flips one into the other:

That's two formulas side by side. The left one is Bayes' rule. The right one works out $P(B)$, the bottom of the fraction.

In plain words: to get the chance of A given B, take the chance of B when A is true. Multiply by how common A was to start with. Then divide by how common B is overall. B can happen in two ways: with A, or without A. The right-hand formula adds those two ways together.

Three words name the parts:

• $P(A)$ is the prior: what you believed before the evidence.
• $P(B \mid A)$ is the likelihood: how likely the evidence is if A is true.
• $P(A \mid B)$ is the posterior: what you believe after the evidence.

"Prior" means before, and "posterior" means after.

Example. A fraud filter flags 95% of fraudulent orders. It also wrongly flags 5% of legitimate ones. 1% of all orders are fraudulent. An order is flagged. What's the chance it's fraud?

Many people guess about 95%. Count it out with 10,000 orders instead:

• 1% are fraud, so 100 orders. The filter flags 95 of them.
• The other 9,900 are legit. 5% of those get flagged too: 495 orders.
• So 95 + 495 = 590 orders get flagged, and only 95 of them are fraud.

95 out of 590 is about 16%. In Bayes' terms, A is "fraud" and B is "flagged". The prior is 1%, and the likelihood is 95%. Bayes' rule gets the same answer without the counting:

prior = 0.01            # P(fraud)
hit = 0.95              # P(flag | fraud)
false_alarm = 0.05      # P(flag | legit)

p_flag = hit * prior + false_alarm * (1 - prior)
posterior = hit * prior / p_flag
print(f"P(flag) = {p_flag:.4f}")
print(f"P(fraud | flag) = {posterior:.3f}")

p_flag is the right-hand formula: flags from fraud plus flags from legit orders. It prints 0.0590, which is 590 flags out of 10,000 orders. The posterior is 0.161, only about 16%.

Most flags are false alarms because fraud is rare. Ignoring the prior like this is called the base-rate fallacy. The base rate is how common something is to start with. It's why a "95% accurate" detector can still be wrong most of the time it fires.

For marketers: the same trap applies to lead scoring. Say 2% of leads buy and your model is "90% accurate". Ask what share of the leads it flags actually buy. That's the posterior, also called precision, and it's the number sales cares about.

Updating step by step

Evidence often arrives in pieces. Think of a detective who updates a hunch each time a new clue turns up. Bayes' rule works the same way. The posterior after one piece of evidence becomes the prior for the next piece.

Say the same order trips two separate checks: the address doesn't match, and the order comes from a new device. Each check catches 95% of fraud and wrongly fires on 5% of legit orders. Assume the two checks work independently of each other. (That's a simplification. Real signals often overlap.)

def update(prior, hit, false_alarm):
    return hit * prior / (hit * prior + false_alarm * (1 - prior))

belief = 0.01
for check in ["address mismatch", "new device"]:
    belief = update(belief, 0.95, 0.05)
    print(f"after {check}: {belief:.3f}")

The first check moves the belief from 1% to 0.161. The second check starts from 0.161 and moves it to 0.785. Two independent warnings push the chance of fraud from 1% to about 78%.

This is the idea behind naive Bayes spam filters. Each word in an email updates the chance that it's spam. "Naive" refers to the assumption that the words are independent.

emails = 1000
spam = emails * 0.10           # 100 spam emails
normal = emails - spam         # 900 normal emails

free_in_spam = spam * 0.40     # 40 spam emails contain "free"
free_in_normal = normal * 0.02 # 18 normal emails contain "free"

share = free_in_spam / (free_in_spam + free_in_normal)
print("emails with 'free':", round(free_in_spam + free_in_normal))
print("P(spam | 'free') =", round(share, 2))