Bayesian rates: trusting small numbers

How strong should the prior be?

The prior's visits are its weight. A prior of 2 in 50 says "I'm about as sure as 50 visits would make me". Choose it from history: if past pages converted around 4%, and their rates varied about as much as 50 visits' worth of noise, then 2 in 50 is a fair summary.

Two failure modes are worth knowing. A prior that's too weak (say 0.04 sales in 1 visit) hardly shrinks anything, and tiny samples fool you again. A prior that's too strong (400 in 10,000) drowns real differences, and a genuinely great page takes forever to be believed. Whatever you pick, enough data always wins: with thousands of real visits, any sensible prior fades into the background.

Be careful where the prior comes from. Taking it from the very data you're judging counts that data twice. A fair prior comes from history, from similar items, or from a deliberate, stated assumption that others can question.

The whole picture: a Beta distribution

The posterior is more than one number. It's a whole spread of plausible rates, a Beta distribution named by two counts: α = prior sales + sales and β = prior non-sales + non-sales. Its mean is α / (α + β), the estimate you've been computing, and its width says how sure you can be.

Python can draw samples from it, which gives an interval with no formulas at all:

import random
random.seed(4)
alpha = 2 + 3            # prior sales + sales
beta = (50 - 2) + (4 - 3)   # prior non-sales + non-sales
draws = sorted(random.betavariate(alpha, beta) for _ in range(4000))
print(round(draws[100], 3), round(draws[3899], 3))   # the middle 95%

Bayesians call this a credible interval: given the prior and the data, the true rate is probably in there. With 4 visitors it's wide; with 5,000 it would be narrow. That width is useful in its own right: two pages whose intervals barely overlap are probably different, while two whose intervals mostly overlap may not be.

alpha, beta = 5, 49
print(round(alpha / (alpha + beta), 3))

import random
random.seed(5)
prior_sales, prior_visits = 2, 50
sales, visits = 12, 200

alpha = 0
beta = 0
low = 0
high = 0

print(alpha, beta, round(low, 3), round(high, 3))

Where this sits in Bayes' rule

The conditional probability lesson wrote Bayes' rule as P(A given B) = P(B given A) × P(A) / P(B): an updated belief is the old belief, P(A), reweighted by how well it explains the evidence. Adding counts to a prior is that same rule, worked out once and for all for rates. Statisticians call a prior that keeps its shape after updating "conjugate"; the Beta prior for rates is the classic example, which is why the update is just addition.

Bayesian rates in AI and product work

Rankings use a Bayesian average so a product with one perfect review doesn't top the list. A/B testing tools report the probability that B beats A from two Beta posteriors. Thompson sampling decides which variant to show next by drawing from each variant's posterior, so promising newcomers get tried without wasting traffic, and clear losers are quietly dropped as their posteriors settle low. And when you report an eval score from a handful of test cases, the same honesty applies: say how many cases there were, or shrink the score toward what similar systems usually get. Small numbers deserve modest claims, always, however exciting the first few results look.

prior_n, prior_mean = 10, 4.0
n, total = 1, 5.0
print(round((prior_n * prior_mean + total) / (prior_n + n), 2))