Samples and confidence

Confidence intervals

A map app doesn't promise you'll arrive at exactly 6:14. It says "6:10 to 6:20". A range is more honest than one exact number. A confidence interval does the same for an estimate.

With enough data, estimates like averages and rates roughly follow a bell curve around the true value. This fact is called the central limit theorem. So the true value is usually within about two standard errors of your estimate. A 95% confidence interval is

In plain words: start at your estimate. Go 1.96 standard errors down for the low end and 1.96 standard errors up for the high end. The ± sign means "plus or minus".

Worked example. The click rate was 0.8 with a standard error of 0.04. Then 1.96 × 0.04 is about 0.078. The interval runs from 0.8 − 0.078 to 0.8 + 0.078: about 72% to 88%.

se = 0.04
print(round(0.8 - 1.96 * se, 3), round(0.8 + 1.96 * se, 3))

It prints 0.722 0.878.

What does "95%" mean exactly? Imagine repeating the whole study many times, building an interval each time. About 95% of those intervals would contain the true value. In practice, read it as the range of values that fit your data. Put another way: the data are consistent with a true rate anywhere in that range, and the method captures the true rate about 95% of the time.

Here is the same recipe on real-sized numbers: a conversion rate and an eval score.

import numpy as np

conversions, visitors = 64, 2000
p_hat = conversions / visitors
se = np.sqrt(p_hat * (1 - p_hat) / visitors)
low, high = p_hat - 1.96 * se, p_hat + 1.96 * se
print(f"conversion rate {p_hat:.2%}, 95% CI {low:.2%} to {high:.2%}")

# An eval: 42 of 50 test questions correct
p_hat, n = 42 / 50, 50
se = np.sqrt(p_hat * (1 - p_hat) / n)
print(f"eval accuracy {p_hat:.0%}, 95% CI {p_hat - 1.96 * se:.0%} to {p_hat + 1.96 * se:.0%}")

CI is short for confidence interval. The eval interval runs from about 74% to 94%. That's 20 points wide. Now say a new prompt scores 88% on the same 50 questions. It sits inside that range, so it isn't clearly better. With only 50 questions, you can only spot big differences. That's why serious evals use hundreds of examples.

The bootstrap: intervals for anything

The formulas cover averages and rates. But often you want an interval for something else. That might be a median, which is the middle value, or a ratio. It might be the 90th percentile: the value that 90% of results fall below. It might be an F1 score, which is a way to grade a classifier, a model that sorts things into groups. For those, use the bootstrap.

Picture your 300 receipts in a hat. Draw one, write down its amount, and put it back. Repeat 300 times. Putting it back is called sampling with replacement. Some receipts get drawn twice, some not at all. The result is a new, slightly different pile of 300 receipts: a resample.

The recipe:

1. Make a resample.
2. Work out your statistic on it, such as the median.
3. Repeat a couple of thousand times.
4. Sort the results and keep the middle 95%. That range is your interval.

import numpy as np

rng = np.random.default_rng(0)
order_values = rng.exponential(60, size=300)          # skewed, like real order values

boot = [np.median(rng.choice(order_values, size=len(order_values), replace=True)) for _ in range(2000)]
low, high = np.percentile(boot, [2.5, 97.5])
print(f"median order value {np.median(order_values):.2f}, 95% CI {low:.2f} to {high:.2f}")

The 2.5 and 97.5 percentiles cut off the lowest 2.5% and the highest 2.5% of results. What's left is the middle 95%. Notice the interval isn't centered on the median. Order values are lopsided, and the bootstrap picks that up on its own.

Why does it work? Your sample is your best picture of the population. Drawing from it again imitates going out and collecting another sample. It needs no formula, which makes it the everyday tool for eval metrics.

How much data do you need?

You can also run the interval backwards. Decide how precise you want to be, then work out how much data that takes.

The margin of error is how far the interval reaches each side of the estimate: half its width. Call it $m$. Let $p$ be the rate you expect to see, as a decimal:

The ≈ sign means "about equal to".

In plain words: multiply 1.96 squared by the rate and by one minus the rate. Divide by the margin squared. Always round up to a whole number.

Worked example. You expect about half of people to say yes, so $p$ is 0.5. You want to be within 5 points, so $m$ is 0.05. That's 3.8416 × 0.25 ÷ 0.0025 = 384.16. Round up: you need 385 people.

import math

print(math.ceil(1.96 ** 2 * 0.5 * (1 - 0.5) / 0.05 ** 2))

It prints 385. math.ceil always rounds up. Rounding down would leave you a little short of the precision you asked for.

The same formula works for a shop. To estimate a conversion rate near 4% within ±0.5 percentage points, you need about 5,900 visitors. Within ±1 point, you need about 1,500. Doubling the margin cut the data needed to a quarter: the square root rule again.

import math

opened, sent = 220, 1000
p_hat = opened / sent
se = math.sqrt(p_hat * (1 - p_hat) / sent)
print(f"open rate {p_hat:.1%}, SE {se:.2%}")
print(f"95% CI {p_hat - 1.96 * se:.1%} to {p_hat + 1.96 * se:.1%}")
print("emails for +/-1 point:", math.ceil(1.96 ** 2 * p_hat * (1 - p_hat) / 0.01 ** 2))