A/B tests and significance

The two-proportion z-test

The usual test for comparing two rates is the two-proportion z-test. ("Proportion" is another word for a rate or a share.) It needs a few names first.

For version A, $n_A$ is the number of visitors. Of those, $x_A$ converted. So A's measured rate is $\hat{p}_A = x_A / n_A$. That's "p-hat A", like the p-hat in the last lesson. Version B gets the same three names, with a B at the bottom.

The test then runs in three steps.

Step 1: pool the two groups. The null hypothesis says A and B share one true rate. Your best guess at that shared rate lumps both groups together. This is the pooled rate:

Step 2: find the standard error of the gap. You met the standard error (SE) in the last lesson. It's the typical size of the wobble.

Step 3: divide the gap by it. The result is called $z$.

In plain words: add up all the conversions and divide by all the visitors. That's the pooled rate. The SE says how big a gap luck alone would usually make, if both groups had that pooled rate. Then $z$ says how many of those SEs the real gap is. It's the same idea as the z-score in the normal distribution lesson.

Worked example with small numbers. A gets 10 sales from 100 visitors, or 10%. B gets 20 sales from 100 visitors, or 20%.

• Pooled rate: (10 + 20) ÷ (100 + 100) = 30 ÷ 200 = 0.15.
• SE: 0.15 × 0.85 = 0.1275. Next, 1/100 + 1/100 = 0.02. Multiply them to get 0.00255. The square root is about 0.0505.
• z: the gap is 0.20 − 0.10 = 0.10. Divide by 0.0505 and you get about 1.98.

import math

xa, na, xb, nb = 10, 100, 20, 100
pooled = (xa + xb) / (na + nb)
se = math.sqrt(pooled * (1 - pooled) * (1 / na + 1 / nb))
z = (xb / nb - xa / na) / se
print(round(pooled, 2), round(se, 4), round(z, 2))

It prints 0.15 0.0505 1.98. The gap is about two standard errors wide.

From z to a p-value

If the null hypothesis is true, $z$ follows the standard bell curve: mean 0, standard deviation

1. (That's an approximation, and a good one when the groups are large.) The two-sided p-value

is the chance the bell lands at least as far from zero as your $z$. "Two-sided" means a gap in either direction counts, B ahead or A ahead.

Books write that distance as $|z|$, the absolute value of z. It's z with its sign dropped, so $|-2| = 2$.

From the normal distribution lesson, 95% of the bell lies within 1.96 of the middle. So a $|z|$ above 1.96 is the same as a p-value below 0.05. In the small example, 1.98 is barely past that line. Its p-value is about 0.048. Even a 10-point gap is borderline with only 100 visitors each.

Now the real test, with 5,000 visitors per version. SciPy's norm.sf gives the upper tail of the bell. Doubling it covers both sides.

import numpy as np
from scipy.stats import norm

xa, na, xb, nb = 200, 5000, 245, 5000
pa, pb = xa / na, xb / nb
pooled = (xa + xb) / (na + nb)
se = np.sqrt(pooled * (1 - pooled) * (1 / na + 1 / nb))
z = (pb - pa) / se
p_value = 2 * norm.sf(abs(z))
print(f"A {pa:.2%}  B {pb:.2%}  z = {z:.2f}  p = {p_value:.4f}")

se_diff = np.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)
diff = pb - pa
print(f"difference {diff:+.2%}, 95% CI {diff - 1.96 * se_diff:+.2%} to {diff + 1.96 * se_diff:+.2%}")

The first line gives z = 2.18 and p ≈ 0.03. If there were no real difference, a gap this big would turn up about 3 times in 100. That's below 0.05, so the result is significant.

The second line is a 95% confidence interval for the lift. It's built as in the last lesson: the estimate plus or minus 1.96 standard errors. This time the SE uses each version's own rate, not the pooled one. The whole interval, +0.09% to +1.71%, sits above zero. That's more evidence B is really better.

Report the interval along with the p-value. A rise from 4.0% to 4.9% is 0.9 percentage points. "B lifts conversion by 0.9 points, plausibly between 0.1 and 1.7" says much more than "significant".

gap, se = 0.02, 0.01
print(round(gap / se, 1))

Power and sample size: plan before you start

Picture someone talking to you across a noisy room. A shout is easy to hear. A whisper needs you to listen much longer. Tests work the same way. A big lift shows up with little data. A small lift needs a lot.

Power is the chance a test detects a real effect of a given size. Most teams plan for 80% power. That means if the lift is real and as big as you planned for, the test catches it 4 times in 5. Small lifts on small baselines need a lot of traffic. (The baseline is the rate you have now.)

For a two-sided test at $\alpha = 0.05$ with 80% power, the visitors needed per variant are roughly:

Here $n$ is the number of visitors each version needs. The rate you have now is $p_A$. The rate you hope to reach is $p_B$. The 1.96 comes from the 5% significance level. The 0.84 comes from 80% power.

In plain words: add 1.96 and 0.84 to get 2.8, and square it to get 7.84. Multiply by how noisy the two rates are: the "rate times one minus the rate" parts. Then divide by the lift, squared. Because the lift is squared, halving the lift you want to catch roughly quadruples the visitors you need.

Worked example with small numbers. You want to detect a rise from 10% to 15%.

• 1.96 + 0.84 = 2.8, and 2.8 squared is 7.84.
• The noise: 0.10 × 0.90 = 0.09, and 0.15 × 0.85 = 0.1275. Together that's 0.2175.
• The lift is 0.05. Squared, that's 0.0025.
• 7.84 × 0.2175 = 1.7052. Divide by 0.0025 to get 682.08.
• Round up: 683 visitors per version.

import math

pa, pb = 0.10, 0.15
n = (1.96 + 0.84) ** 2 * (pa * (1 - pa) + pb * (1 - pb)) / (pb - pa) ** 2
print(math.ceil(n))

It prints 683.

Now the shop's numbers. A lift from 4.0% to 4.4% is 0.4 points. That's a 10% relative lift, because 0.4 is a tenth of 4. It needs about 39,400 visitors per variant. A lift from 4% to 5% needs about 6,700.

So plan in this order. Decide the smallest lift worth detecting. Compute $n$. Run the test until each version reaches it.

import math

xa, na = 50, 400    # subject line A: opens, emails sent
xb, nb = 75, 400    # subject line B
pa, pb = xa / na, xb / nb
pooled = (xa + xb) / (na + nb)
se = math.sqrt(pooled * (1 - pooled) * (1 / na + 1 / nb))
z = (pb - pa) / se
print(f"A {pa:.2%}  B {pb:.2%}  z = {z:.2f}")
print("significant at 0.05" if abs(z) > 1.96 else "no clear difference")

se_diff = math.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)
diff = pb - pa
print(f"difference {diff:+.2%}, 95% CI {diff - 1.96 * se_diff:+.2%} to {diff + 1.96 * se_diff:+.2%}")

Three ways to fool yourself

1. Peeking. Imagine a race where you may stop the clock whenever your favorite runner is ahead. They'll "win" far more often than they should. Tests work the same way. Checking the p-value every day and stopping the first time it dips below 0.05 pushes false positives far above 5%. A false positive is declaring a winner when there's no real difference.

The simulation below runs 2,000 A/A tests. In an A/A test both groups see the same version, so any "winner" is a false positive. Each test runs for 20 days, with 500 visitors a day per group. It peeks at the p-value every day.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
p, visitors_per_day, days, tests = 0.04, 500, 20, 2000
false_wins_peeking = false_wins_once = 0
for _ in range(tests):
    a = rng.random((days, visitors_per_day)) < p
    b = rng.random((days, visitors_per_day)) < p
    ca, cb = a.sum(axis=1).cumsum(), b.sum(axis=1).cumsum()
    n = visitors_per_day * np.arange(1, days + 1)
    pooled = (ca + cb) / (2 * n)
    z = (cb - ca) / n / np.sqrt(pooled * (1 - pooled) * 2 / n)
    pvals = 2 * norm.sf(np.abs(z))
    false_wins_peeking += bool((pvals < 0.05).any())
    false_wins_once += bool(pvals[-1] < 0.05)
print(f"declared a winner when checking only at the end: {false_wins_once / tests:.1%}")
print(f"declared a winner when stopping at the first p < 0.05: {false_wins_peeking / tests:.1%}")

Checking only at the end gives about 4%, close to the 5% you signed up for. Stopping at the first p below 0.05 gives more than 20%. Each daily look is one more chance for noise to cross the line. Repeated looks inflate the false-positive rate well above the 5% you intended. The fix is to set the sample size in advance and look once, at the end. Or use a method built for watching results as they arrive, called sequential testing.

2. Many comparisons. Test 20 headlines against the original at $\alpha = 0.05$. Even if none is better, you expect about one to look "significant" by luck alone: 20 × 0.05 = 1. One simple fix is the Bonferroni correction: divide $\alpha$ by the number of comparisons. For 20 comparisons, that's 0.05 ÷ 20 = 0.0025. Another fix is to confirm any winner in a fresh test.

3. Significant isn't the same as important. A precise enough kitchen scale can show that one bag of flour is 1 gram heavier than another. The difference is real, and it doesn't matter. In the same way, with tens of millions of visitors a 0.01-point lift can be "significant" and worth nothing. Look at the size of the effect and its interval, not only at p.

For marketers: the same rules apply when you compare prompts or models on an eval. Fix the eval set in advance. Use enough examples. Compare both on the same examples. Report the difference with an interval.