A/B testing a model change

Check the split before the result: sample ratio mismatch

If the plan was 50/50 and the data has 5,000 users in one group and 4,700 in the other, something is broken: maybe the new version crashes for some users before it logs their exposure, or a bot filter treats the groups differently. That's a sample ratio mismatch (SRM), and when it happens the groups are no longer comparable, so no result from the experiment can be trusted until the cause is found.

The test is a chi-square goodness-of-fit test: compare the observed counts with the counts the plan expects. Because experiments are large, a tiny imbalance is normal and a real bug produces a very small p-value, so teams use a strict threshold such as 0.001.

from scipy.stats import chisquare

def srm_p_value(counts, shares):
    total = sum(counts)
    expected = [total * s for s in shares]
    return chisquare(counts, f_exp=expected).pvalue

print(round(srm_p_value([5012, 4988], [0.5, 0.5]), 3))    # normal wobble
print(srm_p_value([5000, 4600], [0.5, 0.5]))               # a bug
print(round(srm_p_value([9030, 970], [0.9, 0.1]), 3))       # a 90/10 rollout, on plan

The first split differs by 24 users and gets a p-value around 0.81: nothing to worry about. The second, 400 users apart, gets about 0.00004, far below 0.001: the split is broken. The third shows the check works for any planned split, not just 50/50: 9,030 and 970 sit close to the 9,000 and 1,000 a 90/10 plan expects.

When the check fails, slice it: run the same test separately for each platform, country or app version. A bug usually lives in one segment, such as one phone model where the new screen crashes before it logs the exposure, and the slice that fails points at it.

from scipy.stats import chisquare


def srm(counts, shares, threshold=0.001):
    return 1.0, False


print(srm([10_000, 9_400], [0.5, 0.5]))

Decide with a rule written in advance

Before the data arrives, write down three things, and the decision becomes arithmetic instead of debate:

1. The primary metric: the one number the change is meant to move, such as the share of users who order.
2. Guardrail metrics: things that must not get worse, such as the p95 response time, the cost per conversation, or the rate of thumbs-down clicks. A new model that lifts orders by 1% while doubling cost may not be a win.
3. The decision rule and the sample size: "ship if the primary metric's 95% interval is above 0, and no guardrail is worse than its limit, after 20,000 users per group." The size comes from a power calculation like the one in the A/B testing lesson.

Then don't stop early because the result looks good on day 3: checking every day and stopping at the first significant result inflates false positives the same way running many tests does. Run to the planned size, check the split first, and apply the rule.

Why only one primary metric? Each metric tested at the 5% level has a 5% chance of a significant difference by luck alone. Track ten independent metrics and the chance that at least one shows one is 1 - 0.95 ** 10, about 40%. If any of them could justify a launch, a change that does nothing would often look like a win. Guardrails are different: they can only block a launch, never justify one. Many guardrails do raise the chance of a false block, so give each a limit that matters in practice (10% slower, not merely "significantly" slower).

Two more traps are about time. A novelty effect makes anything new get extra clicks for its first days, and it fades. And weekdays differ from weekends, so run whole weeks: a test that runs Monday to Friday measures weekday users only.

import numpy as np
import pandas as pd


def _simulate(seed=7):
    """Build the experiment's raw logs. Treat this as a black box: find any problems from the data."""
    rng = np.random.default_rng(seed)
    n = 24_000
    users = pd.DataFrame({"user": [f"u{i:05d}" for i in range(n)],
                          "platform": rng.choice(["web", "ios", "android"], size=n, p=[0.5, 0.3, 0.2])})
    users["variant"] = np.where(rng.random(n) < 0.5, "new", "old")
    users["seen"] = pd.Timestamp("2025-09-01") + pd.to_timedelta(rng.integers(0, 14 * 24 * 60, n), unit="min")
    older = (users["platform"] == "android") & (rng.random(n) < 0.5)
    lost = (users["variant"] == "new") & older & (rng.random(n) < 0.8)
    logged = users[~lost]
    again = logged.sample(frac=0.6, random_state=7)
    exposures = pd.concat([
        logged[["user", "variant", "platform", "seen"]].rename(columns={"seen": "time"}),
        again[["user", "variant", "platform"]].assign(time=again["seen"] + pd.Timedelta(hours=30)),
    ]).sort_values("time").reset_index(drop=True)
    rate = users["platform"].map({"web": 0.11, "ios": 0.12, "android": 0.07}) - np.where(older, 0.04, 0.0)
    rate = rate + np.where(users["variant"] == "new", 0.004, 0.0)
    buyers = users[rng.random(n) < rate]
    later = buyers["seen"] + pd.to_timedelta(rng.integers(5, 3 * 24 * 60, len(buyers)), unit="min")
    early = users.sample(n=900, random_state=11)
    orders = pd.concat([pd.DataFrame({"user": buyers["user"], "time": later}),
                        pd.DataFrame({"user": early["user"], "time": early["seen"] - pd.Timedelta(days=2)})])
    return exposures, orders.sort_values("time").reset_index(drop=True)


# The test: "checkout-helper-v2" put a new assistant on the checkout page for half of all users,
# planned 50/50 for two weeks. Primary metric: the share of exposed users who order afterwards.
exposures, orders = _simulate()
print(exposures.head())
print(orders.head())
print(len(exposures), "exposure rows,", exposures["user"].nunique(), "users;", len(orders), "orders")

# Your analysis starts here: one row per exposed user, then the checks, then the result.