Comparing groups: t-tests

Look before you test

A test squeezes two groups into one number. Look at the groups first, so you know what that number is summarizing:

import matplotlib.pyplot as plt
import pandas as pd

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
north = d[d["warehouse"] == "north"]
before = north[north["month"] < "2024-10"]["rate"]
after = north[north["month"] >= "2024-10"]["rate"]

fig, (left, right) = plt.subplots(1, 2, figsize=(10, 3.2))
for name, part in d.groupby("warehouse"):
    left.plot(range(len(part)), part["rate"], marker=".", label=name)
left.axvline(21, color="gray", linestyle=":")
left.set_title("Monthly damage rate (%)")
left.set_xlabel("month number (the dotted line is October 2024)")
left.legend()
right.boxplot([before, after], tick_labels=["north before", "north after"])
right.set_title("North: distribution of monthly rates")

The line chart shows the drop lining up with October 2024 in the north, and no matching drop in the south. It also shows a December spike every year in both, a pattern to keep in mind. The box plot compares the two groups of months. The boxes overlap a little, but the whole "after" box sits lower.

Welch's t-test

The t-test asks the same question as the last lesson's z-test: if nothing really changed, how surprising is a gap this big? The gap is the difference of the two means, $\bar{x}_\text{after} - \bar{x}_\text{before}$. Its standard error combines each group's spread $s$ and size $n$:

$$ t = \frac{\bar{x}_\text{after} - \bar{x}_\text{before}}{\sqrt{s_\text{after}^2 / n_\text{after} + s_\text{before}^2 / n_\text{before}}} $$

In plain words: $t$ counts how many standard errors apart the two averages are. Spread-out groups or few values make the standard error big and $t$ small. A $t$ beyond about ±2 is unusual if there's no real difference.

This version, Welch's t-test, doesn't assume the two groups are equally spread, so it's the safe default. SciPy runs it when you pass equal_var=False:

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
north = d[d["warehouse"] == "north"]
before = north[north["month"] < "2024-10"]["rate"]
after = north[north["month"] >= "2024-10"]["rate"]

result = stats.ttest_ind(after, before, equal_var=False)
low, high = result.confidence_interval(0.95)
print("difference:", round(after.mean() - before.mean(), 3), "percentage points")
print("t =", round(result.statistic, 2), " p =", round(result.pvalue, 4))
print("95% interval:", round(low, 3), "to", round(high, 3))

The rate fell about 0.4 percentage points, with $p$ well under 0.05. The interval says the true drop is plausibly anywhere from about 0.13 to 0.69 points. That's a wide range. Report the interval, not just "significant", because the business decision depends on how big the effect is.

Before and after isn't enough

A before/after comparison credits the change with everything that happened at the same time. In January 2025, the shop also switched couriers for both warehouses. Some of north's drop could be the courier, not the inserts. Seasons matter too: the "after" period has a different mix of months.

The fix is a control group: something that went through the same period without the change. South kept the old inserts, so its before/after change shows what happened anyway. Subtract it:

$$ \text{effect} = (\text{north after} - \text{north before}) - (\text{south after} - \text{south before}) $$

This is called a difference-in-differences. It removes anything that hit both warehouses equally, like the courier and the seasons, and leaves what's special to the north. It rests on one assumption: without the new inserts, the two warehouses would have moved in parallel. Check it on the "before" months, when neither had changed. Their lines should run side by side.

A randomized test, sending a random half of parcels with each insert, needs no such assumption. That's why A/B tests are the gold standard. Real operations often can't randomize, though, so a good control group is the next best thing.

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100

south_diff = None
south_p = None
changed = None

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
wide = d.pivot_table(index="month", columns="warehouse", values="rate")
wide["gap"] = wide["north"] - wide["south"]
before = wide[wide.index < "2024-10"]["gap"]
after = wide[wide.index >= "2024-10"]["gap"]

print("gap before:", round(before.mean(), 3), " gap after:", round(after.mean(), 3))
result = stats.ttest_ind(after, before, equal_var=False)
low, high = result.confidence_interval(0.95)
print("effect:", round(after.mean() - before.mean(), 3), "points, p =", round(result.pvalue, 4))
print("95% interval:", round(low, 3), "to", round(high, 3))
print("before, north vs south:", round(wide.loc[wide.index < "2024-10", "north"].mean(), 3), round(wide.loc[wide.index < "2024-10", "south"].mean(), 3))