Comparing groups: t-tests
Welch's t-test, pooled vs average rates, and a control group for before-and-after changes.
What it covers8
Why it matters
"Did the change work?" is the question behind every A/B test, every model upgrade and every new process. The A/B testing lesson compared two conversion rates. Often you're comparing averages instead: response times, scores, monthly rates. And often you can't randomize, so the comparison is before against after. This lesson gives you the t-test for averages, and the design habit that matters even more than the test: finding a fair comparison group.
What you'll be able to do
- Choose between a pooled rate and an average of rates by the question you're answering
- Compare two groups' distributions with box plots and histograms before testing anything
- Run Welch's t-test, and read its p-value and confidence interval for the difference
- Use a control group to separate a change's effect from everything else that changed
Show the answer
How much the weekly numbers vary, and how many weeks you have
A drop of 0.5 means little if the weekly counts swing by 2, and a lot if they barely move. A t-test weighs the gap against the spread and the sample size.
The brief
Customers sometimes return orders that arrived damaged. In October 2024, the north warehouse switched to new recycled-pulp packing inserts. The south warehouse kept the old ones. A year later, the operations team asks: did the new inserts cut damage, and by how much?
import pandas as pd d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 d["period"] = (d["month"] >= "2024-10").map({False: "before", True: "after"}) print(d.head(4)) print(d.groupby(["warehouse", "period"])["rate"].agg(["mean", "std", "size"]).round(3))
Each row is one warehouse in one month: orders shipped, orders returned damaged, and whether the new inserts were in use. The damage rate is damaged ÷ orders, shown as a percentage. North's average monthly rate fell from about 3.1% to 2.7%. South's barely moved. Is that drop real, or the kind of wobble you'd see anyway?
Show the answer
A drop this big would be rare if nothing had changed
A p-value starts from the null hypothesis, that nothing really changed, and asks how often luck alone would give a gap at least this big. Here: about 3 times in 100. It is not the chance that the inserts work or don't. The t-test later in this lesson gives a p-value with the same meaning.
Two ways to average a rate
"North's damage rate before the change" has two reasonable meanings:
- The pooled rate: all damaged orders ÷ all orders, over the whole period. Every order counts equally, so it's the rate a customer experienced.
- The mean of the monthly rates: work out each month's rate, then average them. Every month counts equally, however many orders it had.
import pandas as pd d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 north_before = d[(d["warehouse"] == "north") & (d["month"] < "2024-10")] pooled = north_before["damaged"].sum() / north_before["orders"].sum() * 100 mean_of_rates = north_before["rate"].mean() print(round(pooled, 3), round(mean_of_rates, 3)) decembers = north_before[north_before["month"].str.endswith("-12")] print(decembers[["month", "orders", "rate"]].round(2))
They differ only a little here: December is both the busiest month and the worst for damage, because of rushed packing, and the pooled rate gives its many orders more weight, so it comes out higher. Neither is wrong; they answer different questions. For "what share of parcels arrived damaged?", pool. To test whether the typical month changed, use the monthly rates. Each month is then one measurement, which is what the t-test below needs.
orders = [100, 900] damaged = [10, 9] pooled = sum(damaged) / sum(orders) mean_of_rates = (10 / 100 + 9 / 900) / 2 print(round(pooled, 3), round(mean_of_rates, 3))
Show the answer
0.019 0.055
Pooled: 19 of 1,000 orders, 1.9%. The month rates are 10% and 1%, averaging 5.5%: the small month counts as much as the big one.
Pooled rate or mean of rates
Write pooled_rate(damaged, orders) (total damaged over total orders, in percent) and mean_of_rates(damaged, orders) (the average of each month's percentage), both from lists of monthly counts. Then set gap to pooled minus mean for the months given.
damaged = [12, 9, 30, 8] orders = [400, 300, 1500, 250] def pooled_rate(damaged, orders): ... def mean_of_rates(damaged, orders): ... gap = None
Write a sentence or two, then compare it with a good answer in the lesson.
Look before you test
A test squeezes two groups into one number. Look at the groups first, so you know what that number is summarizing:
import matplotlib.pyplot as plt import pandas as pd d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 north = d[d["warehouse"] == "north"] before = north[north["month"] < "2024-10"]["rate"] after = north[north["month"] >= "2024-10"]["rate"] fig, (left, right) = plt.subplots(1, 2, figsize=(10, 3.2)) for name, part in d.groupby("warehouse"): left.plot(range(len(part)), part["rate"], marker=".", label=name) left.axvline(21, color="gray", linestyle=":") left.set_title("Monthly damage rate (%)") left.set_xlabel("month number (the dotted line is October 2024)") left.legend() right.boxplot([before, after], tick_labels=["north before", "north after"]) right.set_title("North: distribution of monthly rates")
The line chart shows the drop lining up with October 2024 in the north, and no matching drop in the south. It also shows a December spike every year in both, a pattern to keep in mind. The box plot compares the two groups of months. The boxes overlap a little, but the whole "after" box sits lower.
Show the answer
It makes the drop look bigger than it is
The "before" average carries two bad months and the "after" average carries one, so "before" comes out worse for a reason that has nothing to do with the inserts. The chart is where you notice it. A test only sees two lists of numbers, so it can't know which months they came from.
Welch's t-test
The t-test asks the same question as the z-test in the A/B testing lesson: if nothing really changed, how surprising is a gap this big? The gap is the difference of the two means, . Its standard error combines each group's spread and size :
In plain words: counts how many standard errors apart the two averages are. Spread-out groups or few values make the standard error big and small. A beyond about ±2 is unusual if there's no real difference.
This version, Welch's t-test, doesn't assume equal spread, so it's the safe default (equal_var=False in SciPy). It treats months as independent draws, though seasons repeat and trends carry over, so read a small p-value as a hint:
import pandas as pd from scipy import stats d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 north = d[d["warehouse"] == "north"] before = north[north["month"] < "2024-10"]["rate"] after = north[north["month"] >= "2024-10"]["rate"] result = stats.ttest_ind(after, before, equal_var=False) low, high = result.confidence_interval(0.95) print("difference:", round(after.mean() - before.mean(), 3), "percentage points") print("t =", round(result.statistic, 2), " p =", round(result.pvalue, 4)) print("95% interval:", round(low, 3), "to", round(high, 3))
The rate fell about 0.4 percentage points, with well under 0.05. The interval says the true drop is plausibly anywhere from about 0.13 to 0.69 points. That's a wide range, and the honest one is probably wider still: the test treats each month as an independent draw, but neighboring months share a season and any slow trend. Report the interval, not just "significant", because the business decision depends on how big the effect is.
import math gap = 0.4 se = math.sqrt(0.3 ** 2 / 9 + 0.4 ** 2 / 16) print(round(se, 3), round(gap / se, 2))
Show the answer
0.141 2.83
Each group adds its spread squared over its size: 0.09 ÷ 9 = 0.01 and 0.16 ÷ 16 = 0.01. Their sum is 0.02, and its square root is about 0.141. The gap of 0.4 is 2.83 of those standard errors, well past 2, so a gap like this would be unusual if nothing had changed.
Double the spread and t halves. Quadruple the months and t doubles. The same gap can be convincing or meaningless depending on how noisy and how plentiful the data is.
Write a sentence or two, then compare it with a good answer in the lesson.
Before and after isn't enough
A before/after comparison credits the change with everything that happened at the same time. In January 2025, the shop also switched couriers for both warehouses. Some of north's drop could be the courier, not the inserts. Seasons matter too: the "after" period has a different mix of months.
The fix is a control group: something that went through the same period without the change. South kept the old inserts, so its before/after change shows what happened anyway. Subtract it:
This is called a difference-in-differences. It removes anything that hit both warehouses equally, like the courier and the seasons, and leaves what's special to the north. It rests on one assumption: without the new inserts, the two warehouses would have moved in parallel. Check it on the "before" months, when neither had changed. Their lines should run side by side.
A randomized test, sending a random half of parcels with each insert, needs no such assumption. That's why A/B tests are the gold standard. Real operations often can't randomize, though, so a good control group is the next best thing.
Show the answer
The two weren't moving in parallel before
The method assumes that without the change, both warehouses would have moved the same way, so south's change can stand in for what north would have done anyway. If north was already on a different path, subtracting south's change removes the wrong amount. That's why you check the "before" months first.
The control warehouse
Run Welch's t-test on the south warehouse's monthly rates, after against before. Set south_diff to the difference of means (after − before) rounded to 3, south_p to the p-value rounded to 3, and changed to True if p is below 0.05.
import pandas as pd from scipy import stats d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 south_diff = None south_p = None changed = None
Worked example: the effect, with a control
Estimate the inserts' effect as a difference-in-differences, and test it with one value per month: north's rate minus south's.
import pandas as pd from scipy import stats d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv") d["rate"] = d["damaged"] / d["orders"] * 100 wide = d.pivot_table(index="month", columns="warehouse", values="rate") wide["gap"] = wide["north"] - wide["south"] before = wide[wide.index < "2024-10"]["gap"] after = wide[wide.index >= "2024-10"]["gap"] print("gap before:", round(before.mean(), 3), " gap after:", round(after.mean(), 3)) result = stats.ttest_ind(after, before, equal_var=False) low, high = result.confidence_interval(0.95) print("effect:", round(after.mean() - before.mean(), 3), "points,", "p =", round(result.pvalue, 4)) print("95% interval:", round(low, 3), "to", round(high, 3)) early = wide[wide.index < "2024-10"] print("before, north vs south:", round(early["north"].mean(), 3), round(early["south"].mean(), 3))
Pivoting puts north and south side by side for each month, and the gap column is north minus south. Anything that hit both warehouses in a month, like December's rush or the new courier, cancels out of the gap. Comparing the gap before and after is the difference-in-differences, and the t-test on the monthly gaps tests it. The effect comes out a little smaller than the plain before/after drop, because part of that drop was the courier. The last line checks the parallel assumption: before the change, the two warehouses had similar rates. That's the number to report: a drop of about a third of a percentage point, with its interval.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 5 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
A before/after comparison shows damage fell after new inserts. What does a control warehouse add?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write