Index Ai Academy
Phase 3 · Data analysis

Comparing groups: t-tests

Welch's t-test, pooled vs average rates, and a control group for before-and-after changes.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

"Did the change work?" is the question behind every A/B test, every model upgrade and every new process. The A/B testing lesson compared two conversion rates. Often you're comparing averages instead: response times, scores, monthly rates. And often you can't randomize, so the comparison is before against after. This lesson gives you the t-test for averages, and the design habit that matters even more than the test: finding a fair comparison group.

What you'll be able to do

  • Choose between a pooled rate and an average of rates by the question you're answering
  • Compare two groups' distributions with box plots and histograms before testing anything
  • Run Welch's t-test, and read its p-value and confidence interval for the difference
  • Use a control group to separate a change's effect from everything else that changed
Quick check
A warehouse switched to new packaging, and damaged returns fell from an average of 3.1 a week to 2.6. What do you need before crediting the packaging?
  1. Which warehouse worker suggested the change
  2. Whether the new packaging is recycled
  3. How much the weekly numbers vary, and how many weeks you have
  4. Only the two averages, which already show the drop
Show the answer

How much the weekly numbers vary, and how many weeks you have

A drop of 0.5 means little if the weekly counts swing by 2, and a lot if they barely move. A t-test weighs the gap against the spread and the sample size.

The brief

Customers sometimes return orders that arrived damaged. In October 2024, the north warehouse switched to new recycled-pulp packing inserts. The south warehouse kept the old ones. A year later, the operations team asks: did the new inserts cut damage, and by how much?

import pandas as pd

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
d["period"] = (d["month"] >= "2024-10").map({False: "before", True: "after"})
print(d.head(4))
print(d.groupby(["warehouse", "period"])["rate"].agg(["mean", "std", "size"]).round(3))
It runs right here in your browser, with nothing to install.

Each row is one warehouse in one month: orders shipped, orders returned damaged, and whether the new inserts were in use. The damage rate is damaged ÷ orders, shown as a percentage. North's average monthly rate fell from about 3.1% to 2.7%. South's barely moved. Is that drop real, or the kind of wobble you'd see anyway?

Quick check
From the lesson on A/B tests: suppose a test of north's drop gives p = 0.03. What does that number say?
  1. A drop this big would be rare if nothing had changed
  2. There is a 97% chance that the new inserts work
  3. There is a 3% chance that the inserts did nothing
  4. The damage rate fell by 3% of what it was before
Show the answer

A drop this big would be rare if nothing had changed

A p-value starts from the null hypothesis, that nothing really changed, and asks how often luck alone would give a gap at least this big. Here: about 3 times in 100. It is not the chance that the inserts work or don't. The t-test later in this lesson gives a p-value with the same meaning.

Two ways to average a rate

"North's damage rate before the change" has two reasonable meanings:

  • The pooled rate: all damaged orders ÷ all orders, over the whole period. Every order counts equally, so it's the rate a customer experienced.
  • The mean of the monthly rates: work out each month's rate, then average them. Every month counts equally, however many orders it had.
import pandas as pd

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
north_before = d[(d["warehouse"] == "north") & (d["month"] < "2024-10")]

pooled = north_before["damaged"].sum() / north_before["orders"].sum() * 100
mean_of_rates = north_before["rate"].mean()
print(round(pooled, 3), round(mean_of_rates, 3))
decembers = north_before[north_before["month"].str.endswith("-12")]
print(decembers[["month", "orders", "rate"]].round(2))

They differ only a little here: December is both the busiest month and the worst for damage, because of rushed packing, and the pooled rate gives its many orders more weight, so it comes out higher. Neither is wrong; they answer different questions. For "what share of parcels arrived damaged?", pool. To test whether the typical month changed, use the monthly rates. Each month is then one measurement, which is what the t-test below needs.

Quick check
What does this print?
orders = [100, 900]
damaged = [10, 9]
pooled = sum(damaged) / sum(orders)
mean_of_rates = (10 / 100 + 9 / 900) / 2
print(round(pooled, 3), round(mean_of_rates, 3))
  1. 0.055 0.019
  2. 0.019 0.019
  3. 0.055 0.055
  4. 0.019 0.055
Show the answer

0.019 0.055

Pooled: 19 of 1,000 orders, 1.9%. The month rates are 10% and 1%, averaging 5.5%: the small month counts as much as the big one.

Drill

Pooled rate or mean of rates

Write pooled_rate(damaged, orders) (total damaged over total orders, in percent) and mean_of_rates(damaged, orders) (the average of each month's percentage), both from lists of monthly counts. Then set gap to pooled minus mean for the months given.

damaged = [12, 9, 30, 8]
orders = [400, 300, 1500, 250]

def pooled_rate(damaged, orders):
    ...

def mean_of_rates(damaged, orders):
    ...

gap = None
In the lesson, your code is checked as you write it.
In your own words
When do you pool a rate and when do you average rates, and what does a t-test weigh?

Write a sentence or two, then compare it with a good answer in the lesson.

Look before you test

A test squeezes two groups into one number. Look at the groups first, so you know what that number is summarizing:

import matplotlib.pyplot as plt
import pandas as pd

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
north = d[d["warehouse"] == "north"]
before = north[north["month"] < "2024-10"]["rate"]
after = north[north["month"] >= "2024-10"]["rate"]

fig, (left, right) = plt.subplots(1, 2, figsize=(10, 3.2))
for name, part in d.groupby("warehouse"):
    left.plot(range(len(part)), part["rate"], marker=".", label=name)
left.axvline(21, color="gray", linestyle=":")
left.set_title("Monthly damage rate (%)")
left.set_xlabel("month number (the dotted line is October 2024)")
left.legend()
right.boxplot([before, after], tick_labels=["north before", "north after"])
right.set_title("North: distribution of monthly rates")

The line chart shows the drop lining up with October 2024 in the north, and no matching drop in the south. It also shows a December spike every year in both, a pattern to keep in mind. The box plot compares the two groups of months. The boxes overlap a little, but the whole "after" box sits lower.

Quick check
Damage spikes every December. Suppose the "before" months include two Decembers and the "after" months only one. What does that do to a plain before-and-after comparison?
  1. It only changes the box plot, not the test
  2. Nothing: a t-test corrects for the seasons
  3. It makes the drop look smaller than it is
  4. It makes the drop look bigger than it is
Show the answer

It makes the drop look bigger than it is

The "before" average carries two bad months and the "after" average carries one, so "before" comes out worse for a reason that has nothing to do with the inserts. The chart is where you notice it. A test only sees two lists of numbers, so it can't know which months they came from.

Welch's t-test

The t-test asks the same question as the z-test in the A/B testing lesson: if nothing really changed, how surprising is a gap this big? The gap is the difference of the two means, xˉafter−xˉbefore\bar{x}_\text{after} - \bar{x}_\text{before}. Its standard error combines each group's spread ss and size nn:

t=xˉafter−xˉbeforesafter2/nafter+sbefore2/nbeforet = \frac{\bar{x}_\text{after} - \bar{x}_\text{before}}{\sqrt{s_\text{after}^2 / n_\text{after} + s_\text{before}^2 / n_\text{before}}}

In plain words: tt counts how many standard errors apart the two averages are. Spread-out groups or few values make the standard error big and tt small. A tt beyond about ±2 is unusual if there's no real difference.

This version, Welch's t-test, doesn't assume equal spread, so it's the safe default (equal_var=False in SciPy). It treats months as independent draws, though seasons repeat and trends carry over, so read a small p-value as a hint:

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
north = d[d["warehouse"] == "north"]
before = north[north["month"] < "2024-10"]["rate"]
after = north[north["month"] >= "2024-10"]["rate"]

result = stats.ttest_ind(after, before, equal_var=False)
low, high = result.confidence_interval(0.95)
print("difference:", round(after.mean() - before.mean(), 3), "percentage points")
print("t =", round(result.statistic, 2), " p =", round(result.pvalue, 4))
print("95% interval:", round(low, 3), "to", round(high, 3))

The rate fell about 0.4 percentage points, with pp well under 0.05. The interval says the true drop is plausibly anywhere from about 0.13 to 0.69 points. That's a wide range, and the honest one is probably wider still: the test treats each month as an independent draw, but neighboring months share a season and any slow trend. Report the interval, not just "significant", because the business decision depends on how big the effect is.

Quick check
The "after" group has 9 monthly rates with a spread of 0.3. The "before" group has 16 with a spread of 0.4. The means are 0.4 apart. What does this print: the standard error of the gap, and t?
import math

gap = 0.4
se = math.sqrt(0.3 ** 2 / 9 + 0.4 ** 2 / 16)
print(round(se, 3), round(gap / se, 2))
  1. 0.02 20.0
  2. 0.141 0.35
  3. 0.141 2.83
  4. 0.5 0.8
Show the answer

0.141 2.83

Each group adds its spread squared over its size: 0.09 ÷ 9 = 0.01 and 0.16 ÷ 16 = 0.01. Their sum is 0.02, and its square root is about 0.141. The gap of 0.4 is 2.83 of those standard errors, well past 2, so a gap like this would be unusual if nothing had changed.

Try it now

Double the spread and t halves. Quadruple the months and t doubles. The same gap can be convincing or meaningless depending on how noisy and how plentiful the data is.

In your own words
Welch's t-test gives t = 2.1 and p = 0.04 for a gap of 0.3 points. How do you read that?

Write a sentence or two, then compare it with a good answer in the lesson.

Before and after isn't enough

A before/after comparison credits the change with everything that happened at the same time. In January 2025, the shop also switched couriers for both warehouses. Some of north's drop could be the courier, not the inserts. Seasons matter too: the "after" period has a different mix of months.

The fix is a control group: something that went through the same period without the change. South kept the old inserts, so its before/after change shows what happened anyway. Subtract it:

effect=(north after−north before)−(south after−south before)\text{effect} = (\text{north after} - \text{north before}) - (\text{south after} - \text{south before})

This is called a difference-in-differences. It removes anything that hit both warehouses equally, like the courier and the seasons, and leaves what's special to the north. It rests on one assumption: without the new inserts, the two warehouses would have moved in parallel. Check it on the "before" months, when neither had changed. Their lines should run side by side.

A randomized test, sending a random half of parcels with each insert, needs no such assumption. That's why A/B tests are the gold standard. Real operations often can't randomize, though, so a good control group is the next best thing.

Quick check
In the year before the change, north's damage rate was climbing month after month while south's stayed flat. Why does that weaken a difference-in-differences estimate?
  1. South has fewer orders than north does
  2. The two weren't moving in parallel before
  3. A control group has to be picked at random
  4. The two groups must be the same size
Show the answer

The two weren't moving in parallel before

The method assumes that without the change, both warehouses would have moved the same way, so south's change can stand in for what north would have done anyway. If north was already on a different path, subtracting south's change removes the wrong amount. That's why you check the "before" months first.

Drill

The control warehouse

Run Welch's t-test on the south warehouse's monthly rates, after against before. Set south_diff to the difference of means (after − before) rounded to 3, south_p to the p-value rounded to 3, and changed to True if p is below 0.05.

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100

south_diff = None
south_p = None
changed = None
In the lesson, your code is checked as you write it.
Worked example

Worked example: the effect, with a control

Estimate the inserts' effect as a difference-in-differences, and test it with one value per month: north's rate minus south's.

import pandas as pd
from scipy import stats

d = pd.read_csv("https://api.gradient.test/v1/datasets/damaged-returns.csv")
d["rate"] = d["damaged"] / d["orders"] * 100
wide = d.pivot_table(index="month", columns="warehouse", values="rate")
wide["gap"] = wide["north"] - wide["south"]
before = wide[wide.index < "2024-10"]["gap"]
after = wide[wide.index >= "2024-10"]["gap"]

print("gap before:", round(before.mean(), 3), " gap after:", round(after.mean(), 3))
result = stats.ttest_ind(after, before, equal_var=False)
low, high = result.confidence_interval(0.95)
print("effect:", round(after.mean() - before.mean(), 3), "points,",
      "p =", round(result.pvalue, 4))
print("95% interval:", round(low, 3), "to", round(high, 3))
early = wide[wide.index < "2024-10"]
print("before, north vs south:", round(early["north"].mean(), 3),
      round(early["south"].mean(), 3))
In the lesson you can change this and run it.

Pivoting puts north and south side by side for each month, and the gap column is north minus south. Anything that hit both warehouses in a month, like December's rush or the new courier, cancels out of the gap. Comparing the gap before and after is the difference-in-differences, and the t-test on the monthly gaps tests it. The effect comes out a little smaller than the plain before/after drop, because part of that drop was the courier. The last line checks the parallel assumption: before the change, the two warehouses had similar rates. That's the number to report: a drop of about a third of a percentage point, with its interval.

In your own words
Why isn't a before-and-after comparison enough to show a change worked?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A before/after comparison shows damage fell after new inserts. What does a control warehouse add?

  1. It shows what changed anyway, so you can subtract effects that hit both
  2. It doubles the sample size, so the difference becomes significant
  3. It makes the p-value smaller, so the result looks more convincing
  4. Nothing, once the before-and-after test is already significant

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write