Index Ai Academy
Phase 3 · Data analysis

More tests: groups, categories, relationships

Choose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

A/B tests and t-tests compare two groups. Real questions are often messier: three landing pages at once, whether the sign-up rate differs by region, whether customers who get more emails spend more. Each has its own standard test, and using the wrong one, or running ten t-tests where one test would do, is how false "findings" get into reports. This lesson gives you a short map from question to test, and builds each test by hand before checking it against SciPy, so the numbers stop being magic.

What you'll be able to do

  • Pick the right test from the question: groups, categories or a relationship
  • Compare three or more groups with one ANOVA instead of many t-tests, and compute it by hand
  • Test whether two categories are related with a chi-square test on a table of counts
  • Measure a relationship with Pearson or Spearman correlation, and know when an outlier or a confounder misleads
Quick check
You want to know whether the sign-up rate differs across four regions. What are you comparing?
  1. The average of one number across two groups
  2. One group against a fixed target
  3. Counts in categories: signed up or not, in each region
  4. Two numbers measured on the same people
Show the answer

Counts in categories: signed up or not, in each region

Signed up or not, by region, is a table of counts, which calls for a chi-square test rather than a t-test. Start from what you're comparing, and the right test follows.

Which test answers this question?

Start from what you're comparing, not from the test's name:

your questionwhat you havethe usual test
Is B's average different from A's?a number per person, two groupsWelch's t-test (the comparing-means lesson)
Is B's rate different from A's?yes/no per person, two groupstwo-proportion z-test (the A/B testing lesson)
Do three or more groups' averages differ?a number per person, three or more groupsone-way ANOVA
Are two categories related?counts in a table, such as region by signed up or notchi-square test of independence
Do two numbers rise and fall together?two numbers per personPearson or Spearman correlation

Each test still asks the same question you've met before: if nothing were going on (the null hypothesis), how surprising would data like this be? The p-value is that surprise, and the usual threshold is still 0.05.

Quick check
From the lesson on comparing groups: a pooled rate and a mean of rates answer different questions. What does this print for a quiet month and a busy month?
orders = [100, 900]
damaged = [1, 45]
pooled = sum(damaged) / sum(orders)
mean_of_rates = (damaged[0] / orders[0] + damaged[1] / orders[1]) / 2
print(round(pooled, 3), round(mean_of_rates, 3))
  1. 0.03 0.046
  2. 0.046 0.046
  3. 0.046 0.03
  4. 0.03 0.03
Show the answer

0.046 0.03

Pooled, 46 of 1,000 orders were damaged: 0.046. The two months' own rates are 0.01 and 0.05, and their mean is 0.03. The busy month was also the worse one, and pooling gives its 900 orders more weight. Picking by the question is the habit of this lesson too: the test follows from what you are comparing.

Try it now

Ask: how many groups, and is each person's answer a number, a yes/no, or a category?

Three groups: why not three t-tests?

Three landing pages give three pairs to compare (A-B, A-C, B-C). Each t-test has a 5% chance of a false alarm, so running all three gives about a 14% chance that at least one "wins" by luck alone. With ten groups there are 45 pairs, and the chance of at least one false alarm is about 90% (treating the tests as independent). ANOVA (analysis of variance) asks one question instead: is there more spread between the group averages than the spread within the groups would explain?

It divides two numbers. The spread between groups: how far each group's mean sits from the overall mean, weighted by group size. The spread within groups: how far each person sits from their own group's mean. Each is divided by its degrees of freedom, the number of values that are free to vary once the totals are fixed: groups minus 1 between groups, and people minus groups within them. That makes the two spreads comparable. Their ratio is F. If the groups are really alike, F is around 1; a big F means the means differ more than chance explains.

import numpy as np
from scipy import stats

a = np.array([42, 51, 38, 47, 55, 44, 49, 40])     # seconds on page, design A
b = np.array([58, 49, 61, 54, 66, 52, 57, 60])     # design B
c = np.array([45, 50, 43, 52, 48, 46, 51, 44])     # design C
groups = [a, b, c]

everyone = np.concatenate(groups)
grand = everyone.mean()
between = sum(len(g) * (g.mean() - grand) ** 2 for g in groups) / (len(groups) - 1)
within = (sum(((g - g.mean()) ** 2).sum() for g in groups)
          / (len(everyone) - len(groups)))
F = between / within
p = stats.f.sf(F, len(groups) - 1, len(everyone) - len(groups))
print("means:", [round(float(g.mean()), 1) for g in groups])
print("F =", round(F, 2), " p =", round(p, 5))
print(stats.f_oneway(a, b, c))
It runs right here in your browser, with nothing to install.

F is 12.2 and p is about 0.0003, matching SciPy's f_oneway: the designs differ. But ANOVA only says some group differs, not which. The next step is to compare pairs with a correction for testing several, such as the Bonferroni correction from the A/B testing lesson, and here B is clearly ahead of both.

Quick check
Four ad designs are compared with t-tests on every pair (six tests) at the 0.05 level, and nothing really differs. About how likely is at least one "significant" result?
print(round(1 - 0.95 ** 6, 2))
  1. 0.5
  2. 0.05
  3. 0.26
  4. 0.01
Show the answer

0.26

The chance that all six tests stay quiet is 0.95 to the sixth power, about 0.74, so the chance of at least one false alarm is about 0.26. One ANOVA keeps it at 0.05.

In your own words
Why not compare four groups with t-tests on every pair?

Write a sentence or two, then compare it with a good answer in the lesson.

Two categories: the chi-square test

Do sign-ups depend on the region a visitor came from? Count them in a table, one row per region and one column per outcome. If region and signing up were unrelated, each row would sign up at the overall rate, so every cell has an expected count: its row's total times its column's total, divided by the grand total. The chi-square statistic adds up how far each observed count is from its expected count, squared and scaled by the expected count: the sum of (observed - expected)² / expected. In the code, keepdims=True keeps the row totals as a column (shape 3 × 1), so multiplying by the row of column totals makes every row-and-column pair at once.

import pandas as pd
from scipy import stats

table = pd.DataFrame({"signed_up": [48, 30, 22], "did_not": [152, 170, 78]},
                     index=["west", "mountain", "south"])
observed = table.to_numpy()
expected = (observed.sum(axis=1, keepdims=True) * observed.sum(axis=0)
            / observed.sum())
chi2 = ((observed - expected) ** 2 / expected).sum()
dof = (observed.shape[0] - 1) * (observed.shape[1] - 1)
print(pd.DataFrame(expected, index=table.index, columns=table.columns))
print("chi-square =", round(chi2, 2), " degrees of freedom =", dof,
      " p =", round(stats.chi2.sf(chi2, dof), 4))
print(stats.chi2_contingency(observed, correction=False)[:3])

The west signs up at 24% and the mountain region at 15%, which looks like a real gap, but chi-square is 5.38 with a p of 0.068: with these counts, a spread like this turns up by chance more than 5% of the time. Two rules for using it: it works on counts, never percentages, and every expected count should be at least about 5, or the p-value isn't trustworthy (merge small categories, or use an exact test such as Fisher's).

Quick check
In the table, 200 visitors came from the west, 100 of all 500 visitors signed up, and 48 of the west's visitors did. What does this print: the west's expected sign-ups, and that cell's share of chi-square?
row_total, column_total, grand_total = 200, 100, 500
expected = row_total * column_total / grand_total
observed = 48
print(expected, round((observed - expected) ** 2 / expected, 2))
  1. 48.0 0.0
  2. 40.0 8.0
  3. 20.0 39.2
  4. 40.0 1.6
Show the answer

40.0 1.6

If region made no difference, the west would sign up at the overall rate, 100 in 500, so 200 visitors would give 40. The west had 48, which is 8 more. Squared and divided by the expected count, that is 64 ÷ 40 = 1.6. Chi-square adds a number like this for every cell of the table.

Drill

The counts you'd expect if nothing were going on

Write expected_counts(observed) for a 2-D NumPy array of counts: each cell is its row total times its column total, divided by the grand total. Then chi_square(observed), returning the statistic, the sum of (observed - expected)² / expected. Use keepdims=True on the row totals so they broadcast across the columns.

import numpy as np


def expected_counts(observed):
    return np.full(observed.shape, observed.mean())


def chi_square(observed):
    return 0.0


observed = np.array([[48, 152], [30, 170], [22, 78]])
print(expected_counts(observed))
print(round(chi_square(observed), 2))
In the lesson, your code is checked as you write it.
In your own words
What question does a chi-square test answer, and what does it compare?

Write a sentence or two, then compare it with a good answer in the lesson.

Relationships: Pearson and Spearman

Correlation measures how two numbers move together, from -1 (one always falls as the other rises) through 0 (no pattern) to 1 (they rise together). Pearson's r measures a straight-line relationship using the values themselves. Spearman's rho first replaces each value by its rank (1st, 2nd, 3rd...) and then computes Pearson's r on the ranks, so it measures whether the relationship always goes the same way, straight or curved, and one wild value can't drag it far.

import numpy as np
from scipy import stats

rng = np.random.default_rng(4)
emails = np.arange(1, 21)                                 # emails sent to 20 members
spend = 20 + 3 * emails + rng.normal(0, 6, 20)            # dollars spent, mostly rising

print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3),
      " Spearman", round(stats.spearmanr(emails, spend)[0], 3))
spend[0] = 400                                            # one member's huge purchase
print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3),
      " Spearman", round(stats.spearmanr(emails, spend)[0], 3))

One extreme purchase turns Pearson's 0.93 into -0.18, a negative relationship that isn't there; Spearman only drops from 0.91 to 0.63. So plot the data first, and prefer Spearman when values are skewed or have outliers.

Then remember what correlation can't tell you. Members who get more emails may spend more because the shop emails its keenest customers most: keenness drives both, a confounder. Only an experiment, such as emailing a random half, separates cause from coincidence.

Quick check
Take y = x³ for x from 1 to 10: y always rises with x, on a curve. What do the two correlations say?
  1. Spearman is exactly 1, and Pearson is a little below 1
  2. Both are near 0, since the points aren't on a line
  3. Pearson is exactly 1, and Spearman is a little below 1
  4. Both are exactly 1, since y always rises with x
Show the answer

Spearman is exactly 1, and Pearson is a little below 1

Spearman works on ranks. The smallest x has the smallest y, the next has the next, and so on, so the ranks match perfectly and rho is 1. Pearson measures how close the points are to one straight line. A curve that only bends one way is close to a line, but not on one, so r comes out a little under 1.

Drill

Spearman is Pearson on ranks

Write spearman(x, y): rank each list with scipy.stats.rankdata (it gives tied values the average of their ranks), then return the Pearson correlation of the two rank lists, computed with np.corrcoef(rx, ry)[0, 1]. The checks compare it with SciPy's own spearmanr.

import numpy as np
from scipy import stats


def spearman(x, y):
    return float(np.corrcoef(x, y)[0, 1])


miles = [3, 7, 12, 5, 20, 9]
rentals = [1, 2, 4, 2, 30, 3]
print(round(spearman(miles, rentals), 3))
In the lesson, your code is checked as you write it.
In your own words
Why can one extreme value flip Pearson's sign but barely move Spearman's?

Write a sentence or two, then compare it with a good answer in the lesson.

One-sided tests, used honestly

A reminder from the A/B testing lesson, because it matters for every test here. The tests so far are two-sided: they flag a difference in either direction. If, before you collect data, you only care about one direction (the new page is better, or you'll keep the old one either way), a one-sided test looks only there, and its p-value is half the two-sided one when the data lean that way. In SciPy, pass alternative="greater" or "less".

The honest rule is about timing: choose one-sided in your plan, before seeing results. Switching after a two-sided p of 0.08 to get a one-sided 0.04 is a classic way to fool yourself, and reviewers know to ask.

Quick check
A two-sided test gives p = 0.08 for "design B differs from A", with B ahead. The team now wants to report a one-sided p = 0.04 for "B is better". What's the problem?
  1. One-sided tests only work for proportions
  2. Nothing: with B ahead, one-sided is right
  3. Picking the direction after seeing the data
  4. One-sided p-values are always the larger ones
Show the answer

Picking the direction after seeing the data

A one-sided test is legitimate when the plan said so before the data arrived. Picking it afterward, because B happened to be ahead, is a way of moving the goalposts.

Worked example

Worked example: four questions, four tests

The trail club has four questions about one season's data. Each gets its own test, and each answer is reported with the number behind it.

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
# 1. Did the new booking page change the average group size?
#    Two groups, numbers: Welch's t-test.
old, new = rng.normal(4.1, 1.5, 60), rng.normal(4.6, 1.5, 60)
print(f"1. t-test p = {stats.ttest_ind(new, old, equal_var=False).pvalue:.2g}")

# 2. Do the three regions' trips differ in miles? Three groups, numbers: one-way ANOVA.
w, m, s = rng.normal(8, 2, 30), rng.normal(8.4, 2, 30), rng.normal(10, 2, 30)
print(f"2. ANOVA p = {stats.f_oneway(w, m, s).pvalue:.2g}")

# 3. Does preferred difficulty depend on age band? Two categories, counts: chi-square.
# rows: under 40, 40 and over; columns: easy, moderate, hard
counts = np.array([[30, 25, 10], [20, 30, 25]])
print(f"3. chi-square p = {stats.chi2_contingency(counts, correction=False)[1]:.2g}")

# 4. Do members who hike more also rent more gear? Two numbers, skewed: Spearman.
hikes = rng.poisson(6, 50)
rentals = np.maximum(0, hikes + rng.poisson(1, 50) - 3)
rho, p = stats.spearmanr(hikes, rentals)
print(f"4. Spearman rho = {rho:.2f}, p = {p:.2g}")
In the lesson you can change this and run it.

Each line pairs a question with the test its data calls for (:.2g prints a tiny p-value like 1.2e-05 instead of rounding it to 0). Read the results with the earlier warnings in mind: a small p says the pattern is unlikely to be chance, not that it's big or that one thing causes another. Report the size too, such as the difference in means, the sign-up rates by row, or rho, and for question 2, follow the ANOVA with corrected pair comparisons to say which region differs.

In your own words
When is a one-sided test honest?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

An ANOVA on five regions gives p = 0.002. What can you conclude?

  1. At least one region's average differs from the rest
  2. Every region differs from every other region
  3. The highest region beats all the rest significantly
  4. Region causes the differences in the averages

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write