More tests: groups, categories, relationships
Choose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.
What it covers8
Why it matters
A/B tests and t-tests compare two groups. Real questions are often messier: three landing pages at once, whether the sign-up rate differs by region, whether customers who get more emails spend more. Each has its own standard test, and using the wrong one, or running ten t-tests where one test would do, is how false "findings" get into reports. This lesson gives you a short map from question to test, and builds each test by hand before checking it against SciPy, so the numbers stop being magic.
What you'll be able to do
- Pick the right test from the question: groups, categories or a relationship
- Compare three or more groups with one ANOVA instead of many t-tests, and compute it by hand
- Test whether two categories are related with a chi-square test on a table of counts
- Measure a relationship with Pearson or Spearman correlation, and know when an outlier or a confounder misleads
Show the answer
Counts in categories: signed up or not, in each region
Signed up or not, by region, is a table of counts, which calls for a chi-square test rather than a t-test. Start from what you're comparing, and the right test follows.
Which test answers this question?
Start from what you're comparing, not from the test's name:
| your question | what you have | the usual test |
|---|---|---|
| Is B's average different from A's? | a number per person, two groups | Welch's t-test (the comparing-means lesson) |
| Is B's rate different from A's? | yes/no per person, two groups | two-proportion z-test (the A/B testing lesson) |
| Do three or more groups' averages differ? | a number per person, three or more groups | one-way ANOVA |
| Are two categories related? | counts in a table, such as region by signed up or not | chi-square test of independence |
| Do two numbers rise and fall together? | two numbers per person | Pearson or Spearman correlation |
Each test still asks the same question you've met before: if nothing were going on (the null hypothesis), how surprising would data like this be? The p-value is that surprise, and the usual threshold is still 0.05.
orders = [100, 900] damaged = [1, 45] pooled = sum(damaged) / sum(orders) mean_of_rates = (damaged[0] / orders[0] + damaged[1] / orders[1]) / 2 print(round(pooled, 3), round(mean_of_rates, 3))
Show the answer
0.046 0.03
Pooled, 46 of 1,000 orders were damaged: 0.046. The two months' own rates are 0.01 and 0.05, and their mean is 0.03. The busy month was also the worse one, and pooling gives its 900 orders more weight. Picking by the question is the habit of this lesson too: the test follows from what you are comparing.
Ask: how many groups, and is each person's answer a number, a yes/no, or a category?
Three groups: why not three t-tests?
Three landing pages give three pairs to compare (A-B, A-C, B-C). Each t-test has a 5% chance of a false alarm, so running all three gives about a 14% chance that at least one "wins" by luck alone. With ten groups there are 45 pairs, and the chance of at least one false alarm is about 90% (treating the tests as independent). ANOVA (analysis of variance) asks one question instead: is there more spread between the group averages than the spread within the groups would explain?
It divides two numbers. The spread between groups: how far each group's mean sits from the overall mean, weighted by group size. The spread within groups: how far each person sits from their own group's mean. Each is divided by its degrees of freedom, the number of values that are free to vary once the totals are fixed: groups minus 1 between groups, and people minus groups within them. That makes the two spreads comparable. Their ratio is F. If the groups are really alike, F is around 1; a big F means the means differ more than chance explains.
import numpy as np from scipy import stats a = np.array([42, 51, 38, 47, 55, 44, 49, 40]) # seconds on page, design A b = np.array([58, 49, 61, 54, 66, 52, 57, 60]) # design B c = np.array([45, 50, 43, 52, 48, 46, 51, 44]) # design C groups = [a, b, c] everyone = np.concatenate(groups) grand = everyone.mean() between = sum(len(g) * (g.mean() - grand) ** 2 for g in groups) / (len(groups) - 1) within = (sum(((g - g.mean()) ** 2).sum() for g in groups) / (len(everyone) - len(groups))) F = between / within p = stats.f.sf(F, len(groups) - 1, len(everyone) - len(groups)) print("means:", [round(float(g.mean()), 1) for g in groups]) print("F =", round(F, 2), " p =", round(p, 5)) print(stats.f_oneway(a, b, c))
F is 12.2 and p is about 0.0003, matching SciPy's f_oneway: the designs differ. But ANOVA only says some group differs, not which. The next step is to compare pairs with a correction for testing several, such as the Bonferroni correction from the A/B testing lesson, and here B is clearly ahead of both.
print(round(1 - 0.95 ** 6, 2))
Show the answer
0.26
The chance that all six tests stay quiet is 0.95 to the sixth power, about 0.74, so the chance of at least one false alarm is about 0.26. One ANOVA keeps it at 0.05.
Write a sentence or two, then compare it with a good answer in the lesson.
Two categories: the chi-square test
Do sign-ups depend on the region a visitor came from? Count them in a table, one row per region and one column per outcome. If region and signing up were unrelated, each row would sign up at the overall rate, so every cell has an expected count: its row's total times its column's total, divided by the grand total. The chi-square statistic adds up how far each observed count is from its expected count, squared and scaled by the expected count: the sum of (observed - expected)² / expected. In the code, keepdims=True keeps the row totals as a column (shape 3 × 1), so multiplying by the row of column totals makes every row-and-column pair at once.
import pandas as pd from scipy import stats table = pd.DataFrame({"signed_up": [48, 30, 22], "did_not": [152, 170, 78]}, index=["west", "mountain", "south"]) observed = table.to_numpy() expected = (observed.sum(axis=1, keepdims=True) * observed.sum(axis=0) / observed.sum()) chi2 = ((observed - expected) ** 2 / expected).sum() dof = (observed.shape[0] - 1) * (observed.shape[1] - 1) print(pd.DataFrame(expected, index=table.index, columns=table.columns)) print("chi-square =", round(chi2, 2), " degrees of freedom =", dof, " p =", round(stats.chi2.sf(chi2, dof), 4)) print(stats.chi2_contingency(observed, correction=False)[:3])
The west signs up at 24% and the mountain region at 15%, which looks like a real gap, but chi-square is 5.38 with a p of 0.068: with these counts, a spread like this turns up by chance more than 5% of the time. Two rules for using it: it works on counts, never percentages, and every expected count should be at least about 5, or the p-value isn't trustworthy (merge small categories, or use an exact test such as Fisher's).
row_total, column_total, grand_total = 200, 100, 500 expected = row_total * column_total / grand_total observed = 48 print(expected, round((observed - expected) ** 2 / expected, 2))
Show the answer
40.0 1.6
If region made no difference, the west would sign up at the overall rate, 100 in 500, so 200 visitors would give 40. The west had 48, which is 8 more. Squared and divided by the expected count, that is 64 ÷ 40 = 1.6. Chi-square adds a number like this for every cell of the table.
The counts you'd expect if nothing were going on
Write expected_counts(observed) for a 2-D NumPy array of counts: each cell is its row total times its column total, divided by the grand total. Then chi_square(observed), returning the statistic, the sum of (observed - expected)² / expected. Use keepdims=True on the row totals so they broadcast across the columns.
import numpy as np def expected_counts(observed): return np.full(observed.shape, observed.mean()) def chi_square(observed): return 0.0 observed = np.array([[48, 152], [30, 170], [22, 78]]) print(expected_counts(observed)) print(round(chi_square(observed), 2))
Write a sentence or two, then compare it with a good answer in the lesson.
Relationships: Pearson and Spearman
Correlation measures how two numbers move together, from -1 (one always falls as the other rises) through 0 (no pattern) to 1 (they rise together). Pearson's r measures a straight-line relationship using the values themselves. Spearman's rho first replaces each value by its rank (1st, 2nd, 3rd...) and then computes Pearson's r on the ranks, so it measures whether the relationship always goes the same way, straight or curved, and one wild value can't drag it far.
import numpy as np from scipy import stats rng = np.random.default_rng(4) emails = np.arange(1, 21) # emails sent to 20 members spend = 20 + 3 * emails + rng.normal(0, 6, 20) # dollars spent, mostly rising print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3), " Spearman", round(stats.spearmanr(emails, spend)[0], 3)) spend[0] = 400 # one member's huge purchase print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3), " Spearman", round(stats.spearmanr(emails, spend)[0], 3))
One extreme purchase turns Pearson's 0.93 into -0.18, a negative relationship that isn't there; Spearman only drops from 0.91 to 0.63. So plot the data first, and prefer Spearman when values are skewed or have outliers.
Then remember what correlation can't tell you. Members who get more emails may spend more because the shop emails its keenest customers most: keenness drives both, a confounder. Only an experiment, such as emailing a random half, separates cause from coincidence.
Show the answer
Spearman is exactly 1, and Pearson is a little below 1
Spearman works on ranks. The smallest x has the smallest y, the next has the next, and so on, so the ranks match perfectly and rho is 1. Pearson measures how close the points are to one straight line. A curve that only bends one way is close to a line, but not on one, so r comes out a little under 1.
Spearman is Pearson on ranks
Write spearman(x, y): rank each list with scipy.stats.rankdata (it gives tied values the average of their ranks), then return the Pearson correlation of the two rank lists, computed with np.corrcoef(rx, ry)[0, 1]. The checks compare it with SciPy's own spearmanr.
import numpy as np from scipy import stats def spearman(x, y): return float(np.corrcoef(x, y)[0, 1]) miles = [3, 7, 12, 5, 20, 9] rentals = [1, 2, 4, 2, 30, 3] print(round(spearman(miles, rentals), 3))
Write a sentence or two, then compare it with a good answer in the lesson.
One-sided tests, used honestly
A reminder from the A/B testing lesson, because it matters for every test here. The tests so far are two-sided: they flag a difference in either direction. If, before you collect data, you only care about one direction (the new page is better, or you'll keep the old one either way), a one-sided test looks only there, and its p-value is half the two-sided one when the data lean that way. In SciPy, pass alternative="greater" or "less".
The honest rule is about timing: choose one-sided in your plan, before seeing results. Switching after a two-sided p of 0.08 to get a one-sided 0.04 is a classic way to fool yourself, and reviewers know to ask.
Show the answer
Picking the direction after seeing the data
A one-sided test is legitimate when the plan said so before the data arrived. Picking it afterward, because B happened to be ahead, is a way of moving the goalposts.
Worked example: four questions, four tests
The trail club has four questions about one season's data. Each gets its own test, and each answer is reported with the number behind it.
import numpy as np from scipy import stats rng = np.random.default_rng(7) # 1. Did the new booking page change the average group size? # Two groups, numbers: Welch's t-test. old, new = rng.normal(4.1, 1.5, 60), rng.normal(4.6, 1.5, 60) print(f"1. t-test p = {stats.ttest_ind(new, old, equal_var=False).pvalue:.2g}") # 2. Do the three regions' trips differ in miles? Three groups, numbers: one-way ANOVA. w, m, s = rng.normal(8, 2, 30), rng.normal(8.4, 2, 30), rng.normal(10, 2, 30) print(f"2. ANOVA p = {stats.f_oneway(w, m, s).pvalue:.2g}") # 3. Does preferred difficulty depend on age band? Two categories, counts: chi-square. # rows: under 40, 40 and over; columns: easy, moderate, hard counts = np.array([[30, 25, 10], [20, 30, 25]]) print(f"3. chi-square p = {stats.chi2_contingency(counts, correction=False)[1]:.2g}") # 4. Do members who hike more also rent more gear? Two numbers, skewed: Spearman. hikes = rng.poisson(6, 50) rentals = np.maximum(0, hikes + rng.poisson(1, 50) - 3) rho, p = stats.spearmanr(hikes, rentals) print(f"4. Spearman rho = {rho:.2f}, p = {p:.2g}")
Each line pairs a question with the test its data calls for (:.2g prints a tiny p-value like 1.2e-05 instead of rounding it to 0). Read the results with the earlier warnings in mind: a small p says the pattern is unlikely to be chance, not that it's big or that one thing causes another. Report the size too, such as the difference in means, the sign-up rates by row, or rho, and for question 2, follow the ANOVA with corrected pair comparisons to say which region differs.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
An ANOVA on five regions gives p = 0.002. What can you conclude?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write