More tests: groups, categories, relationships

Two categories: the chi-square test

Do sign-ups depend on the region a visitor came from? Count them in a table, one row per region and one column per outcome. If region and signing up were unrelated, each row would sign up at the overall rate, so every cell has an expected count: its row's total times its column's total, divided by the grand total. The chi-square statistic adds up how far each observed count is from its expected count, squared and scaled by the expected count: the sum of (observed - expected)² / expected. In the code, keepdims=True keeps the row totals as a column (shape 3 × 1), so multiplying by the row of column totals makes every row-and-column pair at once.

import numpy as np
import pandas as pd
from scipy import stats

table = pd.DataFrame({"signed_up": [48, 30, 22], "did_not": [152, 170, 78]}, index=["west", "mountain", "south"])
observed = table.to_numpy()
expected = observed.sum(axis=1, keepdims=True) * observed.sum(axis=0) / observed.sum()
chi2 = ((observed - expected) ** 2 / expected).sum()
dof = (observed.shape[0] - 1) * (observed.shape[1] - 1)
print(pd.DataFrame(expected, index=table.index, columns=table.columns))
print("chi-square =", round(chi2, 2), " degrees of freedom =", dof, " p =", round(stats.chi2.sf(chi2, dof), 4))
print(stats.chi2_contingency(observed, correction=False)[:3])

The west signs up at 24% and the mountain region at 15%, which looks like a real gap, but chi-square is 5.38 with a p of 0.068: with these counts, a spread like this turns up by chance more than 5% of the time. Two rules for using it: it works on counts, never percentages, and every expected count should be at least about 5, or the p-value isn't trustworthy (merge small categories, or use an exact test such as Fisher's).

import numpy as np


def expected_counts(observed):
    return np.full(observed.shape, observed.mean())


def chi_square(observed):
    return 0.0


observed = np.array([[48, 152], [30, 170], [22, 78]])
print(expected_counts(observed))
print(round(chi_square(observed), 2))

Relationships: Pearson and Spearman

Correlation measures how two numbers move together, from -1 (one always falls as the other rises) through 0 (no pattern) to 1 (they rise together). Pearson's r measures a straight-line relationship using the values themselves. Spearman's rho first replaces each value by its rank (1st, 2nd, 3rd...) and then computes Pearson's r on the ranks, so it measures whether the relationship always goes the same way, straight or curved, and one wild value can't drag it far.

import numpy as np
from scipy import stats

rng = np.random.default_rng(4)
emails = np.arange(1, 21)                                 # emails sent to 20 members
spend = 20 + 3 * emails + rng.normal(0, 6, 20)            # dollars spent, roughly rising

print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3), " Spearman", round(stats.spearmanr(emails, spend)[0], 3))
spend[0] = 400                                            # one member made a huge one-off purchase
print("Pearson ", round(stats.pearsonr(emails, spend)[0], 3), " Spearman", round(stats.spearmanr(emails, spend)[0], 3))

One extreme purchase turns Pearson's 0.93 into -0.18, a negative relationship that isn't there; Spearman only drops from 0.91 to 0.63. So plot the data first, and prefer Spearman when values are skewed or have outliers.

Then remember what correlation can't tell you. Members who get more emails may spend more because the shop emails its keenest customers most: keenness drives both, a confounder. Only an experiment, such as emailing a random half, separates cause from coincidence.

import numpy as np
from scipy import stats


def spearman(x, y):
    return float(np.corrcoef(x, y)[0, 1])


miles = [3, 7, 12, 5, 20, 9]
rentals = [1, 2, 4, 2, 30, 3]
print(round(spearman(miles, rentals), 3))

One-sided tests, used honestly

A reminder from the A/B testing lesson, because it matters for every test here. The tests so far are two-sided: they flag a difference in either direction. If, before you collect data, you only care about one direction (the new page is better, or you'll keep the old one either way), a one-sided test looks only there, and its p-value is half the two-sided one when the data lean that way. In SciPy, pass alternative="greater" or "less".

The honest rule is about timing: choose one-sided in your plan, before seeing results. Switching after a two-sided p of 0.08 to get a one-sided 0.04 is a classic way to fool yourself, and reviewers know to ask.

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
# 1. Did the new booking page change the average group size? Two groups, numbers: Welch's t-test.
old, new = rng.normal(4.1, 1.5, 60), rng.normal(4.6, 1.5, 60)
print(f"1. t-test p = {stats.ttest_ind(new, old, equal_var=False).pvalue:.2g}")

# 2. Do the three regions' trips differ in miles? Three groups, numbers: one-way ANOVA.
w, m, s = rng.normal(8, 2, 30), rng.normal(8.4, 2, 30), rng.normal(10, 2, 30)
print(f"2. ANOVA p = {stats.f_oneway(w, m, s).pvalue:.2g}")

# 3. Does preferred difficulty depend on age band? Two categories, counts: chi-square.
counts = np.array([[30, 25, 10], [20, 30, 25]])        # rows: under 40, 40 and over; columns: easy, moderate, hard
print(f"3. chi-square p = {stats.chi2_contingency(counts, correction=False)[1]:.2g}")

# 4. Do members who hike more also rent more gear? Two numbers, skewed: Spearman.
hikes = rng.poisson(6, 50)
rentals = np.maximum(0, hikes + rng.poisson(1, 50) - 3)
rho, p = stats.spearmanr(hikes, rentals)
print(f"4. Spearman rho = {rho:.2f}, p = {p:.2g}")