Index Ai Academy
Phase 2 · Math, when you need it · Probability & statistics

Correlation and the shape of data

Covariance and correlation, curves, outliers and common causes, rank correlation, skew and kurtosis.

About 30 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

"Customers who view more pages spend more." "Longer prompts get better answers." Claims like these are correlations, and they drive product decisions every week. Correlation is easy to compute and easy to misread: it misses curved relationships, one odd point can create or destroy it, and two things can move together only because something else drives both. This lesson builds correlation from the variance you know, then looks at the shape of data, because skewed and heavy-tailed numbers (order values, response times, token counts) break the averages people report.

What you'll be able to do

  • Compute covariance and correlation, and say why correlation is the one to compare
  • Spot the three ways correlation misleads: curves, outliers and a hidden common cause
  • Use rank (Spearman) correlation when a relationship is steady but not straight
  • Describe a distribution's shape with skew and kurtosis, and choose the median when tails are long

What it covers

  1. Covariance: moving together
  2. Correlation: covariance without units
  3. Find the bug: correlation without the means
  4. Three ways correlation misleads
  5. When the groups disagree with the whole
  6. Rank correlation for steady but curved relationships
  7. The shape of a distribution: skew and kurtosis
  8. Working with long tails
Quick check
A guess first. Over eight summer weeks, the shop's tent sales and sunscreen sales rose and fell together almost perfectly. What's the most likely explanation?
  1. Warm weather drives both, so they move together
  2. Sunscreen advertising sells tents
  3. Buying a tent makes people buy sunscreen
  4. It's a coincidence across eight weeks
Show the answer

Warm weather drives both, so they move together

A third thing, the weather, pushes both up and down. Two series moving together says nothing about one causing the other.

Covariance: moving together

You know variance: the average squared distance from the mean. Covariance does the same for two variables at once, multiplying their distances from their means instead of squaring one:

cov(x,y)=1n−1∑i(xi−xˉ)(yi−yˉ)\text{cov}(x, y) = \frac{1}{n-1}\sum_i (x_i - \bar x)(y_i - \bar y)

In plain words: each pair of values votes positive when both sit on the same side of their averages, and negative when they sit on opposite sides.

When both are above their means together, or below together, the products are positive and the covariance is positive. When one is high while the other is low, it's negative. Near zero means no straight-line tendency.

import statistics as st

tents = [12, 15, 22, 30, 41, 38, 25, 14]
sunscreen = [40, 52, 70, 95, 130, 118, 80, 45]   # bottles
print("covariance:", round(st.covariance(tents, sunscreen), 1))
cases = [s / 12 for s in sunscreen]              # the same sales, counted in cases of 12
print("in cases:", round(st.covariance(tents, cases), 2))

Same sales, two very different covariances: its size depends on the units, so "368.6" alone means nothing.

Correlation: covariance without units

Divide the covariance by both standard deviations and the units cancel. That's the correlation rr, always between -1 and 1:

r=cov(x,y)sx syr = \frac{\text{cov}(x, y)}{s_x \, s_y}

r=1r = 1 means the points sit on a rising straight line, −1-1 on a falling one, and 0 means no straight-line relationship. Because units cancel, correlations can be compared across very different pairs of measurements.

There's a neat link to the embeddings lessons: subtract each variable's mean, and the correlation is exactly the cosine similarity of the two centered vectors. Correlation is cosine similarity after centering.

Squaring rr gives a useful reading: r2r^2 is the share of the variation in one variable that a straight line on the other accounts for. A correlation of 0.5 sounds strong, yet the line explains only a quarter of the variation; three quarters comes from everything else.

Quick check
Study hours and quiz scores have a correlation of 0.3. How much of the variation in scores does a straight line on hours account for?
  1. About 3%
  2. About 70%
  3. About 9%
  4. About 30%
Show the answer

About 9%

r² = 0.3² = 0.09. A correlation of 0.3 is real but weak: most of the differences in scores come from other things.

Quick check
Sunscreen is now counted in cases of 12 instead of bottles. What happens to the correlation with tent sales?
import statistics as st
tents = [12, 15, 22, 30, 41, 38, 25, 14]
bottles = [40, 52, 70, 95, 130, 118, 80, 45]
cases = [b / 12 for b in bottles]
print(round(st.correlation(tents, bottles), 3), round(st.correlation(tents, cases), 3))
  1. 368.6 30.72
  2. 0.999 0.083
  3. 0.999 0.999
  4. 0.999 12.0
Show the answer

0.999 0.999

Rescaling one variable changes its covariance but not the correlation: the standard deviation shrinks by the same factor and cancels it out.

Drill

Find the bug: correlation without the means

correlation(xs, ys) should return Pearson's r, but it says [1, 2, 3] and [3, 2, 1] are positively related, when one goes down exactly as the other goes up. It's computing cosine similarity of the raw values. Fix it.

import math


def correlation(xs, ys):
    num = sum(x * y for x, y in zip(xs, ys))
    den = math.sqrt(sum(x * x for x in xs) * sum(y * y for y in ys))
    return num / den


print(round(correlation([1, 2, 3], [3, 2, 1]), 3))
In the lesson, your code is checked as you write it.
In your own words
Why compare correlations rather than covariances?

Write a sentence or two, then compare it with a good answer in the lesson.

Three ways correlation misleads

Curves. Correlation measures straight-line relationships only. Satisfaction that rises with delivery speed up to a point and then doesn't change, or a U shape, can have a correlation near zero while one variable almost fully determines the other. Always plot before you trust rr.

Outliers. One extreme point can create a strong correlation out of noise, or wipe out a real one, because distances from the mean are squared in spirit. The cell below adds a single unusual week to six unrelated ones.

import statistics as st

visits = [1, 2, 3, 4, 5, 6]
orders = [3, 1, 4, 1, 5, 2]
print("six weeks:", round(st.correlation(visits, orders), 3))
print("plus one launch week:", round(st.correlation(visits + [30], orders + [30]), 3))

A common cause. Tents and sunscreen correlate because of weather; page views and spending may both be driven by how much someone already wanted to buy. Only an experiment, changing one thing on purpose and watching the other, shows cause, which is why the A/B tests lesson matters.

Quick check
What does this print?
import statistics as st
x = list(range(11))
y = [(v - 5) ** 2 for v in x]
print(round(st.correlation(x, y), 3))
  1. 1.0
  2. -1.0
  3. 0.5
  4. 0.0
Show the answer

0.0

y is completely determined by x, but in a U shape: the falling half and the rising half cancel. Correlation 0 means no straight-line relationship, not no relationship.

Try it now

A straight-line fit with R², which for one variable is the correlation squared. Drag a single point far from the others.

When the groups disagree with the whole

A fourth trap catches experienced analysts. Within every group, a relationship can point one way while the pooled data points the other: Simpson's paradox. Suppose that in both of the shop's warehouses, faster packing comes with fewer complaints. But the busy city warehouse packs fast and gets many complaints from its sheer volume of awkward orders, while the small rural one packs slowly and gets few. Pool the two and "faster packing" seems to go with "more complaints".

The pooled correlation isn't wrong arithmetic; it's answering a different question, mixing the effect of packing speed with the difference between warehouses. When groups differ a lot, compute the relationship within each group before trusting the total.

Quick check
In each warehouse, faster packing goes with fewer complaints. Pooled, faster packing goes with more. What explains it?
  1. There were too few orders to tell either way
  2. The fast warehouse also gets more complaints
  3. One of the two calculations must be wrong
  4. Packing speed has no real effect at all
Show the answer

The fast warehouse also gets more complaints

The busy warehouse is both faster and gets more complaints, so pooling mixes the warehouse difference into the speed relationship. Within each warehouse, the real pattern shows.

In your own words
Name three ways a correlation can mislead.

Write a sentence or two, then compare it with a good answer in the lesson.

Rank correlation for steady but curved relationships

Often you only care whether one thing goes up when the other does, not whether the line is straight. Spearman's rank correlation replaces each value by its rank (1st, 2nd, 3rd...) and correlates the ranks. Any relationship that always rises gets 1, however curved, and one huge value can't dominate, because its rank is just "the largest".

import statistics as st

prompt_words = [1, 2, 3, 4, 5, 6, 7, 8]
cost = [1, 2, 3, 4, 5, 6, 7, 100]
print("Pearson:", round(st.correlation(prompt_words, cost), 3))
print("Spearman:", round(st.correlation(prompt_words, cost, method="ranked"), 3))

Pearson is pulled around by the single 100; Spearman sees a perfectly steady rise. Use Spearman for ratings, rankings and anything with long tails, and Pearson when you expect a straight line.

Quick check
A reminder from the variance lesson. Correlation divides covariance by the two standard deviations. What is a standard deviation?
  1. The average distance from the median
  2. The square root of the variance
  3. The variance divided by the count
  4. The difference between the largest and smallest values
Show the answer

The square root of the variance

Variance is the average squared distance from the mean; its square root, the standard deviation, is back in the data's own units.

The shape of a distribution: skew and kurtosis

The mean and standard deviation describe a distribution well only when it's roughly symmetric. Two more numbers describe the shape.

Skew measures lopsidedness. Order values, response times, salaries and token counts are right-skewed: most values are modest, a few are huge, and the long tail drags the mean far above the median. Report the median (and percentiles such as the 90th) for skewed data, and consider analyzing the logarithm, which often makes it close to symmetric.

Kurtosis measures how heavy the tails are: how often extreme values turn up compared with a normal distribution. High kurtosis means rare but large surprises, the kind that break averages, alerts and capacity plans built on "typical" days. Here are the shop's order values for one morning:

import statistics as st

orders = [18, 22, 25, 25, 30, 34, 41, 48, 60, 95, 240, 610]
mean, sd = st.mean(orders), st.pstdev(orders)
skew = sum(((x - mean) / sd) ** 3 for x in orders) / len(orders)
kurtosis = sum(((x - mean) / sd) ** 4 for x in orders) / len(orders) - 3
print("mean", round(mean, 1), "median", st.median(orders))
print("skew", round(skew, 2), "excess kurtosis", round(kurtosis, 2))

A skew of 2.45 and excess kurtosis of 4.72 say clearly: long right tail, and rare large orders. A normal distribution scores 0 on both.

Quick check
For the morning's orders, the mean is 104 dollars and the median 37.5. A manager asks what a typical order is. Which number should you give?
  1. The mean, 104
  2. The largest order, 610
  3. The median, 37.5
  4. The midpoint of 18 and 610
Show the answer

The median, 37.5

Ten of the twelve orders are below 100; the mean sits above almost all of them because of two large ones. For skewed data, the median describes the typical case.

Working with long tails

Long tails are everywhere in AI products. A few long conversations use most of the tokens, a few slow requests set how long users wait, and a few heavy customers produce most of the cost. Three habits help.

Report percentiles alongside the median: the 50th, 90th and 99th percentiles of response time say far more than the mean. Analyze on a log scale when values span orders of magnitude, so a model or a chart isn't dominated by the largest few. And look at the tail itself, because that's often where the money and the complaints are: the top 1% of conversations by cost deserves its own report.

Quick check
You're reporting token cost per conversation to the team. Which summary is most useful?
  1. The mean, since it counts every token
  2. The median, with the 90th and 99th percentiles
  3. The standard deviation of the cost
  4. The most expensive conversation this month
Show the answer

The median, with the 90th and 99th percentiles

The median shows a typical conversation and the percentiles show how heavy the tail is, which is where most of the cost hides. The mean blends the two into a number that describes neither.

In your own words
When do you reach for Spearman's rank correlation, and when for the median instead of the mean?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 6 review cards for spaced repetition

Create a free account

Try a quiz question

Why compare correlations rather than covariances?

  1. Correlation has no units
  2. Covariance can't be negative
  3. Correlation ignores outliers
  4. Covariance only works for two variables

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson