Index Ai Academy
Foundation · Foundation mathProbability

Two variables at once: covariance and correlation

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

So far every random variable has stood alone. Real data comes in pairs and more: temperature and rentals, height and weight, two pixels of the same picture. The question is whether they move together, and how strongly. Covariance measures it in the data's units; correlation rescales it to a number from −1 to 1 that's easy to read. In ML this tells you which features carry the same information, which ones a model can do without, and it ties back to two earlier lessons: the covariance matrix whose eigenvectors gave PCA's directions (lessons 62 and 63), and correlation, which turns out to be exactly lesson 51's cosine similarity, applied to centered data.

  • Read a joint table, get each variable's own distribution (the marginals) by summing, and test independence
  • Compute covariance by centering, multiplying pair by pair and averaging, and correlation from −1 to 1
  • Build a covariance matrix as XᵀX/n on centered data, and check it against np.cov and torch.cov
  • Find features that carry the same information, and see that correlation is the cosine of centered data
Quick check
Have a guess before you start. On hotter days the tent shop rents more tents. What sign would their covariance have?

The math: moving together

Purpose. Do two quantities move together?

A joint table counts two things at once. 100 days, by weather and by rentals:

blankhigh rentalslow rentals
sunny3010
rainy1545

The marginals are each variable on its own: sum each row for weather (40 sunny, 60 rainy) and each column for rentals (45 high, 55 low). If weather and rentals were independent, P(sunny and high) would equal P(sunny) × P(high) = 0.4 × 0.45 = 0.18. It's actually 0.30, so they're not independent: sun goes with high rentals.