So far every random variable has stood alone. Real data comes in pairs and more: temperature and rentals, height and weight, two pixels of the same picture. The question is whether they move together, and how strongly. Covariance measures it in the data's units; correlation rescales it to a number from −1 to 1 that's easy to read. In ML this tells you which features carry the same information, which ones a model can do without, and it ties back to two earlier lessons: the covariance matrix whose eigenvectors gave PCA's directions (lessons 62 and 63), and correlation, which turns out to be exactly lesson 51's cosine similarity, applied to centered data.
What you'll be able to do
Read a joint table, get each variable's own distribution (the marginals) by summing, and test independence
Compute covariance by centering, multiplying pair by pair and averaging, and correlation from −1 to 1
Build a covariance matrix as XᵀX/n on centered data, and check it against np.cov and torch.cov
Find features that carry the same information, and see that correlation is the cosine of centered data
Quick check
Have a guess before you start. On hotter days the tent shop rents more tents. What sign would their covariance have?
It can't be known
Positive: they rise together
Negative
Zero
Show the answer
Positive: they rise together
When one is above its average the other tends to be too, so their centered products are mostly positive.
The math: moving together
Purpose. Do two quantities move together?
A joint table counts two things at once. 100 days, by weather and by rentals:
blank
high rentals
low rentals
sunny
30
10
rainy
15
45
The marginals are each variable on its own: sum each row for weather (40 sunny, 60 rainy) and each column for rentals (45 high, 55 low). If weather and rentals were independent, P(sunny and high) would equal P(sunny) × P(high) = 0.4 × 0.45 = 0.18. It's actually 0.30, so they're not independent: sun goes with high rentals.
Quick check
From lesson 92: what is the variance of the values 1, 2, 3, 4, each equally likely?
2.5
1.25
1.118
5
Show the answer
1.25
The mean is 2.5; squared distances 2.25, 0.25, 0.25, 2.25 average to 1.25. Covariance is the same idea with two variables.
Covariance. Center each variable (subtract its mean), multiply the pairs, and average. Picture the points on a scatter plot with a cross drawn through the two means. Points in the top-right and bottom-left quarters, above both means or below both, give positive products; points in the other two quarters give negative ones. If most points fall in the first pair of quarters, the average is positive and the variables rise together; if they're spread evenly among all four, the products cancel toward 0. The picture draws exactly that for four days: the points, the dashed cross through the two means, and for each pair a rectangle from the cross to its point. Its sides are the pair's centered values and its area is their product, so points far from both means, with the biggest rectangles, count most.
Try it now
Then set y values to 9, 5, 4, 2 and press Use these numbers: the points move into the − quarters (one sits on a dashed line), and the covariance turns negative.
Symbols and formula. Covariance has awkward units (degrees times tents). Dividing by both standard deviations gives the correlation r, always between −1 and 1:
cov(x,y)=n1i∑(xi−xˉ)(yi−yˉ)r=σxσycov(x,y)
In plain words: r = 1 is a perfect rising line, −1 a perfect falling line, 0 no straight-line link.
For the four pairs: cov = 2.75, σₓ = √1.25 ≈ 1.118 and σᵧ = √6.5 ≈ 2.55, so r = 2.75 ÷ (1.118 × 2.55) ≈ 0.965: strongly rising together. Correlation is not cause. Ice-cream sales and sunburns correlate because both follow the sun; neither causes the other.
Quick check
x = (1, 2, 3, 4) and y = (8, 6, 4, 2). What is their correlation?
0
−1
1
−0.5
Show the answer
−1
y falls by exactly 2 for every step up in x: a perfect falling line, r = −1.
The covariance matrix. Every pair of variables has a covariance, and a variable paired with itself gives its variance: centering x and multiplying it by itself squares the distances. Written as a table, one row and one column per variable, x and y need four entries. Press Next to see which entry is which.
In your own words
In your own words: why does a correlation always land between −1 and 1, while a covariance can be any size?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: center, multiply, average
Purpose. Compute covariance by hand-written lines, then the whole covariance matrix at once.
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([2.0, 4.0, 5.0, 9.0])
dx = x - x.mean()
dy = y - y.mean()
cov = np.mean(dx * dy)
r = cov / (x.std() * y.std())
print(cov, round(float(r), 3), round(float(np.corrcoef(x, y)[0, 1]), 3))
print(np.cov(x, y, bias=True).tolist())
np.corrcoef returns a table of correlations; entry [0, 1] is between x and y, and it matches the hand-written r. np.cov returns the covariance matrix from the math part's picture, 1.25 and 6.5 on the diagonal and 2.75 off it (bias=True is explained below). With many features, center a data matrix X (one row per example) and Xᵀ X / n gives every covariance at once (lesson 62 found PCA's directions in this matrix).
Code flow. Step through center, multiply, average one line at a time. The arrays show as rows of boxes, the same boxes as the flow in the math part: dx and dy are the centered columns, products their pairs multiplied.
A named mistake: n or n − 1.np.cov divides by n − 1 unless told bias=True, which divides by n. Both are used: n − 1 corrects a small bias when estimating from a sample (lesson 97). They differ noticeably only for small samples, but mixing them makes hand checks fail. Here, everything divides by n, so the check uses bias=True.
import numpy as np
X = np.array([[20, 30], [25, 45], [30, 60], [22, 40], [28, 55]], dtype=float)
Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / 5print(C.round(2).tolist(), np.allclose(C, np.cov(X.T, bias=True)), np.allclose(C, np.cov(X.T)))
Dividing by n, 5 here, matches np.cov with bias=True (True) and not its default (False).
Worked example
Worked example: a 2 × 2 covariance matrix by hand-written lines
Build the covariance matrix of temperature and rentals entry by entry, and check it against np.cov.
import numpy as np
t = np.array([20.0, 25.0, 30.0, 22.0, 28.0])
r = np.array([30.0, 45.0, 60.0, 40.0, 55.0])
tc = t - t.mean()
rc = r - r.mean()
C = np.array([[np.mean(tc * tc), np.mean(tc * rc)],
[np.mean(rc * tc), np.mean(rc * rc)]])
print(C.round(2).tolist())
print(np.allclose(C, np.cov(np.array([t, r]), bias=True)))
print(round(float(C[0, 1] / (C[0, 0] ** 0.5 * C[1, 1] ** 0.5)), 3))
In the lesson you can change this and run it.
The diagonal holds each variable's variance, 13.6 and 114; the off-diagonal entries are the covariance, 39, twice, since cov(t, r) = cov(r, t): the matrix is symmetric (lesson 57). Dividing the covariance by the two standard deviations, the roots of the diagonal, gives r = 0.99: on these days, temperature and rentals rise almost in lockstep.
Quick check
What does this print?
import numpy as np
a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 2.0, 2.0])
print(np.mean((a - a.mean()) * (b - b.mean())))
0.0
2.0
1.0
6.0
Show the answer
0.0
b never changes, so every centered b is 0, and so is every product: no covariance with a constant.
Plot it. Centering moves the cross of the means to (0, 0), so the four quarters become the plot's own. Draw the five tent days centered, with the cross as two dashed lines:
import numpy as np
import matplotlib.pyplot as plt
t = np.array([20, 25, 30, 22, 28])
r = np.array([30, 45, 60, 40, 55])
plt.plot(t - t.mean(), r - r.mean(), "o")
plt.plot([-6, 6], [0, 0], "--")
plt.plot([0, 0], [-20, 20], "--")
plt.xlabel("temperature, centered")
plt.ylabel("rentals, centered")
plt.show()
Quick check
Read the plot. How many of the five days sit in the top-right or bottom-left quarter?
5
4
3
2
Show the answer
4
Two days sit up and right, two down and left, and the 25-degree day is exactly on the upright dashed line: its centered temperature is 0, so its product is 0. Four positive products and one 0 give the positive covariance, 39.
Units change covariance, not correlation. Measure temperature in Fahrenheit instead of Celsius and every value is multiplied by 1.8 and shifted by 32. The shift vanishes when the data is centered, and the 1.8 multiplies both the covariance and σₓ, so it cancels in r. The covariance itself grows by the same factor 1.8, which is why covariances of differently measured variables can't be compared and correlations can. A negative stretch flips r's sign and nothing else. One limit is worth knowing early: r measures straight-line links only. Values that rise and then fall, like y = x² for x from −2 to 2, have r = 0 even though x settles y completely.
Quick check
What does this print?
import numpy as np
c = np.array([10.0, 15.0, 20.0, 25.0])
r = np.array([30.0, 45.0, 40.0, 60.0])
f = c * 1.8 + 32print(round(float(np.corrcoef(c, r)[0, 1]), 3), round(float(np.corrcoef(f, r)[0, 1]), 3))
0.878 1.58
0.878 0.878
0.878 0.488
1.0 1.0
Show the answer
0.878 0.878
Changing units stretches and shifts c, and r ignores both: 0.878 either way. The covariance grows 1.8 times; r doesn't.
Drill
Correlation by hand
Fix corr(x, y) so it divides the covariance by both standard deviations. Then set r1 for the pairs below, rounded to 3 places, and same to whether it matches np.corrcoef to 3 places.
import numpy as np
defcorr(x, y):
dx = x - x.mean()
dy = y - y.mean()
return np.mean(dx * dy)
x = np.array([2.0, 4.0, 6.0, 8.0])
y = np.array([1.0, 3.0, 2.0, 6.0])
r1 = round(float(corr(x, y)), 3)
same = Falseprint(r1, same)
In the lesson, your code is checked as you write it.
In your own words
Why is a covariance matrix always symmetric?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: features that carry the same information
Purpose. Spot redundant features, and see correlation as cosine similarity.
Correlation is lesson 51's cosine similarity, applied to centered data:
r=∥xc∥∥yc∥xc⋅yc
In plain words: center both columns, then take the cosine of the angle between them.
import numpy as np
t = np.array([20.0, 25.0, 30.0, 22.0, 28.0])
r = np.array([30.0, 45.0, 60.0, 40.0, 55.0])
tc = t - t.mean()
rc = r - r.mean()
print(round(float(tc @ rc / (np.linalg.norm(tc) * np.linalg.norm(rc))), 3))
0.99, the same as before. A feature almost perfectly correlated with another adds almost nothing new: it's nearly a rescaled copy, the redundancy lessons 59 and 61 warned about.
In PyTorch.torch.cov and torch.corrcoef take one row per variable, like their NumPy namesakes, so torch.stack stacks the two columns as rows. correction=0 divides by n, like bias=True in np.cov.
import torch
t = torch.tensor([20.0, 25.0, 30.0, 22.0, 28.0])
r = torch.tensor([30.0, 45.0, 60.0, 40.0, 55.0])
X = torch.stack([t, r])
print(torch.cov(X, correction=0))
print(round(torch.corrcoef(X)[0, 1].item(), 3))
The matrix is the one the worked example built by hand, 13.6 and 114 on the diagonal and 39 off it, and entry [0, 1] of the correlations is 0.99, the number the NumPy cell above printed as a cosine.
Quick check
Two pixels in a set of pictures always have the same value. What is their correlation, and what does it mean for a model?
r = 1: one of them is redundant
r can't be computed for pixels
r = 0: they're unrelated
r = −1: they cancel
Show the answer
r = 1: one of them is redundant
Identical columns line up perfectly: r = 1. Keeping both adds no information.
What to do with redundant features. A table of correlations, one entry for every pair of features, is usually one of the first things to look at in a new data set. Pairs near 1 or −1 point to features that say the same thing twice. Keeping both rarely breaks anything, but it can make a linear model's weights unstable: with two nearly identical columns, many ways of splitting the weight between them fit equally well, lesson 61's dependent columns again. The usual fixes are to drop one of the pair, or to combine them, which is what PCA does in a principled way (lesson 63). Pairs near 0 are the opposite case: each feature carries a straight-line signal the other doesn't, so both are worth keeping.
Drill
The most and least correlated features
Four features (temperature, rentals, humidity, wind) are the columns of X. Set R to their correlation table with np.corrcoef, then strongest and weakest to the pairs (i, j), with i < j, whose correlations are biggest and smallest in size.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write