Lesson 58 showed a matrix moving the whole plane. Watch the arrows as it moves and almost every one swings to a new direction. But a few special directions don't turn at all: arrows along them only get longer, shorter or reversed. Those are the matrix's eigenvectors, its natural axes, and the stretch factor along each is its eigenvalue. They reveal what a matrix really does once you look along the right directions. In ML they find the directions in which data spreads the most, the idea behind PCA in the next lesson, and they explain why some training runs settle smoothly while others blow up.
What you'll be able to do
Tell an eigenvector from an ordinary vector: the matrix only stretches it, never turns it
Check Av = λv by hand and read the eigenvalue λ
Find and verify eigenvectors with np.linalg.eig, checking A @ v against λ * v for every pair
Find the main direction of a cloud of points as the top eigenvector of its covariance matrix, in NumPy and with torch.linalg.eigh
Quick check
Have a guess before you start. The matrix diag(3, 1) stretches x by 3 and leaves y alone. Which arrow keeps its direction?
(2, 1): it becomes (6, 1)
(1, 1): it becomes (3, 1)
(1, 0): it just becomes (3, 0)
None of them
Show the answer
(1, 0): it just becomes (3, 0)
(1, 0) becomes (3, 0), three times longer in the same direction. (1, 1) becomes (3, 1), which points somewhere new.
The math: arrows that stay on their line
Purpose. Find the natural axes of a matrix.
Multiply a matrix A by a vector v and usually the result points somewhere new. Sometimes it doesn't: Av lands on the same line as v, just scaled. Then v is an eigenvector of A and the scale factor λ (lambda) is its eigenvalue:
Av=λv
In plain words: the matrix does to v exactly what multiplying by one number would do.
Take A = [[2, 1], [1, 2]]. Then A(1, 1) = (3, 3) = 3 × (1, 1), so (1, 1) is an eigenvector with eigenvalue 3. And A(1, −1) = (1, −1), an eigenvector with eigenvalue 1. But A(1, 0) = (2, 1), not a multiple of (1, 0): an ordinary vector, turned.
Quick check
From lesson 55: what is [[4, 1], [2, 3]] times (1, −2)?
(6, −4)
(4, −6)
(2, −4)
(2, 8)
Show the answer
(2, −4)
Row 1: 4 − 2 = 2. Row 2: 2 − 6 = −4. And (2, −4) = 2 × (1, −2), so (1, −2) is an eigenvector with eigenvalue 2.
Check a candidate step by step: compute Av, then ask whether it's a multiple of v.
The picture. Under the formula, the arrows show the same check. The dashed line through the origin is v's line: every multiple of v lies on it. Each Step works out one entry of Av, and the picture marks where that puts Av's tip. If Av's arrow lands on the dashed line, v is an eigenvector.
Try it now
Then set vector v to 1, -1 (eigenvalue 1), and then to 1, 0: Av lands off the dashed line, so (1, 0) is turned, not an eigenvector.
Symbols and formula. An eigenvector must be nonzero (the zero vector is trivially "scaled"), and any multiple of an eigenvector is one too: if (1, 1) works, so do (2, 2) and (−5, −5). So an eigenvector is really a direction, a whole line through the origin.
What λ says. λ > 1 stretches along that line; between 0 and 1 shrinks; a negative λ flips the arrow to point the other way; λ = 0 squashes the line to the origin, which happens exactly when the determinant is 0 (lesson 59).
Symmetric matrices are the friendly case. A symmetric matrix (lesson 57) always has real eigenvalues and eigenvectors that are perpendicular: (1, 1) and (1, −1) above meet at a right angle. A quarter-turn rotation, by contrast, turns every arrow, so it has no real eigenvectors at all.
Triangular matrices show theirs. For a triangular matrix (lesson 57) the eigenvalues sit right on the diagonal: [[3, 1], [0, 2]] has eigenvalues 3 and 2. And (1, 0) is an eigenvector of it, since the matrix sends (1, 0) to (3, 0). For other matrices the eigenvalues are hidden, and finding them takes a calculation like the one NumPy does.
Quick check
What are the eigenvalues of the triangular matrix [[5, 2], [0, −1]]?
5 and −1
2 and −1
7 and −1
5 and 2
Show the answer
5 and −1
A triangular matrix's eigenvalues are its diagonal entries: 5 and −1.
Quick check
Is (0, 1) an eigenvector of [[4, 1], [2, 3]]?
Yes, with eigenvalue 3
Yes, with eigenvalue 1
No: A(0, 1) = (1, 3), off its line
Only if the matrix is symmetric
Show the answer
No: A(0, 1) = (1, 3), off its line
A multiple of (0, 1) has first entry 0, but (1, 3) has first entry 1: the arrow turned.
In your own words
In your own words: why is any multiple of an eigenvector also an eigenvector, with the same eigenvalue?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: np.linalg.eig
Purpose. Find every eigenvalue and eigenvector at once, and check them.
import numpy as np
A = np.array([[4, 1], [2, 3]])
lam, V = np.linalg.eig(A)
lam, V = lam.real, V.real # newer NumPy writes these as complex numbersprint(np.round(lam, 6).tolist())
for i inrange(len(lam)):
v = V[:, i]
print(np.allclose(A @ v, lam[i] * v))
np.linalg.eig returns two things: the eigenvalues, and a matrix whose columns are the eigenvectors, the first column going with the first eigenvalue. Newer versions of NumPy hand both back as complex numbers, written like (5+0j), because some matrices, such as a quarter turn, have eigenvalues that aren't real numbers. .real keeps the real part; for these matrices the other part is 0, so nothing else changes. NumPy scales each eigenvector to length 1 (lesson 49), so (1, 1) comes back as about (0.707, 0.707), possibly with its signs flipped. The loop checks each pair the honest way: does A @ v equal λ × v?
Code flow. Step through that loop one line at a time. Under Variables, V shows as a grid: watch v take one column of it, then Av appear beside it. Each pass prints the eigenvalue and whether the check passed.
Symmetric matrices: use eigh. For a symmetric matrix, np.linalg.eigh is faster and more accurate, and it returns the eigenvalues sorted from smallest to largest:
import numpy as np
S = np.array([[2, 1], [1, 2]])
lam, V = np.linalg.eigh(S)
print(np.round(lam, 6).tolist(), round(abs(V[:, 0] @ V[:, 1]), 6))
The second number is the dot product of the two eigenvectors: 0, so they're perpendicular, as promised.
A named mistake: reading rows instead of columns.V[0] is the first row of V: one entry from each eigenvector. The first eigenvector is the first column, V[:, 0]. With a symmetric matrix like S the slip can hide, because eigh often hands back a V whose rows equal its columns. With A = [[4, 1], [2, 3]] it can't, as the second check below shows.
Worked example
Worked example: verify every pair for two matrices
For two matrices, list the eigenvalues and confirm each eigenvector stays on its line.
import numpy as np
for A in [np.array([[2, 1], [1, 2]]), np.array([[3, 0], [0, 0.5]])]:
lam, V = np.linalg.eig(A)
lam, V = lam.real, V.real
for i inrange(len(lam)):
v = V[:, i]
ok = np.allclose(A @ v, lam[i] * v)
print(round(lam[i], 6), ok)
In the lesson you can change this and run it.
The first matrix has eigenvalues 3 and 1 (in whatever order NumPy finds them), along (1, 1) and (1, −1). The second is diagonal, so its eigenvectors are the axes, with eigenvalues 3 and 0.5, its diagonal entries. Every check prints True. The order of eigenvalues from eig isn't guaranteed, which is why the code pairs each one with its own column instead of assuming an order.
Quick check
What does this print?
import numpy as np
A = np.array([[2, 0], [0, 5]])
v = np.array([0, 1])
print((A @ v).tolist(), np.allclose(A @ v, 5 * v))
[0, 1] True
[0, 5] True
[2, 5] True
[0, 5] False
Show the answer
[0, 5] True
A(0, 1) = (0, 5) = 5 × (0, 1): an eigenvector with eigenvalue 5.
Quick check
What does this print?
import numpy as np
A = np.array([[4, 1], [2, 3]])
lam, V = np.linalg.eig(A)
lam, V = lam.real, V.real
print(np.allclose(A @ V[:, 0], lam[0] * V[:, 0]), np.allclose(A @ V[0], lam[0] * V[0]))
True True
False True
True False
False False
Show the answer
True False
V[:, 0] is the first column, the eigenvector that goes with lam[0], so its check passes. V[0] is the first row, one entry from each eigenvector: A turns it off its line, so its check fails.
Drill
Test a candidate
Fix eigen_value(A, v) so it returns the eigenvalue if v is an eigenvector of A and None if it isn't. Use the first nonzero entry of v to read λ, then check the whole vector with np.allclose.
import numpy as np
defeigen_value(A, v):
Av = A @ v
return Av[0] / v[0]
A = np.array([[4, 1], [2, 3]])
print(eigen_value(A, np.array([1, 1])), eigen_value(A, np.array([0, 1])))
In the lesson, your code is checked as you write it.
In your own words
Why check A @ v against λ * v instead of trusting the output of np.linalg.eig?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the direction data spreads most
Purpose. Find the main direction of a cloud of points.
The covariance matrix of some 2-D points is a 2 × 2 table of how much each coordinate spreads and how much they move together (lesson 94 builds it properly). It's symmetric, and its eigenvectors have a meaning: the top one, with the largest eigenvalue, is the direction along which the points spread the most.
Cv1=λ1v1,λ1=the largest⇒v1=the main direction
In plain words: the eigenvalue says how much spread there is along its eigenvector.
import numpy as np
P = np.array([[1, 1.2], [2, 1.4], [3, 2.1], [4, 2.3], [5, 2.4], [6, 3.3],
[7, 3.4], [8, 4.1], [9, 4.4], [10, 5.2], [11, 5.3], [12, 6.0]])
C = np.cov(P.T)
lam, V = np.linalg.eigh(C)
v = V[:, 1]
if v[0] < 0:
v = -v
print(np.round(lam, 3).tolist(), np.round(v, 3).tolist(), round(v[1] / v[0], 3))
np.cov wants one row per feature, so it gets P.T. The top eigenvalue, 15.501, dwarfs the other, 0.03: nearly all the spread runs along one direction, (0.916, 0.402), which climbs 0.439 for every 1 across, just like the points.
In PyTorch. The same calculation with tensors. torch.cov is NumPy's np.cov, also one row per feature, and torch.linalg.eigh is np.linalg.eigh: eigenvalues smallest first, eigenvectors as columns. torch.from_numpy keeps NumPy's float64 (lesson 47), so the digits match, and .round(decimals=3) rounds a whole tensor.
import numpy as np
import torch
P = torch.from_numpy(np.array([[1, 1.2], [2, 1.4], [3, 2.1], [4, 2.3], [5, 2.4], [6, 3.3],
[7, 3.4], [8, 4.1], [9, 4.4], [10, 5.2], [11, 5.3], [12, 6.0]]))
C = torch.cov(P.T)
lam, V = torch.linalg.eigh(C)
v = V[:, 1]
if v[0] < 0:
v = -v
print(lam.round(decimals=3).tolist(), v.round(decimals=3).tolist(), round((v[1] / v[0]).item(), 3))
It prints what the NumPy cell printed: eigenvalues 0.03 and 15.501, the main direction (0.916, 0.402), and its climb of 0.439 for every 1 across.
Why it matters. If almost all the spread lies along one direction, one number per point, its position along that direction, captures nearly everything. That's how high-dimensional data gets squeezed into a few numbers for plotting or for faster models: PCA, built from these eigenvectors, in lesson 63. Eigenvalues also show up in training: a step size that's too big for the largest eigenvalue of a loss's curvature makes gradient descent overshoot and blow up (lesson 83).
Drill
The main direction of a cloud
Set C to the covariance matrix of the points (use np.cov on P.T), top to the eigenvector with the largest eigenvalue, flipped if needed so its first entry is positive, and share to the largest eigenvalue's share of the total, rounded to 4 places.
import numpy as np
P = np.array([[0, 0.1], [2, 0.9], [4, 2.1], [6, 2.9], [8, 4.1], [10, 4.9]])
C = np.cov(P)
lam, V = np.linalg.eigh(C)
top = V[0]
share = 0print(C, top, share)
In the lesson, your code is checked as you write it.
In your own words
A cloud's two eigenvalues are 15.5 and 0.03. What does that say about the data?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write