Lesson 61 showed that a matrix of rank r is a sum of r column-times-row pieces. The singular value decomposition picks those pieces in the best possible way: in order of importance, each with a size, its singular value. Keep the biggest few and drop the rest and you have the best small summary of the whole table that any method can give. That one idea compresses pictures, removes noise, and, applied to a data set with its average taken away, becomes PCA, the standard way to squeeze many features into two or three numbers you can plot. Lesson 62's main direction of a cloud is exactly PCA's first piece.
What you'll be able to do
Read a matrix as a sum of rank-1 pieces, ordered by size: the singular value decomposition
Rebuild a matrix from its first k pieces and watch the error fall as k rises
Use np.linalg.svd and torch.linalg.svd, plot the sizes, and pick how many pieces to keep
Compress a picture and map 25-number signs down to 2 numbers with PCA
Quick check
Have a guess before you start. A 1,000 × 1,000 picture is stored as 20 column-times-row pieces instead of all its pixels. Roughly how many numbers is that?
About 1,000,000
About 2,000
About 40,000
About 20
Show the answer
About 40,000
Each piece is a column of 1,000 and a row of 1,000 (plus one size): 20 × 2,001 ≈ 40,000, against 1,000,000 pixels.
The math: a matrix as pieces, biggest first
Purpose. The best short summary of a table of numbers.
Every matrix can be written as a sum of rank-1 pieces, each a size times a column times a row:
A=s1u1v1T+s2u2v2T+⋯s1≥s2≥⋯≥0
In plain words: each piece is an outer product (lesson 61), scaled by its size. The sizes come biggest first.
The sizes s₁, s₂, … are the singular values. The columns u and the rows v each have length 1 and are perpendicular to one another, the orthonormal directions of lesson 52, so the pieces never overlap: each adds something the others lack.
Quick check
From lesson 61: what is the rank of a sum of two rank-1 pieces whose columns point in different directions?
2
0
1
4
Show the answer
2
Each piece adds one independent direction, so two different ones give rank 2.
The picture. Watch the arrow sign from lesson 53 rebuilt from its three pieces, sizes 2.449, 1.414 and 1. The picture shades each cell by its number, darker for bigger, so the table reads as the sign it stores. A cell with a thick outline is a pixel that would still round the wrong way at 0.5: on the empty start, the outlines trace the whole arrow. Watch how many outlines each piece clears.
Try it now
The sizes are the arrow's singular values. A photo has many pieces, and most of them are tiny.
Symbols and formula. Stacking the columns u into U, the sizes into a diagonal Σ (sigma) and the rows v into Vᵀ gives the singular value decomposition:
A=UΣVT
In plain words: U holds the columns, Σ the sizes on its diagonal, and Vᵀ the rows. Multiplying them out adds up all the pieces.
Keeping the first k. Drop every piece after the k-th and you get A_k, a matrix of rank k. A famous result says no other rank-k matrix is closer to A. The error left behind is set by the sizes you dropped: square them, add, and take the root. Drop only tiny pieces and you lose almost nothing. For the arrow, keeping two pieces drops only the size 1, so the error is 1. Keeping one drops 1.414 and 1, whose squares are 2 (1.414 is √2) and 1, so the error is √3 = 1.732.
Quick check
A matrix has singular values 10, 4, 0.1 and 0.05. How many pieces would you keep for a close copy?
1: only the biggest matters
2: the last two are tiny next to 10 and 4
0: the sizes are too different
4: every piece matters equally
Show the answer
2: the last two are tiny next to 10 and 4
10 and 4 carry almost everything; 0.1 and 0.05 add very little. Two pieces give a close, rank-2 copy.
In your own words
In your own words: why is keeping the biggest pieces the best way to shrink a matrix?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: np.linalg.svd
Purpose. Get every piece at once, then rebuild from the first k.
It runs right here in your browser, with nothing to install.
np.linalg.svd returns three things: U, whose columns are the u's; s, the sizes, biggest first; and Vt, whose rows are the v's. Piece k is s[k] * np.outer(U[:, k], Vt[k]). np.linalg.norm of the difference measures the error, here 3.
Code flow. Step through the rebuild, one piece per pass of the loop. To keep every number exact, the code types in the two pieces the SVD found: sizes 6 and 3, columns (1, 0) and (0, 1), and rows (2, 1, 2) ÷ 3 and (2, −2, −1) ÷ 3. (Your computer may hand back the first column and row with both signs flipped. Flipping both leaves the piece the same.) Watch Ak gain row 1 on the first pass and row 2 on the second.
The first k pieces in one line. Slices keep the first k columns of U, sizes and rows of Vt:
import numpy as np
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
U, s, Vt = np.linalg.svd(arrow)
print(np.round(s, 3).tolist())
for k in [1, 2, 3]:
Ak = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
print(k, round(np.linalg.norm(arrow - Ak), 3))
The arrow sign's sizes are 2.449, 1.414, 1 and then zeros: its rank is 3. The error falls as k rises and hits 0 at k = 3. A named mistake: Vt is already transposed. Its rows are the v's, so the k-th is Vt[k], not Vt[:, k].
The shapes of the three pieces. For an m × n matrix, U is m × m, s holds as many sizes as the smaller of m and n, and Vt is n × n. Only the first few columns of U and rows of Vt matter when you keep only a few pieces, which is why rebuilding uses the slices U[:, :k] and Vt[:k]. A 1,000 × 1,000 picture kept at k = 20 needs just the first 20 columns of U, 20 sizes, and the first 20 rows of Vt.
Quick check
What does this print?
import numpy as np
U, s, Vt = np.linalg.svd(np.array([[2, 0], [0, 1], [0, 0]]))
print(np.round(s, 6).tolist(), U.shape, Vt.shape)
[2.0, 1.0] (3, 2) (2, 3)
[2.0, 1.0] (3, 3) (2, 2)
[1.0, 2.0] (3, 3) (2, 2)
[2.0, 1.0, 0.0] (3, 3) (2, 2)
Show the answer
[2.0, 1.0] (3, 3) (2, 2)
A 3 × 2 matrix has 2 sizes (the smaller of 3 and 2), biggest first. U is 3 × 3 and Vt is 2 × 2.
Worked example
Worked example: rank 1, rank 2 and full rebuilds
Rebuild the arrow from 1, 2 and 5 pieces, and test whether each is still readable: round each pixel to 0 or 1 at 0.5 and compare with the original.
import numpy as np
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
U, s, Vt = np.linalg.svd(arrow)
for k in [1, 2, 5]:
Ak = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
readable = np.array_equal(Ak > 0.5, arrow == 1)
print(k, round(np.linalg.norm(arrow - Ak), 3), readable)
In the lesson you can change this and run it.
One piece gives a blur, error 1.732, and some pixels land on the wrong side of 0.5. Two pieces are already readable: every pixel rounds the right way, though the error is still 1. Five pieces rebuild the sign exactly. Ak > 0.5 gives True wherever a pixel is inked after rounding, and arrow == 1 does the same for the original, so np.array_equal compares the two pictures pixel by pixel.
Quick check
What does this print?
import numpy as np
A = np.array([[3, 0], [0, 1]])
U, s, Vt = np.linalg.svd(A)
print(np.round(s, 6).tolist(), A.shape)
[1.0, 3.0] (2, 2)
[3.0, 1.0] (2, 2)
[4.0] (2, 2)
[3.0, 0.0] (2, 2)
Show the answer
[3.0, 1.0] (2, 2)
For a diagonal matrix with positive entries the sizes are its diagonal, biggest first: 3 then 1.
Plot the sizes. Drawing the sizes in order shows at a glance how many pieces matter. Here are the arrow's:
import numpy as np
import matplotlib.pyplot as plt
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
U, s, Vt = np.linalg.svd(arrow)
plt.plot([1, 2, 3, 4, 5], s)
plt.xlabel("piece k")
plt.ylabel("size of piece k")
plt.show()
A photo's plot has the same shape over many more pieces: a steep drop, then a long, low tail of pieces you can drop.
Quick check
Read the plot. Where does the line first reach 0, and what does that say about the arrow?
At piece 5: all five pieces matter, so its rank is 5
Nowhere: the sizes only shrink, so its rank is 5
At piece 4: only three pieces have any size, so its rank is 3
At piece 3: only two pieces have any size, so its rank is 2
Show the answer
At piece 4: only three pieces have any size, so its rank is 3
The sizes are 2.449, 1.414 and 1, then 0 and 0. The line reaches 0 at piece 4, so three pieces carry the whole arrow: its rank is 3, and three pieces rebuild it exactly.
Drill
Rebuild from k pieces
Fix rebuild(A, k) so it returns the sum of the first k pieces. Then set errs to the rebuild error for k = 1, 2 and 3 on the cross sign, each rounded to 3 places. .item() turns NumPy's number into a plain Python one, so the list prints as plain numbers.
In the lesson, your code is checked as you write it.
In your own words
Why can a 5 × 5 sign need only 2 or 3 pieces, while a photo needs many?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: compression and maps of data
Purpose. Squeeze many numbers into a few while keeping what matters.
The pieces do two jobs in machine learning. They compress a matrix, whether it holds a picture or a layer's weights, and they map data with many features down to a few numbers you can plot.
Compression. The smallest k that keeps a sign readable depends on the sign: the box needs 1 piece, the arrow 2, and the cross 3. A photo works the same way with bigger numbers: a few dozen pieces often look almost like the original. Weight matrices too: when a layer's sizes fall fast, a few pieces can stand in for the whole layer, the idea behind lesson 61's low-rank updates.
In PyTorch. You met torch.linalg.svd in lesson 52, where the rows of its Vh passed the orthonormal test. It hands back the same three things as NumPy, named U, S and Vh (Vt by another name). Here is the arrow kept at two pieces, in the 64-bit decimals NumPy uses:
The sizes are the ones NumPy printed, 2.449, 1.414, 1 and two zeros, and two pieces leave the same error, 1.0. torch.equal asks whether two tensors match everywhere: True, so two pieces keep the arrow readable.
PCA: principal component analysis. Take a data matrix X (one row per example), subtract the average row from every row so the cloud is centered on the origin, and take the SVD. The rows of Vt are the principal directions, the first being lesson 62's main direction of spread. Each example's position along the first two gives it two numbers, a point you can plot:
coordinates=XcV2V2=the first two principal directions, as columns
Here are nine signs, three of each type, some with one pixel flipped, each 25 numbers long, mapped to 2 numbers:
import numpy as np
cross = np.array([[1,0,0,0,1],[0,1,0,1,0],[0,0,1,0,0],[0,1,0,1,0],[1,0,0,0,1]])
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
box = np.array([[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,1]])
rows = []
for S in [cross, arrow, box]:
rows.append(S.flatten())
for spot in [(0, 0), (2, 2)]:
T = S.copy()
T[spot[0], spot[1]] = 1 - T[spot[0], spot[1]]
rows.append(T.flatten())
X = np.array(rows, dtype=float)
mean = np.zeros(25)
for i inrange(9):
mean = mean + X[i] / 9
Xc = X.copy()
for i inrange(9):
Xc[i] = X[i] - mean
U, s, Vt = np.linalg.svd(Xc)
coords = Xc @ Vt[:2].T
print(np.round(coords, 2).tolist())
share = (s[0] ** 2 + s[1] ** 2) / (s @ s)
print(np.round(s[:4], 3).tolist(), round(share, 3))
dtype=float makes the array hold decimals, so the centered values aren't cut to whole numbers. The nine rows land in three tight groups of three, one group per sign type, though each sign started as 25 numbers. (Which way each axis points can flip from one computer to another; the groups don't change.) A size squared measures the spread along its direction, so share is the part of all the spread the first two directions keep: 0.915.
Quick check
Why is the average row subtracted from every row before the SVD in PCA?
To remove the redundant features first
To make every number positive
To make the matrix square
So directions measure spread around the center
Show the answer
So directions measure spread around the center
Without centering, the first direction would mostly point at the average itself. Centering puts the cloud at the origin, so the directions measure how examples differ.
Drill
The smallest readable k
Write smallest_k(S) returning the smallest k for which the rebuild from k pieces, rounded at 0.5, matches the sign exactly. Then set ks to its answers for the box, the arrow and the cross, in that order.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write