SVD and PCA: the important directions first
Why this matters
Lesson 61 showed that a matrix of rank r is a sum of r column-times-row pieces. The singular value decomposition picks those pieces in the best possible way: in order of importance, each with a size, its singular value. Keep the biggest few and drop the rest and you have the best small summary of the whole table that any method can give. That one idea compresses pictures, removes noise, and, applied to a data set with its average taken away, becomes PCA, the standard way to squeeze many features into two or three numbers you can plot. Lesson 62's main direction of a cloud is exactly PCA's first piece.
- Read a matrix as a sum of rank-1 pieces, ordered by size: the singular value decomposition
- Rebuild a matrix from its first k pieces and watch the error fall as k rises
- Use np.linalg.svd and torch.linalg.svd, plot the sizes, and pick how many pieces to keep
- Compress a picture and map 25-number signs down to 2 numbers with PCA
The math: a matrix as pieces, biggest first
Purpose. The best short summary of a table of numbers.
Every matrix can be written as a sum of rank-1 pieces, each a size times a column times a row:
In plain words: each piece is an outer product (lesson 61), scaled by its size. The sizes come biggest first.
The sizes s₁, s₂, … are the singular values. The columns u and the rows v each have length 1 and are perpendicular to one another, the orthonormal directions of lesson 52, so the pieces never overlap: each adds something the others lack.