Index Ai Academy
Foundation · Foundation mathVectors and matrices

SVD and PCA: the important directions first

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 61 showed that a matrix of rank r is a sum of r column-times-row pieces. The singular value decomposition picks those pieces in the best possible way: in order of importance, each with a size, its singular value. Keep the biggest few and drop the rest and you have the best small summary of the whole table that any method can give. That one idea compresses pictures, removes noise, and, applied to a data set with its average taken away, becomes PCA, the standard way to squeeze many features into two or three numbers you can plot. Lesson 62's main direction of a cloud is exactly PCA's first piece.

  • Read a matrix as a sum of rank-1 pieces, ordered by size: the singular value decomposition
  • Rebuild a matrix from its first k pieces and watch the error fall as k rises
  • Use np.linalg.svd and torch.linalg.svd, plot the sizes, and pick how many pieces to keep
  • Compress a picture and map 25-number signs down to 2 numbers with PCA
Quick check
Have a guess before you start. A 1,000 × 1,000 picture is stored as 20 column-times-row pieces instead of all its pixels. Roughly how many numbers is that?

The math: a matrix as pieces, biggest first

Purpose. The best short summary of a table of numbers.

Every matrix can be written as a sum of rank-1 pieces, each a size times a column times a row:

A=s1u1v1T+s2u2v2T+⋯s1≥s2≥⋯≥0A = s_1 \mathbf{u}_1 \mathbf{v}_1^{\mathsf{T}} + s_2 \mathbf{u}_2 \mathbf{v}_2^{\mathsf{T}} + \cdots \qquad s_1 \ge s_2 \ge \cdots \ge 0

In plain words: each piece is an outer product (lesson 61), scaled by its size. The sizes come biggest first.

The sizes s₁, s₂, … are the singular values. The columns u and the rows v each have length 1 and are perpendicular to one another, the orthonormal directions of lesson 52, so the pieces never overlap: each adds something the others lack.