Independence, span and rank
Why this matters
A table can hold many columns and still say very little. If one column is just another one doubled, or the sum of two others, it adds no new information: anything it could tell you, the others already do. Rank counts how many columns really carry information. It explains when a system has one answer (lesson 60), when features are redundant (lesson 59), and why huge weight matrices can often be stored as two thin ones. That last idea, the low-rank update, is how modern language models are fine-tuned cheaply, with a method called LoRA that trains a tiny fraction of the numbers a full update would need.
- Say when a vector is redundant: it can be built as a mix of the others
- Tell independent vectors from dependent ones, and describe their span
- Find a matrix's rank, its number of independent columns, by eye, with np.linalg.matrix_rank and with torch.linalg.matrix_rank
- Build a low-rank weight update W + B @ A and count the numbers it saves, the idea behind LoRA
The math: how much real information?
Purpose. Count how much real information a set of vectors holds.
A vector is redundant in a set if it can be built from the others, as a mix c₁v₁ + c₂v₂ + ⋯ (lesson 48). With v₁ = (1, 0, 1), v₂ = (0, 1, 1) and v₃ = (1, 1, 2), the third is redundant: v₃ = v₁ + v₂.
Vectors are independent when none of them can be built from the others. (1, 0) and (0, 1) are independent; so are (2, 1) and (1, 2). But (2, 1) and (4, 2) aren't: the second is the first doubled.
In plain words: independent means every vector brings a direction the others can't reach.