A table can hold many columns and still say very little. If one column is just another one doubled, or the sum of two others, it adds no new information: anything it could tell you, the others already do. Rank counts how many columns really carry information. It explains when a system has one answer (lesson 60), when features are redundant (lesson 59), and why huge weight matrices can often be stored as two thin ones. That last idea, the low-rank update, is how modern language models are fine-tuned cheaply, with a method called LoRA that trains a tiny fraction of the numbers a full update would need.
What you'll be able to do
Say when a vector is redundant: it can be built as a mix of the others
Tell independent vectors from dependent ones, and describe their span
Find a matrix's rank, its number of independent columns, by eye, with np.linalg.matrix_rank and with torch.linalg.matrix_rank
Build a low-rank weight update W + B @ A and count the numbers it saves, the idea behind LoRA
Quick check
Have a guess before you start. A table has three columns: price in dollars, the same price in cents, and weight in grams. How many columns carry separate information?
2
3
0
1
Show the answer
2
Cents are just dollars × 100, so they add nothing new. Dollars and grams are the two real pieces of information.
The math: how much real information?
Purpose. Count how much real information a set of vectors holds.
A vector is redundant in a set if it can be built from the others, as a mix c₁v₁ + c₂v₂ + ⋯ (lesson 48). With v₁ = (1, 0, 1), v₂ = (0, 1, 1) and v₃ = (1, 1, 2), the third is redundant: v₃ = v₁ + v₂.
Vectors are independent when none of them can be built from the others. (1, 0) and (0, 1) are independent; so are (2, 1) and (1, 2). But (2, 1) and (4, 2) aren't: the second is the first doubled.
In plain words: independent means every vector brings a direction the others can't reach.
Quick check
From lesson 48: what is 2 × (1, 0, 1) + 3 × (0, 1, 1)?
(2, 3, 5)
(2, 3, 2)
(3, 2, 5)
(1, 1, 2)
Show the answer
(2, 3, 5)
(2, 0, 2) + (0, 3, 3) = (2, 3, 5). Every mix of these two vectors lies in one flat plane: their span.
The span. Every mix of a set of vectors is called their span. One nonzero vector spans a line. Two independent vectors span a plane. Add a redundant third and the span doesn't grow, since it was already inside the plane. Add an independent third and you reach all of 3-D space.
The picture. Here are u = (2, 1) and w = (1, 2), the independent pair from above, with their mixes drawn as dots. Press Next to see the mixes, then to swap w for (4, 2), u doubled, and watch what happens to the dots.
Try it now
The large dot is u + w both times: w's arrow laid from u's tip, dashed.
Rank. The rank of a matrix is the number of independent columns it has. A column that's a mix of the others doesn't count. A neat fact: the number of independent rows is always the same, so rank says how much real information the matrix holds either way you read it.
A rank-1 matrix. The thinnest matrix there is: one column times one row. Every column is a multiple of the same column, so only one is independent. Watch (1, 2) times the row (3, 4) build a 2 × 2 matrix. In the picture each cell is shaded by its value, darker for bigger, so you can see each row's pattern: the second row is the first one doubled, the same pattern in deeper shades.
Symbols and formula. The outer product, and what rank r saves:
uvT has rank 1rank(A)=r⇒A=u1v1T+⋯+urvrT
In plain words: a matrix of rank r can be built from r column-times-row pieces, and no fewer.
Rank from elimination. Run elimination (lesson 60) on a matrix and count the rows that survive with a pivot. A redundant row becomes a row of zeros. [[1, 2, 3], [2, 4, 6], [1, 0, 1]] has rank 2: row 2 is twice row 1, and R₂ − 2R₁ wipes it out.
Quick check
What is the rank of [[1, 2], [2, 4]]?
2
1
0
4
Show the answer
1
Row 2 is twice row 1 (and column 2 is twice column 1): one independent direction.
In your own words
In your own words: why does a redundant column not raise a matrix's rank?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: np.linalg.matrix_rank
Purpose. Count independent columns in one call, and build low-rank matrices.
import numpy as np
A = np.array([[1, 2, 3], [2, 4, 6], [1, 0, 1]])
print(np.linalg.matrix_rank(A))
print(np.linalg.matrix_rank(np.eye(3)), np.linalg.matrix_rank(np.zeros((2, 2))))
It runs right here in your browser, with nothing to install.
matrix_rank counts independent columns, allowing for tiny float differences: two columns that are copies up to float noise still count as one.
Code flow. Here are three matrices from this lesson, ranked in one loop: the matrix with a doubled row, the identity and an outer product. Step through and watch M take each matrix in turn, shown as a grid under Variables, and r take its rank. r is NumPy's own kind of whole number, so the stepper shows it wrapped in its type, while print writes just the number.
Adding a redundant column keeps the rank. Build a third column as a mix of the first two and the rank stays put:
np.array([c1, c2, c3]) stacks them as rows, so .T turns them into columns (lesson 54). A named mistake: reading rank as "the number of columns". This matrix has 3 columns but rank 2.
The outer product in code.np.outer(u, v) builds the column-times-row matrix, each cell uᵢ × vⱼ.
Worked example
Worked example: rebuild a matrix from two thin pieces
Make a rank-2 matrix from two outer products, and confirm its rank and its storage cost.
import numpy as np
u1 = np.array([1, 0, 1])
v1 = np.array([1, 2, 1])
u2 = np.array([0, 1, 1])
v2 = np.array([1, 0, 2])
A = np.outer(u1, v1) + np.outer(u2, v2)
print(A.tolist())
print(np.linalg.matrix_rank(A))
print(4 * 3, "numbers in the pieces;", 3 * 3, "in A")
In the lesson you can change this and run it.
Each outer product is rank 1, and their sum has rank 2. Here the pieces take 12 numbers to store and A only 9, so there's no saving: the thin pieces only pay off when the matrix is big and the rank is small. A 25 × 25 matrix of rank 2 needs 100 numbers as pieces against 625 in full.
Quick check
What does this print?
import numpy as np
A = np.outer(np.array([1, 2, 3]), np.array([1, -1]))
print(A.shape, np.linalg.matrix_rank(A), A[2].tolist())
(3, 2) 2 [3, -3]
(2, 3) 1 [3, -3]
(3, 2) 1 [3, -3]
(3, 2) 1 [1, -1]
Show the answer
(3, 2) 1 [3, -3]
A column of 3 times a row of 2 is 3 × 2, rank 1. Row index 2 is 3 × (1, −1) = (3, −3).
Quick check
What does this print?
import numpy as np
A = np.array([[1, 0, 1], [0, 1, 1], [1, 1, 2]])
B = np.outer(np.array([1, 2, 3]), np.array([1, 1]))
print(np.linalg.matrix_rank(A), np.linalg.matrix_rank(B))
3 2
3 1
2 1
2 2
Show the answer
2 1
A's third column is the sum of the first two, so its rank is 2. Every outer product has rank 1.
Drill
Spot the redundant column
Fix the lines so r is the rank of A, c3 is a column built as 3 × c1 + c2 (so B's rank stays 2), and thin is the outer product of u and v.
import numpy as np
A = np.array([[1, 2, 0], [2, 4, 1], [3, 6, 1]])
r = A.shape[1]
c1 = np.array([1, 0, 2])
c2 = np.array([0, 1, 1])
c3 = np.array([1, 1, 1])
B = np.array([c1, c2, c3]).T
u = np.array([2, 1])
v = np.array([1, 3, 5])
thin = u * v
print(r, np.linalg.matrix_rank(B), thin)
In the lesson, your code is checked as you write it.
In your own words
Why does matrix_rank allow for tiny float differences instead of testing exact equality?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: redundant features, and cheap updates
Purpose. Redundant features add nothing, and big weight changes can be stored cheaply when their rank is low.
Redundant features. A data matrix X whose rank is less than its number of columns has redundant features: some column can be rebuilt from the others, like the price in cents. They add storage and, as lesson 59 showed, leave the best weights undecided.
Low-rank updates. Fine-tuning a language model changes its weight matrices, each maybe 4,096 × 4,096: almost 17 million numbers per matrix. Researchers found that the useful change usually has low rank. So LoRA (low-rank adaptation) stores the change as two thin pieces, B (d × r) and A (r × d), and trains only those:
ΔW=BAnumbers: 2dr instead of d2
In plain words: instead of learning a full d × d change, learn r columns and r rows and multiply them.
for d, r in [(25, 2), (1000, 8), (4096, 16)]:
print(d, r, 2 * d * r, d * d, round(2 * d * r / (d * d), 4))
At d = 4,096 and r = 16, the pieces need 131,072 numbers instead of 16,777,216: under 1% of the full update.
Quick check
In LoRA, the change to a d × d weight matrix is stored as B (d × r) times A (r × d). What is the largest rank that change can have?
d
d × r
2r
r
Show the answer
r
B A is a sum of r column-times-row pieces, so its rank is at most r, however big d is.
One small update, in numbers. Here is LoRA at toy size, with d = 4 and r = 1: a 4 × 4 weight matrix W, a column B (4 × 1) and a row A (1 × 4). With r = 1, B @ A is a column times a row: the outer product from the math part.
import numpy as np
W = np.array([[2, 0, 1, 0], [0, 1, 0, 2], [1, 0, 3, 0], [0, 2, 0, 1]])
B = np.array([[1.0], [2.0], [-1.0], [0.5]])
A = np.array([[2.0, -1.0, 0.5, 1.0]])
W_new = W + B @ A
print(W_new.tolist())
print(np.linalg.matrix_rank(B @ A), B.size + A.size, W.size)
.size counts an array's numbers. The change B @ A has rank 1 and is stored in 8 numbers, yet it moves all 16 of W's entries. Here are the three tables.
In PyTorch. The same update with tensors. torch.linalg.matrix_rank counts independent columns like NumPy's, numel() counts a tensor's numbers like .size (you met it in lesson 12), and .item() takes the rank out of its one-number tensor.
import torch
W = torch.tensor([[2, 0, 1, 0], [0, 1, 0, 2], [1, 0, 3, 0], [0, 2, 0, 1]])
B = torch.tensor([[1.0], [2.0], [-1.0], [0.5]])
A = torch.tensor([[2.0, -1.0, 0.5, 1.0]])
W_new = W + B @ A
print(W_new.tolist())
print(torch.linalg.matrix_rank(B @ A).item(), B.numel() + A.numel(), W.numel())
It prints what the NumPy cell printed: the same table, rank 1, and 8 numbers against 16. In a real fine-tune W is frozen and only B and A are trained, and at d = 4,096 with r = 16 the pieces are the 131,072 numbers from the loop above.
Why low rank works. A full update can point anywhere among millions of directions, but fine-tuning on one task needs only a few of them. Limiting the change to rank r keeps it to the most important directions, trains far fewer numbers, and lets many small task-specific updates share one big base model. Lesson 63 shows how to find a matrix's most important rank-1 pieces in order.
Drill
Numbers saved by a low-rank update
Write lora_count(d, r) returning the numbers stored in a rank-r update of a d × d matrix (two thin pieces), and fraction(d, r) returning that count divided by the full d × d. Then set big to lora_count(2048, 4).
deflora_count(d, r):
return d * r
deffraction(d, r):
return d * d / lora_count(d, r)
big = 0print(lora_count(25, 2), fraction(25, 2), big)
In the lesson, your code is checked as you write it.
In your own words
Why might a rank-1 or rank-2 update be enough to adapt a model to one narrow task?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
PCA and SVD: the most important directions of a matrix.
LoRA: fine-tuning real models with low-rank updates.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write