Most matrices are just tables of numbers, but a few have patterns that make them behave simply, and they come up so often that they have names. The identity changes nothing, like multiplying by 1. A diagonal matrix stretches each coordinate by its own amount. A triangular one has zeros on one side of its diagonal, and a symmetric one is its own mirror image. Each has a job in ML: rescaling features, storing distances, and the triangular mask inside every model that writes text one word at a time, which is what stops it peeking at words it hasn't written yet. This lesson teaches you to spot each pattern, build it in NumPy, and say what it does.
What you'll be able to do
Recognize the identity, diagonal, triangular and symmetric matrices by their pattern of entries
Predict what multiplying by I or by a diagonal matrix does, without working every entry
Build each kind with np.eye, np.diag and np.tril, and test symmetry with np.allclose(A, A.T)
Build the causal mask that stops a language model looking at words that come later, in NumPy and with PyTorch's torch.tril and masked_fill
Quick check
Have a guess before you start. Which number leaves any number unchanged when you multiply by it?
0
−1
1
10
Show the answer
1
Multiplying by 1 changes nothing. Matrices have a 1 of their own, the identity matrix.
The math: four patterns worth naming
Purpose. A few matrices with simple patterns, and what each one does.
The main diagonal of a square matrix runs from the top left to the bottom right: a₁₁, a₂₂, a₃₃. The patterns are all about it.
Identity, I: 1s on the diagonal, 0s everywhere else. AI = A and IA = A: it changes nothing.
Diagonal: anything on the diagonal, 0s off it. Multiplying a vector by it scales each coordinate by its own diagonal entry.
I=100010001D=20000.50003
In plain words: D times (4, 6, 1) is (2 × 4, 0.5 × 6, 3 × 1) = (8, 3, 3). Each coordinate is scaled on its own, and none of them mix.
Quick check
From lesson 56: what is entry (1, 1) of [[2, 7], [0, 3]] times [[1, 0], [0, 1]]?
2
7
1
3
Show the answer
2
Row 1, (2, 7), dotted with column 1, (1, 0): 2. Multiplying by this matrix, the identity, gives back the same matrix.
Triangular: zeros on one side of the diagonal. Lower triangular has zeros above it; upper has zeros below it.
Symmetric: equal to its own transpose, A = Aᵀ, so aᵢⱼ = aⱼᵢ: the entries mirror across the diagonal.
The picture. Here are the four patterns as pictures, one per frame. Each cell is shaded by its number, and the cells that make the pattern are outlined: the diagonal of I and of D, the places above the diagonal in the triangular one, and a mirrored pair in the symmetric one.
Try it now
One lower-triangular matrix has a famous job, and the AI part comes back to it: the causal mask of a model that writes text one word at a time. It has one row per word, down the left, and one column per word that word might look at, numbered along the top. A 1 means "may look" and a 0 means "may not", and a word may look only at itself and the words before it. So the mask is lower triangular: 1s up to the diagonal, 0s after it. The flow builds it row by row, and each step fills the next word's row: dark 1s for the words it may see, light 0s for the words after it. The PyTorch tab builds the same mask with torch.tril.
Symbols and formula. The four patterns in symbols:
Iij={10i=ji=jDij=0 for i=jLij=0 for j>iA=AT
In plain words: the identity is 1 on the diagonal and 0 off it; a diagonal matrix is 0 off the diagonal; a lower-triangular one is 0 above it; a symmetric one equals its transpose.
A free symmetric matrix. For any matrix A, the product AᵀA is symmetric. With A = [[1, 2], [3, 4], [5, 6]], AᵀA = [[35, 44], [44, 56]]: the two off-diagonal entries match. Entry (i, j) is column i of A dotted with column j, and swapping the two columns gives the same dot product.
Quick check
Which matrix is symmetric?
[[1, 0, 0], [4, 2, 0], [5, 3, 6]]
[[2, 7, 1], [7, 3, 4], [1, 4, 5]]
[[1, 2], [3, 4]]
[[0, 1, 0], [1, 0, 0], [0, 0, 2], [1, 1, 1]]
Show the answer
[[2, 7, 1], [7, 3, 4], [1, 4, 5]]
Its entries mirror across the diagonal: a₁₂ = a₂₁ = 7, a₁₃ = a₃₁ = 1 and a₂₃ = a₃₂ = 4.
Which side the diagonal matrix goes on. D A scales each row of A by its diagonal entry, and A D scales each column. With D = diag(3, 1) and A = [[1, 2], [3, 4]], DA = [[3, 6], [3, 4]] triples row 1, while AD = [[3, 2], [9, 4]] triples column 1. That's lesson 56's order rule again, now with a picture of what each order does.
In your own words
In your own words: why does multiplying by the identity change nothing?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a builder for each
Purpose. Make each special matrix in one call, and test for symmetry.
import numpy as np
print(np.eye(3).tolist())
print(np.diag([2, 0.5, 3]).tolist())
print(np.tril(np.ones((3, 3))).tolist())
It runs right here in your browser, with nothing to install.
np.eye(n) is the n × n identity (say it as "I"). np.diag puts a list on the diagonal of a matrix of zeros. np.tril keeps the lower triangle of any matrix and zeros the rest; np.ones((n, n)) is a matrix of 1s to start from. np.triu does the same for the upper triangle.
Code flow. A builder for each pattern, one line at a time. Under Variables each matrix shows as a grid of boxes, so you can check its pattern at a glance. The last line makes a symmetric matrix the free way from the math part, AᵀA, with the triangle low as A.
Testing symmetry. A matrix is symmetric when it equals its transpose. np.allclose(A, A.T) checks that, allowing for tiny float differences:
import numpy as np
S = np.array([[2, 7, 1], [7, 3, 4], [1, 4, 5]])
L = np.array([[1, 0, 0], [4, 2, 0], [5, 3, 6]])
A = np.array([[1, 2], [3, 4], [5, 6]])
print(S.T.tolist(), np.allclose(S, S.T))
print(L.T.tolist(), np.allclose(L, L.T))
print((A.T @ A).tolist(), np.allclose(A.T @ A, (A.T @ A).T))
S.T prints the same rows as S, so S is symmetric. L's transpose is upper triangular, a different matrix, so L isn't. And AᵀA passes, as the math part promised.
A named mistake: np.ones for the identity.np.ones((3, 3)) is all 1s, not the identity. Multiplying by it adds up rows or columns instead of leaving the matrix alone. The identity is np.eye(3).
Worked example
Worked example: check each pattern in code
Write one test per pattern and run them on three matrices.
import numpy as np
defis_diagonal(A):
return np.allclose(A, np.diag(np.diag(A)))
defis_lower(A):
return np.allclose(A, np.tril(A))
defis_symmetric(A):
return A.shape[0] == A.shape[1] and np.allclose(A, A.T)
L = np.array([[1, 0, 0], [4, 2, 0], [5, 3, 6]])
S = np.array([[2, 7, 1], [7, 3, 4], [1, 4, 5]])
D = np.diag([2, 0.5, 3])
for M in [L, S, D]:
print(is_diagonal(M), is_lower(M), is_symmetric(M))
In the lesson you can change this and run it.
Each test rebuilds the pattern from the matrix and compares. np.diag used on a matrix reads its diagonal, and used again rebuilds a diagonal matrix from it; if that equals A, A was diagonal. np.tril(A) zeros A's upper part; if nothing changed, A was lower triangular. L passes only the lower test, S only the symmetric one, and D passes all three: a diagonal matrix is triangular and symmetric too.
Quick check
What does this print?
import numpy as np
A = np.array([[1, 2], [3, 4]])
print((A @ np.eye(2)).tolist(), (A @ np.ones((2, 2))).tolist())
[[1.0, 2.0], [3.0, 4.0]] [[1.0, 2.0], [3.0, 4.0]]
[[1.0, 0.0], [0.0, 4.0]] [[3.0, 3.0], [7.0, 7.0]]
[[1.0, 2.0], [3.0, 4.0]] [[3.0, 3.0], [7.0, 7.0]]
[[3.0, 3.0], [7.0, 7.0]] [[1.0, 2.0], [3.0, 4.0]]
Show the answer
[[1.0, 2.0], [3.0, 4.0]] [[3.0, 3.0], [7.0, 7.0]]
The identity gives A back (as decimals, since np.eye makes floats). The matrix of 1s adds up each row: 1 + 2 = 3 and 3 + 4 = 7.
Quick check
Which line builds the 4 × 4 identity?
np.eye(4)
np.tril(4)
np.diag(4)
np.ones((4, 4))
Show the answer
np.eye(4)
np.eye(n) gives 1s on the diagonal and 0s elsewhere.
Drill
Build each kind
Fix the four parts: I4 is the 4 × 4 identity, D the diagonal matrix with 5, 2 and 0.1 on its diagonal, low the lower triangle of A, and is_sym(M) returns whether a matrix M is symmetric. The starter's is_sym makes a named mistake: it calls every square matrix symmetric.
import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
I4 = np.ones((4, 4))
D = np.array([5, 2, 0.1])
low = A
defis_sym(M):
return M.shape[0] == M.shape[1]
print(I4, D, low, is_sym(A), is_sym(A.T @ A))
In the lesson, your code is checked as you write it.
In your own words
Why is a diagonal matrix both lower triangular and symmetric?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: each pattern has a job
Purpose. Rescale features, store distances, and stop a model looking ahead.
Diagonal: rescaling features. A day described as (temperature, takings in cents) has wildly different scales (lesson 41). A diagonal matrix rescales each feature on its own: diag(0.1, 0.01) times (25, 4800) is (2.5, 48).
Symmetric: tables of distances. The distance from camp A to camp B equals the distance from B to A, so a table of all the distances is symmetric, with 0s on its diagonal.
Lower triangular: the causal mask. A language model writes one word at a time, so while it works on word i it may use words 1 to i, never later ones. Its attention scores (one per pair of words) are multiplied by the mask:
Everything above the diagonal, word i looking at a later word, is wiped out. (Real models put −∞ there instead of 0, for a reason lesson 104's softmax will make clear, but the triangle is the same.)
Quick check
Here * multiplies cell by cell, which is exactly what a mask needs. What does this print?
import numpy as np
s = np.array([[5, 9], [7, 1]])
m = np.tril(np.ones((2, 2)))
print((s * m).tolist())
[[5.0, 9.0], [7.0, 1.0]]
[[5.0, 9.0], [0.0, 1.0]]
[[5.0, 0.0], [7.0, 1.0]]
[[68.0, 9.0], [36.0, 1.0]]
Show the answer
[[5.0, 0.0], [7.0, 1.0]]
The mask keeps the lower triangle, so the 9 above the diagonal becomes 0.
Why multiply cells here and not use @? A mask isn't a movement of space; it's a stencil laid over the scores, keeping some and blanking others. That's cell-by-cell work, so * is right, and @ would be the mistake. Matrix code needs both: @ to combine, * to filter.
In PyTorch. The same builders exist in PyTorch: torch.eye(n), torch.diag and torch.tril, and torch.ones(n, n) takes the two sizes without the extra brackets. Real models add one more call, masked_fill: it writes a value into every place where a test is true. Here the test is mask == 0, the places above the diagonal, and the value is float("-inf"), minus infinity, written −∞.
The first line is the masked scores the NumPy cell printed. The second is what a real model uses instead: −∞ above the diagonal rather than 0, so that softmax (lesson 104) gives those later words no weight at all. The last line checks two facts from the math part: diag(2, 0.5, 3) scales (4, 6, 1) to (8, 3, 3), and multiplying by the identity gives the scores back unchanged (torch.equal is True when two tensors match everywhere).
Quick check
What does this print?
import torch
s = torch.tensor([[5., 9.], [7., 1.]])
m = torch.tril(torch.ones(2, 2))
print(s.masked_fill(m == 0, float("-inf")).tolist())
[[5.0, 9.0], [-inf, 1.0]]
[[5.0, 0.0], [7.0, 1.0]]
[[5.0, -inf], [7.0, 1.0]]
[[-inf, 9.0], [-inf, -inf]]
Show the answer
[[5.0, -inf], [7.0, 1.0]]
The mask is 0 only above the diagonal, so only the 9 is replaced, by −∞. Everything on and below the diagonal stays.
Drill
The mask for five words
Build mask, the causal mask for 5 words. Set sees3 to the list of word numbers (counting from 1) that word 3 may see, read from row 3 of the mask, and blocked to how many entries the mask blocks.
import numpy as np
mask = np.ones((5, 5))
sees3 = []
for j inrange(5):
if mask[3, j] == 1:
sees3.append(j)
blocked = 0print(mask, sees3, blocked)
In the lesson, your code is checked as you write it.
In your own words
What would go wrong if a language model were trained without the causal mask?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write