Before the big operation of the next lessons, matrix multiplication, come the easy ones. Adding two matrices, scaling one by a number and flipping one over its diagonal are all done entry by entry, with nothing new to learn except the rules about size. They're also everywhere in ML code: an average picture is an add and a scale, a brightness change is a scale, and a data table that arrives stored the wrong way around is fixed with a transpose. By the end of this lesson you'll do all three on paper and in one short line of NumPy each, and you'll know the shape error NumPy gives when two sizes don't match. The transpose in particular comes back in almost every lesson that follows, from matrix products to the weights of a layer.
What you'll be able to do
Add and subtract matrices of the same size, entry by entry
Scale a matrix by multiplying every entry by one number
Transpose a matrix, turning rows into columns, and predict the new size
Average pictures and turn a table the right way around with A + B, 2 * A and A.T, and do both on a batch in PyTorch with X.mean(dim=0) and X.T
Quick check
Have a guess before you start. Each morning two shops record the tents they rent, in the same 2 × 2 table layout. How do you get the combined table?
Multiply the matching entries
Put one table under the other
Add the matching entries
Add every entry of both into one number
Show the answer
Add the matching entries
The same position means the same day and tent type in both tables, so the combined count adds the matching entries.
The math: entry by entry
Purpose. The easy operations first, each done one entry at a time.
Adding and subtracting. Two matrices of the same size add entry by entry:
(1240)+(35−12)=(4732)
In plain words: each entry of the answer is the sum of the two entries in the same place.
Subtracting works the same way. Two matrices of different sizes can't be added: some entries would have no partner, just like vectors of different lengths in lesson 48.
Scaling. Multiplying a matrix by a number, a scalar, multiplies every entry: 3 × [[1, 4], [2, 0]] = [[3, 12], [6, 0]].
Quick check
From lesson 48: what is 2 × (3, −1) + (1, 4)?
(4, 3)
(7, −2)
(8, 6)
(7, 2)
Show the answer
(7, 2)
2 × (3, −1) = (6, −2), and adding (1, 4) gives (7, 2). Matrices follow the same rules, entry by entry.
The transpose. Flip a matrix over its main diagonal, the line from top left down to the right, and its rows become its columns. The transpose of A is written Aᵀ. Watch a 2 × 3 matrix become a 3 × 2 one.
The picture. The flow draws A on the left and, beside it, Aᵀ, whose places start empty. Each cell is shaded by its number, small numbers light and big ones dark, so a number keeps its shade when it moves. Each step lights one row of A and the column of Aᵀ it becomes. By the end the shading runs from light to dark along A's rows but down Aᵀ's columns: the whole pattern has turned over the diagonal. The PyTorch tab shows the same line on a tensor.
Try it now
Then set the matrix to three rows, 1 2 3, 4 5 6 and 7 8 9, and press Use these numbers. The diagonal, 1, 5 and 9, stays put, and everything else swaps across it.
Symbols and formula. The transpose swaps the two indexes:
(AT)ij=Aji(A+B)T=AT+BT(AT)T=A
In plain words: row i, column j of Aᵀ is row j, column i of A. Transposing a sum is the sum of the transposes, and transposing twice gets you back where you started.
Sizes. An m × n matrix has an n × m transpose: 2 × 3 becomes 3 × 2. A square matrix stays the same size, and the entries on its diagonal, a₁₁, a₂₂ and so on, never move.
Quick check
What is the size of the transpose of a 4 × 7 matrix?
7 × 4
4 × 7
28 × 1
7 × 7
Show the answer
7 × 4
Rows and columns swap: 4 rows of 7 become 7 rows of 4.
Mixing the operations. Because everything here works entry by entry, the usual rules of arithmetic carry over: A + B = B + A, and 2A − B means double every entry of A, then subtract B's matching entry. With A = [[2, 0, 1], [1, 3, 2]] and B = [[1, 1, 0], [0, 2, 5]], 2A − B = [[3, −1, 2], [2, 4, −1]]. Check one entry by hand, say row 2, column 3: 2 × 2 − 5 = −1.
Transposes and sums. Transposing a sum gives the same answer as adding the transposes: (A + B)ᵀ = Aᵀ + Bᵀ, because both put the entry from row i, column j of each matrix into row j, column i. And scaling commutes with transposing too: (3A)ᵀ = 3Aᵀ. Rules like these let you move a transpose around an expression without working out every entry, which is how the formulas for training a network are usually simplified. You'll meet them again in lesson 81, where the transpose carries a model's error backward through each layer.
In your own words
In your own words: why can't a 2 × 3 matrix be added to a 3 × 2 one, even though both hold 6 numbers?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: one operator each
Purpose. NumPy does each operation on the whole matrix in one line.
import numpy as np
A = np.array([[1, 4], [2, 0]])
B = np.array([[3, -1], [5, 2]])
print((A + B).tolist())
print((A - B).tolist())
print((3 * A).tolist())
print(A.T.tolist())
It runs right here in your browser, with nothing to install.
+, - and * by a number work entry by entry, exactly as on paper, and .T is the transpose. The .tolist() calls just print each result as plain lists, one row inside each pair of brackets.
A named mistake: shapes that don't match. Add a 2 × 3 matrix to a 3 × 2 one and NumPy stops with an error that ends "operands could not be broadcast together with shapes (2,3) (3,2)". Read it as "these sizes don't match". Print both .shapes, and if one of them is the other's transpose, .T is usually the fix:
import numpy as np
C = np.array([[2, 0, 1], [1, 3, 2]])
D = np.array([[1, 0], [1, 2], [0, 5]])
print(C.shape, D.shape)
print((C + D.T).tolist())
Worked example
Worked example: brighten, dim and flip a picture
A tiny 2 × 3 grayscale picture has pixels from 0 (black) to 1 (white). Brighten it, dim it, and flip it on its side.
import numpy as np
P = np.array([[0.2, 0.5, 0.1],
[0.4, 0.9, 0.3]])
brighter = P + 0.1
dimmer = P * 0.5
flipped = P.T
print(brighter.round(2).tolist())
print(dimmer.tolist())
print(flipped.shape, flipped.tolist())
In the lesson you can change this and run it.
Adding 0.1 to every pixel brightens the picture evenly; .round(2) tidies the float noise that adding 0.1 leaves behind. Multiplying by 0.5 halves every pixel, so the whole picture is half as bright. The transpose turns the 2 × 3 picture into a 3 × 2 one, laid on its side: the top row becomes the left column. Each is one operation on all six pixels at once.
Code flow. Step through the three edits one line at a time. Under Variables each picture shows as a grid of boxes, so you can see every pixel change in one line, and see flipped come out a different shape from P.
Quick check
What does this print?
import numpy as np
A = np.array([[1, 2, 3],
[4, 5, 6]])
print(A.T.shape, (2 * A.T).tolist()[0])
(3, 2) [1, 4]
(3, 2) [2, 8]
(2, 3) [2, 4, 6]
(3, 2) [2, 4]
Show the answer
(3, 2) [2, 8]
A.T is 3 × 2 with first row (1, 4); doubled, that's (2, 8).
Quick check
Which line gives the transpose of a NumPy matrix M?
M.T
M[::-1]
M.flatten()
M * -1
Show the answer
M.T
.T swaps rows and columns.
Drill
Three edits to a picture
Fix the three lines so brighter adds 0.2 to every pixel, half halves every pixel, and turned is the transpose of P.
import numpy as np
P = np.array([[0.1, 0.6],
[0.3, 0.8],
[0.0, 0.4]])
brighter = P * 0.2
half = P - 0.5
turned = P
print(brighter.round(2).tolist(), half.tolist(), turned.shape)
In the lesson, your code is checked as you write it.
In your own words
Why is printing both shapes the first thing to do when NumPy says it can't add two matrices?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: averages and tables the right way around
Purpose. An average is add-then-scale, and data often arrives transposed.
The mean image of several pictures is their sum scaled by one over the count:
Xˉ=n1(X1+X2+⋯+Xn)
In plain words: add the pictures entry by entry, then divide every entry by how many there were.
Pixels inked in both signs average to 1, in one sign to 0.5, and in neither to 0. A mean image is the first thing people look at in a new picture data set: it shows where the ink usually is.
Quick check
Three signs are averaged. A pixel is inked (1) in two of them and blank (0) in one. What is that pixel in the mean image, to 2 places?
1
2
0.67
0.5
Show the answer
0.67
(1 + 1 + 0) ÷ 3 = 2/3 ≈ 0.67.
Tables the wrong way around. Data sometimes arrives with one row per feature: a row of temperatures and a row of humidities, one column per day. Models expect one row per example (lesson 53). One transpose fixes it:
import numpy as np
by_feature = np.array([[20, 25, 30],
[50, 60, 30]])
X = by_feature.T
print(by_feature.shape, X.shape)
print(X.tolist())
Now each row is one day, (temperature, humidity). Checking .shape against "examples first" catches the mix-up before a model trains on it.
Quick check
What does this print?
import numpy as np
a = np.array([[2, 0], [0, 4]])
b = np.array([[0, 2], [4, 0]])
print(((a + b) / 2).tolist(), (a - b).T.tolist())
[[2.0, 2.0], [4.0, 4.0]] [[2, -4], [-2, 4]]
[[1.0, 1.0], [2.0, 2.0]] [[2, -4], [-2, 4]]
[[1.0, 1.0], [2.0, 2.0]] [[2, -2], [-4, 4]]
[[1.0, 1.0], [2.0, 2.0]] [[-2, 4], [2, -4]]
Show the answer
[[1.0, 1.0], [2.0, 2.0]] [[2, -4], [-2, 4]]
The average adds entry by entry and halves: (2 + 0) / 2 = 1, and so on. a − b is [[2, −2], [−4, 4]], and its transpose swaps the off-diagonal entries.
In PyTorch. Both of this part's calculations, in the real tool. A model takes its examples in a batch, many stacked into one tensor, so here the two signs go in together. torch.tensor([cross, box]) stacks two 5 × 5 pictures into one tensor of shape (2, 5, 5): 2 pictures, each 5 rows by 5 columns. .float() makes its numbers decimals, which an average needs. .mean(dim=0) averages along the first number of that shape, the 2: for each of the 25 pixel places it averages the two pictures, so out comes one 5 × 5 picture.
The mean image comes out 5 × 5, and its rows are the ones the NumPy cell printed: 1.0, 0.5, 0.5, 0.5, 1.0 across the top. .T turns the table to one row per day, (temperature, humidity), exactly as NumPy's did. In PyTorch, dim names the direction to work along, and dim=0 is the direction a batch is stacked in, so .mean(dim=0) gives the average example.
A batch of two 2 × 2 pictures. dim=0 averages across the two, pixel by pixel: (1 + 1) / 2 = 1, (0 + 1) / 2 = 0.5, (0 + 0) / 2 = 0 and (1 + 0) / 2 = 0.5.
Drill
The mean of three signs
Set mean3 to the mean image of the cross, the arrow and the box, center to its center pixel rounded to 2 places, and X to the table by_feature turned so there's one row per day.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write