Data doesn't stop at two directions. A batch of pictures has three: which picture, which row, which column. A batch of sentences has three too: which sentence, which word, which number in the word's vector. NumPy and PyTorch call any such block of numbers a tensor, and its shape says how long each direction is. Most bugs in model code are shape bugs, so reading shapes, changing them with reshape, collapsing one direction with axis=, and knowing how NumPy broadcasts a small array across a big one are everyday skills. This lesson puts all of them together in one small pipeline, in NumPy and then in PyTorch, where the same shapes carry every model you'll meet.
What you'll be able to do
Name scalars, vectors, matrices and tensors by their number of axes, and read a shape
Count a tensor's numbers as the product of its shape, and reshape without changing the count
Sum or average along one axis with axis=, and predict the shape that's left
Use broadcasting to apply one row to a whole table, and follow shapes through a small model in NumPy and PyTorch
Quick check
Have a guess before you start. A batch holds 8 pictures, each 5 × 5 pixels. How many numbers is that?
40
200
25
18
Show the answer
200
8 × 5 × 5 = 200: multiply the length of every direction.
The math: axes and shapes
Purpose. Data with more than two directions.
Count the directions, called axes:
A scalar, like 7, has 0 axes. Shape ().
A vector, like (2, 3, 1), has 1 axis. Shape (3,).
A matrix has 2 axes, rows and columns. Shape (2, 3).
A tensor has 3 or more. Eight 5 × 5 pictures stacked up have shape (8, 5, 5).
The number of entries is the product of the shape: (8, 5, 5) holds 200 numbers. Axes are numbered from 0, so in (8, 5, 5) axis 0 picks the picture, axis 1 the row and axis 2 the column. Model code often names the axes instead: a batch of sentences is (batch, seq, dim), for which sentence, which word in the sequence, and which number of the word's vector.
Quick check
From lesson 53: what is the shape of a data matrix for 60 signs of 25 pixels, one row per sign?
(1500,)
(25, 60)
(60, 25)
(60, 5, 5)
Show the answer
(60, 25)
One row per example, one column per feature. Keep each sign as a grid instead and the shape is (60, 5, 5): the same 1,500 numbers.
Collapsing an axis. Summing or averaging along an axis adds up the entries in that direction and removes it. For T = [[1, 2, 3], [4, 5, 6]], shape (2, 3):
along axis 0 (down the rows) leaves one number per column: (5, 7, 9), shape (3,);
along axis 1 (across the columns) leaves one number per row: (6, 15), shape (2,);
along both leaves one number, 21.
In plain words: the axis you name is the one that disappears.
Try it now
Switch between axis 0 and axis 1 and watch which direction collapses.
The picture. Shapes are how you follow data through a model. Below, a batch of 4 sentences, 6 words each, every word a vector of 16 numbers, goes through a layer, an average and a last layer. The first box is the input x, the middle three are the steps, and each arrow is labeled with the shape that travels along it. Watch which axis each step changes and which ones ride along untouched.
Broadcasting. When NumPy combines arrays of different shapes, it stretches the smaller one to fit, when it can, without copying it. A single row subtracted from a table is reused for every row. Watch the average day taken away from each day of a 3 × 2 table. The picture draws each day as a point, temperature across and hikers up, and the row being taken away as a ring. Each step moves one point by the same arrow.
Symbols and formula. The count and the broadcasting rule:
count=d0×d1×⋯(3,2) with (2,)→(3,2)
In plain words: to broadcast, line the two shapes up from the right. Each pair of sizes must be equal, or one of them must be 1 (or missing); the 1s are stretched to match.
So (3, 2) with (2,) works, and so does (3, 2) with (3, 1), a column reused for each column. But (3, 2) with (3,) fails: lined up from the right, 2 meets 3.
Quick check
Which pair of shapes can NumPy broadcast together?
(4, 3) and (4,)
(4, 3) and (2, 3)
(4, 3) and (3, 4)
(4, 3) and (3,)
Show the answer
(4, 3) and (3,)
From the right, 3 matches 3, and the missing axis is filled by reusing the row.
In your own words
In your own words: what does "the axis you name is the one that disappears" mean for a (5, 3) table summed along axis 0?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: shape, reshape, axis= and broadcasting
Purpose. Change and collapse shapes in one call each.
import numpy as np
v = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
M = v.reshape(3, 4)
T = v.reshape(2, 2, 3)
print(v.shape, M.shape, T.shape, T.size)
print(T[1].tolist())
It runs right here in your browser, with nothing to install.
reshape lays the same numbers out in a new shape, row by row; the count must stay 12. .size is the count. T[1] picks along axis 0, leaving a 2 × 3 matrix. A named mistake: a reshape whose count doesn't match, like v.reshape(5, 3), stops with "cannot reshape array of size 12 into shape (5,3)".
Code flow. Step through six lines that use every idea so far: a reshape, a sum along each axis, and an average row taken away by broadcasting. Watch the boxes as you go. T holds v's six numbers in two rows, down has one box per column, across one per row, and C keeps T's shape.
axis= and broadcasting. Most NumPy reductions take an axis:
import numpy as np
T = np.array([[1, 2, 3], [4, 5, 6]])
print(T.sum(axis=0).tolist(), T.sum(axis=1).tolist(), T.sum())
X = np.array([[20, 4], [25, 2], [30, 6]])
mean = X.mean(axis=0)
print(mean.tolist(), (X - mean).tolist())
X.mean(axis=0) is the average day, one number per feature. X - mean broadcasts it over every row in one line, centering the table, which lesson 63's PCA needed and did with a loop. Broadcasting also covers the earliest case of all: P + 0.1 in lesson 54 was a single number stretched over a whole picture.
Worked example
Worked example: the mean pixel per sign
A batch of 8 signs has shape (8, 5, 5). Find how much ink each sign has, and the average picture.
Stacking 8 grids gives shape (8, 5, 5). Reshaping to (8, 25) unrolls each sign into a row; the mean along axis 1 is each sign's share of inked pixels: 0.36 for the cross and the arrow (9 of 25), 0.64 for the box (16 of 25). The mean along axis 0 averages the 8 signs pixel by pixel, leaving one 5 × 5 average picture; its middle row is shown.
Quick check
What does this print?
import numpy as np
A = np.array([[1, 2], [3, 4]])
print((A + np.array([10, 20])).tolist(), (A * np.array([[1], [10]])).tolist())
[[11, 12], [23, 24]] [[1, 2], [30, 40]]
[[11, 22], [13, 24]] [[1, 20], [3, 40]]
[[11, 22], [13, 24]] [[1, 2], [30, 40]]
An error: the shapes differ
Show the answer
[[11, 22], [13, 24]] [[1, 2], [30, 40]]
A row (10, 20) is added to each row. A column (1, 10) multiplies each column: row 1 by 1, row 2 by 10.
Drill
Center a table in one line
Fix the lines so means is each column's average, centered is the table with each column's average taken away (by broadcasting), and ink is each sign's mean pixel from the (3, 5, 5) batch.
In the lesson, your code is checked as you write it.
In your own words
Why is X - X.mean(axis=0) right for centering, while X - X.mean(axis=1) usually isn't?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: real inputs are tensors
Purpose. Follow the shapes through a model.
Model inputs come in batches, so they're tensors: a batch of 8 signs is (8, 5, 5), and a batch of 4 sentences of 6 words, each word a vector of 16, is (4, 6, 16), the x of the math part's picture.
Here's lesson 56's sign layer on a batch, with the shape written after every step:
In plain words: unroll each picture, score it three ways, add the biases by broadcasting, and pick the best score in each row.
import numpy as np
cross = np.array([[1,0,0,0,1],[0,1,0,1,0],[0,0,1,0,0],[0,1,0,1,0],[1,0,0,0,1]])
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
box = np.array([[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,1]])
batch = np.array([cross, arrow, box, cross, arrow, box, cross, arrow])
W = np.array([cross.flatten(), arrow.flatten(), box.flatten()]).T
b = np.array([0, 0, -5])
Z = batch.reshape(8, 25) @ W + b
print(batch.shape, W.shape, Z.shape)
print(np.argmax(Z, axis=1).tolist())
np.argmax(Z, axis=1) gives the position of the biggest number in each row: 0, 1, 2, … names cross, arrow, box. The bias −5 on the box score is broadcast to every row, a small handicap the box still overcomes.
Quick check
A batch of 4 sentences, 6 words each, every word a vector of 16, goes through a layer W of shape (16, 64). What shape comes out?
(4, 6, 64)
(64, 6, 4)
(4, 64)
(4, 6, 16)
Show the answer
(4, 6, 64)
@ acts on the last axis: 16 meets 16 and becomes 64. The batch and word axes ride along unchanged, as they did for the (16, 32) layer in the math part's picture.
Shapes are the first thing to check. Model code is full of comments like # (batch, words, 64), because a shape error, a wrong axis or an accidental broadcast is the most common bug in ML. Broadcasting is the sneakiest: (8, 1) minus (8,) quietly becomes (8, 8) instead of failing. When a result has a surprising shape, print .shape at each step, just as the pipeline above does. A comment with the expected shape on each line turns a surprise into an obvious mismatch.
Quick check
A column and a row broadcast together. What does this print?
import numpy as np
col = np.ones((4, 1))
row = np.array([1, 2, 3])
print((col + row).shape, (col + row)[0].tolist())
(4, 3) [2.0, 3.0, 4.0]
(4, 1) [2.0]
An error: the shapes differ
(3,) [2.0, 3.0, 4.0]
Show the answer
(4, 3) [2.0, 3.0, 4.0]
Lined up from the right: (4, 1) and (3,). The 1 stretches to 3 and the missing axis to 4, so the result is (4, 3), each row 1 + (1, 2, 3).
In PyTorch. The same pipeline on tensors. PyTorch says dim= where NumPy says axis=, and torch.stack piles tensors along a new first axis, as np.array did with the list of grids:
import torch
cross = torch.tensor([[1,0,0,0,1],[0,1,0,1,0],[0,0,1,0,0],[0,1,0,1,0],[1,0,0,0,1]])
arrow = torch.tensor([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
box = torch.tensor([[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,1]])
batch = torch.stack([cross, arrow, box, cross, arrow, box, cross, arrow])
W = torch.stack([cross.flatten(), arrow.flatten(), box.flatten()]).T
b = torch.tensor([0, 0, -5])
Z = batch.reshape(8, 25) @ W + b
print(batch.shape, W.shape, Z.shape)
print(Z.argmax(dim=1).tolist())
print(batch.reshape(8, 25).sum(dim=1).tolist())
one = cross.unsqueeze(0)
print(one.shape, (one.reshape(1, 25) @ W + b).argmax(dim=1).tolist())
The shapes (PyTorch writes them torch.Size([8, 5, 5])) and the predictions are the ones NumPy printed: 0, 1, 2, 0, 1, 2, 0, 1. .sum(dim=1) counts each sign's ink: 9 for the cross and the arrow, 16 for the box. .unsqueeze(0) adds an axis of size 1 at the front, so one sign becomes a batch of one, (1, 5, 5), and the layer reads it as a cross, [0]. A model always expects the batch axis first, even for a single example.
The picture's shapes, in code.torch.zeros makes tensors of any shape, which is all you need to check shapes before any real numbers exist. These are the four boxes of the math part's picture:
import torch
x = torch.zeros(4, 6, 16) # (batch, seq, dim)
W = torch.zeros(16, 32)
h = x @ W
pooled = h.mean(dim=1)
scores = pooled @ torch.zeros(32, 3)
print(x.shape, h.shape, pooled.shape, scores.shape)
print(torch.einsum("bsd,dh->bsh", x, W).shape)
An aside: einsum. The last line writes x @ W with a letter for every axis: b for batch, s for seq, d for dim and h for the 32 outputs. The letter d is in both inputs but not after the arrow, so it's the axis that gets multiplied and added away; the rest come through. You'll meet einsum in model code, and reading its letters is just reading shapes.
Drill
Shapes through a pipeline
A batch of 6 pictures, each 4 × 4, goes through a layer with 5 outputs. Set flat_shape, W_shape, Z_shape and pred_shape to the shape (as a tuple) at each step: after reshaping, the weights, after @ W + b, and after argmax along axis 1. Then build the arrays with zeros to check them.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write