Index Ai Academy
Foundation · Foundation mathVectors and matrices

Tensors, shapes and broadcasting

About 43 minLooking ahead: some prerequisites aren't done yet

Why this matters

Data doesn't stop at two directions. A batch of pictures has three: which picture, which row, which column. A batch of sentences has three too: which sentence, which word, which number in the word's vector. NumPy and PyTorch call any such block of numbers a tensor, and its shape says how long each direction is. Most bugs in model code are shape bugs, so reading shapes, changing them with reshape, collapsing one direction with axis=, and knowing how NumPy broadcasts a small array across a big one are everyday skills. This lesson puts all of them together in one small pipeline, in NumPy and then in PyTorch, where the same shapes carry every model you'll meet.

  • Name scalars, vectors, matrices and tensors by their number of axes, and read a shape
  • Count a tensor's numbers as the product of its shape, and reshape without changing the count
  • Sum or average along one axis with axis=, and predict the shape that's left
  • Use broadcasting to apply one row to a whole table, and follow shapes through a small model in NumPy and PyTorch
Quick check
Have a guess before you start. A batch holds 8 pictures, each 5 × 5 pixels. How many numbers is that?

The math: axes and shapes

Purpose. Data with more than two directions.

Count the directions, called axes:

  • A scalar, like 7, has 0 axes. Shape ().
  • A vector, like (2, 3, 1), has 1 axis. Shape (3,).
  • A matrix has 2 axes, rows and columns. Shape (2, 3).
  • A tensor has 3 or more. Eight 5 × 5 pictures stacked up have shape (8, 5, 5).

The number of entries is the product of the shape: (8, 5, 5) holds 200 numbers. Axes are numbered from 0, so in (8, 5, 5) axis 0 picks the picture, axis 1 the row and axis 2 the column. Model code often names the axes instead: a batch of sentences is (batch, seq, dim), for which sentence, which word in the sequence, and which number of the word's vector.