Index Ai Academy
Foundation · Foundation mathVectors and matrices

Matrix multiplication

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

A matrix times a vector does several dot products for one vector. A matrix times a matrix does them for many vectors at once: every row of the first meets every column of the second. That one operation is the workhorse of modern AI. A layer of a neural network applied to a whole batch of examples is one matrix multiplication, and the attention inside a transformer is a few more. GPUs are, more or less, machines built to do it fast. This lesson builds it from shopping bills, works it by hand with the size rule as a guide, writes it as three loops and then as one @, and finishes by pushing four pictures through a layer in a single line, in NumPy and in PyTorch.

  • Multiply two matrices: entry (i, j) is row i of the first dotted with column j of the second
  • Use the size rule: inner sizes must match, and the outer sizes give the answer's size
  • Compute A @ B with a triple loop and with @, time the two, and know that A @ B and B @ A usually differ
  • Run a batch of pictures through a layer at once, Z = X W + b, and get the same scores from PyTorch's nn.Linear
Quick check
Have a guess before you start. A 2 × 3 matrix times a 3 × 2 matrix. How many numbers are in the answer?

The math: every row meets every column

Purpose. Many bills at once. Three hikers each buy some energy bars, apples and nuts, recorded in A, one row per hiker. Two shops price the three items, recorded in B, one column per shop:

A=(213041520)B=(1.52.00.50.43.02.5)A = \begin{pmatrix} 2 & 1 & 3 \\ 0 & 4 & 1 \\ 5 & 2 & 0 \end{pmatrix} \qquad B = \begin{pmatrix} 1.5 & 2.0 \\ 0.5 & 0.4 \\ 3.0 & 2.5 \end{pmatrix}

Hiker 1's bill at shop 1 is a dot product, row 1 of A with column 1 of B: 2 × 1.5 + 1 × 0.5 + 3 × 3 = 12.5. Do that for every hiker and every shop and you have six bills in a 3 × 2 table. That table is the matrix product AB.