A single dot product is one weighted sum, the work of one neuron. Most of the time you want several at once: three scores for three kinds of sign, ten for ten digits, thousands for the words a language model might say next. Stack the weight vectors as the rows of a matrix, and one multiplication computes all those dot products together. That's a matrix times a vector, and it's what one layer of a neural network does to one example. This lesson does it by hand, row by row, writes it as a loop and then as one @, and uses it to make a tiny sign recognizer that scores a picture against three templates and picks the best match, first in NumPy and then as a real PyTorch layer.
What you'll be able to do
Multiply a matrix by a vector: each row dotted with the vector gives one output
Check the size rule: the matrix's columns must match the vector's length, and the answer has one number per row
Compute A x with a loop of dot products, then with A @ x
Score one picture against three sign types at once, as a layer of three neurons does, in NumPy and with PyTorch's nn.Linear
Quick check
Have a guess before you start. Three neurons each take the same 25-pixel picture and give one score. How many numbers come out?
25
1
75
3
Show the answer
3
Each neuron turns all 25 pixels into one score, so three neurons give three numbers.
The math: one dot product per row
Purpose. Apply several weighted sums to the same vector at once.
To multiply a matrix A by a vector x, take each row of A, dot it with x, and write the result as one entry of the answer:
In plain words: row 1 times x gives the first answer, row 2 times x the second. Two rows, two answers.
The vector is written standing up, as a column, because it lines up with each row as the row slides down past it.
Quick check
From lesson 50: what is (3, −1, 4) · (2, 1, 3)?
9
17
19
6
Show the answer
17
6 − 1 + 12 = 17: the second entry of the product above. Each entry of A x is one dot product.
The picture. Watch each row of A meet the vector x in turn and drop one number into the answer, as boxes of numbers and as a drawing. In the drawing, x's numbers sit in the circles on the left, and each row of A is a circle on the right, a unit: the row's numbers ride the lines into it. A unit shows Σ until its row has been dotted with x, then the total. Two rows, two units. That's a layer of neurons, and the AI part comes back to it.
Try it now
Then add a third line to matrix A, like 0 1 1, and press Use these numbers: a third unit appears, with a third answer, 4.
Symbols and formula. For an m × n matrix A and a vector x of length n:
(Ax)i=j=1∑nAijxj
In plain words: entry i of the answer is row i of A dotted with x.
The size rule. Each row of A has n entries and must pair with x's n entries, so the columns of A must match the length of x. The answer has one entry per row: (m × n) times (n) gives (m). A 2 × 3 matrix can multiply a 3-vector, but not a 2-vector.
Quick check
A is 4 × 6. Which vector can it multiply, and how long is the answer?
A vector of 4; the answer has 6
A vector of 6; the answer has 6
A vector of 6; the answer has 4 entries
Any vector; the answer has 24
Show the answer
A vector of 6; the answer has 4 entries
Columns of A (6) must match the vector's length; the answer has one entry per row (4).
Another way to see it. A x is also a mix of A's columns, weighted by x's entries: 2 × (1, 3) + 1 × (2, −1) + 3 × (0, 4) = (4, 17), the same answer. The row view says how each output is computed; the column view says what the outputs can possibly be. Both come back in later lessons.
Checking a product by hand. Pick one entry of the answer and work it yourself: row 2 of A dotted with x. If it matches, the rows and the vector are lined up right; if it doesn't, a row or the vector was probably read the wrong way around. One entry is enough to catch most slips.
In your own words
In your own words: why does the answer to A x have as many entries as A has rows?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a loop of dot products, then @
Purpose. Write A x out once as a loop, then use NumPy's one-character version.
import numpy as np
A = np.array([[1, 2, 0],
[3, -1, 4]])
x = np.array([2, 1, 3])
y = np.zeros(2)
for i inrange(2):
y[i] = A[i] @ x
print(y.tolist(), (A @ x).tolist())
It runs right here in your browser, with nothing to install.
A[i] is row i (lesson 53), and @ dots it with x (lesson 50). The loop fills the answer one row at a time. Then A @ x does the whole thing: the same @, now with a matrix on the left, meaning "every row dotted with x".
Code flow. Step through the loop one line at a time. np.zeros(2) makes y an array of two 0s, a place for each answer. Watch each pass replace one 0 with a row's dot product, in the same order as the units fill in the math part's picture.
A named mistake: A * x for A @ x. With a matrix and a vector, * doesn't do the dot products. It multiplies x into each row, entry by entry, and keeps them all, so it returns a matrix instead of the answer:
import numpy as np
A = np.array([[1, 2, 0],
[3, -1, 4]])
x = np.array([2, 1, 3])
print((A * x).tolist())
Those rows are the products before they're added up: (2, 2, 0) sums to 4 and (6, −1, 12) to 17. Check the shape: the answer to A x should have one number per row, not a whole matrix.
Worked example
Worked example: three scores from one input
Three ice-cream stands each estimate sales from (temperature, people on the beach) with their own weights. Score one day for all three at once.
import numpy as np
W = np.array([[0.5, 1.0],
[2.0, -1.0],
[1.0, 1.0]])
day = np.array([4.0, 2.0])
print(W.shape, day.shape)
scores = W @ day
print(scores.tolist(), scores.shape)
In the lesson you can change this and run it.
W has one row per stand and one column per input, so it's 3 × 2, and the day is a 2-vector: the inner sizes match. Each stand's estimate is its row dotted with the day: 0.5 × 4 + 1 × 2 = 4, then 2 × 4 − 1 × 2 = 6, then 4 + 2 = 6. Printing the shapes first is the habit that catches mistakes before they happen: (3, 2) times (2,) gives (3,).
Quick check
What does this print?
import numpy as np
A = np.array([[1, 0, 2],
[0, 3, 1]])
x = np.array([4, 1, 2])
print((A @ x).tolist(), (A @ x).shape)
[8, 5] (2,)
[8, 5, 0] (3,)
[13] (1,)
[4, 3] (2,)
Show the answer
[8, 5] (2,)
Row 1: 4 + 0 + 4 = 8. Row 2: 0 + 3 + 2 = 5. Two rows, so two numbers.
Quick check
W has shape (3, 25) and x has shape (25,). What is the shape of W @ x?
(3,)
(25,)
It's an error
(3, 25)
Show the answer
(3,)
The inner 25s match and disappear; one number per row of W is left: (3,).
Drill
A matrix-vector product by loop
Fix matvec(A, x) so it returns None when the sizes don't fit, and otherwise one dot product per row, in a NumPy array. Then set agrees to whether it matches A @ x.
import numpy as np
defmatvec(A, x):
y = np.zeros(len(x))
for i inrange(len(x)):
y[i] = A[i] * x
return y
A = np.array([[2, 0, 1], [1, 3, 0], [0, 1, 2]])
x = np.array([3, 1, 2])
agrees = Falseprint(matvec(A, x), agrees)
In the lesson, your code is checked as you write it.
In your own words
Why print the shapes of W and x before computing W @ x?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: a layer acting on one example
Purpose. One layer of a network is a matrix times a vector, plus a bias.
Give each of three neurons a weight vector of 25, one per pixel, and stack them as the rows of W (3 × 25). One picture x (25 numbers) then gets three scores at once:
z=Wx+b
In plain words: every neuron dots its weights with the same picture, then adds its own bias.
Here each neuron's weights are simply one sign's pixels, a template. Dotting a template with a picture counts the inked pixels they share, so the best match scores highest:
import numpy as np
cross = np.array([[1,0,0,0,1],[0,1,0,1,0],[0,0,1,0,0],[0,1,0,1,0],[1,0,0,0,1]])
arrow = np.array([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]])
box = np.array([[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,1]])
W = np.array([cross.flatten(), arrow.flatten(), box.flatten()])
b = np.array([-1, -1, -1])
x = arrow.flatten()
print((W @ x).tolist(), (W @ x + b).tolist())
The arrow shares 3 pixels with the cross, all 9 with itself and 4 with the box, so the arrow neuron wins.
Quick check
The box picture is scored the same way, with W @ x giving [4, 4, 16]. Which neuron wins, and why?
The box neuron: all 16 of its pixels match
None: two scores tie
The arrow neuron: it has the most weights
The cross neuron: it comes first
Show the answer
The box neuron: all 16 of its pixels match
16 is the biggest score: every inked pixel of the box matches the box template. The cross and arrow share only 4 pixels each with it.
In PyTorch. The same layer in the real tool, two ways. First this part's line, W @ x + b, on tensors. torch.stack stacks the three unrolled templates as the rows of W, as np.array did, and .float() makes the 0s and 1s decimals, the kind of number a layer works in. Second, nn.Linear(25, 3): a layer with 25 inputs and 3 outputs, which is three neurons with 25 weights each. It starts with random weights, so the cell loads ours, under torch.no_grad() as in lesson 50.
import torch
import torch.nn as nn
cross = torch.tensor([[1,0,0,0,1],[0,1,0,1,0],[0,0,1,0,0],[0,1,0,1,0],[1,0,0,0,1]]).float()
arrow = torch.tensor([[0,0,1,0,0],[0,1,1,1,0],[1,0,1,0,1],[0,0,1,0,0],[0,0,1,0,0]]).float()
box = torch.tensor([[1,1,1,1,1],[1,0,0,0,1],[1,0,0,0,1],[1,0,0,0,1],[1,1,1,1,1]]).float()
W = torch.stack([cross.flatten(), arrow.flatten(), box.flatten()])
b = torch.tensor([-1.0, -1.0, -1.0])
x = arrow.flatten()
print((W @ x + b).tolist())
layer = nn.Linear(25, 3)
with torch.no_grad():
layer.weight.copy_(W)
layer.bias.copy_(b)
print(layer.weight.shape, layer(x).tolist())
Both give 2, 8 and 3, the scores the NumPy cell printed after the bias, now as decimals. Look at the shape, as in lesson 53: PyTorch stores a layer's weights as (out, in), here (3, 25), one row per neuron, exactly the W of this lesson. So nn.Linear(25, 3) names its sizes inputs first, but its weight matrix lists the outputs first.
Quick check
A layer is made with nn.Linear(25, 10). What is layer.weight.shape?
torch.Size([250])
torch.Size([10])
torch.Size([10, 25])
torch.Size([25, 10])
Show the answer
torch.Size([10, 25])
PyTorch stores the weights as (out, in): one row of 25 weights for each of the 10 neurons, so that W @ x gives 10 scores.
Real layers learn their weights. A trained network doesn't copy templates; it learns weights that tell the classes apart, some of them negative to punish pixels that shouldn't be inked. But the computation is exactly this one: W x + b. Lesson 56 does the same for many pictures at once, as a matrix times a matrix. There the pictures are rows of X, so the product needs the weights the other way around, as Wᵀ, a 25 × 3 matrix: lesson 54's transpose. PyTorch keeps the (3, 25) W and turns it around inside the layer.
Quick check
What does this print?
import numpy as np
W = np.array([[1, 0, 1], [0, 1, 1]])
x = np.array([1, 0, 1])
b = np.array([-1, 0])
print((W @ x + b).tolist())
[1, 0, 2]
[0, 1]
[2, 1]
[1, 1]
Show the answer
[1, 1]
Row 1: 1 + 0 + 1 = 2, plus −1 is 1. Row 2: 0 + 0 + 1 = 1, plus 0 is 1.
Drill
Recognize a smudged cross
A cross with one stray inked pixel, at row 1, column 3 (the math's numbering), is noisy; cross.copy() makes a separate copy to change, leaving the cross alone. Fix the stray pixel's index, set scores to W x for it as a list, and set winner to the position of the highest score.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write