The dot product turns two lists of numbers into one number: multiply them pair by pair and add. It's a weighted total, like the bill for a shopping basket, amounts times prices. It's also the single most important operation in machine learning. One neuron computes exactly one dot product, a layer computes many, and a transformer spends most of its time doing them. This lesson builds it by hand, three ways in Python, and as a neuron in NumPy and PyTorch. It also meets two facts that the rest of the module leans on: a vector dotted with itself gives its squared length, and two vectors whose dot product is 0 meet at a right angle.
What you'll be able to do
Compute a dot product: multiply matching components and add
Read it as a weighted total, like a shopping bill, and as a shadow whose sign says whether two arrows lean together; know that a·a is the squared length
Compute it three ways in Python, a loop, sum(a * b) and a @ b, and check the lengths first
See one neuron's calculation, z = w·x + b, as a dot product plus a bias, and get the same z from PyTorch's nn.Linear
Quick check
Have a guess before you start. You buy 3 apples at $1.50, 2 loaves at $4 and 5 lemons at $0.80. What's the bill?
$16.50
$10
$6.30
$63
Show the answer
$16.50
3 × 1.50 + 2 × 4 + 5 × 0.80 = 4.50 + 8 + 4 = 16.50. That multiply-and-add is a dot product of amounts and prices.
The math: multiply in pairs, then add
Purpose. One number that says how much two lists agree, weighted.
The dot product of two vectors of the same length multiplies matching components and adds the results. For a = (2, 3, 1) and b = (4, 1, 5):
a⋅b=2×4+3×1+1×5=16
In plain words: pair them up, multiply each pair, add the products.
The shopping bill is the everyday case: amounts · prices. The answer is a single number, a scalar, not a vector. The two vectors must be the same length, or some numbers have no partner.
Quick check
From an earlier lesson: what is Σ i² for i = 1 to 3?
6
9
36
14
Show the answer
14
1 + 4 + 9 = 14. A dot product is a Σ too: the sum over i of aᵢ × bᵢ.
The picture. Watch (2, 3, 1) · (4, 1, 5) two ways at once: as boxes of numbers, and as a drawing. In the drawing, a's numbers sit in the circles on the left and b's numbers ride on the lines. Each pair multiplies along its line, then the products pour into the circle on the right, marked Σ, which keeps the running total. That circle is one unit of a neural network, and the AI part comes back to it.
Try it now
Then set vector a to 1, 2 and vector b to 2, -1, and press Use these numbers. The two products cancel, and the dot product is 0.
Symbols and formula. The dot product, and two facts worth knowing:
a⋅b=i=1∑naibia⋅a=∥a∥2
In plain words: a vector dotted with itself is its length squared: (3, 4) · (3, 4) = 9 + 16 = 25 = 5².
Order doesn't matter. a · b = b · a, because each product aᵢbᵢ is the same both ways round.
Zero means perpendicular. (1, 2) · (2, −1) = 2 − 2 = 0. When the dot product of two nonzero vectors is 0, they point at right angles: neither has any part along the other. Lesson 51 turns this into a measure of angle.
Reading the sign: shadows. A positive dot product means the two vectors lean the same way; a negative one means they lean apart. In two dimensions you can see it. Take b = (1, 0), an arrow one unit long along the x-axis, and shine a light straight down onto b's line. The arrow a casts a shadow on that line, starting at the origin (the picture draws it just under the line, so b stays in view). Because b is one unit long, a · b is exactly the shadow's length: positive when the shadow falls forward along b, negative when it falls behind.
Quick check
What is (1, 0, 2) · (3, 5, −1)?
−2
8
1
3
Show the answer
1
1 × 3 + 0 × 5 + 2 × (−1) = 3 + 0 − 2 = 1.
Quick check
Which pair of vectors is perpendicular?
(1, 1) and (1, 2)
(2, 1) and (2, 1)
(3, 0) and (2, 0)
(2, 1) and (−1, 2)
Show the answer
(2, 1) and (−1, 2)
2 × (−1) + 1 × 2 = 0: a dot product of 0 means right angles.
In your own words
In your own words: why must two vectors be the same length to have a dot product?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: three ways, one answer
Purpose. See the dot product as a loop, then use NumPy's shorthand.
import numpy as np
a = np.array([2, 3, 1])
b = np.array([4, 1, 5])
total = 0for i inrange(len(a)):
total = total + a[i] * b[i]
print(total, np.sum(a * b), a @ b)
It runs right here in your browser, with nothing to install.
The loop is the definition written out. a * b multiplies matching components (lesson 48), and np.sum adds them: the same two steps in one line. The @ operator does both at once and is what real code uses. All three give 16.
Code flow. Step through the loop one line at a time. It runs on plain lists here; the loop works the same on arrays. Watch i pick each pair in turn, and total grow by that pair's product: the running sum from the picture's circle.
A named mistake: * for @.a * b alone is a vector of products, not their total. It runs without an error, so check what you get: a dot product is a single number.
import numpy as np
a = np.array([2, 3, 1])
b = np.array([4, 1, 5])
print((a * b).tolist(), a @ b)
Check the lengths first.@ on vectors of different lengths stops with an error. A good function says why, before NumPy does, by comparing len(a) with len(b).
Why three ways? The loop is the definition, step by step, and it's the version to write when you need to see what's happening. np.sum(a * b) shows the two stages, multiply then add, as two NumPy operations. And a @ b is the one real code uses: it's shortest and fastest, and lesson 56 will use the same @ to multiply whole matrices. All three must agree, which makes the first two a handy test of the third.
Worked example
Worked example: a shopping bill, three ways
Compute the bill for amounts (3, 2, 5) at prices (1.50, 4.00, 0.80) by loop, by sum, and with @.
import numpy as np
amounts = np.array([3, 2, 5])
prices = np.array([1.5, 4.0, 0.8])
bill = 0for i inrange(len(amounts)):
bill = bill + amounts[i] * prices[i]
print(round(bill, 2), round(np.sum(amounts * prices), 2), round(amounts @ prices, 2))
In the lesson you can change this and run it.
All three give 16.5, the bill from the opening question. Each pair is an item: its amount times its price. The rounding tidies float noise in the money. In ML the "amounts" are a model's inputs and the "prices" its weights: a dot product says how much each input contributes, weighted by how much the model cares about it.
Quick check
What does this print?
import numpy as np
a = np.array([1, 2])
b = np.array([2, -1])
print(a @ b, a @ a)
0 5
0 3
5 0
4 5
Show the answer
0 5
1 × 2 + 2 × (−1) = 0, so they're perpendicular. a · a = 1 + 4 = 5, the squared length.
The shadows, in code. The three arrows from the math part's picture, each dotted with b = (1, 0):
import numpy as np
b = np.array([1, 0])
print(np.array([4, 2]) @ b, np.array([0, 3]) @ b, np.array([-3, 2]) @ b)
Each answer is a's first number: dotting with b = (1, 0) keeps the first number (times 1) and drops the second (times 0). That's the shadow's length with its sign: forward, none, behind. Change [4, 2] to [1, 5] and predict the first answer before you run it.
Quick check
Which line gives the dot product of a and b as one number?
a * b
(a * b).tolist()
a + b
a @ b
Show the answer
a @ b
@ multiplies matching components and adds them up.
Drill
Three ways to the same number
Fix dot(a, b) so it checks the lengths match (returning None if not) and then computes the dot product with a loop. Then set with_at to the same product with @.
import numpy as np
defdot(a, b):
total = 0for i inrange(len(a)):
total = total + a[i] + b[i]
return total
a = np.array([2, -1, 3])
b = np.array([1, 4, 2])
with_at = a * b
print(dot(a, b), with_at, dot(a, np.array([1, 2])))
In the lesson, your code is checked as you write it.
In your own words
Why is it worth writing the dot product as a loop once, even though @ exists?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: one neuron is one dot product
Purpose. The basic unit of a neural network computes a dot product plus a number.
A neuron takes an input vector x, multiplies it by its weight vector w, and adds a bias b:
z=w⋅x+b
In plain words: each input times its weight, all added up, plus the bias. That's lesson 14's ŷ = w·x + b, with many inputs instead of one.
import numpy as np
w = np.array([0.5, -1.0, 2.0])
x = np.array([4.0, 1.0, 3.0])
b = 0.5print((w * x).tolist(), w @ x + b)
The products show each input's contribution: 2, −1 and 6, plus the bias 0.5, making z = 7.5. It's the drawing from the math part with a bias added at the end: the inputs in the circles, the weights on the lines, and z in the circle they feed.
Quick check
A neuron has weights (0.2, 3, −0.5) and gets the input (10, 1, 2). Which input contributes the most to z?
The third: its weight is negative
The second: 3 × 1 = 3
They all contribute equally
The first: it's the biggest input
Show the answer
The second: 3 × 1 = 3
The contributions are 0.2 × 10 = 2, 3 × 1 = 3 and −0.5 × 2 = −1. The second is biggest, because its weight is large.
Reading weights. A weight says how much the neuron cares about its input: a big positive weight makes that input push z up strongly, a negative one pushes it down, and a weight near 0 means the input hardly matters. Training adjusts these weights so the dot product comes out high for some inputs and low for others. A layer is many neurons side by side, which lesson 55 writes as one matrix times one vector.
The bias shifts the whole total. b doesn't multiply any input; it's added at the end, so it moves z up or down by the same amount for every input. Without it, an input of all zeros would always give z = 0. With it, the neuron can be "on" or "off" by default, before any input pushes it either way. Training learns b along with the weights: it's one more number that the update in lesson 48 nudges at every step.
Try both ideas on the neuron below. It has just two inputs, to keep it small: slide each input, each weight and the bias, and watch the sum change.
In PyTorch. The same neuron in the real tool, two ways. First, this lesson's line, w @ x + b, on tensors. Second, nn.Linear(3, 1): PyTorch's layer with 3 inputs and 1 output, which is one neuron with a weight for each input and one bias. A new layer starts with random weights, so the cell loads ours. copy_(w) copies w into the weights and fill_(b) sets the bias; a name ending in _ changes a tensor in place. with torch.no_grad(): tells PyTorch not to record the lines under it for training, which is how you set weights by hand. Without it, PyTorch refuses to change a layer's weights in place.
import torch
import torch.nn as nn
w = torch.tensor([0.5, -1.0, 2.0])
x = torch.tensor([4.0, 1.0, 3.0])
b = 0.5print(w @ x + b)
layer = nn.Linear(3, 1)
with torch.no_grad():
layer.weight.copy_(w)
layer.bias.fill_(b)
print(layer(x).item())
Both give 7.5, the z the NumPy cell printed: first as a tensor, then as a plain number, which .item() takes out of a one-number tensor. In a real model, nobody loads the weights by hand. Training finds them, nudging every weight and the bias a little at each step.
Quick check
What does this print?
import numpy as np
w = np.array([1.5, -2.0, 0.5])
x = np.array([2.0, 1.0, 4.0])
print((w * x).tolist(), w @ x - 1)
[3.0, -2.0, 2.0] 3.0
[3.0, -2.0, 2.0] 2.0
3.0 [3.0, -2.0, 2.0]
[3.5, -1.0, 4.5] 2.0
Show the answer
[3.0, -2.0, 2.0] 2.0
The contributions are 3, −2 and 2; their sum is 3, and the bias −1 makes z = 2.
Drill
One neuron's output
Write neuron(w, x, b) returning w · x + b. Then set z for weights (1.5, −2, 0.5), input (2, 1, 4) and bias −1, and biggest to the position of the input that contributes most.
import numpy as np
defneuron(w, x, b):
return w * x + b
w = np.array([1.5, -2.0, 0.5])
x = np.array([2.0, 1.0, 4.0])
z = neuron(w, x, -1)
contrib = w * x
biggest = 0print(z, contrib.tolist(), biggest)
In the lesson, your code is checked as you write it.
In your own words
What does a negative weight do to its input's effect on a neuron?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write