Vectors can be added, subtracted and stretched, and each operation has a picture: walking one arrow then another, or making an arrow longer, shorter or backward. These few moves are the whole of how a model learns. Every training step takes the weights, a vector, and subtracts a small multiple of another vector, the gradient. This lesson builds those moves by hand, in NumPy, and then as that one line of learning. Along the way come the arrow from one point to another, b − a, and the mix of two vectors, which is how a point partway between two others is found and how word vectors can be blended and compared.
What you'll be able to do
Add and subtract vectors component by component, and picture it tail to head
Scale a vector by a number, to stretch, shrink or reverse it, and mix two vectors
Do vector arithmetic on NumPy arrays, know that sizes must match, and see why one array line beats a loop (vectorization)
Read a training step, w ← w − η·g, as one vector subtraction, and run it in PyTorch, by hand and with an optimizer
Quick check
Have a guess before you start. You walk (3, 1), then (1, 2). Where do you end up, starting from the origin?
(4, 3)
(4, 1)
(3, 2)
(2, −1)
Show the answer
(4, 3)
Walking one arrow then the other adds them: 3 + 1 across and 1 + 2 up, so (4, 3).
The math: combining and resizing arrows
Purpose. Combine movements, and make them bigger, smaller or backward.
Adding. Add matching components: (3, 1) + (1, 2) = (4, 3). As arrows, put the second arrow's tail at the first one's head (its tip), tail to head; the sum goes from the first tail to the last head. Order doesn't matter: either way round you land in the same place.
Scaling. Multiply every component by one number, a scalar: 2 × (3, 1) = (6, 2), twice as long, same direction. A scalar between 0 and 1 shrinks it; a negative scalar reverses it: −1 × (3, 1) = (−3, −1).
Subtracting. b − a is the arrow that goes from a to b: what you'd add to a to reach b.
Quick check
From the last lesson: what vector goes from the point (1, 4) to the point (6, 7)?
(7, 11)
(5, 3)
(−5, −3)
(6, 7)
Show the answer
(5, 3)
End minus start: (6 − 1, 7 − 4) = (5, 3). That's exactly vector subtraction, b − a.
The picture. Here are the three moves drawn on a grid, with a = (3, 1) and b = (1, 2). Press Next to go from adding, to scaling, to subtracting. Watch where each arrow starts: a sum is built tail to head, and a difference runs between two tips.
Try it now
Symbols and formula. Component by component:
a+b=(a1+b1,a2+b2)ca=(ca1,ca2)
In plain words: to add, add matching numbers. To scale, multiply every number by the scalar.
Mixing. Scale two vectors and add them, c₁a + c₂b, and you get a mix of them. With 0.25 and 0.75, the mix sits a quarter of the way from b to a: 0.25 × (8, 4) + 0.75 × (4, 12) = (5, 10). Mixes like this, with scalars that add to 1, are how a point between two others is found.
Any number of components. The pictures need two components to draw, but every move works the same in three dimensions, or three hundred: only the number of components changes. And the order of adding never matters, a + b = b + a, since each component is an ordinary sum: walking a then b ends in the same place as walking b then a. That's the parallelogram you dragged.
Quick check
What is 2a + 3b for a = (3, 1) and b = (1, 2)?
(6, 6)
(5, 3)
(4, 3)
(9, 8)
Show the answer
(9, 8)
2a = (6, 2) and 3b = (3, 6). Their sum is (9, 8).
Quick check
What does multiplying a vector by −0.5 do?
Turns it a quarter turn
Doubles it and reverses it
Halves it and reverses it
Halves it, same direction
Show the answer
Halves it and reverses it
The 0.5 halves its length, and the minus sign points it the opposite way.
In your own words
In your own words: why does b − a point from a to b?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: arithmetic on whole arrays
Purpose. NumPy does a vector operation on every component at once, in one short line.
import numpy as np
a = np.array([3, 1])
b = np.array([1, 2])
print(a + b, a - b, b - a)
print(2 * a, -1 * a, 0.5 * a)
print(2 * a + 3 * b)
It runs right here in your browser, with nothing to install.
+, - and * by a number all work element-wise: on each component, in place. There's no loop to write, and NumPy runs it fast even for vectors with millions of components.
Code flow. Step through the same sum two ways. First a loop adds one pair of components per pass, filling an array that np.zeros(2) starts as two zeros. Then a + b does both pairs in one line, and two more array lines subtract and scale. Watch the boxes: the loop fills total one box at a time, and fast arrives whole.
Vectorization. Writing one array line instead of a loop is called vectorization, and it matters for speed. The loop runs Python once per component; the array line hands the whole job to NumPy's compiled code. Time both on 100,000 components. time.perf_counter() reads a clock in seconds, and np.ones(n) makes an array of n ones:
import time
import numpy as np
n = 100000
a = np.ones(n)
b = np.ones(n)
start = time.perf_counter()
total = np.zeros(n)
for i inrange(n):
total[i] = a[i] + b[i]
loop_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
fast = a + b
array_ms = (time.perf_counter() - start) * 1000print(total[0], fast[0], total[n - 1], fast[n - 1])
print("loop:", round(loop_ms, 1), "ms array line:", round(array_ms, 3), "ms")
Both give 2.0, at the start and at the end. The times change from run to run and from computer to computer, but the loop is far slower every time. Real models do arithmetic on millions of numbers at each step, so their code is written as array lines, not Python loops.
A named mistake: sizes that don't match. Adding a 2-vector to a 3-vector has no meaning, and NumPy stops with an error that names both shapes. That error is your friend: it catches a mismatch the moment it happens. Check shapes with .shape whenever two vectors meet.
Array times array also works element-wise, multiplying matching components. That's useful later, but it isn't scaling, and it isn't the dot product of lesson 50:
.tolist() turns an array back into a list, which prints with commas.
Worked example
Worked example: a step, and a point between two others
Find the step from one point to another, and the point a quarter of the way along it.
import numpy as np
p = np.array([2, 5])
q = np.array([10, 1])
step = q - p
quarter = p + 0.25 * step
print(step.tolist(), quarter.tolist())
In the lesson you can change this and run it.
q − p is the arrow from p to q: (8, −4). Walking a quarter of it from p lands at (4, 4): two units across and one down from (2, 5). This "start, plus a fraction of the step" pattern is how animations move smoothly between two positions, and how a model's weights move partway toward a better spot.
Quick check
What does this print?
import numpy as np
a = np.array([4, -2])
print((a + a).tolist(), (3 * a - a).tolist())
[4, -2, 4, -2] [8, -4]
[8, 4] [8, 4]
[8, -4] [12, -6]
[8, -4] [8, -4]
Show the answer
[8, -4] [8, -4]
a + a doubles each component: [8, −4]. 3a − a is also 2a: [8, −4].
Quick check
Which line gives the vector from point p to point q?
q - p
p - q
p + q
q * p
Show the answer
q - p
End minus start: q − p is what you add to p to reach q.
Drill
A mix and a step
Fix the code: mix should be 0.25 × a + 0.75 × b, and step the vector from a to b.
import numpy as np
a = np.array([8, 4])
b = np.array([4, 12])
mix = 0.25 + a + 0.75 + b
step = a - b
print(mix.tolist(), step.tolist())
In the lesson, your code is checked as you write it.
In your own words
Why is NumPy's element-wise arithmetic a good match for vector math?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: learning is a vector update
Purpose. Every training step is one line of vector arithmetic.
A model's weights form a vector w. Training works out another vector, the gradient g, which points the way the loss climbs fastest. To make the loss smaller, step the other way, by a small multiple η (the learning rate):
w←w−ηg
In plain words: the new weights are the old weights minus a small step along the gradient. The arrow ← means "is replaced by".
Watch one update on two weights, component by component. The picture is the math part's tail to head: the step −0.1 g starts at the tip of w, and the new w runs from the origin to where the step lands.
Three steps in code. Training repeats the update many times. With a fixed gradient, each step moves the weights the same amount:
import numpy as np
w = np.array([1.0, 2.0])
g = np.array([0.5, -1.0])
for step inrange(3):
w = w - 0.2 * g
print(step + 1, w.round(3).tolist())
Real training works out a fresh gradient at every step, so the steps change size and direction as the weights improve. .round(3) tidies the float noise for printing.
The learning rate sets the step. η (eta) scales the whole gradient before it's subtracted. With η = 0.2 each step above moves the weights by 0.2 × (0.5, −1) = (0.1, −0.2). Double η and every step doubles; make it tiny and training crawls. Choosing η is one of the first decisions in any training run, and lesson 79 shows what happens when it's too big: the steps overshoot the low point and the weights bounce away from it.
Quick check
What does this print?
import numpy as np
w = np.array([2.0, -1.0])
g = np.array([4.0, 2.0])
print((w - 0.5 * g).tolist())
[4.0, 0.0]
[0.0, 0.0]
[-2.0, -4.0]
[0.0, -2.0]
Show the answer
[0.0, -2.0]
0.5 × g = (2, 1). w − (2, 1) = (0, −2).
In PyTorch. The same three steps, with tensors in place of arrays. The update line doesn't change at all:
import torch
w = torch.tensor([1.0, 2.0])
g = torch.tensor([0.5, -1.0])
lr = 0.2for step inrange(3):
w = w - lr * g
print(step + 1, w)
w = torch.tensor([1.0, 0.2], requires_grad=True)
w.grad = torch.tensor([2.0, -4.0])
opt = torch.optim.SGD([w], lr=0.1)
opt.step()
print(w)
The first three lines it prints hold the numbers the NumPy loop printed, 0.9 and 2.2, then 0.8 and 2.4, then 0.7 and 2.6, which PyTorch writes with four decimals. In real training nobody writes that line, though: an optimizer does it. The second half of the cell sets it up on the flow's numbers. requires_grad=True marks w as weights that training will change, and w.grad is where PyTorch keeps their gradient (here it's put there by hand; lesson 81 shows how PyTorch finds it). torch.optim.SGD([w], lr=0.1) is the optimizer, and opt.step() does w − 0.1·g to every weight. The last line is the flow's new w, (0.8, 0.6), with a note that it's still a weight training can change. That's the optimizer.step() on the model map.
Drill
Three training steps
Fix the loop so it does three updates w ← w − η·g with η = 0.1, and set final to the rounded result.
import numpy as np
w = np.array([1.0, 1.0])
g = np.array([2.0, -3.0])
eta = 0.1for step inrange(3):
w = w + eta * g
final = w.round(3).tolist()
print(final)
In the lesson, your code is checked as you write it.
In your own words
In w ← w − η·g, what do the minus sign and η each do?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write