Adding, taking away and scaling work on vectors the same way they work on numbers, so most vector formulas turn around one component at a time. Two things are new. You can't divide by a vector, so a scale is found by comparing components. And an angle hides inside the dot product, freed by arccos. These moves recover the gradient a training step used, the mix of two embeddings, and the angle behind a similarity cutoff.
What you'll be able to do
Solve a + x = b for the unknown vector, one component at a time, and check by adding back
Find the scale k in k·v = w by comparing components, and say when no k exists because w isn't a multiple of v
Turn the update rule, the unit vector and the mix of two vectors around for any letter, without dividing by a vector
Free the angle, a length or the projection from a·b = ‖a‖‖b‖cos θ, with arccos clipped to [−1, 1] in code
Recover a gradient from two weight snapshots, a mix of two embeddings, a similarity cutoff's angle and momentum's steady state
Quick check
Have a guess before you start. a + x = b, with a = (2, 5) and b = (7, 3). What is x?
(3.5, 0.6)
(−5, 2)
(9, 8)
(5, −2)
Show the answer
(5, −2)
Take a away from both sides: x = b − a = (7 − 2, 3 − 5) = (5, −2). Check: (2, 5) + (5, −2) = (7, 3).
The math: undo each move, one component at a time
Purpose. Make a vector, a scale or an angle the subject of a vector formula. This is transposing, with arrows.
A quick recap of the notation. A bold letter like v is a vector, a list of numbers called its components, so v = (v₁, v₂). The length of v is ‖v‖ (lesson 49), and a·b is the dot product (lesson 50).
Adding and taking away. A vector equation is one equation per component. a + x = b says a₁ + x₁ = b₁ and a₂ + x₂ = b₂. Undo each one: x = b − a. Nothing new is needed.
Scaling. k·v = w, where k is a plain number, also says two things: k·v₁ = w₁ and k·v₂ = w₂. Each gives its own answer, k = w₁/v₁ and k = w₂/v₂. A k exists only if every component gives the same one. Take v = (2, 6). For w = (5, 15), both give 2.5, so k = 2.5. For w = (5, 14), one gives 2.5 and the other about 2.33. No single k works: w isn't a multiple of v, so it points along a different line. We say w isn't parallel to v.
Quick check
From an earlier lesson: what is (6, 2) − (1, 4)?
(6, 8)
(7, 6)
(−5, 2)
(5, −2)
Show the answer
(5, −2)
Take away matching components: 6 − 1 = 5 and 2 − 4 = −2. So (5, −2).
Each component is an ordinary number, so the machine from lesson 30 works on one at a time. Here is the first weight of a training step, w_new = w − ηg, with w = 0.5 and a learning rate η = 0.1. The gradient g goes in, and the way back recovers it from the new weight.
Try it now
Try a negative gradient, like −3: the weight goes up, and the way back still finds −3.
Symbols and formula. The three moves so far:
x=b−ak=v1w1=v2w2=⋯g=ηw−wnew
In plain words: take away a vector by taking away each part. Find a scale from any part, then check that every part agrees. To get the gradient back, take the new weights from the old ones and divide by the learning rate.
Named mistake: dividing by a vector. From k·v = w it's tempting to write k = w/v. There is no such division: w/v isn't one number. NumPy will happily divide part by part and hand you a list, and if the parts differ, no k exists. The same goes for η. From w − w_new = ηg, the answer isn't (w − w_new)/g. Use one component with g₁ ≠ 0, η = (w₁ − w_new,₁)/g₁, then check the other components give the same η. Dividing by η or by k is fine: they're numbers, as long as they aren't 0.
Quick check
Find k with k·(4, −2) = (−6, 3).
−0.67
−1.5
1.5
No k exists
Show the answer
−1.5
First part: −6 ÷ 4 = −1.5. Second part: 3 ÷ (−2) = −1.5. They agree, so k = −1.5. Check: −1.5 × (4, −2) = (−6, 3).
The unit vector. v̂ = v/‖v‖ divides v by its length, which is a number, so it's allowed when v isn't the zero vector. Undo it by multiplying: v = ‖v‖·v̂. A direction and a length rebuild the arrow: 5 × (0.6, 0.8) = (3, 4).
Mixing. m = (1 − t)a + t·b is the point a fraction t of the way from a to b. At t = 0 it's a, and at t = 1 it's b. To free t, expand: m = a + t(b − a). Take a away: m − a = t(b − a). That's a scale problem: t is the number that turns b − a into m − a. So t = (m₁ − a₁)/(b₁ − a₁), and every component must agree, which says m lies on the line through a and b.
To free a instead, take t·b away: m − t·b = (1 − t)a. Divide by the number 1 − t: a = (m − t·b)/(1 − t).
Conditions. t needs a ≠ b, or there's no line to measure along. a needs t ≠ 1: at t = 1 the mix is just b and holds no trace of a.
Quick check
m = (1 − t)a + t·b with t = 0.25, b = (20, 0) and m = (5, 7.5). What is a?
(0, 7.5)
(0, 30)
(0, 10)
(−20, 10)
Show the answer
(0, 10)
m − t·b = (5 − 5, 7.5 − 0) = (0, 7.5). Divide by 1 − t = 0.75: a = (0, 10). Check: 0.75 × (0, 10) + 0.25 × (20, 0) = (5, 7.5).
The angle in a dot product. From lesson 51, a·b = ‖a‖‖b‖cos θ. The angle θ sits inside a cos, which is multiplied by both lengths. Peel in reverse: divide by the lengths, then take arccos. So θ = arccos(a·b/(‖a‖‖b‖)).
arccos gives an angle from 0 to π, that is 0° to 180°: the principal angle. The angle between two arrows is always in that range, so no ± is needed. The same formula gives a length too: ‖b‖ = a·b/(‖a‖cos θ). That fails at 90°, where a·b = 0 whatever ‖b‖ is.
The projection length, how far a reaches along b, is a·b/‖b‖, which is ‖a‖cos θ. Turned around, a·b = (projection) × ‖b‖, and θ = arccos(projection/‖a‖).
Conditions. A zero vector has no direction: no angle, no unit vector. Watch θ = 4 radians go through below.
In your own words
A square root needs a ± when you undo a square. Why does arccos not need one when you free the angle between two arrows?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: each rearrangement in one line
Purpose. NumPy does each part-by-part move in one line, and a dot product turns a vector equation into a number equation you can divide.
import numpy as np
a = np.array([2.0, 5.0])
b = np.array([7.0, 3.0])
x = b - a
print(x, a + x)
v = np.array([2.0, 6.0])
w = np.array([5.0, 15.0])
print(w / v)
k = (v @ w) / (v @ v)
print(k, np.allclose(k * v, w))
It runs right here in your browser, with nothing to install.
b - a takes away part by part, and adding a back gives b.
w / v divides part by part. That's a list of answers for k, one per part, not a k. Here they agree.
The dot trick. Dot both sides of k·v = w with v: k(v·v) = v·w. That's an equation in numbers, so divide by v·v, which is 0 only for the zero vector. Then np.allclose checks that k·v really is w. If w isn't parallel to v, the trick still gives a number, the scale of w's projection onto v, but the check says False.
Worked example
Worked example: an angle that rounding breaks
θ = arccos(a·b/(‖a‖‖b‖)) with its two conditions in code: no zero vectors, and a cosine clipped to [−1, 1]. np.clip(c, -1, 1) clamps a number into the range from −1 to 1: anything above 1 becomes 1, anything below −1 becomes −1, and anything in between is left alone.
import numpy as np
defangle(a, b):
na = np.linalg.norm(a)
nb = np.linalg.norm(b)
if na == 0or nb == 0:
returnNone
c = (a @ b) / (na * nb)
return np.arccos(np.clip(c, -1, 1))
c = 1.0000000000000002print(c, np.clip(c, -1, 1), np.clip(-3.5, -1, 1), np.clip(0.25, -1, 1))
a = np.array([2.0, 3.0])
print(round(angle(a, 2 * a), 6), round(np.degrees(angle(a, np.array([3.0, -2.0]))), 1), angle(a, np.zeros(2)))
In the lesson you can change this and run it.
Two arrows that point the same way, like a and 2a, should have a cosine of exactly 1. Rounding in the lengths can leave it a hair either side, and a value like 1.0000000000000002 is past 1. arccos of anything past 1 doesn't exist: NumPy would return nan, "not a number". np.clip pulls it back to 1.0, and it leaves 0.25 alone. So the angle between a and 2a comes out as 0.0, rounded to 6 places because a cosine a hair under 1 gives a tiny angle, not exactly 0. (3, −2) is at a right angle to (2, 3), and the function gives 90.0 degrees. The zero vector has no direction, so the answer is None.
Quick check
What does this print?
import numpy as np
v = np.array([2.0, 4.0])
w = np.array([5.0, 12.0])
print((w / v).tolist(), np.allclose(2.5 * v, w))
An error
[2.5, 3.0] True
2.5 True
[2.5, 3.0] False
Show the answer
[2.5, 3.0] False
w / v divides part by part: 5 ÷ 2 = 2.5 and 12 ÷ 4 = 3.0. The parts disagree, so no k works, and 2.5 × (2, 4) = (5, 10) isn't w.
Quick check
a's projection onto b is 3 long, and ‖a‖ = 6. What is the angle between them?
60°
0.5°
120°
30°
Show the answer
60°
The projection is ‖a‖cos θ, so cos θ = 3 ÷ 6 = 0.5 and θ = arccos 0.5 = 60°.
Drill
Fix the scale and the angle
scale_between(v, w) should return the k with k·v = w, or None when there's no such k. angle(a, b) should give the angle in radians, even when rounding pushes the cosine past 1. Fix both.
import numpy as np
defscale_between(v, w):
return w / v
defangle(a, b):
c = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
return np.arccos(c)
print(scale_between(np.array([2.0, 6.0]), np.array([5.0, 15.0])), angle(np.array([1.0, 0.0]), np.array([0.0, 2.0])))
In the lesson, your code is checked as you write it.
In your own words
The dot trick always gives a number. Why check np.allclose(k * v, w) afterward?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: reading training and search backward
Purpose. Training logs and embedding searches give you the result of a vector formula. Rearranging gets back what went in.
The gradient from two snapshots. A training step is w_new = w − ηg (lesson 48). Save the weights before and after a step, and with the learning rate you get the gradient back: g = (w − w_new)/η. That's how you check an optimizer did what you think.
The mix of two embeddings. An embedding is a word as a vector. Blend "tent" and "stove" as m = (1 − t)·tent + t·stove. With a = tent and b = stove, the dot trick recovers t from m: t = (m − a)·(b − a)/((b − a)·(b − a)).
import numpy as np
w_before = np.array([0.50, -1.20, 0.05])
w_after = np.array([0.47, -1.16, 0.05])
lr = 0.1
g = (w_before - w_after) / lr
print(np.round(g, 4), np.allclose(w_before - lr * g, w_after))
tent = np.array([0.9, 0.1, 0.0])
stove = np.array([0.1, 0.9, 0.0])
m = np.array([0.34, 0.66, 0.0])
d = stove - tent
t = ((m - tent) @ d) / (d @ d)
print(round(t, 4), np.allclose((1 - t) * tent + t * stove, m))
The third weight didn't move, so its gradient was 0. The mix sits 70% of the way from tent to stove.
Quick check
Two snapshots: w = (1.0, 2.0), w_new = (0.9, 2.2), and η = 0.5. What was g?
(0.2, −0.4)
(0.05, −0.1)
(−0.2, 0.4)
(0.1, −0.2)
Show the answer
(0.2, −0.4)
w − w_new = (0.1, −0.2). Divide by 0.5: (0.2, −0.4). The second weight went up, so its gradient was negative.
The angle behind a similarity cutoff. A search keeps documents whose cosine similarity to the query is at least 0.8. As an angle, that's θ = arccos 0.8 ≈ 36.87°: anything within about 37° of the query passes. Going the other way, "within 30°" is a cutoff of cos 30° ≈ 0.866. cos falls as θ grows, so a higher cutoff is a narrower cone. In NumPy, np.arccos answers in radians and np.degrees turns radians into degrees. np.radians goes the other way, degrees into radians, which np.cos needs.
Momentum at its steady state. Momentum keeps a running velocity, v = βv + g, with β around 0.9. If g stays the same, v settles. v appears twice, so gather it (lesson 32): v − βv = g, so v(1 − β) = g, and v = g/(1 − β). With β = 0.9 that's 10 times the gradient: momentum takes steps up to 10 times bigger. Turned around, g = (1 − β)v. The condition: β ≠ 1. At β = 1 nothing is ever forgotten and v grows without end.
Quick check
What does this print?
import numpy as np
print(round(np.degrees(np.arccos(0.8)), 2), round(np.cos(np.radians(30)), 3))
0.64 0.154
53.13 0.866
36.87 0.5
36.87 0.866
Show the answer
36.87 0.866
arccos 0.8 is 0.6435 radians, which is 36.87°. cos 30° ≈ 0.866. A cutoff of 0.8 and a cone of 37° say the same thing.
Drill
Read the optimizer backward
Write grad_from(w, w_new, lr), the gradient a step used, and steady_v(g, beta), where v = βv + g settles, returning None when beta is 1.
import numpy as np
defgrad_from(w, w_new, lr):
return (w_new - w) * lr
defsteady_v(g, beta):
return g * (1 - beta)
print(grad_from(np.array([1.0, 2.0]), np.array([0.9, 2.2]), 0.5), steady_v(0.2, 0.9))
In the lesson, your code is checked as you write it.
In your own words
Why does momentum's velocity settle at 10 times the gradient when β = 0.9, and what goes wrong at β = 1?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
lesson 48 for adding, scaling, mixing and the update rule.
lesson 51 for the angle, cosine similarity and projection.
lesson 79 puts the update rule to work, step after step.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
Learn to make any letter of any formula the subject: layers undone in reverse order, fractions, a letter that appears twice, powers, roots, quadratics, logs, inequalities, trigonometry, vectors, matrices, calculus, probability, statistics and the formulas inside AI models, each checked in Python.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write