The dot product mixes two things together: how long the vectors are and how much they point the same way. Often only the direction matters. Two documents about tents should count as similar whether one is a page long and the other is a book. Dividing the dot product by both lengths removes the length and leaves the cosine of the angle between the vectors, a number from −1 to 1. That number, cosine similarity, is how search engines, recommenders and chat assistants find the text that means the same thing as your question. This lesson builds it from lessons 49 and 50, adds the projection (how far one arrow reaches along another), and ends by searching a tiny dictionary of camping words by meaning.
What you'll be able to do
Find the angle between two vectors from cos θ = a·b / (‖a‖‖b‖)
Read cosine similarity from −1 to 1, and know that 0 means perpendicular
Find how far one vector reaches along another: its projection
Rank words by cosine similarity to a query, the core of search by meaning, in NumPy and with PyTorch's F.cosine_similarity and F.normalize
Quick check
Have a guess before you start. The arrows (1, 2) and (2, 4) have different lengths. How similar are their directions?
Identical: one is the other doubled
Half as similar, because one is twice as long
Opposite
Unrelated
Show the answer
Identical: one is the other doubled
(2, 4) is 2 × (1, 2): the same direction, stretched. Cosine similarity says 1, the most similar possible.
The math: the angle hidden in the dot product
Purpose. Compare directions regardless of length.
The dot product has a second formula, using the angle θ (theta) between the two arrows:
a⋅b=∥a∥∥b∥cosθ
In plain words: the dot product is length times length times the cosine of the angle between them.
Divide both sides by the two lengths and the angle is all that's left. For a = (3, 4) and b = (4, 3): the dot product is 12 + 12 = 24, and both lengths are 5, so cos θ = 24 ÷ 25 = 0.96. The arrows lean almost together.
Quick check
From lesson 43: what is cos 90°?
0
0.5
−1
1
Show the answer
0
At 90° the point on the unit circle is straight up, (0, 1), so its x-coordinate, the cosine, is 0.
The picture. Step through (3, 4) and (4, 3). The drawing puts both arrows at the origin, at their true angle. Watch the dot product bring in the angle θ between them, marked by an arc. Then each length is written along its arrow, and the division cancels both lengths and leaves only the angle.
Try it now
Then set vector a to 6, 8 and vector b to 3, 4, and press Use these numbers: one is twice the other, the arrows lie on one line, and the cosine is exactly 1.
Symbols and formula. Cosine similarity is the dot product of the two arrows divided by both lengths:
cosθ=∥a∥∥b∥a⋅b
Reading the scale. 1 means the same direction, 0 means perpendicular (90°), and −1 means opposite (180°). (1, 0) and (1, 1) give 1 ÷ (1 × √2) ≈ 0.707, and the angle whose cosine is 0.707 is 45°. Length never matters: (1, 2) and (2, 4) give 1, and (1, 2) and (−2, −4) give −1.
The named mistake. Comparing raw dot products. (6, 8) · (3, 4) = 50 is bigger than (3, 4) · (3, 4) = 25, yet both pairs point exactly the same way. The bigger number came from longer arrows, not closer directions.
Quick check
What is the cosine similarity of (2, 0) and (0, 5)?
10
0.4
0
1
Show the answer
0
The dot product is 2 × 0 + 0 × 5 = 0: one points along x, the other along y. Perpendicular arrows have cosine 0.
Projection: how far one arrow reaches along another. Shine a light straight down onto b's line, and a casts a shadow on it. The shadow's length is the scalar projection of a onto b (the widget above showed it the other way round, b's shadow on a):
proj length=∥b∥a⋅bproj vector=b⋅ba⋅bb
In plain words: the length is the dot product divided by the length of the arrow you project onto. The vector is that much of b's direction.
Project a = (3, 1) onto b = (1, 1): the dot product is 4 and ‖b‖ = √2, so the shadow is 4 ÷ √2 ≈ 2.83 long, and the vector is (4 ÷ 2) × (1, 1) = (2, 2). Lesson 52 uses these shadows as coordinates.
Quick check
How far does (5, 2) reach along the x-axis, b = (1, 0)?
√29
7
5
2
Show the answer
5
(5, 2) · (1, 0) = 5, and ‖b‖ = 1, so the shadow on the x-axis is 5: just the x-coordinate.
In your own words
In your own words: why does dividing by both lengths make cosine similarity ignore length?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a cosine function from two earlier pieces
Purpose. Write cosine similarity once, from @ and np.linalg.norm, and use it to rank.
import numpy as np
defcosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
a = np.array([3.0, 4.0])
print(round(cosine(a, np.array([4.0, 3.0])), 3))
print(round(cosine(a, np.array([6.0, 8.0])), 3))
print(round(cosine(a, np.array([-4.0, 3.0])), 3))
It runs right here in your browser, with nothing to install.
The three answers are 0.96 (leaning together), 1.0 (same direction, twice as long) and 0.0 (perpendicular). The rounding keeps float noise like 0.9999999999999998 out of the printout.
Code flow. Here is the inside of cosine written out for (3, 4) and (4, 3), one piece per line. Step through it and watch dot, then the two lengths na and nb, then cos: the same cells the math part's flow filled.
Unit vectors make it a plain dot product. Normalize both arrows first (lesson 49) and their lengths are 1, so the cosine is just their dot product. Search systems store their vectors normalized for exactly this reason: one @ per comparison.
import numpy as np
a = np.array([3.0, 4.0])
b = np.array([4.0, 3.0])
a_hat = a / np.linalg.norm(a)
b_hat = b / np.linalg.norm(b)
print(round(a_hat @ b_hat, 3))
A named mistake: a zero vector. (0, 0) has length 0, so the formula divides by 0 and NumPy prints nan, "not a number", with a warning. A zero vector has no direction; check for it before you compare.
Worked example
Worked example: rank four arrows by similarity to a query
Score each of four vectors against the query (1, 1) and print them from most to least similar.
import numpy as np
defcosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
query = np.array([1.0, 1.0])
names = ["east", "north-east", "north", "south-west"]
vecs = [np.array([2.0, 0.0]), np.array([3.0, 3.0]), np.array([0.0, 5.0]), np.array([-1.0, -1.0])]
scores = []
for v in vecs:
scores.append(round(cosine(query, v), 3))
best = 0for i inrange(len(vecs)):
print(names[i], scores[i])
if scores[i] > scores[best]:
best = i
print("most similar:", names[best])
In the lesson you can change this and run it.
north-east scores 1.0 even though it's three times longer than the query: same direction. east and north both sit 45° away and score 0.707. south-west points exactly backward and scores −1.0. Two lists kept side by side, names and vectors, share their positions, so position 1 is both "north-east" and (3, 3). The search for the best is lesson 19's: keep the position of the best so far, and replace it when a score beats it.
Quick check
What does this print?
import numpy as np
a = np.array([2.0, 0.0])
b = np.array([3.0, 3.0])
print(a @ b, round((a @ b) / (np.linalg.norm(a) * np.linalg.norm(b)), 3))
6.0 0.707
12.0 0.707
0.707 6.0
6.0 1.0
Show the answer
6.0 0.707
The dot product is 2 × 3 + 0 × 3 = 6. The lengths are 2 and √18, so the cosine is 6 ÷ (2 × 4.243) ≈ 0.707: a 45° angle.
Drill
A cosine that refuses zero vectors
Fix cosine(a, b): it should return None when either vector has length 0, and otherwise divide the dot product by both lengths. Then set best to the name of the vector most similar to query.
import numpy as np
defcosine(a, b):
return (a @ b) / np.linalg.norm(a)
query = np.array([0.0, 2.0])
names = ["up", "right", "up-right"]
vecs = [np.array([0.0, 7.0]), np.array([3.0, 0.0]), np.array([1.0, 1.0])]
best = "right"print(cosine(query, vecs[0]), cosine(query, np.array([0.0, 0.0])), best)
In the lesson, your code is checked as you write it.
In your own words
Why might a search system normalize every stored vector once, ahead of time?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: search by meaning
Purpose. Find the words, sentences or documents that mean the same as a query.
Models turn words into vectors, called embeddings (lesson 47), placed so that words used in similar ways point in similar directions. Here are six camp-log words as tiny 3-number vectors, with one number each for shelter, cooking and finding the way. Real embeddings have hundreds of numbers, but the search works the same.
import numpy as np
defcosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
names = ["tent", "tarp", "stove", "kettle", "map", "compass"]
vecs = [np.array([0.9, 0.1, 0.0]), np.array([0.8, 0.0, 0.2]), np.array([0.1, 0.9, 0.0]),
np.array([0.0, 0.8, 0.1]), np.array([0.1, 0.0, 0.9]), np.array([0.0, 0.1, 0.8])]
q = vecs[2]
for i inrange(len(names)):
if names[i] != "stove":
print(names[i], round(cosine(q, vecs[i]), 3))
"kettle" scores 0.986 and every other word scores under 0.25: the nearest meaning to "stove" is the other cooking word. That's retrieval, the core of semantic search and of the systems that look up documents before a chatbot answers. A search shows its results ranked, best first. Here are the five scores as bars:
Quick check
Which word is closest in meaning to "map"? This prints the cosine of map with compass, then with tarp.
import numpy as np
defcosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
m = np.array([0.1, 0.0, 0.9])
print(round(cosine(m, np.array([0.0, 0.1, 0.8])), 3), round(cosine(m, np.array([0.8, 0.0, 0.2])), 3))
1.0 0.0
0.986 0.348
0.348 0.986
0.986 0.986
Show the answer
0.986 0.348
compass (printed first) scores 0.986 and tarp 0.348, so compass is closest. Both lean mostly on the third number, finding the way, but compass leans almost entirely on it.
Why cosine, not distance or raw dot product? A long document repeats its words more, so its vector can be longer without meaning anything different. Cosine similarity ignores that. A query vector that's twice as long should find the same answers, and with cosine it does. Many systems normalize every vector once and then use the dot product, which then equals the cosine, as in Part 2. Search engines built on embeddings do exactly this, at a scale of millions of stored vectors.
In PyTorch. The same search with tensors. F.cosine_similarity(x, y, dim=0) is this lesson's formula for two vectors. F.normalize(words, dim=1) divides each row by its own length, so every word becomes a unit vector, as Part 2 did by hand. Then one @ scores every word at once: it dots each row of unit with stove's unit vector, one dot product per row (lesson 55 takes this apart). torch.round(..., decimals=3) rounds like NumPy's round.
The first line is kettle against stove: 0.986, the number the NumPy cell printed. The second scores all six words in the order tent, tarp, stove, kettle, map, compass: 0.22, 0.107, 1.0, 0.986, 0.012 and 0.123, the NumPy cell's numbers again, plus stove against itself, 1.0. Normalize once, then every search is one @.
Drill
The two nearest words
Write nearest_two(query, names, vecs) returning the names of the two words most similar to query by cosine, best first, skipping any word whose vector equals the query.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write