Written math has an alphabet of symbols, and most people stall at the first unfamiliar one. This lesson is a phrase book. It starts with sets, the idea of a collection, and the symbols for "is in", "both" and "either". Then it reads the shorthand that opens almost every machine learning paper: ℝⁿ for a list of numbers, f : ℝⁿ → ℝ for a model, and the hats, bars and subscripts you've already met. By the end you can read a paper's first equation aloud.
What you'll be able to do
Read set notation: ∈, ⊆, ∪, ∩ and ∅, and work out unions and intersections
Name the number sets ℕ, ℤ and ℝ, and read ℝⁿ as lists of n real numbers
Read a function's signature, f : ℝⁿ → ℝᵐ, and the marks ML uses: hats, bars, subscripts and Greek letters
Use Python's set, in, | and &, and describe a model's inputs and outputs as sets
Build a vocabulary as a sorted set and turn words into ids, and read x ∈ ℝᵈ as a tensor's shape
Quick check
Have a guess before you start. A = {cat, dog} and B = {dog, owl}. How many different animals are in A or B, or both?
2
1
3
4
Show the answer
3
cat, dog and owl. dog is in both, but a collection counts each thing once: 3.
The math: collections, and the symbols for them
Purpose. Learn the alphabet of written math, starting with collections.
A set is a collection of different things, written in curly brackets: {2, 4, 6}. Order doesn't matter, and nothing is counted twice. A few symbols do almost all the work:
∈ means "is in": 4 ∈ {2, 4, 6}. ∉ means "isn't in".
⊆ means "is a subset of": every member of the first is in the second. {2, 4} ⊆ {2, 4, 6}.
∪, union, is everything in either set. ∩, intersection, is what's in both.
∅ is the empty set, with nothing in it.
|A|, a set between bars, is its size: how many members it has. |{2, 4, 6}| = 3.
Quick check
From an earlier lesson: what does the subscript in x₃ mean?
x raised to the power 3
Three copies of x side by side
The third item of x
x multiplied by 3
Show the answer
The third item of x
A subscript names a position: x₃ is the third term of x. It's one of the marks this lesson reads.
Union and intersection, one item at a time. To find A ∩ B, go through A and keep each item that's also in B. To find A ∪ B, take all of A, then add each item of B that isn't already there. Watch it on two short sentences, each turned into the set of its words.
The picture is a Venn diagram: one circle for each sentence, overlapping in the middle, where the words in both go. Each step checks one word of A. If B has it too, the word moves into the overlap and B's copy goes, because a set holds each word once. A word that only A has stays on A's side. The last step lights every word in either circle, the union, and counts them.
Try it now
Then put a word from sentence A into sentence B and press Use these numbers: the intersection grows and the union stays the same size.
The number sets. Some sets of numbers have their own letters, written with a double stroke:
ℕ, the natural numbers: the counting numbers 1, 2, 3, … (some books include 0).
ℤ, the integers: whole numbers with their negatives, … −2, −1, 0, 1, 2, ….
ℝ, the real numbers: every point on the number line, decimals included.
So −3 ∈ ℤ but −3 ∉ ℕ, and 0.5 ∈ ℝ but 0.5 ∉ ℤ. Each set sits inside the next: ℕ ⊆ ℤ ⊆ ℝ.
Quick check
Which is true?
−3 ∈ ℤ but −3 ∉ ℕ
0.5 ∈ ℤ
−3 ∈ ℕ
ℝ ⊆ ℕ
Show the answer
−3 ∈ ℤ but −3 ∉ ℕ
−3 is a whole number, so it's an integer, but it isn't a counting number.
Symbols and formula. Two pieces of shorthand open most ML papers:
x∈Rnf:Rn→Rm
In plain words: "x is a list of n real numbers", and "f takes a list of n numbers and gives back a list of m."
ℝⁿ is the set of all lists of n real numbers, so a point in ℝ³ is three numbers, like (1.5, −2, 0). The arrow line is a function's signature: what goes in and what comes out.
The marks on letters. A hat, ŷ, marks a prediction or estimate. A bar, x̄, marks a mean. A subscript, xᵢ, names a position; a superscript, x², is a power. And Greek letters are ordinary names: θ for a model's weights, η for a learning rate, σ for a spread, μ for a mean.
Quick check
How do you read f : ℝ² → ℝ out loud?
f takes one number and gives two
f is the square of a real number
f takes two numbers and gives one number
f is two times a real number
Show the answer
f takes two numbers and gives one number
The left of the arrow is what goes in, ℝ², a pair of numbers. The right is what comes out, ℝ, one number. For example, f(x, y) = x + y.
In your own words
In your own words: what's the difference between A ∪ B and A ∩ B?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: set, in, | and &
Purpose. Python has sets built in, with the same operations as the math.
A = {"cat", "dog"}
B = {"dog", "owl"}
print(len(A | B), sorted(A | B), sorted(A & B))
print("cat"in A, "owl"in A)
It runs right here in your browser, with nothing to install.
| is union and & is intersection; in is ∈. A set prints in no particular order, so sorted turns it into a list in alphabetical order for printing. len counts the members.
Code flow.& does the math part's sweep in one character. Here is the same sweep written out, on the two sentences from the flow: go through a and keep each word that's also inb. in works on a list too, where it checks every item. Step through and watch both collect the words that moved into the overlap. The last two lines build the union and count it.
Turning a list into a set removes repeats, which is how a vocabulary is found:
A named mistake: {} is not an empty set. In Python, {} makes an empty dictionary, a different kind of collection. An empty set is written set(). And a set has no positions, so vocab[0] is an error: to get members in order, use sorted(vocab), which gives a list.
Difference, and subsets.A - B keeps what's in A but not B, and <= asks whether one set is a subset of another: {2, 4} <= {2, 4, 6} is True. Every math symbol in this lesson has a one-character Python partner, which makes sets a handy way to check a hand calculation.
Worked example
Worked example: two vocabularies
Two short documents. Find each one's vocabulary, the words they share, and the combined vocabulary.
Each document has 6 words but only 5 different ones, because "the" appears twice. Together they have 7 different words: the union counts the shared ones once. The intersection, the words both use, is on, sat and the. A language model's vocabulary is built the same way, as the union of the words (or word pieces) across all of its training text.
The union is {1, 2, 3, 4, 5}: 5. The intersection is {2, 3}: 2. And 4 isn't in s1.
Quick check
Which line makes an empty set in Python?
s = ∅
s = {}
s = []
s = set()
Show the answer
s = set()
set() is an empty set. {} is an empty dictionary, and [] an empty list.
Drill
Shared and combined words
Fix the code so shared is the set of words in both lists, combined the number of different words in either, and has_hat whether "hat" is in the first list's set.
In the lesson, your code is checked as you write it.
In your own words
Why does turning a list into a set change its length sometimes?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: reading a paper's first equation
Purpose. Most papers state their setup in this lesson's symbols. Reading them is now just translation.
Here is how the setup of a paper reads, piece by piece:
A vocabulary is a set. V = {all the tokens a model knows}, and |V|, the size of V, might be 50,000.
An input is a point in ℝⁿ. A trail sign drawn on a 5 by 5 grid is 25 brightness values: x ∈ ℝ²⁵.
A model is a function between them. A classifier that sorts signs into three kinds (arrow, tent and water) is f : ℝ²⁵ → ℝ³: 25 numbers in, three scores out, one per kind.
Weights are a set of numbers too. θ ∈ ℝᵈ says the model has d adjustable numbers.
fθ:R25→R3,y^=fθ(x)
In plain words: a function with weights θ takes 25 numbers and gives 3; its output for an input x is the prediction ŷ.
Quick check
A paper says its model is f : ℝ⁷⁸⁴ → ℝ¹⁰. What does it take in and give out?
One number between 10 and 784
784 times 10 numbers
784 numbers in, 10 numbers out
10 numbers in, 784 out
Show the answer
784 numbers in, 10 numbers out
Left of the arrow is the input: a list of 784 numbers, like a 28 × 28 image. Right is the output: 10 numbers, like a score for each digit 0 to 9.
Translating a whole line. Here's a typical sentence from a paper, translated piece by piece. "Given a dataset {(xᵢ, yᵢ)}, i = 1 to n, with xᵢ ∈ ℝᵈ and yᵢ ∈ {0, 1}, we learn f_θ : ℝᵈ → [0, 1]." It says: there are n examples; each input is d numbers; each answer is 0 or 1; and the model takes d numbers and gives a score between 0 and 1. The square brackets [0, 1] are a range with both ends included, the interval from lesson 18. Nothing in it is new.
There are 5 labels in the list, but only 3 different kinds: arrow, tent and water. And arrow is one of them.
Words to ids. A model can't compute with words, only with numbers. So before it reads any text, each word in its vocabulary gets a number of its own, its id. One simple way: sort the vocabulary, a set, into a list, and use each word's position in the list as its id. Here is a line of the camp log, turned into ids:
log = ["the", "tent", "is", "up", "the", "fire", "is", "lit"]
vocab = sorted(set(log))
print(len(log), len(vocab), vocab)
ids = []
for w in log:
ids.append(vocab.index(w))
print(ids)
vocab.index(w) gives the position of w in the list: its id. In this lesson's symbols, V = {fire, is, lit, tent, the, up} and |V| = 6, so every id is a whole number from 0 to 5. The 8 words of the log become 8 numbers, and a word that repeats keeps its id: both copies of "the" are 4. Sorting matters here: a set has no order of its own, so sorting gives the same ids on every run. Real models split text into word pieces, with a |V| near 50,000, and fix each piece's id once, when the vocabulary is built. Lesson 47 then gives each id a list of numbers of its own, an embedding.
In the real tools. A paper's "x ∈ ℝ²⁵" becomes a line real code can check. A tensor's .shape lists how many numbers it holds along each direction. Here is a trail sign flattened to 25 numbers (all zeros, standing in for its brightness values):
Drill
A model's sets, in code
A sign reader takes 5 × 5 pictures and sorts them into the kinds in kinds. Set n_in to its input size n (for ℝⁿ), n_out to its output size (one score per kind), and n_seen to how many different kinds have turned up in labels_seen so far. Careful: not every kind has turned up yet.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write