Foundation math
From whole numbers to the math inside a neural network: each idea by hand, in Python, and in AI. 118 lessons, about 96 hours of study, in nine tracks. All of them are free to try without an account.
Where the math lives.
One diagram of a model learning, the same in every lesson. Each lesson lights the parts its math is used in; here every part links to the math behind it and the lessons that teach it.
The math behind each part
- data: examples and their answers, split before training. In PyTorch:
Dataset. Taught in lesson 9, Ratios, rates and proportion, lesson 97, Describing data: center, spread and scale. - numbers: every input written as numbers: pixels, ids, measurements. In PyTorch:
torch.tensor(x). Taught in lesson 1, Numbers and place value, lesson 7, Decimals and rounding, lesson 12, Bits, bytes and floats: how numbers are stored, lesson 28, Sets, and reading the symbols, lesson 47, Vectors: lists of numbers that point, lesson 53, Matrices: tables of numbers, lesson 65, One-hot vectors and lookup tables: embeddings, lesson 94, Two variables at once: covariance and correlation, lesson 97, Describing data: center, spread and scale. - batch: many examples at once, a matrix with one row each. In PyTorch:
DataLoader(data, batch_size=32). Taught in lesson 2, Multiplying and dividing: groups, lesson 16, Solving equations: keep the balance, lesson 17, Rearranging a formula, lesson 54, Adding, scaling and transposing matrices, lesson 56, Matrix multiplication, lesson 64, Tensors, shapes and broadcasting, lesson 98, Samples, the law of large numbers and the central limit theorem. - vectors: each word or example as a list of numbers with a direction. In PyTorch:
nn.Embedding(vocab, d). Taught in lesson 41, Coordinates and distance, lesson 42, Angles, degrees and radians, lesson 43, Sine and cosine, lesson 44, Waves: height, speed and shift, lesson 47, Vectors: lists of numbers that point, lesson 51, Angle, cosine similarity and projection, lesson 52, Perpendicular directions: bases and coordinates, lesson 62, Eigenvectors: directions that only stretch, lesson 65, One-hot vectors and lookup tables: embeddings. - layer: a matrix product plus a bias: XW + b. In PyTorch:
nn.Linear(d_in, d_out). Taught in lesson 3, Order of operations, and names for numbers, lesson 14, Letters for numbers, lesson 20, Chaining and undoing functions, lesson 21, Straight lines: slope and intercept, lesson 22, Two equations, two unknowns, lesson 50, The dot product, lesson 53, Matrices: tables of numbers, lesson 55, A matrix times a vector, lesson 56, Matrix multiplication, lesson 58, Matrices as movements of space, lesson 59, Determinant and inverse, lesson 66, Sliding dot products: convolution, lesson 110, A layer and a small network as matrices. - activation: a bend after each layer, like relu(z) = max(0, z). In PyTorch:
F.relu(z). Taught in lesson 18, Inequalities and true-or-false, lesson 20, Chaining and undoing functions, lesson 71, Limits: what a value heads toward, lesson 110, A layer and a small network as matrices. - attention: softmax(QKᵀ/√d)V: each word a weighted average of the others. In PyTorch:
F.softmax(q @ k.T / d ** 0.5, dim=-1) @ v. Taught in lesson 57, Identity, diagonal, triangular and symmetric, lesson 87, Counting: choices, orders and groups, lesson 113, Attention as matrix math. - scores: one number per answer, any size (the logits). In PyTorch:
logits = model(x). Taught in lesson 24, Exponentials: growth and decay, lesson 65, One-hot vectors and lookup tables: embeddings, lesson 102, Logistic regression: a line that answers yes or no, lesson 110, A layer and a small network as matrices. - softmax: eᶻ shared out so the scores add to 1. In PyTorch:
F.softmax(logits, dim=-1). Taught in lesson 104, Softmax: scores into probabilities. - chances: a probability for every answer, adding to 1. In PyTorch:
probs. Taught in lesson 6, Multiplying and dividing fractions, lesson 25, Logarithms: the power question, lesson 88, Probability: from counts to chances, lesson 90, Conditional probability and Bayes' rule, lesson 102, Logistic regression: a line that answers yes or no. - targets: the right answers the model is judged against. In PyTorch:
y. Taught in lesson 4, Numbers below zero, lesson 105, Surprise, entropy and cross-entropy. - loss: one number for how wrong: squared error, or −log of the right answer's chance. In PyTorch:
F.mse_loss(pred, y) or F.cross_entropy(logits, y). Taught in lesson 4, Numbers below zero, lesson 10, Powers and roots, lesson 15, Expanding and factoring, lesson 19, Functions: a rule from input to output, lesson 22, Two equations, two unknowns, lesson 23, Quadratics: bowls and their lowest point, lesson 25, Logarithms: the power question, lesson 26, Sequences and Σ, lesson 27, Weighted and moving averages, lesson 60, Solving systems by elimination, lesson 70, Rates of change, lesson 75, Lowest points and curvature, lesson 92, Expected value, variance and standard deviation, lesson 100, Maximum likelihood, lesson 101, Linear regression: the best line, lesson 105, Surprise, entropy and cross-entropy, lesson 106, KL divergence: how far one distribution is from another. - slopes: how the loss changes with each number, by the chain rule. In PyTorch:
loss.backward(). Taught in lesson 72, The derivative: slope at a point, lesson 73, Derivative rules, lesson 74, The product rule and the chain rule, lesson 77, Partial derivatives, lesson 81, The chain rule with many paths: backpropagation by hand, lesson 83, Jacobians and Hessians, lesson 82, Automatic differentiation: how PyTorch finds every slope, lesson 109, One neuron, start to finish. - gradient: all the slopes together, pointing uphill. In PyTorch:
w.grad. Taught in lesson 49, Length and distance: norms, lesson 78, The gradient, lesson 82, Automatic differentiation: how PyTorch finds every slope. - update: w − η·g: a small step downhill. In PyTorch:
optimizer.step(). Taught in lesson 11, Very big and very small numbers, lesson 16, Solving equations: keep the balance, lesson 24, Exponentials: growth and decay, lesson 27, Weighted and moving averages, lesson 48, Adding and scaling vectors, lesson 49, Length and distance: norms, lesson 76, Standing in for a curve near a point, lesson 79, Gradient descent, lesson 80, Faster descent: momentum and Adam, lesson 109, One neuron, start to finish. - weights: the numbers training changes. In PyTorch:
model.parameters(). Taught in lesson 61, Independence, span and rank, lesson 63, SVD and PCA: the important directions first, lesson 93, Bernoulli, binomial, uniform and normal. - train and test split: some examples held back to judge on. In PyTorch:
random_split(data, [0.8, 0.2]). Taught in lesson 9, Ratios, rates and proportion, lesson 101, Linear regression: the best line. - init: the weights' starting values: small random numbers, scaled to the layer's size. In PyTorch:
nn.init.normal_(w, mean=0.0, std=0.02). Taught in lesson 93, Bernoulli, binomial, uniform and normal. - penalty: λ‖w‖² added to the loss, so training prefers small weights. In PyTorch:
loss = loss + lam * (w ** 2).sum(). Taught in lesson 49, Length and distance: norms, lesson 59, Determinant and inverse. - sampling: drawing the next word by its chance. In PyTorch:
torch.multinomial(probs, 1). Taught in lesson 88, Probability: from counts to chances, lesson 91, Random variables and distributions. - metrics and error bars: accuracy and its likely range. In PyTorch:
(pred == y).float().mean(). Taught in lesson 5, Fractions: equal parts, equal fractions, adding, lesson 8, Percentages, lesson 18, Inequalities and true-or-false, lesson 90, Conditional probability and Bayes' rule, lesson 84, Integrals: adding up thin slices, lesson 98, Samples, the law of large numbers and the central limit theorem, lesson 103, Scoring a classifier: confusion table, precision, recall, F1, lesson 99, Is the difference real? Comparing two models. - memory: count × bytes per number. In PyTorch:
x.numel() * x.element_size(). Taught in lesson 10, Powers and roots, lesson 11, Very big and very small numbers, lesson 12, Bits, bytes and floats: how numbers are stored. - cost: how the work grows with length and size. In PyTorch:
x.numel(). Taught in lesson 87, Counting: choices, orders and groups, lesson 89, Combining events: and, or, independent.
A real training loop, line by line.
These eight lines are how most of today's models learn. By the end you'll know the math behind every one of them, and you'll have written the same lines yourself.
model = nn.Sequential(nn.Linear(25, 8), nn.ReLU(), nn.Linear(8, 3))Lesson 110: A layer and a small network as matricesopt = torch.optim.Adam(model.parameters(), lr=1e-3)Lesson 80: Faster descent: momentum and Adamfor x, y in loader:Lesson 64: Tensors, shapes and broadcastinglogits = model(x)Lesson 56: Matrix multiplicationloss = F.cross_entropy(logits, y)Lesson 105: Surprise, entropy and cross-entropyopt.zero_grad()Lesson 82: Automatic differentiation: how PyTorch finds every slopeloss.backward()Lesson 74: The product rule and the chain ruleopt.step()Lesson 79: Gradient descent
Nine tracks, in order.
118 lessons, about 96 hours of study. Open a lesson to read it, or start at the first.
Numbers
- Numbers and place value FreeWhat a digit's place is worth, the number line, and Python as a calculator that never tires.36 min
- Multiplying and dividing: groups FreeEqual groups, quotient and remainder, and cutting training data into batches.35 min
- Order of operations, and names for numbers FreeBrackets, powers, then multiply and divide, then add and subtract; variables hold numbers.36 min
- Numbers below zero FreeThe number line through zero, sign rules, distance from zero, and an error's direction and size.37 min
- Fractions: equal parts, equal fractions, adding FreePieces of a whole, equal fractions, and re-cutting to add; accuracy as a fraction.45 min
- Multiplying and dividing fractions FreeA share of a share, dividing by flipping, and why chances in a row get small.35 min
- Decimals and rounding FreeFractions in place value, rounding, and why 0.1 + 0.2 isn't exactly 0.3.36 min
- Percentages FreePer hundred, percent change, percentage points, and the accuracy and error rates of models.35 min
- Ratios, rates and proportion FreeSharing in a ratio, rates per unit, scaling, and splitting data 80/10/10.35 min
- Powers and roots FreeRepeated multiplying, the laws of exponents, square roots, and squared error.43 min
- Very big and very small numbers FreePowers of ten, e-notation like 3e-4, orders of magnitude, and a model's size in memory.36 min
- Bits, bytes and floats: how numbers are stored FreeBase 2, what n bits can hold, a float as sign, power and digits, and a tensor's memory.35 min
- Review: find the slip (numbers) FreeBroken calculations to fix, mixed practice from the whole module, and its formula sheet.34 min
Algebra
- Letters for numbers FreeVariables, terms and like terms, substitution, and a model's formula as a Python function.41 min
- Expanding and factoring FreeBrackets as areas, pulling out common factors, and a squared error expanded into a bowl.41 min
- Solving equations: keep the balance FreeThe same move on both sides, undoing steps in reverse, and running a model's formula backward.41 min
- Rearranging a formula FreeMaking any letter the subject, round-trip checks, and planning a training run.36 min
- Inequalities and true-or-false FreeRanges, flipping the sign, if and elif, and the thresholds, clips and ReLU inside a model.42 min
- Functions: a rule from input to output Freef(x) notation, tables and graphs, domain and range, loops over lists, and loss as a function.42 min
- Chaining and undoing functions Freeg(f(x)), inverses that undo, scaling and unscaling, and a network as a chain.42 min
- Straight lines: slope and intercept FreeRise over run, the line through two points, and a model's weight and bias as slope and intercept.42 min
- Two equations, two unknowns FreeSubstitution and elimination, where lines cross, and why models are fitted, not solved.42 min
- Quadratics: bowls and their lowest point FreeRoots by factoring and by formula, the discriminant, the vertex, and the best weight of a model.42 min
- Exponentials: growth and decay FreeThe same factor each step, the number e, and decaying learning rates.43 min
- Logarithms: the power question FreeTo what power?, the laws of logs, and adding log-probabilities instead of multiplying.45 min
- Sequences and Σ FreeArithmetic and geometric sequences, Σ as a loop that adds, and mean squared error.40 min
- Weighted and moving averages FreeWeights that add to 1, a window average, the exponential moving average, and smoothing a loss curve.35 min
- Sets, and reading the symbols FreeSets, union and intersection, ℝⁿ and f : ℝⁿ → ℝᵐ, and reading a paper's first equation.41 min
- Review: find the slip (algebra) FreeBroken algebra and code to fix, mixed practice from the whole module, and its formula sheet.34 min
Shapes, angles and waves
- Coordinates and distance FreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
- Angles, degrees and radians FreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
- Sine and cosine FreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
- Waves: height, speed and shift FreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
- Review: find the slip (shapes and waves) FreeBroken geometry and code to fix, mixed practice from the module, and its formula sheet.29 min
Vectors and matrices
- Vectors: lists of numbers that point FreeA list, a point and an arrow; components, dimension, NumPy arrays, and data as vectors.42 min
- Adding and scaling vectors FreeComponent-wise sums, scalar multiples, mixes, and the weight update w − ηg.43 min
- Length and distance: norms FreeL2, L1 and max norms, unit vectors, distance, and the norms inside losses and clipping.42 min
- The dot product FreeMultiply in pairs and add; a·a as squared length; a @ b; one neuron as a dot product.43 min
- Angle, cosine similarity and projection FreeDirection without length, perpendicular vectors, shadows, and search by meaning.45 min
- Perpendicular directions: bases and coordinates FreeOrthonormal directions, coordinates as dot products, rebuilding a vector, and taking a direction out.35 min
- Matrices: tables of numbers FreeRows, columns and entries; two-dimensional arrays and slices; a data set as a matrix.41 min
- Adding, scaling and transposing matrices FreeEntry-by-entry sums and scalings, the transpose, mean images, and tables turned around.36 min
- A matrix times a vector FreeEach row dotted with the vector, the size rule, A @ x, and a layer acting on one example.42 min
- Matrix multiplication FreeRow meets column, the size rule, order matters, and a batch through a layer: X W + b.45 min
- Identity, diagonal, triangular and symmetric FreeFour patterns and their jobs: rescaling, distance tables, and the causal mask.37 min
- Matrices as movements of space FreeStretches, flips, shears and rotations; columns as landing spots; chaining; a layer moving data.42 min
- Determinant and inverse FreeThe area factor ad − bc, singular matrices, undoing a matrix, and redundant features.42 min
- Solving systems by elimination FreeAugmented matrices, row moves, back-substitution, the three outcomes, and fitting weights.45 min
- Independence, span and rank FreeRedundant vectors, span, rank, outer products, and low-rank updates like LoRA.42 min
- Eigenvectors: directions that only stretch FreeAv = λv, checking candidates, np.linalg.eig, and the main direction of a data cloud.41 min
- SVD and PCA: the important directions first FreeA matrix as pieces by size, the best rank-k copy, compression, and PCA maps of data.45 min
- Tensors, shapes and broadcasting FreeAxes and shapes, reshape, axis=, broadcasting, and shapes through a small model.43 min
- One-hot vectors and lookup tables: embeddings FreeA one-hot row picks a table's row, a sentence looked up as vectors, and a score for every word.35 min
- Sliding dot products: convolution FreeA small pattern slid along a list and over a grid, output sizes, stride and padding, and shared weights.40 min
- Review: find the slip (vectors and matrices) FreeShape slips to fix, mixed practice from the module, and its formula sheet with shapes.36 min
Calculus
- Rates of change FreeChange over change, secants, rates from readings with np.diff, and reading a loss curve.41 min
- Limits: what a value heads toward FreeApproaching from both sides, holes, corners and jumps, epsilon, and judging activations.37 min
- The derivative: slope at a point FreeShrinking secants, f′(x) and dy/dx, Δy ≈ f′Δx, and which way a weight should move.45 min
- Derivative rules FreeConstants, powers, multiples, sums, eˣ and ln x, checked numerically, and a squared error's slope.41 min
- The product rule and the chain rule FreeProducts and chains of functions, forward then backward, and one neuron's dL/dw.45 min
- Lowest points and curvature Freef′ = 0, the second-derivative test, tables, local and global minima, and convex losses.43 min
- Standing in for a curve near a point FreeTangent lines, a curvature term, growing error, and why learning rates must be small.37 min
- Partial derivatives FreeOne input at a time, ∂, nudging in Python, and the two slopes of a line's loss.41 min
- The gradient FreeAll the slopes as a vector, steepest climb and descent, a numerical gradient, and the loss gradient.42 min
- Gradient descent FreeStepping against the gradient, the learning rate, a training loop, and fitting a line.45 min
- Faster descent: momentum and Adam FreeMomentum and Adam by hand, the bias correction, and three optimizers down one narrow valley.40 min
- The chain rule with many paths: backpropagation by hand FreeGraphs, multiply along paths and add across them, gradient checks, and a 2-2-1 network by hand.45 min
- Automatic differentiation: how PyTorch finds every slope FreeA tape of steps walked backward, and requires_grad, backward(), .grad, zero_grad() and no_grad().40 min
- Jacobians and Hessians FreeSlopes of vector functions, a layer's Jacobian, second partials, and bowls, domes and saddles.42 min
- Integrals: adding up thin slices FreeStrip and trapezoid sums, summing undoes differencing, probability as area, and AUC.43 min
- Review: find the slip (calculus) FreeCalculus slips to fix, mixed practice from the module, and its formula sheet.36 min
Probability
- Counting: choices, orders and groups FreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
- Probability: from counts to chances FreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
- Combining events: and, or, independent FreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
- Conditional probability and Bayes' rule FreeConditionals from tables, why P(A | B) isn't P(B | A), Bayes' rule, and precision and recall.45 min
- Random variables and distributions FreeUncertain numbers, the probability of a range, sampling by slices, and sampling versus argmax.42 min
- Expected value, variance and standard deviation FreeProbability-weighted averages, spread in the data's own units, and a training loss as an expected value.42 min
- Bernoulli, binomial, uniform and normal FreeYes-or-no trials, counts of yeses, the 68-95-99.7 rule, z-scores, and an accuracy's error bar.45 min
- Two variables at once: covariance and correlation FreeJoint tables and marginals, covariance, correlation, the covariance matrix, and features that carry the same information.45 min
- Review: find the slip (probability) FreeProbability slips to fix, mixed practice from the module, and its formula sheet.36 min
Statistics and information
- Describing data: center, spread and scale FreeMean, median and mode, variance and standard deviation, percentiles, outliers, and standardizing features.42 min
- Samples, the law of large numbers and the central limit theorem FreeSamples and populations, the standard error σ/√n, the bell of averages, and error bars on test scores.43 min
- Is the difference real? Comparing two models FreePaired differences, a bootstrap interval, what a p-value is and isn't, and why many tries make lucky wins.40 min
- Maximum likelihood FreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
- Linear regression: the best line FreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
- Logistic regression: a line that answers yes or no FreeThe sigmoid, the log loss, its slope p − y, a training loop, and choosing a threshold.45 min
- Scoring a classifier: confusion table, precision, recall, F1 FreeThe four cells, accuracy, precision, recall and F1, and what moving a threshold does to each.35 min
- Softmax: scores into probabilities FreeExponentiate and share out, shifting and temperature, the safe softmax, and a classifier's last step.43 min
- Surprise, entropy and cross-entropy FreeSurprise in bits, entropy, cross-entropy and KL divergence, perplexity, and a language model's loss.45 min
- KL divergence: how far one distribution is from another FreeKL by hand, cross-entropy as entropy plus KL, why it isn't symmetric, and a soft-target loss.35 min
- Review: find the slip (statistics and information) FreeStatistics slips to fix, mixed practice from the module, and its formula sheet.38 min
Putting it together
- One neuron, start to finish FreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
- A layer and a small network as matrices FreeTwo layers on a batch, the shape of every matrix, adding a hidden unit, and XOR solved.45 min
- Training a small network FreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
- The same network in PyTorch Freenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
- Attention as matrix math FreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
- Residual connections and layer norm FreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
- A transformer block, start to finish FreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
- Capstone: a sign reader from the formulas FreeBuild, train and test a trail-sign classifier with NumPy alone, and report its accuracy with an error bar.45 min
Transposition of formulas
- Transposing step by step: undo in reverse order FreeA formula's layers, the opposite of each, peeling from the outside in, conditions, and round-trip checks.35 min
- Formulas with fractions: clear them first FreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
- When the letter appears twice: collect and factor FreeGathering the letter on one side, factoring it out, dividing by the bracket, and rearranging precision and F1.35 min
- Squares, roots and powers FreeSquare roots with ±, squaring to undo a root, cube and nth roots, and the √d inside attention.40 min
- When the letter is squared and plain: quadratics FreeRearranging into ax² + bx + c = 0, the formula, completing the square, and choosing the root that fits.35 min
- Letters in the power: exponentials and logarithms FreeTaking logs to bring a power down, exponentiating to free a letter, and the logit that undoes the sigmoid.40 min
- Rearranging inequalities: when the sign flips FreeThe four moves on an inequality, negatives and reciprocals that flip it, logs below 1, and sample sizes.35 min
- Inverse functions, lines and scaling FreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
- Sequences and sums: find n, a, d and r FreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
- Combining formulas: substitute and eliminate FreePutting one formula inside another to remove a letter, and sizing a training run from a time or money budget.35 min
- Review: rearrange anything (algebra) FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
- Angles, triangles and waves: rearranging with sin, cos and tan FreeSides and angles from SOH CAH TOA, principal values, arcs, polar coordinates, and a wave's time and period.45 min
- Vector equations: rearranging with arrows FreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
- Matrix equations: multiply by the inverse, on the correct side FreeNo dividing by a matrix: inverses from the left or right, order, transposes, and the normal equations.40 min
- Rearranging in calculus: slopes, steps and constants FreeSetting a slope to zero, Newton's step, the chain rule turned around, constants from a point, and learning rates.45 min
- Probability formulas: solve for the missing chance FreeConditionals, Bayes' rule for the prior, the or rule, expected values, variance, and z-scores turned around.45 min
- Statistics and information: n, z, logits and bits FreeSample sizes from error bars, inputs from probabilities, logit gaps from softmax, and chances from bits and losses.40 min
- Planning a model: rearranging the formulas of a working network FreeLayer widths from parameter counts, sizes from memory and compute, the attention scale, and a training plan.40 min
- Final test: transpose any formula FreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
The parts of the system this course works on.
Each part, what it does, the real tool for it and the lessons that teach it. In a lesson, the same map lights the part you are working on.
Where this sits in the model
A model learning: forward from data to a loss, backward to an update, and what sits around training.
- data: examples and their answers, split before training PyTorch:
Dataset. Ratios, rates and proportion, Describing data: center, spread and scale - numbers: every input written as numbers: pixels, ids, measurements PyTorch:
torch.tensor(x). Numbers and place value, Decimals and rounding, Bits, bytes and floats: how numbers are stored - batch: many examples at once, a matrix with one row each PyTorch:
DataLoader(data, batch_size=32). Multiplying and dividing: groups, Solving equations: keep the balance, Rearranging a formula - vectors: each word or example as a list of numbers with a direction PyTorch:
nn.Embedding(vocab, d). Coordinates and distance, Angles, degrees and radians, Sine and cosine - layer: a matrix product plus a bias: XW + b PyTorch:
nn.Linear(d_in, d_out). Order of operations, and names for numbers, Letters for numbers, Chaining and undoing functions - activation: a bend after each layer, like relu(z) = max(0, z) PyTorch:
F.relu(z). Inequalities and true-or-false, Chaining and undoing functions, Limits: what a value heads toward - attention: softmax(QKᵀ/√d)V: each word a weighted average of the others PyTorch:
F.softmax(q @ k.T / d ** 0.5, dim=-1) @ v. Identity, diagonal, triangular and symmetric, Counting: choices, orders and groups, Attention as matrix math - scores: one number per answer, any size (the logits) PyTorch:
logits = model(x). Exponentials: growth and decay, One-hot vectors and lookup tables: embeddings, Logistic regression: a line that answers yes or no - softmax: eᶻ shared out so the scores add to 1 PyTorch:
F.softmax(logits, dim=-1). Softmax: scores into probabilities - chances: a probability for every answer, adding to 1 PyTorch:
probs. Multiplying and dividing fractions, Logarithms: the power question, Probability: from counts to chances - targets: the right answers the model is judged against PyTorch:
y. Numbers below zero, Surprise, entropy and cross-entropy - loss: one number for how wrong: squared error, or −log of the right answer's chance PyTorch:
F.mse_loss(pred, y) or F.cross_entropy(logits, y). Numbers below zero, Powers and roots, Expanding and factoring - slopes: how the loss changes with each number, by the chain rule PyTorch:
loss.backward(). The derivative: slope at a point, Derivative rules, The product rule and the chain rule - gradient: all the slopes together, pointing uphill PyTorch:
w.grad. Length and distance: norms, The gradient, Automatic differentiation: how PyTorch finds every slope - update: w − η·g: a small step downhill PyTorch:
optimizer.step(). Very big and very small numbers, Solving equations: keep the balance, Exponentials: growth and decay - weights: the numbers training changes PyTorch:
model.parameters(). Independence, span and rank, SVD and PCA: the important directions first, Bernoulli, binomial, uniform and normal - train and test split: some examples held back to judge on PyTorch:
random_split(data, [0.8, 0.2]). Ratios, rates and proportion, Linear regression: the best line - init: the weights' starting values: small random numbers, scaled to the layer's size PyTorch:
nn.init.normal_(w, mean=0.0, std=0.02). Bernoulli, binomial, uniform and normal - penalty: λ‖w‖² added to the loss, so training prefers small weights PyTorch:
loss = loss + lam * (w ** 2).sum(). Length and distance: norms, Determinant and inverse - sampling: drawing the next word by its chance PyTorch:
torch.multinomial(probs, 1). Probability: from counts to chances, Random variables and distributions - metrics and error bars: accuracy and its likely range PyTorch:
(pred == y).float().mean(). Fractions: equal parts, equal fractions, adding, Percentages, Inequalities and true-or-false - memory: count × bytes per number PyTorch:
x.numel() * x.element_size(). Powers and roots, Very big and very small numbers, Bits, bytes and floats: how numbers are stored - cost: how the work grows with length and size PyTorch:
x.numel(). Counting: choices, orders and groups, Combining events: and, or, independent
Where it starts, and what you earn.
Before you start
Nothing. This phase starts from your very first line of Python, and every lesson in it is free to try. Already know some? The placement test marks off what you know, so you only study what's new to you.
When you finish
This phase leads to:
- Phase 0: Tools
- Phase 1: Building with LLMs
- Phase 2: Math for ML engineers
- Phase 3: Machine learning
- Phase 5: Transformers
- Phase 6: Modern LLM engineering
Finish every lesson in this phase and you earn a certificate of completion, "Foundation math (Foundation)", with its own link you can share. Pass the phase's checkpoint, 12 questions the academy's server draws and marks itself, and it becomes a verified certificate; the checkpoint is also how lessons you skipped in the placement test count.
All thirteen phases.
Every lesson on one page, or the AI engineer roadmap, which shows what you can skip.
How a lesson works
Read a little, run real Python in your browser, change it, and answer a short quiz. Exercises check your code and hint at what went wrong. What you learn comes back as review cards, spaced out over the weeks. See what you'll build.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write.
Start with Numbers and place value.
What a digit's place is worth, the number line, and Python as a calculator that never tires.