The Foundation Math For Ai & ML
Formula sheets
Every formula of the Foundation, one page per module, with the module's key picture on top: the formula, what it means in plain words, its line of Python, the same call in PyTorch, where it is used in a model, and the lesson that teaches it. Print them, or save them as a PDF to keep beside you.
Module 1. Numbers
| Lesson | Formula | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 1 | each digit times its place | 4 * 1000 + 7 * 10 + 2 | x @ 10.0 ** torch.arange(3, -1, -1) | numbers | |
| 2 | full groups, plus what's left | a // b and a % b | torch.div(a, b, rounding_mode="floor"), a % b | batch | |
| 3 | multiply first, then add | w * x + b | w * x + b | layer | |
| 4 | distance = abs(a − b) | how far apart, either way | abs(a - b) | torch.abs(a - b) | targets, loss |
| 5 | add over a common bottom | Fraction(1, 2) + Fraction(1, 3) | a / b + c / d | metrics and error bars | |
| 6 | tops times tops, bottoms times bottoms | Fraction(2, 3) * Fraction(3, 4) | (a / b) * (c / d) | chances | |
| 7 | divide to get a decimal | math.isclose(a, b) to compare | torch.round(x, decimals=2) | numbers | |
| 8 | change % = (new − old) ÷ old × 100 | divide by where you started | (new - old) / old * 100 | (new - old) / old * 100 | metrics and error bars |
| 9 | one portion = T ÷ (a + b + c); y = kx | count portions, then share | total / (8 + 1 + 1) * 8 | parts / parts.sum() | data, train and test split |
| 10 | multiply: add; divide: take away | 2 ** 3 * 2 ** 4 | x ** 3, torch.sqrt(x) | loss, memory | |
| 11 | a short number times a power of ten | 3.4e-4 | torch.tensor(3.4e-4) | update, memory | |
| 12 | each binary place is worth twice the one to its right | int("1101", 2) | torch.finfo(torch.float32).eps | numbers, memory |
Module 2. Algebra
| Lesson | Formula | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 14 | a weight times the input, plus a bias | w * x + b | w * x + b | layer | |
| 15 | every part times every part | (a + b) ** 2 | (a + b) ** 2 | loss | |
| 16 | undo in reverse, both sides | (c - b) / a | torch.linalg.solve(A, c) | batch, update | |
| 17 | make any letter the subject | (y - b) / m | (y - b) / m | batch | |
| 18 | ReLU(x) = 0 if x ≤ 0, else x | a decision by an inequality | if x <= 0: | torch.relu(x) | activation, metrics and error bars |
| 19 | f(x): one output per input | a rule with a name | def f(x): | y = model(x) | loss |
| 20 | the inverse undoes | f_inv(f(x)) == x | torch.log(torch.exp(x)) | layer, activation | |
| 21 | rise over run | (y2 - y1) / (x2 - x1) | (y2 - y1) / (x2 - x1) | layer | |
| 22 | add or take away equations | make a letter cancel | (c1 - c2) / (a1 - a2) | torch.linalg.solve(A, b) | layer, loss |
| 23 | roots, and the vertex at −b/2a | -b / (2 * a) | (-b + torch.sqrt(b * b - 4 * a * c)) / (2 * a) | loss | |
| 24 | the same factor each step | a * r ** t | a * r ** t | scores, update | |
| 25 | a product's log is a sum | math.log(a) + math.log(b) | torch.log(a * b) | chances, loss | |
| 26 | the mean squared error | a loop, then / len(ys) | ((y_hat - y) ** 2).mean() | loss | |
| 27 | blend each new value into the running one | m = beta * m + (1 - beta) * x | m = beta * m + (1 - beta) * x | loss, update | |
| 28 | n numbers in, m out | len(x) is n | x.shape | numbers |
Transposition. Rearranging in algebra
| Lesson | Formula | Rearranged | Condition | In Python |
|---|---|---|---|---|
| 30 | y = 3(x − 2)/4 + 5 | x = 4(y − 5)/3 + 2 | none: the divisors are numbers | 4 * (y - 5) / 3 + 2 |
| 30 | v = u − gt | t = (u − v)/g | g ≠ 0 | (u - v) / g |
| 31 | 1/R = 1/R₁ + 1/R₂ | R = R₁R₂/(R₁ + R₂) | R₁ + R₂ ≠ 0 | R1 * R2 / (R1 + R2) |
| 31 | y = a/(b + x) | x = a/y − b | y ≠ 0 | a / y - b |
| 32 | A = P + Prt | P = A/(1 + rt) | 1 + rt ≠ 0 | A / (1 + r * t) |
| 33 | A = πr² | r = √(A/π) | keep + for a length | math.sqrt(A / math.pi) |
| 33 | T = 2π√(l/g) | l = g(T/2π)² | square the whole bracket | g * (T / (2 * math.pi)) ** 2 |
| 34 | ax² + bx + c = 0 | x = (−b ± √(b² − 4ac))/(2a) | a ≠ 0, b² − 4ac ≥ 0 | (-b + math.sqrt(b ** 2 - 4 * a * c)) / (2 * a) |
| 35 | A = P(1 + r)ⁿ | n = ln(A/P)/ln(1 + r) | A/P > 0, r > −1, r ≠ 0 | math.log(A / P) / math.log(1 + r) |
| 35 | p = 1/(1 + e^(−z)) | z = ln(p/(1 − p)) | 0 < p < 1 | math.log(p / (1 - p)) |
| 36 | √(p(1 − p)/n) ≤ E | n ≥ p(1 − p)/E² | 1 over both sides flips the sign; round up | math.ceil(p * (1 - p) / E ** 2) |
| 36 | lr₀γᵗ ≤ target | t ≥ ln(target/lr₀)/ln γ | flips: ln γ < 0 for γ < 1; round up | math.ceil(math.log(target / lr0) / math.log(gamma)) |
| 37 | f(x) = (2x + 3)/5 | f⁻¹(x) = (5x − 3)/2 | f one-to-one | (5 * x - 3) / 2 |
| 37 | z = (x − μ)/σ | x = μ + zσ | σ ≠ 0 | mu + z * sigma |
| 38 | uₙ = a + (n − 1)d | n = (uₙ − a)/d + 1 | d ≠ 0; n a positive whole number | (u - a) / d + 1 |
| 38 | uₙ = a·rⁿ⁻¹ | n = log(uₙ/a)/log r + 1 | uₙ/a > 0, r > 0, r ≠ 1; n whole | math.log(u / a) / math.log(r) + 1 |
| 39 | C = 2πr and A = πr² | A = C²/(4π) | keep the brackets: π(C/(2π))² | C ** 2 / (4 * math.pi) |
| 39 | v = u + at and s = ut + ½at² | v² = u² + 2as | u² + 2as ≥ 0; v = ±√(u² + 2as) | math.sqrt(u ** 2 + 2 * a * s) |
Module 3. Shapes, angles and waves
| Lesson | Formula | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 41 | square the differences, add, root | math.sqrt(dx ** 2 + dy ** 2) | torch.dist(p, q) | vectors | |
| 42 | radians = degrees × π/180 | a full turn is 2π | math.radians(deg) | torch.deg2rad(d) | vectors |
| 42 | arc = radius × angle in radians | r * theta | torch.deg2rad(d) | vectors | |
| 43 | (cos θ, sin θ), with sin² + cos² = 1 | across and up on the unit circle | math.cos(t), math.sin(t) | torch.sin(theta), torch.cos(theta) | vectors |
| 44 | y = A sin(kx + φ), period 2π/k | height, speed and shift | A * math.sin(k * x + phase) | A * torch.sin(k * x + phi) | vectors |
Module 4. Vectors and matrices
| Lesson | Formula | In plain words | Shapes | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|---|
| 48 | entry by entry | (n,), (n,) → (n,) | a + b, c * a | a + b, c * a | update | |
| 49 | length | (n,) → one number | np.linalg.norm(v) | torch.linalg.norm(v) | gradient, update, penalty | |
| 50 | pairs multiplied, then added | (n,), (n,) → one number | a @ b | a @ b | layer | |
| 51 | direction only | (n,), (n,) → one number | (a @ b) / (norm(a) * norm(b)) | F.cosine_similarity(a, b, dim=0) | vectors | |
| 52 | amounts along perpendicular unit directions | (n, n), (n,) → (n,) | Q.T @ v | c = Q.T @ v; v_back = Q @ c | vectors | |
| 53 | row i, then column j | (m, n) | A[i - 1, j - 1] | A[i, j] | numbers, layer | |
| 54 | rows become columns | (m, n) → (n, m) | A.T | A.T | batch | |
| 55 | each row dotted with x | (m, n), (n,) → (m,) | A @ x | A @ x | layer | |
| 56 | row i meets column j | (m, n), (n, p) → (m, p) | A @ B | A @ B | batch, layer | |
| 59 | area factor; 0 means no inverse | (2, 2) → one number | np.linalg.det(A) | torch.linalg.det(A), torch.linalg.inv(A) | layer, penalty | |
| 60 | elimination, then back-substitution | (n, n), (n,) → (n,) | np.linalg.solve(A, b) | torch.linalg.solve(A, b) | loss | |
| 61 | independent columns | (m, n) → one number | np.linalg.matrix_rank(A) | torch.linalg.matrix_rank(A) | weights | |
| 62 | only stretched | (n, n) → n pairs | np.linalg.eig(A); np.linalg.eigh(A) if A is symmetric | torch.linalg.eig(A) | vectors | |
| 63 | pieces, biggest first | (m, n) → three arrays | np.linalg.svd(A) | torch.linalg.svd(A) | weights | |
| 64 | sum along an axis | that axis disappears | (m, n) → (n,) for axis 0 | X.sum(axis=0) | x.sum(dim=0) | batch |
| 65 | eᵢE = row i of E | a one-hot row picks a row of the table | (vocab, d), ids (n,) → (n, d) | E[ids] | nn.Embedding(vocab, d)(ids) | numbers, vectors, scores |
| 66 | a pattern slid along, a dot product at each stop | (n,), (k,) → (n − k + 1,) | x[j:j + k] @ w | F.conv1d(x, w) | layer |
Module 5. Calculus
| Lesson | Rule | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 70 | change over change | np.diff(v) / h | torch.diff(y) / h | loss | |
| 72 | the slope at a point | (f(x + h) - f(x - h)) / (2 * h) | y.backward(); x.grad | slopes | |
| 73 | power down, power lowered | n * c[n] per term | y.backward(); x.grad | slopes | |
| 74 | multiply the links | dy_du * du_dx | loss.backward() | slopes | |
| 75 | f′ = 0; f″ > 0 bowl, f″ < 0 dome | flat, then which way it bends | xs[np.argmin(f(xs))] | torch.argmin(f(xs)) | loss |
| 76 | the curve near a point | fa + dfa * d + 0.5 * d2fa * d ** 2 | fa + dfa * d | update | |
| 77–78 | all the slopes, steepest climb | nudge copies, one input at a time | loss.backward(); w.grad | slopes | |
| 79 | step downhill, not too far | w = w - lr * dw | torch.optim.SGD(model.parameters(), lr=0.1) | update | |
| 80 | momentum: v ← βv + g, w ← w − ηv; Adam: m ← β₁m + (1 − β₁)g, v ← β₂v + (1 − β₂)g², m̂ = m ÷ (1 − β₁ᵗ), v̂ = v ÷ (1 − β₂ᵗ), w ← w − η·m̂ ÷ (√v̂ + ε) | momentum piles up slopes that keep pointing one way; Adam divides the average slope by its usual size, so every weight moves about η | v = 0.9 * v + g; w = w - lr * mh / (np.sqrt(vh) + eps) | torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999)) | update |
| 81 | multiply along a path, add across paths | dL_dv = dL_dv + ... | loss.backward() | slopes | |
| 82 | ū ← ū + v̄ · ∂v/∂u for each entry v, last to first (ū = ∂L/∂u) | record each step forward, walk the tape back; .grad adds up until zeroed | loss.backward(), then w.grad; optimizer.zero_grad() | loss.backward(); w.grad; opt.zero_grad() | slopes, gradient |
| 83 | all slopes; all curvatures | J[:, j] = ..., np.linalg.eigvalsh(H) | torch.autograd.functional.jacobian(f, x) | slopes | |
| 84 | thin strips, added | dx * (np.sum(y) - (y[0] + y[-1]) / 2) | torch.trapezoid(y, x) | metrics and error bars |
Module 6. Probability
| Lesson | Rule | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 87 | orders; ordered picks; groups | math.factorial(n), math.perm(n, r), math.comb(n, r) | math.comb(n, r) | attention, cost | |
| 88 | P(A) = ways for A ÷ all ways; P(not A) = 1 − P(A) | a share of equally likely outcomes | np.mean(event) over seeded trials | (draws == a).float().mean() | chances, sampling |
| 89 | add, then remove the overlap | pa + pb - pab | (a & b).float().mean() | cost | |
| 89 | P(A and B) = P(A) × P(B), if independent | multiply only when one doesn't change the other | pa * pb | (a & b).float().mean() | cost |
| 90 | count inside the group where B happened | T / T.sum(axis=0) | (a & b).sum() / b.sum() | chances, metrics and error bars | |
| 90 | Bayes: P(A given B) = P(B given A) P(A) ÷ P(B) | turn one conditional into the other | pba * pa / pb | (a & b).sum() / b.sum() | chances, metrics and error bars |
| 91 | Σ pᵢ = 1; P(range) = Σ of its pᵢ | a chance for every value | rng.choice(v, size=n, p=p) | torch.multinomial(p, 1) | sampling |
| 92 | balance point; spread, in the data's units | v @ p, ((v - mu) ** 2) @ p, var ** 0.5 | (x * p).sum(), x.var() | loss | |
| 93 | P(k) = C(n, k) pᵏ(1 − p)ⁿ⁻ᵏ; mean np; z = (x − μ)/σ | yeses in n trials; distance in σ's | math.comb(n, k) * p ** k * (1 - p) ** (n - k) | Binomial(n, p).log_prob(k).exp() | weights, init |
| 94 | cov = mean of (x − x̄)(y − ȳ); r = cov ÷ (σₓσᵧ) | moving together, from −1 to 1 | np.mean(dx * dy), np.corrcoef(x, y) | torch.cov(torch.stack([x, y])) | numbers |
Module 7. Statistics and information
| Lesson | Rule | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 97 | center, spread, distance in s's | x.mean(), np.median(x), x.std(), np.percentile(x, 95) | x.mean(), x.std() | data, numbers | |
| 98 | the wobble of an average | sigma / n ** 0.5 | x.std() / n ** 0.5 | batch, metrics and error bars | |
| 99 | dᵢ = bᵢ − aᵢ; d̄ ± 1.96·s_d/√n; P(at least one false win in k tries) = 1 − 0.95ᵏ | compare item by item; the lead is real when 0 is outside its range | d = b - a, d.mean(), d.std() / len(d) ** 0.5 | d = b - a; d.mean(), d.std() / n ** 0.5 | metrics and error bars |
| 100 | the setting that makes the data least surprising | ps[np.argmax(ll)] | dist.log_prob(x).sum() | loss | |
| 101 | the line with the smallest squared misses | np.linalg.lstsq(X, y, rcond=None) | torch.linalg.lstsq(X, y).solution | loss, train and test split | |
| 102 | σ(z) = 1/(1 + e⁻ᶻ); L = −[y log p + (1 − y) log(1 − p)]; slope p − y | a score into a chance; the cost of a yes-or-no guess | 1 / (1 + np.exp(-z)) | torch.sigmoid(z) | scores, chances |
| 103 | precision = TP/(TP + FP); recall = TP/(TP + FN); F1 = 2PR/(P + R) | right out of the model's yeses; found out of the real yeses | tp = np.sum((y == 1) & (pred == 1)) | (pred & y).sum() / pred.sum() | metrics and error bars |
| 104 | scores into chances that add to 1 | e = np.exp(z - z.max()); e / e.sum() | F.softmax(z, dim=-1) | softmax | |
| 105 | H(p) = −Σ pᵢ log₂ pᵢ; H(p, q) = −Σ pᵢ log₂ qᵢ; perplexity 2^H | average surprise; the model's average surprise | p @ np.log2(1 / q) | F.cross_entropy(logits, y) | targets, loss |
| 106 | the model's extra surprise over the truth's own; 0 only when they match | p @ np.log2(p / q) | F.kl_div(q.log(), p, reduction="sum") | loss |
Transposition. Rearranging in every topic
| Lesson | Formula | Rearranged | Condition | In Python |
|---|---|---|---|---|
| 46 | sin θ = o/h | θ = arcsin(o/h), h = o/sin θ | −1 ≤ o/h ≤ 1; one angle, −90° to 90°; sin θ ≠ 0 for h | math.asin(o / h) |
| 46 | x = r cos θ, y = r sin θ | r = √(x² + y²), θ = atan2(y, x) | atan(y/x) fails when x < 0 | math.atan2(y, x) |
| 46 | y = A sin(ωt + φ) | t = (arcsin(y/A) − φ)/ω | −A ≤ y ≤ A; A, ω ≠ 0; add the period if t < 0 | (math.asin(y / A) - phi) / omega |
| 68 | w_new = w − ηg | g = (w − w_new)/η | η ≠ 0; never divide by a vector | (w - w_new) / lr |
| 68 | a·b = ‖a‖‖b‖cos θ | θ = arccos(a·b/(‖a‖‖b‖)) | no zero vector; θ from 0 to π | np.arccos(np.clip(c, -1, 1)) |
| 68 | v = βv + g, steady | v = g/(1 − β), g = (1 − β)v | β ≠ 1 | g / (1 - beta) |
| 69 | AX = B; XA = B | X = A⁻¹B; X = BA⁻¹ | A square, det A ≠ 0; the inverse on A's side | np.linalg.solve(A, B); np.linalg.solve(A.T, B.T).T |
| 69 | y = Wx + b | x = W⁻¹(y − b) | W square and invertible | np.linalg.solve(W, y - b) |
| 69 | XᵀXw = Xᵀy | w = (XᵀX)⁻¹Xᵀy | XᵀX invertible | np.linalg.solve(X.T @ X, X.T @ y) |
| 86 | C′(x) = 2ax − b = 0 | x = b/(2a) | a > 0 for a lowest point | b / (2 * a) |
| 86 | f(x + h) ≈ f(x) + f′(x)h | x_new = x − f(x)/f′(x) | f′(x) ≠ 0 | x - f(x) / df(x) |
| 86 | F(x) = x² + C, F(x₀) = y₀ | C = y₀ − x₀² | one point fixes C | y0 - x0 ** 2 |
| 96 | P(A and B) = P(A) × P(B given A) | P(B given A) = P(A and B)/P(A) | P(A) > 0; the answer from 0 to 1 | p_and / p_a |
| 96 | P(A given B) = P(B given A) × P(A)/P(B) | P(A) = P(A given B) × P(B)/P(B given A) | P(B given A) > 0; the answer from 0 to 1 | a_given_b * p_b / b_given_a |
| 96 | z = (x − μ)/σ | x = μ + zσ | σ > 0 | mu + z * sigma |
| 108 | SE = σ/√n; m = zσ/√n | n = (σ/SE)²; n = (zσ/m)² | round n up | math.ceil((z * sigma / m) ** 2) |
| 108 | p = σ(wx + b) | x = (ln(p/(1 − p)) − b)/w | 0 < p < 1; w ≠ 0 | (math.log(p / (1 - p)) - b) / w |
| 108 | s = −log₂ p; L = −ln p | p = 2^(−s); p = e^(−L) | bits use log₂, nats use ln | 2 ** -s, math.exp(-L) |
| 116 | P = n_in·n_out + n_out | n_out = P/(n_in + 1) | count the bias; a whole number | P // (n_in + 1) |
| 116 | C ≈ 6ND, D ≈ 20N | N = √(C/120), D = C/(6N) | rules of thumb, not laws | math.sqrt(C / 120) |
| 116 | s = q·k/√d | d = (q·k/s)² | s ≠ 0, same sign as q·k; d a whole number | (qk / s) ** 2 |
| 30 | y = ax + b | x = (y − b)/a | a ≠ 0 | (y - b) / a |
| 33 | y = xⁿ | x = ±y^(1/n) for even n, y^(1/n) for odd n | even n: y ≥ 0; in code, root the size of a negative y | y ** (1 / n) |
| 35 | y = eˣ; p = 1/(1 + e^(−z)) | x = ln y; z = ln(p/(1 − p)) | y > 0; 0 < p < 1 | math.log(p / (1 - p)) |
| 46 | y = sin x | x = arcsin y | −1 ≤ y ≤ 1; one principal angle, −90° to 90° | math.asin(y) |
| 69 | y = Ax | x = A⁻¹y | det A ≠ 0; the inverse on A's side | np.linalg.solve(A, y) |
| 36 | ax < c, a < 0 | x > c/a | the sign flips | c / a |
Module 8. Putting it together
| Lesson | Formula | In plain words | In Python | In PyTorch | Where it's used |
|---|---|---|---|---|---|
| 109 | one neuron: sum, squeeze, blame each weight by its input | dz * x | torch.relu(w @ x + b) | slopes, update | |
| 110 | a whole batch through two layers | np.maximum(X @ W1 + b1, 0) @ W2 + b2 | nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, k)) | layer, activation, scores | |
| 111 | dZ₂ = (P − Y)/n; dW₂ = A₁ᵀdZ₂; dZ₁ = dZ₂W₂ᵀ where Z₁ > 0; dW₁ = XᵀdZ₁ | the backward pass, shape by shape | A1.T @ dZ2 | loss.backward(); opt.step() | — |
| 113 | score, scale, mask, share out, blend | A @ V | F.scaled_dot_product_attention(q, k, v) | attention | |
| 114 | add the input back, then recenter and rescale each vector | x + f(x), g * (x - mu) / (var + eps) ** 0.5 + b | x + sublayer(x); nn.LayerNorm(d) | — | |
| 115 | h = x + Attn(LN(x)); out = h + MLP(LN(h)) | attention, then a small network, each added back | h = x + attn(ln1(x)); h + mlp(ln2(h)) | nn.TransformerEncoderLayer(d, heads) | — |
| 112 | the same two-layer network, as modules | the NumPy formulas, named once each | nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3)) | nn.Sequential(...); opt = torch.optim.SGD(...) | — |
| 118 | an honest test score | (acc * (1 - acc) / n) ** 0.5 | acc ± 1.96 * (acc * (1 - acc) / n) ** 0.5 | — |