Index Ai Academy

The Foundation Math For Ai & ML

Formula sheets

Every formula of the Foundation, one page per module, with the module's key picture on top: the formula, what it means in plain words, its line of Python, the same call in PyTorch, where it is used in a model, and the lesson that teaches it. Print them, or save them as a PDF to keep beside you.

Module 1. Numbers

places, each worth twice the one on its right1 × 81 × 40 × 21 × 113 squares8 + 4 + 1 = 13, which is 1101 in base 2
The key picture: Every place is decided. 8 + 4 + 1 = 13, so 13 in base 2 is 1101. From Bits, bytes and floats: how numbers are stored.
LessonFormulaIn plain wordsIn PythonIn PyTorchWhere it's used
14,072=4×1,000+7×10+24,072 = 4 \times 1,000 + 7 \times 10 + 2each digit times its place4 * 1000 + 7 * 10 + 2x @ 10.0 ** torch.arange(3, -1, -1)numbers
2a=q×b+ra = q \times b + rfull groups, plus what's lefta // b and a % btorch.div(a, b, rounding_mode="floor"), a % bbatch
3y^=w×x+b\hat{y} = w \times x + bmultiply first, then addw * x + bw * x + blayer
4distance = abs(a − b)how far apart, either wayabs(a - b)torch.abs(a - b)targets, loss
5a/b+c/d=(a×d+c×b)/(b×d)a/b + c/d = (a \times d + c \times b) / (b \times d)add over a common bottomFraction(1, 2) + Fraction(1, 3)a / b + c / dmetrics and error bars
6a/b×c/d=(a×c)/(b×d)a/b \times c/d = (a \times c) / (b \times d)tops times tops, bottoms times bottomsFraction(2, 3) * Fraction(3, 4)(a / b) * (c / d)chances
73/8=3÷8=0.3753/8 = 3 \div 8 = 0.375divide to get a decimalmath.isclose(a, b) to comparetorch.round(x, decimals=2)numbers
8change % = (new − old) ÷ old × 100divide by where you started(new - old) / old * 100(new - old) / old * 100metrics and error bars
9one portion = T ÷ (a + b + c); y = kxcount portions, then sharetotal / (8 + 1 + 1) * 8parts / parts.sum()data, train and test split
10am×an=am+na^{m} \times a^{n} = a^{m+n}am÷an=am−na^{m} \div a^{n} = a^{m-n}(am)n=amn(a^{m})^{n} = a^{mn}multiply: add; divide: take away2 ** 3 * 2 ** 4x ** 3, torch.sqrt(x)loss, memory
110.00034=3.4×10−40.00034 = 3.4 \times 10^{-4}a short number times a power of ten3.4e-4torch.tensor(3.4e-4)update, memory
1211012=8+4+1=131101_{2} = 8 + 4 + 1 = 13each binary place is worth twice the one to its rightint("1101", 2)torch.finfo(torch.float32).epsnumbers, memory

From Review: find the slip (numbers).

Module 2. Algebra

xy−1−0.50.511.522.533.5424681012rise 4run 2(1, 5)(3, 9)b = 3y = 2x + 3
The key picture: Intercept: walk back from (1, 5) to x = 0. b = 5 − 2 × 1 = 3. So y = 2x + 3. From Straight lines: slope and intercept.
LessonFormulaIn plain wordsIn PythonIn PyTorchWhere it's used
14y^=w⋅x+b\hat{y} = w\cdot x + ba weight times the input, plus a biasw * x + bw * x + blayer
15(a+b)2=a2+2ab+b2(a + b)^{2} = a^{2} + 2ab + b^{2}every part times every part(a + b) ** 2(a + b) ** 2loss
16ax+b=c, so x=(c−b)/aax + b = c, \ \text{so}\ x = (c - b)/aundo in reverse, both sides(c - b) / atorch.linalg.solve(A, c)batch, update
17y=mx+b, so x=(y−b)/my = mx + b, \ \text{so}\ x = (y - b)/mmake any letter the subject(y - b) / m(y - b) / mbatch
18ReLU(x) = 0 if x ≤ 0, else xa decision by an inequalityif x <= 0:torch.relu(x)activation, metrics and error bars
19f(x): one output per inputa rule with a namedef f(x):y = model(x)loss
20f−1(f(x))=xf^{-1}(f(x)) = xthe inverse undoesf_inv(f(x)) == xtorch.log(torch.exp(x))layer, activation
21m=(y2−y1)/(x2−x1)m = (y_{2} - y_{1})/(x_{2} - x_{1})rise over run(y2 - y1) / (x2 - x1)(y2 - y1) / (x2 - x1)layer
22add or take away equationsmake a letter cancel(c1 - c2) / (a1 - a2)torch.linalg.solve(A, b)layer, loss
23x=(−b±b2−4ac)/2ax = (-b \pm \sqrt{b^{2} - 4ac})/2aroots, and the vertex at −b/2a-b / (2 * a)(-b + torch.sqrt(b * b - 4 * a * c)) / (2 * a)loss
24y=a⋅rty = a\cdot r^{t}the same factor each stepa * r ** ta * r ** tscores, update
25log⁡(ab)=log⁡a+log⁡b\log (ab) = \log a + \log ba product's log is a summath.log(a) + math.log(b)torch.log(a * b)chances, loss
26(1/n)∑(y^i−yi)2(1/n) \sum (\hat{y}_{i} - y_{i})^{2}the mean squared errora loop, then / len(ys)((y_hat - y) ** 2).mean()loss
27m←β⋅m+(1−β)⋅xm \leftarrow \beta \cdot m + (1 - \beta )\cdot xblend each new value into the running onem = beta * m + (1 - beta) * xm = beta * m + (1 - beta) * xloss, update
28f:Rn→Rmf : \mathbb{R}^{n} \to \mathbb{R}^{m}n numbers in, m outlen(x) is nx.shapenumbers

From Review: find the slip (algebra).

Transposition. Rearranging in algebra

LessonFormulaRearrangedConditionIn Python
30y = 3(x − 2)/4 + 5x = 4(y − 5)/3 + 2none: the divisors are numbers4 * (y - 5) / 3 + 2
30v = u − gtt = (u − v)/gg ≠ 0(u - v) / g
311/R = 1/R₁ + 1/R₂R = R₁R₂/(R₁ + R₂)R₁ + R₂ ≠ 0R1 * R2 / (R1 + R2)
31y = a/(b + x)x = a/y − by ≠ 0a / y - b
32A = P + PrtP = A/(1 + rt)1 + rt ≠ 0A / (1 + r * t)
33A = πr²r = √(A/π)keep + for a lengthmath.sqrt(A / math.pi)
33T = 2π√(l/g)l = g(T/2π)²square the whole bracketg * (T / (2 * math.pi)) ** 2
34ax² + bx + c = 0x = (−b ± √(b² − 4ac))/(2a)a ≠ 0, b² − 4ac ≥ 0(-b + math.sqrt(b ** 2 - 4 * a * c)) / (2 * a)
35A = P(1 + r)ⁿn = ln(A/P)/ln(1 + r)A/P > 0, r > −1, r ≠ 0math.log(A / P) / math.log(1 + r)
35p = 1/(1 + e^(−z))z = ln(p/(1 − p))0 < p < 1math.log(p / (1 - p))
36√(p(1 − p)/n) ≤ En ≥ p(1 − p)/E²1 over both sides flips the sign; round upmath.ceil(p * (1 - p) / E ** 2)
36lr₀γᵗ ≤ targett ≥ ln(target/lr₀)/ln γflips: ln γ < 0 for γ < 1; round upmath.ceil(math.log(target / lr0) / math.log(gamma))
37f(x) = (2x + 3)/5f⁻¹(x) = (5x − 3)/2f one-to-one(5 * x - 3) / 2
37z = (x − μ)/σx = μ + zσσ ≠ 0mu + z * sigma
38uₙ = a + (n − 1)dn = (uₙ − a)/d + 1d ≠ 0; n a positive whole number(u - a) / d + 1
38uₙ = a·rⁿ⁻¹n = log(uₙ/a)/log r + 1uₙ/a > 0, r > 0, r ≠ 1; n wholemath.log(u / a) / math.log(r) + 1
39C = 2πr and A = πr²A = C²/(4π)keep the brackets: π(C/(2π))²C ** 2 / (4 * math.pi)
39v = u + at and s = ut + ½at²v² = u² + 2asu² + 2as ≥ 0; v = ±√(u² + 2as)math.sqrt(u ** 2 + 2 * a * s)

From Review: rearrange anything (algebra).

Module 3. Shapes, angles and waves

1cos² 0.25sin² 0.75
The key picture: Across² + up² = 0.25 + 0.75 = 1: Pythagoras on an arm of length 1. The point (cos, sin) is always on the unit circle. From Sine and cosine.
LessonFormulaIn plain wordsIn PythonIn PyTorchWhere it's used
41d=(x2−x1)2+(y2−y1)2d = \sqrt{(x_{2} - x_{1})^{2} + (y_{2} - y_{1})^{2}}square the differences, add, rootmath.sqrt(dx ** 2 + dy ** 2)torch.dist(p, q)vectors
42radians = degrees × π/180a full turn is 2πmath.radians(deg)torch.deg2rad(d)vectors
42s=rθs = r\theta arc = radius × angle in radiansr * thetatorch.deg2rad(d)vectors
43(cos θ, sin θ), with sin² + cos² = 1across and up on the unit circlemath.cos(t), math.sin(t)torch.sin(theta), torch.cos(theta)vectors
44y = A sin(kx + φ), period 2π/kheight, speed and shiftA * math.sin(k * x + phase)A * torch.sin(k * x + phi)vectors

From Review: find the slip (shapes and waves).

Module 4. Vectors and matrices

1.520.50.432.521312.511.904154.15208.510.8B →row 1row 2row 3A ↓col 1col 2
The key picture: Row 3 of A (counts) meets column 2 of B (prices): 5×2 + 2×0.4 + 0×2.5 = 10.8. From Matrix multiplication.
LessonFormulaIn plain wordsShapesIn PythonIn PyTorchWhere it's used
48a+b,caa + b, caentry by entry(n,), (n,) → (n,)a + b, c * aa + b, c * aupdate
49∥v∥=∑vi2\Vert v\Vert = \sqrt{\sum v_{i}^{2}}length(n,) → one numbernp.linalg.norm(v)torch.linalg.norm(v)gradient, update, penalty
50a⋅b=∑aibia \cdot b = \sum a_{i}b_{i}pairs multiplied, then added(n,), (n,) → one numbera @ ba @ blayer
51cos⁡θ=a⋅b÷(∥a∥∥b∥)\cos \theta = a\cdot b \div (\Vert a\Vert \Vert b\Vert )direction only(n,), (n,) → one number(a @ b) / (norm(a) * norm(b))F.cosine_similarity(a, b, dim=0)vectors
52ci=v⋅qi, so v=∑ciqic_{i} = v \cdot q_{i}, \ \text{so}\ v = \sum c_{i}q_{i}amounts along perpendicular unit directions(n, n), (n,) → (n,)Q.T @ vc = Q.T @ v; v_back = Q @ cvectors
53aija_{ij}row i, then column j(m, n)A[i - 1, j - 1]A[i, j]numbers, layer
54A⊤A^{\top}rows become columns(m, n) → (n, m)A.TA.Tbatch
55(Ax)i=∑jAijxj(Ax)_{i} = \sum _{j} A_{ij}x_{j}each row dotted with x(m, n), (n,) → (m,)A @ xA @ xlayer
56(AB)ij=∑kAikBkj(AB)_{ij} = \sum _{k} A_{ik}B_{kj}row i meets column j(m, n), (n, p) → (m, p)A @ BA @ Bbatch, layer
59det⁡=ad−bc\det = ad - bcarea factor; 0 means no inverse(2, 2) → one numbernp.linalg.det(A)torch.linalg.det(A), torch.linalg.inv(A)layer, penalty
60Ax=bAx = belimination, then back-substitution(n, n), (n,) → (n,)np.linalg.solve(A, b)torch.linalg.solve(A, b)loss
61rank\mathrm{rank}independent columns(m, n) → one numbernp.linalg.matrix_rank(A)torch.linalg.matrix_rank(A)weights
62Av=λvAv = \lambda vonly stretched(n, n) → n pairsnp.linalg.eig(A); np.linalg.eigh(A) if A is symmetrictorch.linalg.eig(A)vectors
63A=UΣV⊤A = U\Sigma V^{\top}pieces, biggest first(m, n) → three arraysnp.linalg.svd(A)torch.linalg.svd(A)weights
64sum along an axisthat axis disappears(m, n) → (n,) for axis 0X.sum(axis=0)x.sum(dim=0)batch
65eᵢE = row i of Ea one-hot row picks a row of the table(vocab, d), ids (n,) → (n, d)E[ids]nn.Embedding(vocab, d)(ids)numbers, vectors, scores
66yj=w⋅(xj,…,xj+k−1)y_{j} = w \cdot (x_{j}, \ldots , x_{j+k-1})a pattern slid along, a dot product at each stop(n,), (k,) → (n − k + 1,)x[j:j + k] @ wF.conv1d(x, w)layer

From Review: find the slip (vectors and matrices).

Module 5. Calculus

×4×2×3×6×4t2∂ 36c3∂ 8x4∂ 6y6∂ 4z24∂ 1
The key picture: t feeds more than one step, so its slopes add: 12 + 6 × 4 = 36. From The chain rule with many paths: backpropagation by hand.
LessonRuleIn plain wordsIn PythonIn PyTorchWhere it's used
70Δy/Δx=(f(b)−f(a))/(b−a)\Delta y/\Delta x = (f(b) - f(a))/(b - a)change over changenp.diff(v) / htorch.diff(y) / hloss
72f′(x)=lim⁡(f(x+h)−f(x))/hf'(x) = \lim (f(x + h) - f(x))/hthe slope at a point(f(x + h) - f(x - h)) / (2 * h)y.backward(); x.gradslopes
73(xn)′=nxn−1(x^{n})' = nx^{n-1}(ex)′=ex(e^{x})' = e^{x}(ln⁡x)′=1/x(\ln x)' = 1/xpower down, power loweredn * c[n] per termy.backward(); x.gradslopes
74dy/dx=dy/du⋅du/dxdy/dx = dy/du \cdot du/dxmultiply the linksdy_du * du_dxloss.backward()slopes
75f′ = 0; f″ > 0 bowl, f″ < 0 domeflat, then which way it bendsxs[np.argmin(f(xs))]torch.argmin(f(xs))loss
76f(x)≈f(a)+f′(a)(x−a)+12f′′(a)(x−a)2f(x) \approx f(a) + f'(a)(x - a) + {\tfrac{1}{2}}f''(a)(x - a)^{2}the curve near a pointfa + dfa * d + 0.5 * d2fa * d ** 2fa + dfa * dupdate
77–78∇f=(∂f/∂x,∂f/∂y)\nabla f = (\partial f/\partial x, \partial f/\partial y)all the slopes, steepest climbnudge copies, one input at a timeloss.backward(); w.gradslopes
79θ←θ−η∇L,η<2/L′′\theta \leftarrow \theta - \eta \nabla L, \eta < 2/L''step downhill, not too farw = w - lr * dwtorch.optim.SGD(model.parameters(), lr=0.1)update
80momentum: v ← βv + g, w ← w − ηv; Adam: m ← β₁m + (1 − β₁)g, v ← β₂v + (1 − β₂)g², m̂ = m ÷ (1 − β₁ᵗ), v̂ = v ÷ (1 − β₂ᵗ), w ← w − η·m̂ ÷ (√v̂ + ε)momentum piles up slopes that keep pointing one way; Adam divides the average slope by its usual size, so every weight moves about ηv = 0.9 * v + g; w = w - lr * mh / (np.sqrt(vh) + eps)torch.optim.Adam(params, lr=1e-3, betas=(0.9, 0.999))update
81∂L/∂v=∑k∂L/∂uk⋅∂uk/∂v\partial L/\partial v = \sum _{k} \partial L/\partial u_{k} \cdot \partial u_{k}/\partial vmultiply along a path, add across pathsdL_dv = dL_dv + ...loss.backward()slopes
82ū ← ū + v̄ · ∂v/∂u for each entry v, last to first (ū = ∂L/∂u)record each step forward, walk the tape back; .grad adds up until zeroedloss.backward(), then w.grad; optimizer.zero_grad()loss.backward(); w.grad; opt.zero_grad()slopes, gradient
83Jij=∂fi/∂xjJ_{ij} = \partial f_{i}/\partial x_{j}Hij=∂2f/∂xi∂xjH_{ij} = \partial ^{2}f/\partial x_{i}\partial x_{j}all slopes; all curvaturesJ[:, j] = ..., np.linalg.eigvalsh(H)torch.autograd.functional.jacobian(f, x)slopes
84∫fdx≈∑f(xi)Δx\int f dx \approx \sum f(x_{i})\Delta xthin strips, addeddx * (np.sum(y) - (y[0] + y[-1]) / 2)torch.trapezoid(y, x)metrics and error bars

From Review: find the slip (calculus).

Module 6. Probability

0.010.990.90.10.050.95personsickhealthypositive 0.009negative 0.001positive 0.0495negative 0.9405
The key picture: B happens on two branches; add them: 0.009 + 0.0495 = 0.0585. The four leaves add to 1. Bayes' rule: P(sick | positive) is the sick-then-positive leaf's share of the two lit leaves, 0.009 ÷ 0.0585 = 0.1538. From Conditional probability and Bayes' rule.
LessonRuleIn plain wordsIn PythonIn PyTorchWhere it's used
87n!n!n!/(n−r)!n!/(n - r)!n!/(r!(n−r)!)n!/(r!(n - r)!)orders; ordered picks; groupsmath.factorial(n), math.perm(n, r), math.comb(n, r)math.comb(n, r)attention, cost
88P(A) = ways for A ÷ all ways; P(not A) = 1 − P(A)a share of equally likely outcomesnp.mean(event) over seeded trials(draws == a).float().mean()chances, sampling
89P(A or B)=P(A)+P(B)−P(A and B)P(A \ \text{or}\ B) = P(A) + P(B) - P(A \ \text{and}\ B)add, then remove the overlappa + pb - pab(a & b).float().mean()cost
89P(A and B) = P(A) × P(B), if independentmultiply only when one doesn't change the otherpa * pb(a & b).float().mean()cost
90P(A∣B)=P(A and B)÷P(B)P(A \mid B) = P(A \ \text{and}\ B) \div P(B)count inside the group where B happenedT / T.sum(axis=0)(a & b).sum() / b.sum()chances, metrics and error bars
90Bayes: P(A given B) = P(B given A) P(A) ÷ P(B)turn one conditional into the otherpba * pa / pb(a & b).sum() / b.sum()chances, metrics and error bars
91Σ pᵢ = 1; P(range) = Σ of its pᵢa chance for every valuerng.choice(v, size=n, p=p)torch.multinomial(p, 1)sampling
92E[X]=∑xipiE[X] = \sum x_{i}p_{i}Var=∑(xi−μ)2pi\mathrm{Var} = \sum (x_{i} - \mu )^{2}p_{i}σ=Var\sigma = \sqrt{\mathrm{Var}}balance point; spread, in the data's unitsv @ p, ((v - mu) ** 2) @ p, var ** 0.5(x * p).sum(), x.var()loss
93P(k) = C(n, k) pᵏ(1 − p)ⁿ⁻ᵏ; mean np; z = (x − μ)/σyeses in n trials; distance in σ'smath.comb(n, k) * p ** k * (1 - p) ** (n - k)Binomial(n, p).log_prob(k).exp()weights, init
94cov = mean of (x − x̄)(y − ȳ); r = cov ÷ (σₓσᵧ)moving together, from −1 to 1np.mean(dx * dy), np.corrcoef(x, y)torch.cov(torch.stack([x, y]))numbers

From Review: find the slip (probability).

Module 7. Statistics and information

414/51454/558/10real 1real 0allsaid 1said 0all
The key picture: F1 is the harmonic mean of precision and recall: 2 × 0.8 × 0.8 ÷ (0.8 + 0.8) = 0.8. It is high only when both are. From Scoring a classifier: confusion table, precision, recall, F1.
LessonRuleIn plain wordsIn PythonIn PyTorchWhere it's used
97xˉ=(1/n)∑xi\bar{x} = (1/n) \sum x_{i}s=(1/n)∑(xi−xˉ)2s = \sqrt{(1/n) \sum (x_{i} - \bar{x})^{2}}z=(x−xˉ)/sz = (x - \bar{x})/scenter, spread, distance in s'sx.mean(), np.median(x), x.std(), np.percentile(x, 95)x.mean(), x.std()data, numbers
98SE=σ/n\mathrm{SE} = \sigma /\sqrt{n}xˉ±1.96⋅s/n\bar{x} \pm 1.96\cdot s/\sqrt{n}the wobble of an averagesigma / n ** 0.5x.std() / n ** 0.5batch, metrics and error bars
99dᵢ = bᵢ − aᵢ; d̄ ± 1.96·s_d/√n; P(at least one false win in k tries) = 1 − 0.95ᵏcompare item by item; the lead is real when 0 is outside its ranged = b - a, d.mean(), d.std() / len(d) ** 0.5d = b - a; d.mean(), d.std() / n ** 0.5metrics and error bars
100log⁡L(θ)=∑log⁡P(xi∣θ)\log L(\theta ) = \sum \log P(x_{i} \mid \theta )θ^=argmax⁡\hat{\theta } = \operatorname{argmax}the setting that makes the data least surprisingps[np.argmax(ll)]dist.log_prob(x).sum()loss
101w=cov(x,y)/Var(x)w = \mathrm{cov}(x, y)/\mathrm{Var}(x)b=yˉ−w⋅xˉb = \bar{y} - w\cdot \bar{x}the line with the smallest squared missesnp.linalg.lstsq(X, y, rcond=None)torch.linalg.lstsq(X, y).solutionloss, train and test split
102σ(z) = 1/(1 + e⁻ᶻ); L = −[y log p + (1 − y) log(1 − p)]; slope p − ya score into a chance; the cost of a yes-or-no guess1 / (1 + np.exp(-z))torch.sigmoid(z)scores, chances
103precision = TP/(TP + FP); recall = TP/(TP + FN); F1 = 2PR/(P + R)right out of the model's yeses; found out of the real yesestp = np.sum((y == 1) & (pred == 1))(pred & y).sum() / pred.sum()metrics and error bars
104pi=e(zi)/∑je(zj)p_{i} = e^(z_{i})/\sum _{j} e^(z_{j})scores into chances that add to 1e = np.exp(z - z.max()); e / e.sum()F.softmax(z, dim=-1)softmax
105H(p) = −Σ pᵢ log₂ pᵢ; H(p, q) = −Σ pᵢ log₂ qᵢ; perplexity 2^Haverage surprise; the model's average surprisep @ np.log2(1 / q)F.cross_entropy(logits, y)targets, loss
106KL(p∥q)=∑pilog2(pi/qi)KL(p \Vert q) = \sum p_{i} log_{2}(p_{i}/q_{i})H(p,q)=H(p)+KL(p∥q)H(p, q) = H(p) + KL(p \Vert q)the model's extra surprise over the truth's own; 0 only when they matchp @ np.log2(p / q)F.kl_div(q.log(), p, reduction="sum")loss

From Review: find the slip (statistics and information).

Transposition. Rearranging in every topic

LessonFormulaRearrangedConditionIn Python
46sin θ = o/hθ = arcsin(o/h), h = o/sin θ−1 ≤ o/h ≤ 1; one angle, −90° to 90°; sin θ ≠ 0 for hmath.asin(o / h)
46x = r cos θ, y = r sin θr = √(x² + y²), θ = atan2(y, x)atan(y/x) fails when x < 0math.atan2(y, x)
46y = A sin(ωt + φ)t = (arcsin(y/A) − φ)/ω−A ≤ y ≤ A; A, ω ≠ 0; add the period if t < 0(math.asin(y / A) - phi) / omega
68w_new = w − ηgg = (w − w_new)/ηη ≠ 0; never divide by a vector(w - w_new) / lr
68a·b = ‖a‖‖b‖cos θθ = arccos(a·b/(‖a‖‖b‖))no zero vector; θ from 0 to πnp.arccos(np.clip(c, -1, 1))
68v = βv + g, steadyv = g/(1 − β), g = (1 − β)vβ ≠ 1g / (1 - beta)
69AX = B; XA = BX = A⁻¹B; X = BA⁻¹A square, det A ≠ 0; the inverse on A's sidenp.linalg.solve(A, B); np.linalg.solve(A.T, B.T).T
69y = Wx + bx = W⁻¹(y − b)W square and invertiblenp.linalg.solve(W, y - b)
69XᵀXw = Xᵀyw = (XᵀX)⁻¹XᵀyXᵀX invertiblenp.linalg.solve(X.T @ X, X.T @ y)
86C′(x) = 2ax − b = 0x = b/(2a)a > 0 for a lowest pointb / (2 * a)
86f(x + h) ≈ f(x) + f′(x)hx_new = x − f(x)/f′(x)f′(x) ≠ 0x - f(x) / df(x)
86F(x) = x² + C, F(x₀) = y₀C = y₀ − x₀²one point fixes Cy0 - x0 ** 2
96P(A and B) = P(A) × P(B given A)P(B given A) = P(A and B)/P(A)P(A) > 0; the answer from 0 to 1p_and / p_a
96P(A given B) = P(B given A) × P(A)/P(B)P(A) = P(A given B) × P(B)/P(B given A)P(B given A) > 0; the answer from 0 to 1a_given_b * p_b / b_given_a
96z = (x − μ)/σx = μ + zσσ > 0mu + z * sigma
108SE = σ/√n; m = zσ/√nn = (σ/SE)²; n = (zσ/m)²round n upmath.ceil((z * sigma / m) ** 2)
108p = σ(wx + b)x = (ln(p/(1 − p)) − b)/w0 < p < 1; w ≠ 0(math.log(p / (1 - p)) - b) / w
108s = −log₂ p; L = −ln pp = 2^(−s); p = e^(−L)bits use log₂, nats use ln2 ** -s, math.exp(-L)
116P = n_in·n_out + n_outn_out = P/(n_in + 1)count the bias; a whole numberP // (n_in + 1)
116C ≈ 6ND, D ≈ 20NN = √(C/120), D = C/(6N)rules of thumb, not lawsmath.sqrt(C / 120)
116s = q·k/√dd = (q·k/s)²s ≠ 0, same sign as q·k; d a whole number(qk / s) ** 2
30y = ax + bx = (y − b)/aa ≠ 0(y - b) / a
33y = xⁿx = ±y^(1/n) for even n, y^(1/n) for odd neven n: y ≥ 0; in code, root the size of a negative yy ** (1 / n)
35y = eˣ; p = 1/(1 + e^(−z))x = ln y; z = ln(p/(1 − p))y > 0; 0 < p < 1math.log(p / (1 - p))
46y = sin xx = arcsin y−1 ≤ y ≤ 1; one principal angle, −90° to 90°math.asin(y)
69y = Axx = A⁻¹ydet A ≠ 0; the inverse on A's sidenp.linalg.solve(A, y)
36ax < c, a < 0x > c/athe sign flipsc / a

From Final test: transpose any formula.

Module 8. Putting it together

1111040−811example 421hiddenscores
The key picture: Example 4, output 2: 1×1 + 1×1 − 1 = 1, kept by ReLU. From A layer and a small network as matrices.
LessonFormulaIn plain wordsIn PythonIn PyTorchWhere it's used
109z=w⋅x+bz = w\cdot x + ba=relu(z)a = \mathrm{relu}(z)∂L/∂wj=∂L/∂z⋅xj\partial L/\partial w_{j} = \partial L/\partial z \cdot x_{j}one neuron: sum, squeeze, blame each weight by its inputdz * xtorch.relu(w @ x + b)slopes, update
110Z1=XW1+b1Z_{1} = XW_{1} + b_{1}A1=relu(Z1)A_{1} = \mathrm{relu}(Z_{1})Z2=A1W2+b2Z_{2} = A_{1}W_{2} + b_{2}P=softmax(Z2)P = \mathrm{softmax}(Z_{2})a whole batch through two layersnp.maximum(X @ W1 + b1, 0) @ W2 + b2nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, k))layer, activation, scores
111dZ₂ = (P − Y)/n; dW₂ = A₁ᵀdZ₂; dZ₁ = dZ₂W₂ᵀ where Z₁ > 0; dW₁ = XᵀdZ₁the backward pass, shape by shapeA1.T @ dZ2loss.backward(); opt.step()—
113softmax(QK⊤/d+M)V\mathrm{softmax}(QK^{\top}/\sqrt{d} + M)Vscore, scale, mask, share out, blendA @ VF.scaled_dot_product_attention(q, k, v)attention
114y=x+f(x)y = x + f(x)LN(x)=γ⋅(x−μ)/σ2+ϵ+βLN(x) = \gamma \cdot (x - \mu )/\sqrt{\sigma ^{2} + \epsilon } + \beta add the input back, then recenter and rescale each vectorx + f(x), g * (x - mu) / (var + eps) ** 0.5 + bx + sublayer(x); nn.LayerNorm(d)—
115h = x + Attn(LN(x)); out = h + MLP(LN(h))attention, then a small network, each added backh = x + attn(ln1(x)); h + mlp(ln2(h))nn.TransformerEncoderLayer(d, heads)—
112the same two-layer network, as modulesthe NumPy formulas, named once eachnn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.Linear(4, 3))nn.Sequential(...); opt = torch.optim.SGD(...)—
118acc±1.96⋅acc(1−acc)/n\mathrm{acc} \pm 1.96\cdot \sqrt{\mathrm{acc}(1 - \mathrm{acc})/n}an honest test score(acc * (1 - acc) / n) ** 0.5acc ± 1.96 * (acc * (1 - acc) / n) ** 0.5—

From Capstone: a sign reader from the formulas.