Index Ai Academy
Foundation · Transposition of formulas

Rearranging in calculus: slopes, steps and constants

Setting a slope to zero, Newton's step, the chain rule turned around, constants from a point, and learning rates.

About 45 minutes · Free lesson

What it covers6

Why it matters

Calculus questions often come backward. You know the slope you want and need the x. You know one point on a curve and need its constant. You know the step a training run took and need its learning rate. Once the calculus is done, each one is an ordinary rearrangement, with two conditions to watch: a slope you divide by can't be 0, and a square gives a ± to choose from.

What you'll be able to do

  • Find where a slope is 0 by setting f′(x) = 0 and making x the subject, keeping both roots when a square appears
  • Rearrange the tangent line for the step to a target, and use it as Newton's method for roots and square roots
  • Turn the chain rule around for an inner slope, and fix an integral's constant or upper limit from a known value
  • Solve a gradient step for its learning rate, and find the rate that reaches a bowl's bottom in one step
  • Find a model's best weight in closed form, and see Newton's method behind second-order optimizers
Quick check
Have a guess before you start. A curve's slope is 2x − 10. At which x is the slope 0?
  1. 10
  2. −5
  3. 5
  4. 20
Show the answer

5

2x − 10 = 0, so 2x = 10 and x = 5. Setting a slope to 0 and solving is the first rearrangement in this lesson: it finds a lowest or highest point.

The math: solve the calculus for the letter you need

Purpose. Transposing in calculus: turn each formula of the module around for the letter a question asks about.

A quick recap: f′(x) is the slope of f at x (lesson 73), and f″(x) is how fast that slope changes, the curvature (lesson 75). Once the slope is worked out, the usual moves finish the job.

You knowYou wantRearranged
the slope is 0xsolve f′(x) = 0 for x
the tangent linethe step hh = (target − f(x))/f′(x)
dy/dx and dy/dudu/dxdu/dx = (dy/dx)/(dy/du)
one point on Fthe constant CC = F(x₀) − the rest of F at x₀
a gradient stepthe learning rateη = (w − w_new)/g

A lowest point. A curve f(x) = ax² − bx + c has slope f′(x) = 2ax − b. At the bottom the slope is 0, so 2ax − b = 0. Add b to both sides: 2ax = b. Divide by 2a: x = b/(2a). The conditions: a can't be 0, and a must be above 0 for a bowl with a lowest point.

Quick check
From an earlier lesson: what is the slope of 2x² − 40x + 300?
  1. 4x − 40
  2. 4x² − 40
  3. 4x − 40 + 300
  4. 2x − 40
Show the answer

4x − 40

The power comes down and drops by one: 2x² gives 4x. −40x gives −40, and the constant 300 gives 0.

The tent-rental shop sews its own tents. When it makes x tents a day, the cost per tent is f(x) = 2x² − 40x + 300 dollars: it falls as the workshop gets busy, then climbs with overtime. Step through the slope set to 0.

Try it now

The widget writes the curve as ax² + bx + c, so its b is −40. Set b to −60 and press Use these numbers: x moves to 60 ÷ 4 = 15. Set a to −2: f″ turns negative, and the flat point becomes a highest point.

Symbols and formula.

f(x)=ax2−bx+c  ⇒  f′(x)=2ax−b=0  ⇒  x=b2af(x) = ax^2 - bx + c \;\Rightarrow\; f'(x) = 2ax - b = 0 \;\Rightarrow\; x = \frac{b}{2a}

In plain words: find the slope, set it to 0, then make x the subject with the usual moves.

When a square appears. The slope of x³ is 3x². Where is that slope 12? Divide by 3: x² = 4. Take the square root: x = ±2. Both work, since (−2)² is 4 too. A square hides the sign, so undoing it gives two answers. The question then decides which ones to keep.

Quick check
f(x) = x³ − 12x. Where is its slope 0?
  1. x = 2 or x = −2
  2. x = 2 only
  3. x = 12
  4. x = 4 or x = −4
Show the answer

x = 2 or x = −2

f′(x) = 3x² − 12 = 0, so x² = 4 and x = ±2. Check: 3 × (−2)² − 12 = 0. This curve has a top at −2 and a bottom at 2.

An area that must hit a target. A runner's speed is v = 2t meters per second, t seconds after the start. The distance covered by time a is the area under the speed line from 0 to a (lesson 84). That area is a triangle with base a and height 2a, so ∫₀ᵃ 2t dt = ½ × a × 2a = a². When is the distance 9 meters? a² = 9 gives a = ±3. Both solve the equation, but −3 seconds is before the start, so keep a = 3.

The machine below shows why a square needs a ±. It builds 3x² from x = −2, then peels it back.

A step to a target. Near x, the tangent line stands in for the curve (lesson 76): f(x + h) ≈ f(x) + f′(x)h. To land on a target, set that equal to the target and make h the subject. Take f(x) away, then divide by f′(x): h = (target − f(x))/f′(x).

Newton's method. To find where f(x) = 0, use a target of 0. Then h = −f(x)/f′(x), and Newton's step is

xnew=x−f(x)f′(x)x_{\text{new}} = x - \frac{f(x)}{f'(x)}

The tangent is only a stand-in, so one step lands close, not exactly. Repeat it.

A square root. √10 is the x above 0 with f(x) = x² − 10 = 0, and f′(x) = 2x. From x = 3: 3 − (9 − 10)/6 = 3.166667. Once more gives 3.162281, and √10 = 3.162278.

The condition. f′(x) can't be 0. At x = 0 the tangent of x² − 10 is flat: it never reaches 0, and the step divides by 0.

Quick check
Newton's step for x² − 10 = 0 starts at x = 0. What goes wrong?
  1. x² − 10 has no root
  2. It divides by f′(0) = 0
  3. It lands exactly on √10
  4. It finds −√10 instead
Show the answer

It divides by f′(0) = 0

f′(x) = 2x is 0 at x = 0. The tangent there is flat, so it never meets 0, and x − f(x)/f′(x) divides by 0. Start anywhere else.

The chain rule, turned around. dy/dx = (dy/du)(du/dx) multiplies two slopes (lesson 74): y depends on an inner u, and u on x. If you know the whole slope and the outer one, divide: du/dx = (dy/dx)/(dy/du), as long as dy/du isn't 0. For y = u² at u = 3, dy/du = 2u = 6. If dy/dx = 12, then du/dx = 12 ÷ 6 = 2.

A constant from a point. An antiderivative of f is a function F whose slope is f: it undoes taking the slope, the way summing undoes differencing in lesson 84. The runner's distance t² is one for the speed 2t. But x² + 4 and x² − 7 also have slope 2x, since a constant adds no slope. So every antiderivative of 2x is F(x) = x² + C, where C, the constant of integration, is unknown. One known point fixes it. If F(1) = 5, then 1² + C = 5, so C = 4: the curve x² lifted 4 units.

Named mistakes. Dropping the − root of x² = 4. Writing Newton's step with a plus. Turning the chain rule upside down, (dy/du)/(dy/dx). Taking C to be 5, the value of F, without taking away the x² part.

In your own words
Solving 3x² = 12 gives two answers, but Newton's method for √10 from x = 3 finds only one. Why the difference?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: rearranged formulas as functions

Purpose. Write each turned-around formula as a function. Put its condition first, and check it with a round trip.

def newton_sqrt(n, x, steps):
    for i in range(steps):
        x = x - (x * x - n) / (2 * x)
        print(i + 1, round(x, 6))
    return x

r = newton_sqrt(10, 3, 3)
print(round(r * r, 9))
It runs right here in your browser, with nothing to install.
  • Each pass is one Newton step for f(x) = x² − n, with f′(x) = 2x.
  • The error shrinks fast: about 0.004 after one step, 0.000003 after two.
  • Squaring the answer gives back 10. That's the round trip.
Worked example

Worked example: a constant and a time from integrals

Two rearrangements, each with its condition: C = F(x₀) − x₀² for F(x) = x² + C, and the time T with T² = distance, keeping T at or above 0.

import math

def constant_from_point(x0, F0):
    return F0 - x0 ** 2

def time_to_cover(distance):
    if distance < 0:
        return None
    return math.sqrt(distance)

C = constant_from_point(1, 5)
print(C, 1 ** 2 + C)
print(time_to_cover(9), time_to_cover(0), time_to_cover(-4))
In the lesson you can change this and run it.

C = 5 − 1 = 4, and putting x = 1 back into x² + 4 gives 5: the point is on the curve. math.sqrt returns only the root at or above 0, which is the one a time needs, so 9 meters takes 3.0 seconds. No square is negative, so a distance of −4 has no time at all, and the function says None instead of crashing.

Quick check
What does this print?
x = 3
print(round(x - (x * x - 10) / (2 * x), 4), round(x + (x * x - 10) / (2 * x), 4))
  1. 2.8333 3.1667
  2. 3.1667 3.1667
  3. 3.0 3.0
  4. 3.1667 2.8333
Show the answer

3.1667 2.8333

f(3) = −1 and f′(3) = 6, so f/f′ = −0.1667. Subtracting it moves up to 3.1667, toward √10 ≈ 3.1623. Adding it moves down to 2.8333, away from the root.

Quick check
dy/dx = 12 and dy/du = 6 are stored as dy_dx and dy_du. Which line gives du/dx?
  1. dy_dx * dy_du
  2. dy_du / dy_dx
  3. dy_dx / dy_du
  4. dy_dx - dy_du
Show the answer

dy_dx / dy_du

dy/dx = (dy/du)(du/dx), so divide both sides by dy/du: 12 ÷ 6 = 2. Check: 6 × 2 = 12.

Drill

Fix the Newton step and the inner slope

newton_step adds the correction, and inner_slope divides upside down. Fix both, and return None when the number you divide by is 0.

def newton_step(x, f_x, slope):
    return x + f_x / slope

def inner_slope(dy_dx, dy_du):
    return dy_du / dy_dx

print(newton_step(3, -1, 6), inner_slope(12, 6))
In the lesson, your code is checked as you write it.
In your own words
Each function here checks the number it divides by before dividing. What would go wrong without that check?

Write a sentence or two, then compare it with a good answer in the lesson.

In AI and ML: the best weight and the learning rate

Purpose. Training runs the calculus forward. Planning a run, or reading its logs, runs it backward.

The learning rate from a step. A gradient step is w_new = w − ηg, with g the slope of the loss (lesson 79). Make η the subject: take w away and change signs, so ηg = w − w_new. Divide by g: η = (w − w_new)/g, as long as g isn't 0. Read backward, a logged step tells you the rate. Read forward: a step's size is η × |g|, so a step of a size s you choose needs η = s/|g|.

Quick check
A log shows a weight moving from 1.0 to 0.94 with gradient 1.5. What does this print?
w, w_new, g = 1.0, 0.94, 1.5
eta = (w - w_new) / g
print(round(eta, 4), round(w - eta * g, 4))
  1. -0.04 1.06
  2. 0.04 0.94
  3. 0.04 1.06
  4. 0.09 0.94
Show the answer

0.04 0.94

w − w_new = 0.06, and 0.06 ÷ 1.5 = 0.04. Putting it back, 1.0 − 0.04 × 1.5 = 0.94: the round trip lands on the logged weight.

The best weight in one line. A model ŷ = w·x predicts tent rentals from temperature, scored by L(w) = Σ(wx − y)². Its slope is L′(w) = 2Σx(wx − y) = 2(wΣx² − Σxy). Set it to 0 and make w the subject:

w=∑xy∑x2w = \frac{\sum xy}{\sum x^2}

In plain words: the best weight is the inputs times the targets, added up, over the inputs squared, added up.

This is a closed-form answer: no training loop at all. Its condition: Σx² isn't 0, so not every x is 0. Here it is on the three days of tent data from lesson 72. That lesson averaged the squared errors; adding them instead doesn't move the bottom, which it found between 2 and 3:

import numpy as np

x = np.array([1.0, 2.0, 3.0])
y = np.array([2.0, 4.0, 7.0])

def slope(w):
    return 2 * np.sum(x * (w * x - y))

best = np.sum(x * y) / np.sum(x * x)
rate = 1 / (2 * np.sum(x * x))
print(round(best, 4), slope(0.0))
print(round(0.0 - rate * slope(0.0), 4))
Quick check
In w = Σxy/Σx², when is there no best weight?
  1. When every x is 0
  2. When w comes out below 0
  3. When there are 3 points
  4. When every y is 0
Show the answer

When every x is 0

Σx² is 0 only when every x is 0. Then the loss doesn't change with w at all, so no weight is better than another.

One step to the bottom. That second print used a special rate. A loss L = a(w − t)² is a bowl with its bottom at t, and slope 2a(w − t). Ask for the η whose step lands on t: t = w − η × 2a(w − t). Then w − t = 2aη(w − t). Divide by w − t (when w isn't already t): η = 1/(2a). Our tent loss is this bowl plus a constant, with a = Σx² = 14, so η = 1/28, and one step from 0 lands on 2.2143. The widget calls the weight x.

Newton's idea in optimizers. A lowest point is where the slope L′ is 0. So use Newton's step on the slope itself: w_new = w − L′(w)/L″(w). That's gradient descent with η = 1/L″. For a bowl, L″ = 2a: Newton reaches a bowl's bottom in one step. Optimizers that use L″ are called second-order methods. With a million weights, L″ is the Hessian (lesson 83), a table of second slopes with a million rows and a million columns: far too big to store. Quasi-Newton methods such as L-BFGS build a rough stand-in from the last few steps and gradients. They suit smaller models; large networks mostly train with gradient descent and its variants. The condition is Newton's again: L″ can't be 0, and it must be above 0 to head for a bottom.

Drill

The best weight and the one-step rate

Write best_weight(x, y), the w that makes Σ(wx − y)² smallest (return None when Σx² is 0), and one_step_rate(x), the rate 1/(2Σx²) that reaches it in one step.

import numpy as np

def best_weight(x, y):
    return np.sum(x * y) / np.sum(x)

def one_step_rate(x):
    return 2 * np.sum(x * x)

x = np.array([1.0, 2.0, 3.0])
y = np.array([2.0, 4.0, 7.0])
print(best_weight(x, y), one_step_rate(x))
In the lesson, your code is checked as you write it.
In your own words
Someone says a second-order optimizer needs no learning rate. In what sense is that true, and what does it cost?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

f(x) = ax² − bx + c. Make x the subject of f′(x) = 2ax − b = 0.

  1. x = b/(2a)
  2. x = 2a/b
  3. x = −b/(2a)
  4. x = b − 2a

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Foundation · about 96 hours of study

Foundation math

From whole numbers to the math inside a neural network: each idea by hand, in Python, and in AI 118 lessons All free
  1. Numbers and place value FreeWhat a digit's place is worth, the number line, and Python as a calculator that never tires.36 min
  2. Multiplying and dividing: groups FreeEqual groups, quotient and remainder, and cutting training data into batches.35 min
  3. Order of operations, and names for numbers FreeBrackets, powers, then multiply and divide, then add and subtract; variables hold numbers.36 min
  4. Numbers below zero FreeThe number line through zero, sign rules, distance from zero, and an error's direction and size.37 min
  5. Fractions: equal parts, equal fractions, adding FreePieces of a whole, equal fractions, and re-cutting to add; accuracy as a fraction.45 min
  6. Multiplying and dividing fractions FreeA share of a share, dividing by flipping, and why chances in a row get small.35 min
  7. Decimals and rounding FreeFractions in place value, rounding, and why 0.1 + 0.2 isn't exactly 0.3.36 min
  8. Percentages FreePer hundred, percent change, percentage points, and the accuracy and error rates of models.35 min
  9. Ratios, rates and proportion FreeSharing in a ratio, rates per unit, scaling, and splitting data 80/10/10.35 min
  10. Powers and roots FreeRepeated multiplying, the laws of exponents, square roots, and squared error.43 min
  11. Very big and very small numbers FreePowers of ten, e-notation like 3e-4, orders of magnitude, and a model's size in memory.36 min
  12. Bits, bytes and floats: how numbers are stored FreeBase 2, what n bits can hold, a float as sign, power and digits, and a tensor's memory.35 min
  13. Review: find the slip (numbers) FreeBroken calculations to fix, mixed practice from the whole module, and its formula sheet.34 min
  14. Letters for numbers FreeVariables, terms and like terms, substitution, and a model's formula as a Python function.41 min
  15. Expanding and factoring FreeBrackets as areas, pulling out common factors, and a squared error expanded into a bowl.41 min
  16. Solving equations: keep the balance FreeThe same move on both sides, undoing steps in reverse, and running a model's formula backward.41 min
  17. Rearranging a formula FreeMaking any letter the subject, round-trip checks, and planning a training run.36 min
  18. Inequalities and true-or-false FreeRanges, flipping the sign, if and elif, and the thresholds, clips and ReLU inside a model.42 min
  19. Functions: a rule from input to output Freef(x) notation, tables and graphs, domain and range, loops over lists, and loss as a function.42 min
  20. Chaining and undoing functions Freeg(f(x)), inverses that undo, scaling and unscaling, and a network as a chain.42 min
  21. Straight lines: slope and intercept FreeRise over run, the line through two points, and a model's weight and bias as slope and intercept.42 min
  22. Two equations, two unknowns FreeSubstitution and elimination, where lines cross, and why models are fitted, not solved.42 min
  23. Quadratics: bowls and their lowest point FreeRoots by factoring and by formula, the discriminant, the vertex, and the best weight of a model.42 min
  24. Exponentials: growth and decay FreeThe same factor each step, the number e, and decaying learning rates.43 min
  25. Logarithms: the power question FreeTo what power?, the laws of logs, and adding log-probabilities instead of multiplying.45 min
  26. Sequences and Σ FreeArithmetic and geometric sequences, Σ as a loop that adds, and mean squared error.40 min
  27. Weighted and moving averages FreeWeights that add to 1, a window average, the exponential moving average, and smoothing a loss curve.35 min
  28. Sets, and reading the symbols FreeSets, union and intersection, ℝⁿ and f : ℝⁿ → ℝᵐ, and reading a paper's first equation.41 min
  29. Review: find the slip (algebra) FreeBroken algebra and code to fix, mixed practice from the whole module, and its formula sheet.34 min
  30. Transposing step by step: undo in reverse order FreeA formula's layers, the opposite of each, peeling from the outside in, conditions, and round-trip checks.35 min
  31. Formulas with fractions: clear them first FreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
  32. When the letter appears twice: collect and factor FreeGathering the letter on one side, factoring it out, dividing by the bracket, and rearranging precision and F1.35 min
  33. Squares, roots and powers FreeSquare roots with ±, squaring to undo a root, cube and nth roots, and the √d inside attention.40 min
  34. When the letter is squared and plain: quadratics FreeRearranging into ax² + bx + c = 0, the formula, completing the square, and choosing the root that fits.35 min
  35. Letters in the power: exponentials and logarithms FreeTaking logs to bring a power down, exponentiating to free a letter, and the logit that undoes the sigmoid.40 min
  36. Rearranging inequalities: when the sign flips FreeThe four moves on an inequality, negatives and reciprocals that flip it, logs below 1, and sample sizes.35 min
  37. Inverse functions, lines and scaling FreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
  38. Sequences and sums: find n, a, d and r FreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
  39. Combining formulas: substitute and eliminate FreePutting one formula inside another to remove a letter, and sizing a training run from a time or money budget.35 min
  40. Review: rearrange anything (algebra) FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
  41. Coordinates and distance FreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
  42. Angles, degrees and radians FreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
  43. Sine and cosine FreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
  44. Waves: height, speed and shift FreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
  45. Review: find the slip (shapes and waves) FreeBroken geometry and code to fix, mixed practice from the module, and its formula sheet.29 min
  46. Angles, triangles and waves: rearranging with sin, cos and tan FreeSides and angles from SOH CAH TOA, principal values, arcs, polar coordinates, and a wave's time and period.45 min
  47. Vectors: lists of numbers that point FreeA list, a point and an arrow; components, dimension, NumPy arrays, and data as vectors.42 min
  48. Adding and scaling vectors FreeComponent-wise sums, scalar multiples, mixes, and the weight update w − ηg.43 min
  49. Length and distance: norms FreeL2, L1 and max norms, unit vectors, distance, and the norms inside losses and clipping.42 min
  50. The dot product FreeMultiply in pairs and add; a·a as squared length; a @ b; one neuron as a dot product.43 min
  51. Angle, cosine similarity and projection FreeDirection without length, perpendicular vectors, shadows, and search by meaning.45 min
  52. Perpendicular directions: bases and coordinates FreeOrthonormal directions, coordinates as dot products, rebuilding a vector, and taking a direction out.35 min
  53. Matrices: tables of numbers FreeRows, columns and entries; two-dimensional arrays and slices; a data set as a matrix.41 min
  54. Adding, scaling and transposing matrices FreeEntry-by-entry sums and scalings, the transpose, mean images, and tables turned around.36 min
  55. A matrix times a vector FreeEach row dotted with the vector, the size rule, A @ x, and a layer acting on one example.42 min
  56. Matrix multiplication FreeRow meets column, the size rule, order matters, and a batch through a layer: X W + b.45 min
  57. Identity, diagonal, triangular and symmetric FreeFour patterns and their jobs: rescaling, distance tables, and the causal mask.37 min
  58. Matrices as movements of space FreeStretches, flips, shears and rotations; columns as landing spots; chaining; a layer moving data.42 min
  59. Determinant and inverse FreeThe area factor ad − bc, singular matrices, undoing a matrix, and redundant features.42 min
  60. Solving systems by elimination FreeAugmented matrices, row moves, back-substitution, the three outcomes, and fitting weights.45 min
  61. Independence, span and rank FreeRedundant vectors, span, rank, outer products, and low-rank updates like LoRA.42 min
  62. Eigenvectors: directions that only stretch FreeAv = λv, checking candidates, np.linalg.eig, and the main direction of a data cloud.41 min
  63. SVD and PCA: the important directions first FreeA matrix as pieces by size, the best rank-k copy, compression, and PCA maps of data.45 min
  64. Tensors, shapes and broadcasting FreeAxes and shapes, reshape, axis=, broadcasting, and shapes through a small model.43 min
  65. One-hot vectors and lookup tables: embeddings FreeA one-hot row picks a table's row, a sentence looked up as vectors, and a score for every word.35 min
  66. Sliding dot products: convolution FreeA small pattern slid along a list and over a grid, output sizes, stride and padding, and shared weights.40 min
  67. Review: find the slip (vectors and matrices) FreeShape slips to fix, mixed practice from the module, and its formula sheet with shapes.36 min
  68. Vector equations: rearranging with arrows FreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
  69. Matrix equations: multiply by the inverse, on the correct side FreeNo dividing by a matrix: inverses from the left or right, order, transposes, and the normal equations.40 min
  70. Rates of change FreeChange over change, secants, rates from readings with np.diff, and reading a loss curve.41 min
  71. Limits: what a value heads toward FreeApproaching from both sides, holes, corners and jumps, epsilon, and judging activations.37 min
  72. The derivative: slope at a point FreeShrinking secants, f′(x) and dy/dx, Δy ≈ f′Δx, and which way a weight should move.45 min
  73. Derivative rules FreeConstants, powers, multiples, sums, eˣ and ln x, checked numerically, and a squared error's slope.41 min
  74. The product rule and the chain rule FreeProducts and chains of functions, forward then backward, and one neuron's dL/dw.45 min
  75. Lowest points and curvature Freef′ = 0, the second-derivative test, tables, local and global minima, and convex losses.43 min
  76. Standing in for a curve near a point FreeTangent lines, a curvature term, growing error, and why learning rates must be small.37 min
  77. Partial derivatives FreeOne input at a time, ∂, nudging in Python, and the two slopes of a line's loss.41 min
  78. The gradient FreeAll the slopes as a vector, steepest climb and descent, a numerical gradient, and the loss gradient.42 min
  79. Gradient descent FreeStepping against the gradient, the learning rate, a training loop, and fitting a line.45 min
  80. Faster descent: momentum and Adam FreeMomentum and Adam by hand, the bias correction, and three optimizers down one narrow valley.40 min
  81. The chain rule with many paths: backpropagation by hand FreeGraphs, multiply along paths and add across them, gradient checks, and a 2-2-1 network by hand.45 min
  82. Automatic differentiation: how PyTorch finds every slope FreeA tape of steps walked backward, and requires_grad, backward(), .grad, zero_grad() and no_grad().40 min
  83. Jacobians and Hessians FreeSlopes of vector functions, a layer's Jacobian, second partials, and bowls, domes and saddles.42 min
  84. Integrals: adding up thin slices FreeStrip and trapezoid sums, summing undoes differencing, probability as area, and AUC.43 min
  85. Review: find the slip (calculus) FreeCalculus slips to fix, mixed practice from the module, and its formula sheet.36 min
  86. Rearranging in calculus: slopes, steps and constants FreeSetting a slope to zero, Newton's step, the chain rule turned around, constants from a point, and learning rates.45 min
  87. Counting: choices, orders and groups FreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
  88. Probability: from counts to chances FreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
  89. Combining events: and, or, independent FreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
  90. Conditional probability and Bayes' rule FreeConditionals from tables, why P(A | B) isn't P(B | A), Bayes' rule, and precision and recall.45 min
  91. Random variables and distributions FreeUncertain numbers, the probability of a range, sampling by slices, and sampling versus argmax.42 min
  92. Expected value, variance and standard deviation FreeProbability-weighted averages, spread in the data's own units, and a training loss as an expected value.42 min
  93. Bernoulli, binomial, uniform and normal FreeYes-or-no trials, counts of yeses, the 68-95-99.7 rule, z-scores, and an accuracy's error bar.45 min
  94. Two variables at once: covariance and correlation FreeJoint tables and marginals, covariance, correlation, the covariance matrix, and features that carry the same information.45 min
  95. Review: find the slip (probability) FreeProbability slips to fix, mixed practice from the module, and its formula sheet.36 min
  96. Probability formulas: solve for the missing chance FreeConditionals, Bayes' rule for the prior, the or rule, expected values, variance, and z-scores turned around.45 min
  97. Describing data: center, spread and scale FreeMean, median and mode, variance and standard deviation, percentiles, outliers, and standardizing features.42 min
  98. Samples, the law of large numbers and the central limit theorem FreeSamples and populations, the standard error σ/√n, the bell of averages, and error bars on test scores.43 min
  99. Is the difference real? Comparing two models FreePaired differences, a bootstrap interval, what a p-value is and isn't, and why many tries make lucky wins.40 min
  100. Maximum likelihood FreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
  101. Linear regression: the best line FreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
  102. Logistic regression: a line that answers yes or no FreeThe sigmoid, the log loss, its slope p − y, a training loop, and choosing a threshold.45 min
  103. Scoring a classifier: confusion table, precision, recall, F1 FreeThe four cells, accuracy, precision, recall and F1, and what moving a threshold does to each.35 min
  104. Softmax: scores into probabilities FreeExponentiate and share out, shifting and temperature, the safe softmax, and a classifier's last step.43 min
  105. Surprise, entropy and cross-entropy FreeSurprise in bits, entropy, cross-entropy and KL divergence, perplexity, and a language model's loss.45 min
  106. KL divergence: how far one distribution is from another FreeKL by hand, cross-entropy as entropy plus KL, why it isn't symmetric, and a soft-target loss.35 min
  107. Review: find the slip (statistics and information) FreeStatistics slips to fix, mixed practice from the module, and its formula sheet.38 min
  108. Statistics and information: n, z, logits and bits FreeSample sizes from error bars, inputs from probabilities, logit gaps from softmax, and chances from bits and losses.40 min
  109. One neuron, start to finish FreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
  110. A layer and a small network as matrices FreeTwo layers on a batch, the shape of every matrix, adding a hidden unit, and XOR solved.45 min
  111. Training a small network FreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
  112. The same network in PyTorch Freenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
  113. Attention as matrix math FreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
  114. Residual connections and layer norm FreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
  115. A transformer block, start to finish FreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
  116. Planning a model: rearranging the formulas of a working network FreeLayer widths from parameter counts, sizes from memory and compute, the attention scale, and a training plan.40 min
  117. Final test: transpose any formula FreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
  118. Capstone: a sign reader from the formulas FreeBuild, train and test a trail-sign classifier with NumPy alone, and report its accuracy with an error bar.45 min
Part of a free course 118 lessons

The Foundation Math For Ai & ML: from arithmetic to attention

Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.

Part of a free course 19 lessons

Transposition of Formulas: algebraic manipulation in every topic

Learn to make any letter of any formula the subject: layers undone in reverse order, fractions, a letter that appears twice, powers, roots, quadratics, logs, inequalities, trigonometry, vectors, matrices, calculus, probability, statistics and the formulas inside AI models, each checked in Python.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write