Calculus is the math of change, and it starts with a question you already answer every day: how fast? A hiker covering 7 km in 2 hours averaged 3.5 km an hour. That number, change in one thing divided by change in another, is a rate, and this whole module is built on it. Shrink the interval and you get the slope at a single point, the derivative; follow the slope downhill and you get gradient descent, which is how every model in this academy is trained. This first lesson stays with the average rate: from tables, from graphs and from formulas, then in a few lines of NumPy, and finally on a real training run, where the rate of the loss says whether learning is still happening or has stalled.
What you'll be able to do
Compute an average rate of change, Δy/Δx, from a table, a graph or a formula
Read the average rate between two points as the slope of the line joining them
Turn a list of readings into a list of rates with np.diff, dividing by the step
Read a training run's loss curve: where it falls fastest, and where it has flattened
Quick check
Have a guess before you start. A hiker is 4 km along the trail after 1 hour and 10 km along after 3 hours. How fast did they walk, on average, between those times?
10 km an hour
6 km an hour
3 km an hour
3.3 km an hour
Show the answer
3 km an hour
They covered 10 − 4 = 6 km in 3 − 1 = 2 hours: 6 ÷ 2 = 3 km an hour.
The math: change over change
Purpose. Turn "how fast" into a number.
A rate of change compares how much an output changed with how much its input changed. Between two points (a, f(a)) and (b, f(b)):
ΔxΔy=b−af(b)−f(a)
In plain words: the change in y divided by the change in x. Δ ("delta") means "change in".
For f(x) = x² between x = 1 and x = 3: (9 − 1) ÷ (3 − 1) = 4. Over that stretch, y grows 4 times as fast as x, on average.
Quick check
From lesson 21: what is the slope of the line through (1, 1) and (3, 9)?
2
4
5
8
Show the answer
4
Rise over run: (9 − 1) ÷ (3 − 1) = 8 ÷ 2 = 4. The average rate of x² from 1 to 3 is exactly this slope.
Rates from a table. Readings taken at equal steps give a whole list of rates: subtract each reading from the next, then divide by the step. Here a hiker's distance is read every half hour: 0, 2.5, 4.5, 6 and 7 km. Watch the neighbors subtract, then divide by 0.5.
The picture. Each reading is a bar, labeled with its time in hours. A step lights two neighbors and shades the band between their tops: the band's height is the difference, written on the later bar. On the last step the bars become the rates themselves, one for each half hour.
Try it now
Then set the time between readings to 1 and press Use these numbers: the same differences, read as if the readings were an hour apart, give speeds half as big.
Symbols and formula. For readings y₀, y₁, … taken a step h apart:
ratei=hyi+1−yi
In plain words: each neighbor minus the one before, divided by the step.
The slope of a secant. On a graph, the average rate between two points is the slope of the straight line, a secant, joining them. A steep secant means fast change; a flat one, slow change; one sloping down, a falling value. The hiker's speeds, 5, 4, 3 and 2 km an hour, say they slowed down steadily, which the curve of distance shows as a line bending flatter.
Quick check
A temperature reads 12, 15, 19, 22 and 23 degrees, every 2 hours. What is the fastest rate of warming?
1.5 degrees an hour
2 degrees an hour
4 degrees an hour
11 degrees an hour
Show the answer
2 degrees an hour
The differences are 3, 4, 3 and 1; divided by 2 hours, the rates are 1.5, 2, 1.5 and 0.5. The fastest is 2.
Differences of differences. Take the rates of a list of rates and you learn whether the change is speeding up or slowing down. For x² at x = 1, 2, 3, 4, 5 the values are 1, 4, 9, 16 and 25; the differences are 3, 5, 7 and 9, growing; and the differences of those are all 2. A constant second difference is the mark of a quadratic, just as a constant first difference is the mark of a straight line.
In your own words
In your own words: why must a difference be divided by its step to be a rate?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: np.diff, then divide by the step
Purpose. Turn a whole list of readings into rates in two lines.
import numpy as np
d = np.array([0, 2.5, 4.5, 6, 7])
h = 0.5print(np.diff(d).tolist())
print((np.diff(d) / h).tolist())
It runs right here in your browser, with nothing to install.
np.diff subtracts each value from the next, so a list of 5 readings gives 4 differences. Dividing by the step turns them into speeds in km per hour. The average rate across the whole walk is the total change over the total time, (7 − 0) ÷ 2 = 3.5, which is also the average of the four speeds when the steps are equal.
Code flow. Here is what those two lines do, written out as a loop on a plain list, with NumPy's answer at the end to compare. Step through it: i picks each pair of neighbors in turn, and rates grows by one box each pass, the same boxes as the flow's bottom row.
A named mistake: forgetting to divide by the step. The differences alone are only rates when the step is 1. With readings every 0.5 hours they're half the true speeds; with readings every 100 training steps they're 100 times too big. The two lines differ only by / h, which makes this an easy slip, and the result still looks plausible.
Second differences. Apply np.diff twice to see whether the rate itself is changing:
import numpy as np
v = np.array([1, 4, 9, 16, 25])
print(np.diff(v).tolist(), np.diff(np.diff(v)).tolist())
Worked example
Worked example: rates for three tables
Three sets of readings were taken with different steps. Compute each one's rates and its fastest change.
import numpy as np
tables = [(np.array([12, 15, 19, 22, 23]), 2),
(np.array([3.0, 3.4, 4.0, 4.2]), 0.25),
(np.array([0, 2.5, 4.5, 6, 7]), 0.5)]
for values, h in tables:
rates = np.diff(values) / h
print(np.round(rates, 4).tolist(), round(float(np.max(rates)), 4))
In the lesson you can change this and run it.
Each tuple holds the readings and their step, unpacked by the for line. The temperatures rise fastest, 2 degrees an hour, between readings 2 and 3. The second table rises fastest at 2.4 per second, in its middle step. The hike's fastest speed is its first, 5 km an hour. np.max picks the biggest rate; round and float tidy it for printing.
Quick check
What does this print?
import numpy as np
v = np.array([2, 5, 11, 20])
print(np.diff(v).tolist(), (np.diff(v) / 3).tolist())
[3, 6, 9] [1.0, 2.0, 3.0]
[5, 11, 20] [1.0, 2.0, 3.0]
[3, 6, 9, 20] [1.0, 2.0, 3.0, 6.67]
[3, 6, 9] [3, 6, 9]
Show the answer
[3, 6, 9] [1.0, 2.0, 3.0]
np.diff gives 5 − 2, 11 − 5 and 20 − 11. Dividing by the step of 3 gives the rates 1, 2 and 3.
Quick check
Readings are taken every 10 minutes and you want a rate per minute. Which line is right?
np.diff(v)
np.diff(v) / 10
v / 10
np.diff(v) * 10
Show the answer
np.diff(v) / 10
Each difference covers 10 minutes, so dividing by 10 gives the change per minute.
Drill
Rates from readings
Fix rates(values, h) so it returns the list of rates for readings taken a step h apart. Then set fast to the position of the fastest rate for the temperatures below.
import numpy as np
defrates(values, h):
return np.diff(values)
temps = np.array([12, 15, 19, 22, 23])
fast = 0print(rates(temps, 2), fast)
In the lesson, your code is checked as you write it.
In your own words
Why does a list of n readings give only n − 1 rates?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: is training still improving?
Purpose. Read a training run by the rate of its loss.
While a model trains, its loss, how wrong it is, is recorded every so often. The loss itself says how good the model is; its rate of change says whether training is still helping:
rate of loss=steps betweenLlater−Learlier
In plain words: a big negative rate means the loss is falling fast; a rate near 0 means it has flattened.
import numpy as np
loss = np.array([2.0, 1.2, 0.8, 0.6, 0.55, 0.53, 0.52])
r = np.diff(loss) / 100print(np.round(r, 6).tolist())
print(np.argmin(r), round(float(r[-1]), 6))
The loss was recorded every 100 steps. Its rate starts at −0.008 per step and shrinks to −0.0001: training improved fast at first and has nearly stopped. np.argmin gives the position of the most negative rate, the fastest fall: position 0, the first 100 steps.
Quick check
The loss rate over the last 1,000 steps is −0.00001 per step. What does that suggest?
The model is getting worse
Training is improving fast
Training has flattened
The learning rate must be 0
Show the answer
Training has flattened
A rate that close to 0 means the loss has almost stopped falling.
Reading the rate, not just the loss. Two runs can end at the same loss and still differ: one may have flattened long ago, the other still falling steadily. The rate tells them apart, which is why training dashboards plot both. Draw them yourself, the loss and its change over each 100 steps, on one set of axes:
import numpy as np
import matplotlib.pyplot as plt
steps = np.array([0, 100, 200, 300, 400, 500, 600])
loss = np.array([2.0, 1.2, 0.8, 0.6, 0.55, 0.53, 0.52])
plt.plot(steps, loss)
plt.plot(steps[1:], np.diff(loss))
plt.xlabel("training step")
plt.ylabel("loss (top), change per 100 steps (bottom)")
plt.show()
The lower line is np.diff(loss), each change drawn at the end of its 100 steps, so it starts at step 100. Divided by 100 it would be the rate per step: the same shape, squeezed toward 0.
Quick check
Read the plot. Where does the lower line, the change, come closest to 0?
Nowhere: it stays the same height throughout
In the middle, at step 300
At the left end, where the loss is highest
At the right end, where the loss has flattened
Show the answer
At the right end, where the loss has flattened
The change rises toward 0 as the loss curve flattens: by step 600 each 100 steps barely lower the loss. The steepest stretch of the loss, at the start, is where the change is most negative.
What people do with it. A flattened loss is the usual signal to stop training, or to lower the learning rate and squeeze out a little more. A loss whose rate turns positive, rising, is a warning: the steps may be too big (lesson 79). And a rate that bounces between negative and positive usually means the loss readings are noisy, so people smooth them first, with a moving average (lesson 27), before reading rates off them.
In PyTorch. A training run keeps its losses as it goes, often in a tensor, and torch.diff is np.diff for tensors:
import torch
loss = torch.tensor([2.0, 1.2, 0.8, 0.6, 0.55, 0.53, 0.52])
r = torch.diff(loss) / 100print([round(v, 6) for v in r.tolist()])
print(torch.argmin(r).item(), round(r[-1].item(), 6))
The same rates as the NumPy cell, −0.008 down to −0.0001, and the same fastest fall, at position 0. .tolist() and .item() take plain Python numbers out of the tensor, and each is rounded to 6 places before printing, as in the NumPy cell.
Quick check
A loss is recorded every 50 steps. What does this print?
import numpy as np
loss = np.array([0.9, 0.7, 0.6, 0.58])
print(np.round(np.diff(loss) / 50, 4).tolist())
[-0.2, -0.1, -0.02]
[-0.004, -0.002, -0.0004]
[0.004, 0.002, 0.0004]
[-0.004, -0.002]
Show the answer
[-0.004, -0.002, -0.0004]
The differences −0.2, −0.1 and −0.02, divided by 50 steps: the fall slows as training goes on.
Drill
Where the loss falls fastest
The loss was recorded every 50 steps. Set r to its rates per step, fastest to the position of the fastest fall (the most negative rate), and flat to True if the last rate is closer to 0 than −0.001.
import numpy as np
loss = np.array([1.5, 1.1, 0.95, 0.9, 0.88, 0.875])
r = np.diff(loss)
fastest = 0
flat = Falseprint(r, fastest, flat)
In the lesson, your code is checked as you write it.
In your own words
A loss curve's rate is still clearly negative when training stops. What might you do, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write