Every number a model is judged by comes from a sample. A test set is a few thousand questions out of all the questions the model will ever meet; a training step's gradient is computed on a batch of 32 examples out of millions. So every measured accuracy and every gradient wobbles: a different sample would have given a slightly different number. This lesson answers how much. Three facts carry the whole idea. A sample's average lands near the population's mean, and nearer as the sample grows: the law of large numbers. The size of its wobble is σ/√n, so four times the data halves it. And the averages of many samples pile up in a bell shape even when the population is lopsided: the central limit theorem. Together they say when a difference between two models is real, and why bigger batches give steadier training.
What you'll be able to do
Tell a population from a sample, and see a sample's average wobble around the population's mean
Use the standard error σ/√n: the wobble shrinks with the square root of the sample size
Simulate many samples, measure their wobble, and draw their averages turning bell-shaped, whatever the population looks like
Read a test score and a mini-batch gradient as samples, put a 95% range on an accuracy two ways, and measure a batch gradient's wobble in NumPy and PyTorch
Quick check
Have a guess before you start. One poll asks 100 people, another 10,000. Repeated many times, whose result wobbles more from poll to poll?
The poll of 100
Neither wobbles
The poll of 10,000
They wobble the same
Show the answer
The poll of 100
A small sample can land far from the truth by chance; a big one averages that chance away. Here the wobble is ten times smaller for 10,000.
The math: what a part says about the whole
Purpose. Know how far a sample's average can be from the truth, before you see the truth.
Population and sample. The population is everything you care about: every wait at a trailhead shuttle, say. A sample is the part you actually measure: the 16 waits you timed last week. The population has a true mean μ and standard deviation σ that you usually never see; the sample has its own average x̄, which you can compute.
Take a lopsided population: a shuttle wait is 2 minutes 80% of the time and 10 minutes 20% of the time. Its mean is μ = 2 × 0.8 + 10 × 0.2 = 3.6 minutes, and its standard deviation σ = 3.2 (lesson 92's formula). One wait is never 3.6: it's 2 or 10. But average 16 waits and you'll usually land close to 3.6, and average 64 and closer still. That's the law of large numbers.
Quick check
From lesson 93: an accuracy of 0.9 measured on 100 questions has standard deviation √(p(1 − p)/n). What is it?
0.3
0.0009
0.03
0.09
Show the answer
0.03
√(0.9 × 0.1 ÷ 100) = √0.0009 = 0.03. That was a standard error already: the wobble of an average of 100 yes-or-no answers.
How big is the wobble? The average of n independent draws has standard deviation σ/√n, called the standard error. The √n is the key: the wobble shrinks, but slowly.
The picture. Step through it for the shuttle, as boxes of numbers and as a drawing. The drawing shows where the averages of n waits land: each bar is the share of all possible samples whose average falls there, worked out exactly, with μ as a dashed line and an error bar one standard error either side of it. Watch the bars crowd in toward μ as n grows.
Try it now
Then set the sample sizes to 100, 400 and press Use these numbers: going from 100 to 400 waits buys the same halving as 1 to 4, and the bars squeeze into a spike.
Symbols and formula.
SE(xˉ)=nσxˉ±1.96⋅ns covers μ about 95% of the time
In plain words: the average of n values wobbles by σ/√n; about 95% of samples land within 1.96 standard errors of the true mean.
The bell appears. Look back at the picture's bars. With n = 1 they're just the population: two spikes, at 2 and 10. With n = 4 there are five possible averages, still lopsided. By n = 64 the averages form a smooth, nearly symmetric bell centered on 3.6, with spread 0.4. That's the central limit theorem: averages of many independent draws are close to normal, whatever the population's shape. It's why the 68–95–99.7 rule from lesson 93 works for averages, and why 1.96 appears in the range above.
Quick check
σ = 12. What is the standard error of the average of 36 values?
2
6
12
0.33
Show the answer
2
12 ÷ √36 = 12 ÷ 6 = 2.
In your own words
In your own words: why does the average of a sample wobble less than a single value does?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: draw many samples, measure the wobble
Purpose. Check σ/√n by simulation: the formula is proved, and trying it shows what it means.
import numpy as np
rng = np.random.default_rng(0)
waits = np.array([2.0, 10.0])
chances = np.array([0.8, 0.2])
for n in [1, 4, 16, 64]:
means = rng.choice(waits, size=(2000, n), p=chances).mean(axis=1)
print(n, round(float(means.std()), 2), round(3.2 / n ** 0.5, 2))
It runs right here in your browser, with nothing to install.
size=(2000, n) draws 2,000 samples at once, one per row, and .mean(axis=1) averages each row: 2,000 sample averages. Their spread matches σ/√n to within a few hundredths each time: 3.24 against 3.2, then 1.61, 0.78 and 0.4 against 1.6, 0.8 and 0.4.
Code flow. Here is the same experiment at a size you can watch: five samples of 4 waits, one at a time. Step through it and watch sample change on every pass, and means collect one average per sample.
A named mistake: forgetting the √. It's tempting to think 100 times the data gives 100 times the precision. It gives 10 times: the standard error divides by √100 = 10. Plans built on the wrong rule run out of budget: cutting a wobble of 2 points to 0.2 points needs 100 times the data, not 10.
Worked example
Worked example: the bell from a lopsided population
Average 64 shuttle waits, 5,000 times, and check where the averages fall.
The averages center on the true 3.6 with spread close to 3.2 ÷ 8 = 0.4. The last line counts the share of the 5,000 averages within 1.96 standard errors of 3.6: about 96%, close to the 95% a normal distribution would give, though every single wait was either 2 or 10. That's the central limit theorem doing its work, and it's what lets a 95% range be built from a formula without knowing the population's shape.
Quick check
What does this print?
import numpy as np
n = np.array([1, 4, 16, 64])
print((3.2 / np.sqrt(n)).tolist())
[3.2, 0.8, 0.2, 0.05]
[0.4, 0.8, 1.6, 3.2]
[3.2, 1.6, 0.8, 0.4]
[3.2, 1.6, 1.07, 0.8]
Show the answer
[3.2, 1.6, 0.8, 0.4]
√n is 1, 2, 4 and 8, so the standard error halves at every step: four times the data, half the wobble.
Draw it. A histogram of 2,000 averages for n = 4 and for n = 64, on one plot. plt.hist sorts values into bins and draws a bar for each, and label= names each one for the legend.
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
for n in [4, 64]:
means = rng.choice([2.0, 10.0], size=(2000, n), p=[0.8, 0.2]).mean(axis=1)
plt.hist(means, bins=np.arange(1.75, 10.5, 0.25), label=f"n = {n}")
plt.xlabel("average wait (minutes)")
plt.ylabel("samples")
plt.legend()
plt.show()
Quick check
Read the plot. How do the two histograms differ?
They look the same, just shifted apart
The n = 4 histogram is the narrower one
The n = 64 histogram centers further right
Both center near 3.6, and n = 64 is far narrower
Show the answer
Both center near 3.6, and n = 64 is far narrower
A bigger sample doesn't move where the averages center, μ = 3.6; it squeezes them closer to it. The n = 4 averages can only be 2, 4, 6, 8 or 10, so they stand in separate spikes; the n = 64 averages make one narrow bell.
Drill
Measure the wobble
With seed 4, for each n in 1, 4, 16 and 64, draw 2,000 samples of n shuttle waits, average each, and append the standard deviation of the 2,000 averages (rounded to 2 places) to wobble. Append σ/√n, rounded to 2, to theory.
import numpy as np
rng = np.random.default_rng(4)
waits = np.array([2.0, 10.0])
chances = np.array([0.8, 0.2])
wobble = []
theory = []
for n in [1, 4, 16, 64]:
means = rng.choice(waits, size=(2000, n), p=chances).mean(axis=1)
print(wobble, theory)
In the lesson, your code is checked as you write it.
In your own words
The simulation matched σ/√n closely at every n. What would you expect if the 2,000 repeats were cut to 20?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: test scores and mini-batches are samples
Purpose. Know how much a measured accuracy and a batch's gradient can wobble.
A test score is a sample. A model that is right on 85 of 100 test questions has a measured accuracy of 0.85, but a different 100 questions could have given 0.82 or 0.88. Treat each question as a yes-or-no draw: the standard error is √(0.85 × 0.15 ÷ 100) = 0.036, and a 95% range is 0.85 ± 1.96 × 0.036, from 0.78 to 0.92. On 10,000 questions the same accuracy is pinned to 0.843 to 0.857. Two models scoring 0.85 and 0.83 on 100 questions can't be told apart; on 10,000 they can.
Quick check
What does this print?
import numpy as np
se = (0.85 * 0.15 / np.array([100, 10000])) ** 0.5print((1.96 * se).round(3).tolist())
[0.036, 0.0036]
[0.07, 0.007]
[0.7, 0.07]
[0.07, 0.0007]
Show the answer
[0.07, 0.007]
The 95% half-widths: 1.96 × 0.0357 ≈ 0.07 for 100 questions, and ten times smaller for 10,000, since √100 = 10.
A second way: resample. Without any formula, draw 100 answers from the model's own 100 answers, with replacement, and score them; repeat 2,000 times. The middle 95% of those scores, from the 2.5th to the 97.5th percentile, is again 0.78 to 0.92. This bootstrap works for scores that have no simple formula, like a median latency or an F1 score.
A mini-batch is a sample. A training step averages the gradient over a batch of B examples instead of all of them, so the step direction wobbles by about σ/√B around the full gradient. A batch of 128 wobbles a quarter as much as a batch of 8, since √(128 ÷ 8) = 4: steadier steps, at sixteen times the cost per step.
Measure it. Here are 4,096 tent-shop days, made up with a seeded random generator: rentals rise with temperature, plus noise. A one-weight model ŷ = w × temperature starts at w = 1, and its loss is the mean squared error, so the gradient on any set of days is the mean of 2(wx − y)x. The cell finds it on all the days, then on every batch of 8 and every batch of 64 in turn, and measures how much the batch gradients spread.
import numpy as np
rng = np.random.default_rng(3)
t = rng.uniform(10, 35, size=4096) # temperatures
y = 1.8 * t + 3 + rng.normal(0, 4, size=4096) # rentals
w = 1.0defgrad(tb, yb):
returnfloat(np.mean(2 * (w * tb - yb) * tb))
print(round(grad(t, y), 1))
for B in [8, 64]:
gs = []
for k inrange(0, 4096, B):
gs.append(grad(t[k:k + B], y[k:k + B]))
print(B, len(gs), round(float(np.mean(gs)), 1), round(float(np.std(gs)), 1))
On all 4,096 days the gradient is −1023.7. The 512 batches of 8 average exactly that, but one batch can land far from it: their spread is 216.3. The 64 batches of 64 average the same −1023.7 with a spread of 76.6, about 2.8 times smaller, as √(64 ÷ 8) = √8 ≈ 2.83 predicts.
Quick check
Why is a mini-batch gradient noisy?
It averages a random sample, not every example
Batches contain mistakes
The model itself changes between the examples
Gradients are always rounded
Show the answer
It averages a random sample, not every example
Each batch is a sample, so its average gradient wobbles around the full-data gradient by about σ/√B.
In PyTorch. Training code doesn't slice batches by hand. A TensorDataset holds the days as pairs, a DataLoader serves them a batch at a time, and loss.backward() finds each batch's gradient, which lands in w.grad. Setting w.grad = None clears it before the next batch, as optimizer.zero_grad() does in a training loop. The days are the same as in the NumPy cell, from the same seeded generator, and float64 keeps every digit NumPy keeps.
import numpy as np
import torch
from torch.utils.data import DataLoader, TensorDataset
rng = np.random.default_rng(3)
t = rng.uniform(10, 35, size=4096)
y = 1.8 * t + 3 + rng.normal(0, 4, size=4096)
days = TensorDataset(torch.tensor(t), torch.tensor(y))
w = torch.tensor(1.0, dtype=torch.float64, requires_grad=True)
for B in [8, 64]:
gs = []
for tb, yb in DataLoader(days, batch_size=B):
loss = ((w * tb - yb) ** 2).mean()
loss.backward()
gs.append(w.grad.item())
w.grad = Noneprint(B, len(gs), round(float(np.mean(gs)), 1), round(float(np.std(gs)), 1))
The same numbers as NumPy: batches of 8 spread 216.3 around −1023.7, batches of 64 only 76.6. batch_size is the n of this lesson. Raising it steadies each step by √n, and costs n examples of work per step.
Drill
An accuracy's range, two ways
A model got 85 of 100 test questions right (1 = right, 0 = wrong). Set formula to the 95% range p ± 1.96·√(p(1 − p)/n), each end rounded to 3 places, and boot to the 2.5th and 97.5th percentiles of 2,000 bootstrap accuracies (seed 1; resample 100 answers with rng.choice), each rounded to 2.
import numpy as np
answers = np.zeros(100)
answers[:85] = 1
p = answers.mean()
n = len(answers)
formula = (0, 0)
rng = np.random.default_rng(1)
boot = (0, 0)
print(formula, boot)
In the lesson, your code is checked as you write it.
In your own words
A colleague says model B beats model A because it scored 0.86 against 0.84 on a 200-question test. What would you say?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write