Samples, the law of large numbers and the central limit theorem
Why this matters
Every number a model is judged by comes from a sample. A test set is a few thousand questions out of all the questions the model will ever meet; a training step's gradient is computed on a batch of 32 examples out of millions. So every measured accuracy and every gradient wobbles: a different sample would have given a slightly different number. This lesson answers how much. Three facts carry the whole idea. A sample's average lands near the population's mean, and nearer as the sample grows: the law of large numbers. The size of its wobble is σ/√n, so four times the data halves it. And the averages of many samples pile up in a bell shape even when the population is lopsided: the central limit theorem. Together they say when a difference between two models is real, and why bigger batches give steadier training.
- Tell a population from a sample, and see a sample's average wobble around the population's mean
- Use the standard error σ/√n: the wobble shrinks with the square root of the sample size
- Simulate many samples, measure their wobble, and draw their averages turning bell-shaped, whatever the population looks like
- Read a test score and a mini-batch gradient as samples, put a 95% range on an accuracy two ways, and measure a batch gradient's wobble in NumPy and PyTorch
The math: what a part says about the whole
Purpose. Know how far a sample's average can be from the truth, before you see the truth.
Population and sample. The population is everything you care about: every wait at a trailhead shuttle, say. A sample is the part you actually measure: the 16 waits you timed last week. The population has a true mean μ and standard deviation σ that you usually never see; the sample has its own average x̄, which you can compute.
Take a lopsided population: a shuttle wait is 2 minutes 80% of the time and 10 minutes 20% of the time. Its mean is μ = 2 × 0.8 + 10 × 0.2 = 3.6 minutes, and its standard deviation σ = 3.2 (lesson 92's formula). One wait is never 3.6: it's 2 or 10. But average 16 waits and you'll usually land close to 3.6, and average 64 and closer still. That's the law of large numbers.