Index Ai Academy
Foundation · Foundation mathStatistics and information

Samples, the law of large numbers and the central limit theorem

About 43 minLooking ahead: some prerequisites aren't done yet

Why this matters

Every number a model is judged by comes from a sample. A test set is a few thousand questions out of all the questions the model will ever meet; a training step's gradient is computed on a batch of 32 examples out of millions. So every measured accuracy and every gradient wobbles: a different sample would have given a slightly different number. This lesson answers how much. Three facts carry the whole idea. A sample's average lands near the population's mean, and nearer as the sample grows: the law of large numbers. The size of its wobble is σ/√n, so four times the data halves it. And the averages of many samples pile up in a bell shape even when the population is lopsided: the central limit theorem. Together they say when a difference between two models is real, and why bigger batches give steadier training.

  • Tell a population from a sample, and see a sample's average wobble around the population's mean
  • Use the standard error σ/√n: the wobble shrinks with the square root of the sample size
  • Simulate many samples, measure their wobble, and draw their averages turning bell-shaped, whatever the population looks like
  • Read a test score and a mini-batch gradient as samples, put a 95% range on an accuracy two ways, and measure a batch gradient's wobble in NumPy and PyTorch
Quick check
Have a guess before you start. One poll asks 100 people, another 10,000. Repeated many times, whose result wobbles more from poll to poll?

The math: what a part says about the whole

Purpose. Know how far a sample's average can be from the truth, before you see the truth.

Population and sample. The population is everything you care about: every wait at a trailhead shuttle, say. A sample is the part you actually measure: the 16 waits you timed last week. The population has a true mean μ and standard deviation σ that you usually never see; the sample has its own average x̄, which you can compute.

Take a lopsided population: a shuttle wait is 2 minutes 80% of the time and 10 minutes 20% of the time. Its mean is μ = 2 × 0.8 + 10 × 0.2 = 3.6 minutes, and its standard deviation σ = 3.2 (lesson 92's formula). One wait is never 3.6: it's 2 or 10. But average 16 waits and you'll usually land close to 3.6, and average 64 and closer still. That's the law of large numbers.