Standard error
How much an estimate from a sample would wobble if you took a fresh sample. Bigger samples give smaller standard errors.
What it means
An average from a sample is only an estimate. Take a different sample and you'd get a slightly different number. The standard error is the typical size of that wobble: how far your estimate is likely to be from the true value.
For an average, it's the standard deviation of the data divided by the square root of the sample size. For a share, such as a conversion rate p measured on n people, it's the square root of p × (1 − p) ÷ n. About 95% of the time the true value lies within two standard errors of your estimate, which gives you a confidence interval.
An example in Python
import math import random random.seed(0) # What 400 customers spent: around $60 on average, with a spread of about $20. spend = [random.gauss(60, 20) for _ in range(400)] mean = sum(spend) / len(spend) sd = math.sqrt(sum((x - mean) ** 2 for x in spend) / (len(spend) - 1)) se = sd / math.sqrt(len(spend)) print(f"average {mean:.2f}, standard deviation {sd:.2f}, standard error {se:.2f}") print(f"about 95% confidence interval: {mean - 2 * se:.2f} to {mean + 2 * se:.2f}") # Four times the data only halves the standard error. for n in [100, 400, 1600]: print(f"{n:>5} customers: standard error {sd / math.sqrt(n):.2f}")
average 59.77, standard deviation 20.51, standard error 1.03 about 95% confidence interval: 57.72 to 61.82 100 customers: standard error 2.05 400 customers: standard error 1.03 1600 customers: standard error 0.51
The last three lines show the catch. The standard error shrinks with the square root of the sample size, so four times the data only halves it. Going from ±2 to ±1 costs four times as much data.
Standard error vs. standard deviation
They answer different questions. The standard deviation describes how spread out the individual values are, and more data won't shrink it: these customers really do vary by about $20. The standard error describes how uncertain your estimate is, and more data does shrink it: their average is known to within about $1.
Why it matters for AI
Every model evaluation is a sample. A model that scores 90% on 100 test questions has a standard error of about 3 points, so a rival on 92% may be no better at all. Before you trust a small difference, check it against the standard error.
Try it now
Each run is 1,000 experiments. Each one measures the conversion rate from n visitors. Raise the visitors per experiment and watch the spread of measured rates shrink.
Learn it properly
Samples and confidence · Estimate from a sample and say how sure you are. About 30 minutes.
Other terms from the same lesson
- bootstrap: Estimating uncertainty by resampling your own data, with replacement, many times and seeing how much the answer moves.
- confidence interval: A range that probably contains the true value, given the data you sampled.
- margin of error: The plus-or-minus around an estimate, such as 3.2% ± 0.8%. It is usually about two standard errors.
- resample: Draw a new sample from your data, picking rows at random with replacement, to mimic collecting fresh data.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.