The normal distribution

The 68–95–99.7 rule

Every normal distribution shares three handy facts. Count outward from the mean in steps of one standard deviation:

The sign ± means "plus or minus": the range runs from the mean minus that amount to the mean plus that amount.

Back at the bakery, say the mean is 500 grams and the standard deviation is 10 grams:

• about 68% of loaves weigh between 490 and 510 grams;
• about 95% weigh between 480 and 520 grams;
• about 99.7% weigh between 470 and 530 grams.

That leaves 0.3% outside 470 to 530. So a loaf more than 3 standard deviations from the mean turns up about 3 times in 1,000.

In plain words: about two thirds of values sit within one standard deviation of the mean. Nearly all sit within two. Almost none go past three.

z-scores: how far out is this?

A day of 30 °C (86 °F) is hot for London in April. In Dubai in July, it's a mild day. A raw number means little until you know what's normal for that place.

A z-score solves this. It counts how many standard deviations a value sits from the mean. Then everything is on the same ruler, whatever the units:

Here $x$ is the value you're checking. The answer, $z$, is its z-score.

In plain words: subtract the average, then divide by the standard deviation. The answer is "how many standard deviations away". A positive z is above average and a negative z is below.

A small example. A class test averages 70 points with a standard deviation of 10. You scored 85. Your distance from the mean is 85 − 70 = 15 points. Divide by 10 and $z = 1.5$. You're one and a half standard deviations above average.

Now a business example. A shop's daily orders average 420, with a standard deviation of 35. Today it got 520. That's $z = (520 - 420) / 35 \approx 2.9$, nearly three standard deviations up. Fewer than 0.5% of ordinary days reach that high. So it's worth a look: maybe an influencer mentioned the shop, or maybe a tracking bug is counting orders twice.

mu, sigma = 70, 10   # class average and standard deviation

for score in [85, 60, 70, 98]:
    z = (score - mu) / sigma
    print(score, "z =", z)

mu, sigma = 50, 5
print((40 - mu) / sigma)

Probabilities and percentiles with SciPy

The rule of thumb covers 1, 2 and 3 standard deviations. For any other value, let the computer work out the area under the bell. The SciPy library's scipy.stats.norm has three functions for this:

• cdf gives the chance of a value at most x. (The name stands for "cumulative distribution function".)
• sf gives the chance of a value at least x: the upper tail. (Short for "survival function".)
• ppf goes the other way. You give it a percentile and it gives back the value. It undoes cdf.

A percentile is a value with a given share of results below it. The 95th percentile is the value that 95% of days fall below.

from scipy.stats import norm

mu, sigma = 420, 35
print("P(orders <= 450):", round(norm.cdf(450, mu, sigma), 4))
print("P(orders >= 520):", round(norm.sf(520, mu, sigma), 4))
print("95th percentile:", round(norm.ppf(0.95, mu, sigma), 1))
print("z for the middle 95%:", round(norm.ppf(0.975), 3))

Read the four answers like this:

• About 80% of ordinary days have 450 orders or fewer.
• Only about 2 days in 1,000 reach 520 or more.
• 95% of days stay below about 478 orders.
• The middle 95% of any normal distribution runs from 1.96 standard deviations below the mean to 1.96 above.

When you leave out the mean and standard deviation, SciPy uses the standard bell (mean 0, standard deviation 1). That's why the last line gives a z-score. That number, 1.96, is the one behind every "95% confidence interval", which you'll meet when you estimate from samples.

Why it shows up everywhere: the central limit theorem

Roll one die. Each face from 1 to 6 is equally likely, so a chart of the results is flat. Now roll ten dice and take their average. Averages near 3.5 are common, because highs and lows tend to cancel out. Averages near 1 or 6 need every die to come up low (or high) together, and that almost never happens. Chart many of those averages and you get a bell.

The same thing happens with almost any random effects. Add up or average many of them, and the total comes out roughly normal, whatever shape each single effect has. The effects must be independent, meaning one doesn't sway another. This is the central limit theorem.

In plain words: single values can have any shape. Averages of many independent values come out close to a bell.

Single orders are often lopsided: many small ones and a few huge ones. But the average of a day's orders is close to normal. That's why statistics built on the bell curve works for conversion rates and averages. The code below makes 5,000 pretend days of 40 orders each. On the left are single orders. On the right are the daily averages.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
orders = rng.exponential(scale=60, size=(5000, 40))     # lopsided: many small orders, a few large
fig, axes = plt.subplots(1, 2, figsize=(8, 3))
axes[0].hist(orders[:, 0], bins=50)
axes[0].set_title("single orders (skewed)")
axes[1].hist(orders.mean(axis=1), bins=50)
axes[1].set_title("averages of 40 orders (bell-shaped)")

The left chart piles up near zero and trails off to the right. The right chart is a bell. Keep in mind that the theorem is about sums and averages. Single orders, incomes and page views often aren't normal at all, so check the shape before you lean on the 68–95–99.7 rule.

For later: neural network weights usually start as small random draws from a normal distribution (scaled by layer size), and diffusion image models work by gradually adding and then removing normal noise.