A distribution is a whole table of chances, but often two numbers say most of what matters: where it's centered and how spread out it is. The expected value is the center, the long-run average you'd see if you could repeat the experiment forever. The standard deviation is the spread, the typical distance from that center, in the same units as the values. Both are weighted averages, which makes them dot products, and both have direct meanings in ML: the loss a model reports while training is an average over examples, an estimate of an expected value, and spreads decide how noisy every such estimate is.
What you'll be able to do
Find a distribution's expected value: the probability-weighted average of its values
Find its variance, the expected squared distance from that average, and its standard deviation, in the data's own units
Compute both as dot products in NumPy, and compare them with the average of many samples
See that a training loss is an expected value, estimated by averaging over examples, and that PyTorch's F.cross_entropy reports exactly that average
Quick check
Have a guess before you start. A game pays $10 with chance 0.1 and nothing otherwise. Over many plays, what does it pay per play on average?
$5
$10
$1
$0.10
Show the answer
$1
10 × 0.1 + 0 × 0.9 = 1: on average $1 per play, though no single play ever pays exactly $1.
The math: center and spread
Purpose. The center and the spread of a distribution.
Tomorrow's tent rentals X are 10, 20 or 30 with chances 0.2, 0.5 and 0.3. The expected value, written E[X] or μ ("mu"), weights each value by its chance:
E[X]=i∑xipi=10(0.2)+20(0.5)+30(0.3)=21
In plain words: a weighted average, with the probabilities as the weights.
21 isn't a value X can even take; it's the balance point of the bars, the average over many days.
Quick check
From lesson 50: what is (10, 20, 30) · (0.2, 0.5, 0.3)?
20
21
1
60
Show the answer
21
2 + 10 + 9 = 21. An expected value is exactly this dot product: values with probabilities.
The picture. Watch the values meet their probabilities as boxes of numbers and as a drawing. In the drawing, each rental count is a bar standing at its place on the rentals line, as tall as its chance. Each step weights one bar. Then the weighted values add up along the line, and the total lands on the balance point: if each bar were a weight as heavy as its chance, a pivot there would hold them level. The last step also shades the spread, which the next formula measures.
Try it now
Then set probabilities to 0.6, 0.3, 0.1 and press Use these numbers: the balance point slides down toward 10, and the band moves with it.
Symbols and formula. The variance is the expected squared distance from μ, and the standard deviation σ ("sigma") is its square root:
Var(X)=i∑(xi−μ)2piσ=Var(X)
In plain words: how far values typically land from the center, squared and averaged; then rooted, to get back to the original units.
For the rentals: (10 − 21)² × 0.2 + (20 − 21)² × 0.5 + (30 − 21)² × 0.3 = 24.2 + 0.5 + 24.3 = 49, so σ = 7 tents. Squaring stops distances above and below the mean from canceling; the root undoes the squaring of the units.
Quick check
A fair die has mean 3.5 and variance about 2.92. What is its standard deviation, roughly?
8.5
1.71
2.92
3.5
Show the answer
1.71
σ = √2.92 ≈ 1.71: the typical distance of a roll from 3.5.
Quick check
Which distribution has the bigger standard deviation: always 5, or 0 and 10 with chance 0.5 each?
They're equal: both have mean 5
Always 5
0 and 10: values sit 5 from the mean
Neither has one
Show the answer
0 and 10: values sit 5 from the mean
Both have mean 5. "Always 5" never strays, so σ = 0; "0 or 10" always lands 5 away, so σ = 5.
In your own words
In your own words: why does the variance square the distances before averaging them?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: dot products, and samples
Purpose. Compute both from the table, and check them against many samples.
import numpy as np
v = np.array([10, 20, 30])
p = np.array([0.2, 0.5, 0.3])
mu = v @ p
var = ((v - mu) ** 2) @ p
print(mu, var, var ** 0.5)
rng = np.random.default_rng(0)
s = rng.choice(v, size=10000, p=p)
print(round(float(s.mean()), 2), round(float(s.var()), 2), round(float(s.std()), 2))
It runs right here in your browser, with nothing to install.
v @ p is the weighted average. (v - mu) ** 2 broadcasts μ across the values (lesson 64) and squares each distance; @ p weights them. Ten thousand samples' plain .mean(), .var() and .std() land close: 20.95, 49.59 and 7.04.
Code flow. Step through both calculations from the table, one line at a time. float() turns NumPy's one-number answer into a plain Python number, so mu shows as 21.0. Watch gaps fill with every value's distance from the mean at once, then var weight their squares by the chances.
A named mistake: the plain average of the values.np.mean(v) gives 20, ignoring the probabilities; it's right only when every value is equally likely. With a table of values and chances, the expected value is v @ p. The plain mean is right for samples, because there each value already appears as often as it's likely.
Variance or standard deviation? Report σ when you want a number in the data's units, "give or take 7 tents". Variance is the one that adds up nicely in formulas, which is why both are used.
Worked example
Worked example: theory against 10,000 samples, three times
For three distributions, compute E[X] and σ from the table, and compare with 10,000 samples.
import numpy as np
rng = np.random.default_rng(8)
tables = [(np.array([1, 2, 3, 4, 5, 6]), np.array([1, 1, 1, 1, 1, 1]) / 6),
(np.array([0, 10]), np.array([0.9, 0.1])),
(np.array([-1, 0, 5]), np.array([0.5, 0.3, 0.2]))]
for v, p in tables:
mu = v @ p
sd = (((v - mu) ** 2) @ p) ** 0.5
s = rng.choice(v, size=10000, p=p)
print(round(float(mu), 3), round(float(sd), 3), round(float(s.mean()), 3), round(float(s.std()), 3))
In the lesson you can change this and run it.
A fair die: mean 3.5, σ 1.708. The 10% game: mean 1, σ 3, a big spread for a small mean, since the payout is rare. The third: mean 0.5, σ 2.291. Each sample column sits within a few hundredths of the theory, a little closer with more samples.
Quick check
What does this print?
import numpy as np
v = np.array([1, 3])
p = np.array([0.5, 0.5])
mu = v @ p
print(mu, ((v - mu) ** 2) @ p)
2.0 2.0
4.0 1.0
2.0 1.0
2.0 0.0
Show the answer
2.0 1.0
The mean is 2. Each value is 1 away, so the variance is 1² × 0.5 + 1² × 0.5 = 1.
A shortcut for the variance. Expanding the square gives a second formula that's often quicker: Var(X) = E[X²] − (E[X])², the mean of the squares minus the square of the mean. In NumPy that's (v ** 2) @ p - (v @ p) ** 2: two dot products and no centering. It gives the same number as the definition, which makes it a handy check on hand-written code, but it has a weakness worth knowing: when the values are huge and the spread is small, the two terms are nearly equal, and subtracting them loses digits to rounding. For the small numbers here, both agree exactly.
Quick check
What does this print?
import numpy as np
v = np.array([0.0, 1.0, 2.0])
p = np.array([0.25, 0.5, 0.25])
mu = v @ p
print(((v - mu) ** 2) @ p, (v ** 2) @ p - mu ** 2)
0.25 0.5
0.5 0.5
0.5 1.5
1.0 0.5
Show the answer
0.5 0.5
The mean is 1. Both formulas give 0.5: the squared distances 1, 0, 1 weighted by the chances, or 1.5 − 1².
Drill
Center and spread from a table
Write mean_sd(v, p) returning the expected value and the standard deviation of a table. Then set rent to its result for rentals (10, 20, 30) with chances (0.6, 0.3, 0.1), each rounded to 3 places, as a tuple.
In the lesson, your code is checked as you write it.
In your own words
Why is the plain average correct for samples but not for a table of values and chances?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: a loss is an expected value
Purpose. The loss reported in training estimates an expected value.
What a model's builders really care about is the expected loss: the average loss over every example the model might ever see. That can't be computed, so training averages the loss over the examples it has, a sample, which estimates it:
E[L]≈n1i=1∑nLi
In plain words: the average loss over the data stands in for the average over all possible data.
A classifier's results fall into three kinds: confident and right (loss 0.1), unsure (0.5) and confidently wrong (2.0), with chances 0.7, 0.2 and 0.1. Its expected loss is 0.37, and the rare confident mistakes contribute more than half of it.
Quick check
Why does a training loss computed on a small batch jump around from batch to batch?
It averages a few random examples, so it scatters around the expected loss
Loss can't be averaged
The expected loss itself keeps changing
The model's weights change completely from one batch to the next
Show the answer
It averages a few random examples, so it scatters around the expected loss
A batch average estimates the expected loss with some spread; fewer examples mean a bigger spread.
In PyTorch. The same expected loss with tensors is (chances * losses).sum(): each loss times its chance, added up, the dot product written out. In training there's no table of chances, only a batch of examples, each with its own loss. F.cross_entropy, the loss classifiers train on (lesson 105 builds it), takes a model's scores for each example (logits, one row per example) and the right answers (y). Asked with reduction="none", it gives one loss per example; asked plainly, their mean.
import torch
import torch.nn.functional as F
losses = torch.tensor([0.1, 0.5, 2.0])
chances = torch.tensor([0.7, 0.2, 0.1])
print(round((chances * losses).sum().item(), 3))
logits = torch.tensor([[2.0, 0.0], [0.0, 0.0], [0.0, 2.0]])
y = torch.tensor([0, 0, 0])
each = F.cross_entropy(logits, y, reduction="none")
print(each.round(decimals=3))
print(round(each.mean().item(), 3), round(F.cross_entropy(logits, y).item(), 3))
The first line is 0.37, the expected loss the NumPy cell printed. Then three examples of the same three kinds: scores that favor the right answer (a loss of 0.127), scores that can't decide (0.693) and scores that favor the wrong one (2.127). The plain F.cross_entropy gives 0.982, exactly their mean: each example counts equally, as each value of a sample does, so the batch's mean estimates the expected loss.
Quick check
A batch of four examples has losses 0.2, 0.4, 0.1 and 1.3. What does F.cross_entropy report for the batch, with its default settings?
0.4, the middle one
0.5, their mean
2.0, their sum
1.3, the worst one
Show the answer
0.5, their mean
By default it averages: (0.2 + 0.4 + 0.1 + 1.3) ÷ 4 = 2.0 ÷ 4 = 0.5. reduction="sum" would give 2.0, and reduction="none" all four.
Spread matters too. Two models with the same expected loss can behave very differently: one makes many small errors, the other mostly perfect predictions and a few disasters. The standard deviation of the loss across examples tells them apart:
Both means are 0.5; the variances are 0 and 0.75. correction=0 divides by n, as NumPy's .var() does. Left out, PyTorch divides by n − 1 and prints 1.0 (lesson 97 says why both exist), so say which one you mean. And the noise in a batch's loss shrinks as the batch grows (lesson 98 says exactly how), which is one reason batch size matters in training.
Averages pass through sums. E[X + Y] = E[X] + E[Y] for any two random variables, dependent or not, and E[aX + b] = aE[X] + b. That's why a batch's loss can be built example by example: the expected value of the batch's average is the average of each example's expected loss. Variance is less forgiving: Var(aX + b) = a²Var(X). Stretching every value by a stretches every distance from the mean by a, and every squared distance by a², while shifting by b moves the mean along with the values and changes no distance at all. So doubling every loss doubles the expected loss but quadruples its variance.
Drill
A classifier's expected loss
Set expected to the expected loss of the classifier below, share_wrong to the part of it that comes from the confidently wrong case (its loss × chance ÷ expected), and sd to the standard deviation of the loss, each rounded to 3 places.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write