Index Ai Academy
Foundation · Foundation mathProbability

Expected value, variance and standard deviation

About 42 minLooking ahead: some prerequisites aren't done yet

Why this matters

A distribution is a whole table of chances, but often two numbers say most of what matters: where it's centered and how spread out it is. The expected value is the center, the long-run average you'd see if you could repeat the experiment forever. The standard deviation is the spread, the typical distance from that center, in the same units as the values. Both are weighted averages, which makes them dot products, and both have direct meanings in ML: the loss a model reports while training is an average over examples, an estimate of an expected value, and spreads decide how noisy every such estimate is.

  • Find a distribution's expected value: the probability-weighted average of its values
  • Find its variance, the expected squared distance from that average, and its standard deviation, in the data's own units
  • Compute both as dot products in NumPy, and compare them with the average of many samples
  • See that a training loss is an expected value, estimated by averaging over examples, and that PyTorch's F.cross_entropy reports exactly that average
Quick check
Have a guess before you start. A game pays $10 with chance 0.1 and nothing otherwise. Over many plays, what does it pay per play on average?

The math: center and spread

Purpose. The center and the spread of a distribution.

Tomorrow's tent rentals X are 10, 20 or 30 with chances 0.2, 0.5 and 0.3. The expected value, written E[X] or μ ("mu"), weights each value by its chance:

E[X]=∑ixi pi=10(0.2)+20(0.5)+30(0.3)=21E[X] = \sum_i x_i\,p_i = 10(0.2) + 20(0.5) + 30(0.3) = 21

In plain words: a weighted average, with the probabilities as the weights.

21 isn't a value X can even take; it's the balance point of the bars, the average over many days.