Expected value, variance and standard deviation
Why this matters
A distribution is a whole table of chances, but often two numbers say most of what matters: where it's centered and how spread out it is. The expected value is the center, the long-run average you'd see if you could repeat the experiment forever. The standard deviation is the spread, the typical distance from that center, in the same units as the values. Both are weighted averages, which makes them dot products, and both have direct meanings in ML: the loss a model reports while training is an average over examples, an estimate of an expected value, and spreads decide how noisy every such estimate is.
- Find a distribution's expected value: the probability-weighted average of its values
- Find its variance, the expected squared distance from that average, and its standard deviation, in the data's own units
- Compute both as dot products in NumPy, and compare them with the average of many samples
- See that a training loss is an expected value, estimated by averaging over examples, and that PyTorch's F.cross_entropy reports exactly that average
The math: center and spread
Purpose. The center and the spread of a distribution.
Tomorrow's tent rentals X are 10, 20 or 30 with chances 0.2, 0.5 and 0.3. The expected value, written E[X] or μ ("mu"), weights each value by its chance:
In plain words: a weighted average, with the probabilities as the weights.
21 isn't a value X can even take; it's the balance point of the bars, the average over many days.