Describing data: center, spread and scale
Why this matters
Lesson 92 summarized a distribution by its expected value and spread. Real data comes as a column of numbers instead: nine days of tent rentals, ten thousand response times, a million house prices. Nobody reads the whole column, so a few numbers have to stand in for it: where it sits, how widely it spreads, and what its edges look like. Choosing the right few numbers is a skill. One very large value can drag the average far from anything typical, and a feature measured in thousands can drown out one measured in tenths, so a model trained on raw columns may barely learn at all. This lesson builds each summary, shows when each one misleads, and ends with the two jobs it does every day in machine learning: putting features on the same scale, and reporting how fast a system is with percentiles rather than an average.
- Summarize a column of numbers by its center (mean, median, mode) and its spread (range, variance, standard deviation, percentiles), and read all three off a histogram
- See why deviations are squared, what an outlier does to each summary, and the difference between dividing by n and by n − 1
- Compute each summary with a loop, then with np.mean, np.median, np.std and np.percentile, and draw the column with Matplotlib
- Standardize features so training works, in NumPy and PyTorch, and report response times as percentiles
The math: a column in a few numbers
Purpose. Say where a column of numbers sits and how widely it spreads, in a few numbers.
The center. Nine days of rentals: 12, 15, 11, 14, 15, 15, 17, 16, 11. Three numbers describe their middle:
- The mean adds them and divides by how many: 126 ÷ 9 = 14.
- The median sorts them (11, 11, 12, 14, 15, 15, 15, 16, 17) and takes the middle one: 15. With an even count, it's the average of the two middle values.
- The mode is the most common value: 15, which came up three times.
The mean is lesson 92's expected value with every day weighted equally, 1/9 each. A weighted mean gives some values more say than others, and it comes back in lesson 113, where attention averages words by weights.