Weighted and moving averages
Why this matters
A plain average treats every number alike. Models rarely do. Attention mixes words with weights. A training loss bounces from batch to batch, so engineers judge a smoothed copy of it. And the optimizers that train almost every model, momentum and Adam, step along a moving average of the slopes. This lesson builds all three averages from the Σ you just learned: weighted, windowed and exponential.
- Compute a weighted average, and say why its weights must add to 1
- Compute a moving average over a window, and an exponential moving average one value at a time
- Show that an exponential moving average's weights form a geometric sequence that adds to 1, and that it remembers about 1/(1 − β) values
- Smooth a noisy training loss and read it, and see momentum and Adam as moving averages of slopes
The math: averages with weights
Purpose. Average numbers that don't all count the same, then keep a running average that follows new numbers as they arrive.
A plain average adds the values and divides by how many there are. A weighted average gives each value its own weight instead, and the weights add to 1:
In plain words: multiply each value by its weight, then add. A weight is the value's share of the vote, and the shares add up to the whole vote, 1.
For the tent shop, half the rentals were the $20 tent, so its weight is 0.5. The $35 tent gets 0.3 and the $50 tent 0.2: the average rental is 0.5 × 20 + 0.3 × 35 + 0.2 × 50 = 30.5 dollars.