Index Ai Academy
Foundation · Foundation mathAlgebra

Weighted and moving averages

About 35 minLooking ahead: some prerequisites aren't done yet

Why this matters

A plain average treats every number alike. Models rarely do. Attention mixes words with weights. A training loss bounces from batch to batch, so engineers judge a smoothed copy of it. And the optimizers that train almost every model, momentum and Adam, step along a moving average of the slopes. This lesson builds all three averages from the Σ you just learned: weighted, windowed and exponential.

  • Compute a weighted average, and say why its weights must add to 1
  • Compute a moving average over a window, and an exponential moving average one value at a time
  • Show that an exponential moving average's weights form a geometric sequence that adds to 1, and that it remembers about 1/(1 − β) values
  • Smooth a noisy training loss and read it, and see momentum and Adam as moving averages of slopes
Quick check
Have a guess before you start. Last weekend the tent shop rented its $20 tent 5 times, its $35 tent 3 times and its $50 tent twice. What did an average rental bring in?

The math: averages with weights

Purpose. Average numbers that don't all count the same, then keep a running average that follows new numbers as they arrive.

A plain average adds the values and divides by how many there are. A weighted average gives each value its own weight instead, and the weights add to 1:

xˉw=∑iwixiwith∑iwi=1\bar{x}_w = \sum_i w_i x_i \qquad \text{with} \qquad \sum_i w_i = 1

In plain words: multiply each value by its weight, then add. A weight is the value's share of the vote, and the shares add up to the whole vote, 1.

For the tent shop, half the rentals were the $20 tent, so its weight is 0.5. The $35 tent gets 0.3 and the $50 tent 0.2: the average rental is 0.5 × 20 + 0.3 × 35 + 0.2 × 50 = 30.5 dollars.