A plain average treats every number alike. Models rarely do. Attention mixes words with weights. A training loss bounces from batch to batch, so engineers judge a smoothed copy of it. And the optimizers that train almost every model, momentum and Adam, step along a moving average of the slopes. This lesson builds all three averages from the Σ you just learned: weighted, windowed and exponential.
What you'll be able to do
Compute a weighted average, and say why its weights must add to 1
Compute a moving average over a window, and an exponential moving average one value at a time
Show that an exponential moving average's weights form a geometric sequence that adds to 1, and that it remembers about 1/(1 − β) values
Smooth a noisy training loss and read it, and see momentum and Adam as moving averages of slopes
Quick check
Have a guess before you start. Last weekend the tent shop rented its $20 tent 5 times, its $35 tent 3 times and its $50 tent twice. What did an average rental bring in?
$30.50
$101.67
$35
$105
Show the answer
$30.50
5 of the 10 rentals were $20, 3 were $35 and 2 were $50, so the cheap tent counts most: 0.5 × 20 + 0.3 × 35 + 0.2 × 50 = 30.5. That's a weighted average.
The math: averages with weights
Purpose. Average numbers that don't all count the same, then keep a running average that follows new numbers as they arrive.
A plain average adds the values and divides by how many there are. A weighted average gives each value its own weight instead, and the weights add to 1:
xˉw=i∑wixiwithi∑wi=1
In plain words: multiply each value by its weight, then add. A weight is the value's share of the vote, and the shares add up to the whole vote, 1.
For the tent shop, half the rentals were the $20 tent, so its weight is 0.5. The $35 tent gets 0.3 and the $50 tent 0.2: the average rental is 0.5 × 20 + 0.3 × 35 + 0.2 × 50 = 30.5 dollars.
Quick check
From an earlier lesson: the mean of 4, 7 and 10 is (1/3) Σ xᵢ. Read as a weighted average, what weight does each value get?
1/10
1
3
1/3
Show the answer
1/3
(1/3) Σ xᵢ is (1/3) × 4 + (1/3) × 7 + (1/3) × 10. Every value gets the same weight, 1/3, and three of them add to 1. A plain mean is a weighted average with equal weights.
Why the weights must add to 1. Suppose every value were 30. Any fair average of them should be 30. A weighted average gives Σ wᵢ × 30, which is 30 × Σ wᵢ: that's 30 only when the weights add to 1. Weights that add to more inflate every average, and weights that add to less shrink it.
A window average. Here is a week of the tent shop's daily rentals: 12, 18, 9, 15, 21, 12, 18. They jump around. A moving average over a window of 3 averages each day with the two days before it. Day 3 gets (12 + 18 + 9) ÷ 3 = 13. Then the window slides one day along and drops the oldest day: 14, 15, 16 and 17. The jumps cancel out, and a steady climb of one rental a day shows through. Inside the window each day weighs 1/3, and every day outside it weighs 0.
Quick check
Three scores, 70, 80 and 90, get the weights 0.5, 0.3 and 0.4. What's wrong with this average?
The weights add to 1.2, so it comes out too big
The biggest weight must go on the biggest score
Nothing: any three weights will do
The weights add to 1.2, so it comes out too small
Show the answer
The weights add to 1.2, so it comes out too big
0.5 + 0.3 + 0.4 = 1.2, so every score is counted 20% too much and the result lands above all three. Weights must add to 1.
A running average that follows new values. A window has to keep its last few values, and it has a sharp edge: a day counts fully for three days, then not at all. The exponential moving average, or EMA, keeps just one number, m, and blends each new value x into it:
m←βm+(1−β)x
In plain words: keep a share β of the old average and take a share 1 − β of the new value. The arrow means "becomes": the result replaces the old m.
β is the Greek letter beta, a number from 0 up to, but not including, 1. With β = 0.5, each new value pulls m half of the way toward it. Watch the first five days, with m starting at the first value. The picture draws each day as a point and m as a line that grows one day at a time. On each step a dashed line runs from the old m, a ring, to the new day, and the new m lands partway along it.
Try it now
Then set β (beta) to 0.9 and press Use these numbers: m keeps 0.9 of itself each day, so the line barely climbs away from 12.
Where the weights went. The flow's last step unrolled the final m: day 5 weighs 0.5, day 4 0.25, day 3 0.125 and day 2 0.0625. (Day 1 gets the last 0.0625 too, because m started as day 1.) Each step back, a value's weight is multiplied by β once more, so the weights form a geometric sequence from lesson 26, with ratio β. A value k steps back weighs (1 − β)βᵏ. Do those weights add to 1? Their total is (1 − β) times this sum, which never ends:
1+β+β2+β3+⋯=1−β1
In plain words: call the total S. Multiplying S by β moves every term one place along, so βS is S without its first 1. That means S − βS = 1, and S = 1 ÷ (1 − β).
So the weights add to (1 − β) × 1/(1 − β) = 1. An EMA is a true weighted average: newest first, and every older value a little less.
It remembers about 1/(1 − β) values. The newest value weighs 1 − β. A window of n values gives each one 1/n. With β = 0.9 the newest weighs 0.1, like each value in a window of 10, so this EMA acts roughly like an average of the last 10 values. The picture draws its first ten weights; press Next to lay a window of 10 over them.
Quick check
An EMA with β = 0.99 remembers about how many values?
100
99
10
0.01
Show the answer
100
The newest value weighs 1 − 0.99 = 0.01, as each value does in a window of 100: 1 ÷ (1 − β) = 1 ÷ 0.01 = 100.
In your own words
In your own words: why must a weighted average's weights add to 1, and how does an EMA's weights manage it?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: three averages as loops
Purpose. Write each average as a loop, then check the EMA's weights with one.
A weighted average is the Σ loop from lesson 26 with a weight in every term:
prices = [20, 35, 50]
weights = [0.5, 0.3, 0.2]
total = 0for i inrange(len(prices)):
total = total + weights[i] * prices[i]
print(total, sum(weights))
print(sum(prices) / len(prices))
It runs right here in your browser, with nothing to install.
The weighted average is 30.5, and the weights add to 1.0. The plain average of the three prices, 35.0, is what an average rental would bring if every tent rented equally often.
Now the EMA on the first five days, written with a loop. Step through it and watch m and the boxes of smooth: they're the same numbers as the flow's cells.
Worked example
Worked example: a window and an EMA on the same week
Smooth the week's rentals both ways, and compare what each one shows.
rentals = [12, 18, 9, 15, 21, 12, 18]
window = []
for i inrange(2, len(rentals)):
window.append((rentals[i - 2] + rentals[i - 1] + rentals[i]) / 3)
m = rentals[0]
ema = []
for x in rentals:
m = 0.5 * m + 0.5 * x
ema.append(m)
print(window)
print(ema)
In the lesson you can change this and run it.
The window starts at i = 2, day 3, because its first window needs three days; rentals[i - 2] and rentals[i - 1] are the two days before day i. It climbs 13.0, 14.0, 15.0, 16.0, 17.0, a steady rise hidden under the jumps. The EMA has a value from day 1, but it still jumps: 17.25 down to 14.625 and back up to 16.3125. With β = 0.5 it remembers about 1 ÷ (1 − 0.5) = 2 values, fewer than the window's 3, so it smooths less. A bigger β smooths more.
Quick check
What does this print?
m = 12for x in [12, 18, 9]:
m = 0.5 * m + 0.5 * x
print(m)
13.0
9.0
15.0
12.0
Show the answer
12.0
12 blended with 12 stays 12.0. Then 18 pulls it halfway to 15.0, and 9 pulls it halfway back down: 0.5 × 15 + 0.5 × 9 = 12.0.
The weights, unrolled. The math part says a value k steps back weighs (1 − β)βᵏ, and that all the weights add to 1. A loop can check both for β = 0.9. For the first ten values it prints how far back each is, its weight and the running total of the weights; then the total of a hundred weights, and 1/(1 − β):
beta = 0.9
total = 0for k inrange(100):
w = (1 - beta) * beta ** k
total = total + w
if k < 10:
print(k, round(w, 3), round(total, 3))
print(round(total, 3), round(1 / (1 - beta), 3))
The weights run 0.1, 0.09, 0.081 and on down to 0.039, each 0.9 times the last: the bars in the math part's picture. The first ten add to only 0.651, because the rest of the weight sits on older values. A hundred of them add to 1.0, to three places. And 1/(1 − β) is 10.0, so this EMA remembers about 10 values.
A named mistake: starting at 0. It's tempting to start m at 0, and real optimizers such as Adam do. Here's what that does to the five days, with β = 0.9:
rentals = [12, 18, 9, 15, 21]
m = 0
smooth = []
for x in rentals:
m = 0.9 * m + 0.1 * x
smooth.append(round(m, 2))
print(smooth)
print(round(1 - 0.9 ** 5, 4), round(m / (1 - 0.9 ** 5), 2))
The running average crawls up from 1.2 and reaches only 6.28 by day 5, though no day rented fewer than 9. The starting 0 still holds most of the weight: the five days' weights add to only 1 − 0.9⁵ = 0.4095. Read the early values of an EMA started at 0 and the shop looks nearly empty. Either start at the first value, or divide by the days' share, 1 − βᵗ after t values, as Adam does: m ÷ 0.4095 gives 15.33, a fair average of the five days.
A noisy curve and two smoothings. Here is a short training loss that zigzags as it falls. The function ema is the stepper's loop, wrapped up as a function so it can smooth the loss twice, with β = 0.5 and with β = 0.9. It rounds each value to 2 places for reading:
defema(xs, beta):
m = xs[0]
out = []
for x in xs:
m = beta * m + (1 - beta) * x
out.append(round(m, 2))
return out
loss = [2.9, 2.2, 2.6, 1.8, 2.3, 1.5, 2.0, 1.3, 1.7, 1.1, 1.5, 0.9]
print(ema(loss, 0.5))
print(ema(loss, 0.9))
The plot types in those two lists and draws them over the loss. label= names each line, and plt.legend() shows the names:
The β = 0.5 line still wobbles a little. The β = 0.9 line is smooth, but look where it ends.
Quick check
Read the plot at the last training step. Which line sits highest, and why?
β = 0.5: it reacts fastest to each new loss
The raw loss: it's never smoothed
β = 0.9: it still remembers the early, higher losses
None: all three end at the same point
Show the answer
β = 0.9: it still remembers the early, higher losses
The loss falls from 2.9 to 0.9. The β = 0.9 line remembers about 10 values, so it still carries the early highs and ends at 2.0, far above the last loss. Heavy smoothing lags behind a trend. The β = 0.5 line, remembering about 2 values, keeps up better and ends at 1.17.
Drill
A weighted average and a window
Fix weighted so it returns the weighted average of xs with the weights ws, and fix the loop so window holds the window-of-3 averages of the week's rentals. There's one slip in each.
defweighted(xs, ws):
total = 0for i inrange(len(xs)):
total = total + ws[i] * xs[i]
return total / len(xs)
rentals = [12, 18, 9, 15, 21, 12, 18]
window = []
for i inrange(2, len(rentals)):
window.append((rentals[i - 1] + rentals[i]) / 3)
print(weighted([20, 35, 50], [0.5, 0.3, 0.2]), window)
In the lesson, your code is checked as you write it.
In your own words
When would you pick a window average, and when an EMA?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: smoothing, momentum and Adam
Purpose. Models average all the time, and rarely evenly. Attention, in lesson 113, mixes words with weights that add to 1. A loss curve is smoothed before anyone judges it, and the optimizer keeps moving averages of the slopes.
Smoothing a training loss. The trail club's sign reader trains on a batch of signs at a time, and its loss bounces from batch to batch: some batches are easy, some hard. Here are 20 batches. Is training still improving?
loss = [2.31, 1.98, 2.12, 1.71, 1.89, 1.52, 1.66, 1.38, 1.55, 1.22,
1.41, 1.15, 1.29, 1.08, 0.97, 1.21, 1.02, 1.13, 0.93, 1.08]
m = loss[0]
smooth = []
for x in loss:
m = 0.8 * m + 0.2 * x
smooth.append(round(m, 3))
print(loss[14], loss[19])
print(smooth[14], smooth[19])
Batches 15 and 20 raw read 0.97, then 1.08, as if training got worse. The smoothed loss, with β = 0.8, says 1.303, then 1.14: still falling. One batch is mostly luck; the trend is in the average. That's why training dashboards draw a smoothed curve over the raw one. The smoothed curve also lags: at batch 20 it sits above four of the last five raw losses, because it still remembers older, higher ones.
Quick check
Why judge training by the smoothed loss, not by the last two raw losses?
Raw losses are computed wrongly
One batch is mostly luck; the average shows the trend
The smoothed loss is always the lower one
The smoothed loss reacts faster to every change
Show the answer
One batch is mostly luck; the average shows the trend
An easy or hard batch moves the raw loss up or down by luck. Averaging many batches cancels most of that, so the smoothed curve shows which way training is really going.
Momentum: an EMA of the slopes. Gradient descent, in lesson 79, steps each weight against its slope. When the slopes bounce from batch to batch, the steps zigzag. Momentum steps along an EMA of the slopes instead. A slope that stays steady keeps its size in the average, and one that flips back and forth mostly cancels out. Predict what a steady slope of 0.4 does to an average that starts at 0.5:
Quick check
What does this print?
m = 0.5for g in [0.4, 0.4, 0.4]:
m = 0.9 * m + 0.1 * g
print(round(m, 4))
0.5
0.4
0.4729
0.49
Show the answer
0.4729
Each step moves m a tenth of the way toward 0.4: 0.49, then 0.481, then 0.4729. With β = 0.9 the average follows a change slowly, over many steps, so one odd slope can't swing it either.
In the real tools. In PyTorch, the library most models are built with, every weight has its own slope, and the momentum line works on a whole tensor of them at once. Here, two weights:
Adam keeps two EMAs. Adam, the optimizer behind most large models, keeps an EMA of each weight's slopes and another of their squares. Its two β's sit in one setting, betas:
Drill
Judge a run by its smoothed loss
Fix smooth so it returns the EMA of xs with the given beta, starting at the first value. Then set still_falling to whether the smoothed loss at batch 20 is lower than at batch 15. The starter has three slips.
defsmooth(xs, beta):
m = 0
out = []
for x in xs:
m = (1 - beta) * m + beta * x
out.append(m)
return out
loss = [2.31, 1.98, 2.12, 1.71, 1.89, 1.52, 1.66, 1.38, 1.55, 1.22,
1.41, 1.15, 1.29, 1.08, 0.97, 1.21, 1.02, 1.13, 0.93, 1.08]
s = smooth(loss, 0.8)
still_falling = s[19] > s[14]
print(s[14], s[19], still_falling)
In the lesson, your code is checked as you write it.
In your own words
A colleague says a run got worse because batch 20's loss is above batch 15's. What would you check, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
Optimizers: momentum and Adam, built from the moving averages in this lesson.
Normalization layers: batch norm keeps a moving average of each feature's mean and spread to use at test time.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write