Some things grow by adding the same amount each step; others grow by multiplying by the same factor. The second kind, exponential growth, starts slowly and then outruns any straight line. Its mirror image, decay, shrinks by a factor each step. Both are everywhere in machine learning: learning rates decay over training, and the exponential function turns a model's raw scores into positive numbers it can compare. You'll also rebuild the number e from scratch, with a loop that pays interest more and more often, and see why exponential growth always overtakes a straight line in the end.
What you'll be able to do
Recognize exponential growth and decay: the same factor each step
Compare exponential growth with a straight line, and see where e = 2.718… comes from
Compute exponentials in Python with ** and math.exp, and with a loop that multiplies each step
Build a decaying learning-rate schedule, and see how e^score makes any score positive
Quick check
Have a guess before you start. Which is bigger after 10 steps: starting at 1 and doubling each step, or starting at 1 and adding 100 each step?
Doubling: 1,024 against 1,001
They're equal
Doubling, but only just: 20 against 1,001
Adding: 1,001 against 1,024
Show the answer
Doubling: 1,024 against 1,001
Doubling ten times gives 2¹⁰ = 1,024. Adding 100 ten times gives 1 + 1,000 = 1,001. Doubling starts slow and wins anyway, and after 20 steps it's over a million.
The math: the same factor each step
Purpose. Describe things that multiply by the same amount each step.
Linear growth adds the same amount each step: 100, 150, 200, 250. Exponential growth multiplies by the same factor each step: 100, 200, 400, 800. In symbols:
y=a⋅rt
In plain words: start at a, and multiply by r once for each of t steps. With r = 2 it doubles every step.
When r is above 1 the value grows; when r is between 0 and 1 it shrinks, which is decay. Halving each step, r = 0.5, takes 80 to 40, 20, 10. It never quite reaches zero: there's always half of something left.
Quick check
From an earlier lesson: what is 2¹⁰, and roughly what is it?
1,000,000, a million
100, a hundred
20, about twenty
1,024, about a thousand
Show the answer
1,024, about a thousand
Ten 2s multiplied make 1,024, close to a thousand. So ten doublings multiply by about a thousand.
Watch ten doublings, starting from 1. Each step multiplies by the same factor, 2, so each step adds more than the one before.
The picture sets steady multiplying beside steady adding. Both start at 1 and take the same first step, up 1. Then one keeps multiplying by 2 (filled dots) and the other keeps adding 1 (hollow dots). Watch how far apart they end up.
Try it now
Then set the factor each step to 0.5 and the starting value to 80: the same flow shows decay, halving each step. Taking away the first drop, 40, each step hits 0 after two steps; halving never gets there.
The number e. One base is special: e = 2.718…. Here's where it comes from. Put 1 dollar in a bank that pays 100% interest a year. Paid once, you end with 2. Paid as two halves, 50% each time, you end with (1 + 1/2)² = 2.25. Paid monthly, (1 + 1/12)¹² ≈ 2.61. The more often it's paid, the closer you get to a limit: 2.718…, the number e.
The function eˣ, also written exp(x), grows exponentially, and e⁻ˣ decays. Their graphs are mirror images.
Quick check
A population of 80 halves every year. How many are left after 3 years?
Which grows faster in the long run: 1,000 + 50t or 2ᵗ?
1,000 + 50t: it starts much higher
2ᵗ: it overtakes any straight line
Neither grows
They grow at the same rate
Show the answer
2ᵗ: it overtakes any straight line
A line adds 50 each step; 2ᵗ doubles. 2ᵗ passes it soon after t = 10, and the gap then grows without end.
Spotting growth in a table. For a straight line, the differences between rows are equal. For an exponential, the ratios are: each row divided by the one before gives the same factor. 3, 6, 12, 24 has ratios of 2, and 100, 90, 81, 72.9 has ratios of 0.9. Checking the ratios is the quickest way to tell exponential data from linear.
In your own words
In your own words: what's the difference between linear and exponential growth?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: powers and a loop that multiplies
Purpose. Two ways to compute growth: in one line with a power, or step by step in a loop.
It runs right here in your browser, with nothing to install.
math.exp(x) is eˣ, and math.e is e itself. The first print, 100 × 2³, does three doublings in one go. A loop does the same three doublings one step at a time. Step through it and watch history grow by one box each pass.
A loop can also rebuild e from the bank example, paying the interest more and more often:
for n in [1, 2, 12, 365, 10000]:
print(n, (1 + 1 / n) ** n)
Paid 10,000 times a year it reaches 2.71814…, closing in on e = 2.71828….
Counting steps with a loop. When the question is "how many steps until…?", a loop that multiplies and counts answers it directly:
value = 1
steps = 0for i inrange(20):
if value < 1000:
value = value * 2
steps = steps + 1print(steps, value)
A named mistake: adding the rate instead of multiplying. Growing 5% a step means multiplying by 1.05, not adding 0.05. And decaying 10% a step means multiplying by 0.9, not taking away 10% of the starting amount each time.
Worked example
Worked example: savings at 5% a year
1,000 dollars grows 5% a year. Find its value after 10 years, then compare with adding 50 a year.
print(round(1000 * 1.05 ** 10, 2))
print(1000 + 50 * 10)
value = 1000for year inrange(10):
value = value * 1.05print(round(value, 2))
In the lesson you can change this and run it.
5% a year is a factor of 1.05, so after 10 years it's 1,000 × 1.05¹⁰ ≈ 1,628.89. Adding a flat 50 a year (5% of the start) gives only 1,500: the exponential earns interest on its interest. The loop reaches the same 1,628.89 by multiplying once a year, which is how a program would update a balance as time passes.
Quick check
What does this print?
value = 81for i inrange(3):
value = value / 3print(value)
3.0
72.0
27.0
1.0
Show the answer
3.0
Dividing by 3 three times: 81 → 27 → 9 → 3.0. That's decay by a factor of 1/3.
Quick check
Which line grows a balance by 3% for one step?
balance = balance + 0.03
balance = balance * 0.03
balance = balance + 3
balance = balance * 1.03
Show the answer
balance = balance * 1.03
Keep all of it and add 3%: a factor of 1.03.
Doubling time. A handy rule: something growing by r% a step doubles in about 70 ÷ r steps. At 7% a year, money doubles in about 10 years; at 1% a step, about 70 steps. It's only approximate, but it turns a percentage into a feel for speed. The loop above is the exact way: multiply and count until the value passes double.
Multiplying against adding, drawn. Here is the race from the guess at the start: doubling from 1, against adding 100 each step from 1. Matplotlib draws both with the same lines as in lesson 19. The adding one is a straight line, so the second plt.plot gives just its two ends, step 0 and step 12:
Read the graph. Between which two steps does the doubling curve pass the straight line?
It never passes the line
Between steps 11 and 12
Between steps 4 and 5
Between steps 9 and 10
Show the answer
Between steps 9 and 10
Doubling hugs the bottom for most of the way, then shoots up: it is still under the line at step 9 and above it at step 10, as the guess at the start found. After that the gap only grows.
Drill
Growth and decay in code
Fix the code: grown should be 500 after 4 years of 10% growth, and steps the number of halvings that take 1,000 below 1.
grown = 500for year inrange(4):
grown = grown + 0.1
value = 1000
steps = 0for i inrange(20):
if value >= 1:
value = value * 2
steps = steps + 1print(round(grown, 2), steps)
In the lesson, your code is checked as you write it.
In your own words
Why does 5% growth a year beat adding 5% of the starting amount each year?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: decaying steps, and e to the score
Purpose. Exponentials do two everyday jobs in training.
A learning-rate schedule. Training takes big steps at first and smaller ones later. A common schedule multiplies the learning rate η by a factor r below 1 after each round:
ηt=η0⋅rt
In plain words: start at η₀, and shrink by the same factor every round.
lr = 0.1
r = 0.9for t inrange(0, 10):
print(t, round(lr * r ** t, 4))
The picture plots those ten rates, round across and learning rate up, with a dashed line at half the start.
Quick check
A learning rate starts at 0.1 and is multiplied by 0.9 each round. What is it after one round?
0.19
0.9
0.01
0.09
Show the answer
0.09
0.1 × 0.9 = 0.09: 10% smaller.
In the real tools. PyTorch, the library most models are built with, has this schedule built in. You give it the optimizer, the part of PyTorch that takes the steps downhill, and the factor, which it calls gamma. After each round of training, sched.step() multiplies the learning rate by gamma:
e to the score. A model often gives raw scores that can be negative. Raising e to each score makes them all positive, while keeping their order, and it stretches the gaps: a score 2 higher becomes about 7.4 times bigger, since e² ≈ 7.39. This is the first step of softmax, which turns scores into probabilities in lesson 104.
import math
for s in [1.0, 2.0, 4.0]:
print(s, round(math.exp(s), 2))
print(round(math.exp(4) / math.exp(2), 2))
The score of 4 isn't just twice the score of 2 after exponentiating; it's e² ≈ 7.39 times as big. The highest score ends up dominating. In PyTorch, torch.exp does the same to every score in a tensor at once, negative scores included:
Why decay instead of a fixed step? A learning rate big enough for fast early progress tends to bounce around the bottom of the loss bowl at the end, overshooting each time. One small enough to settle is painfully slow at the start. Decay gets both: big steps early, small steps late. Other schedules exist, but the exponential one is the simplest: one factor, applied every round. Predict three halvings of 0.1.
Quick check
What does this print?
lr = 0.1for t inrange(3):
lr = lr * 0.5print(lr)
0.025
-0.05
0.0125
0.05
Show the answer
0.0125
Halving three times: 0.1 → 0.05 → 0.025 → 0.0125.
Drill
A schedule and three scores
Set rate5 to a learning rate of 0.2 after 5 rounds of decay by 0.8, and ratio to how many times bigger e³ is than e¹ (use math.exp).
import math
rate5 = 0.2 - 0.8 * 5
ratio = math.exp(3) - math.exp(1)
print(rate5, ratio)
In the lesson, your code is checked as you write it.
In your own words
Why does exponentiating scores make the highest score dominate?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write