Exponentials: growth and decay
Why this matters
Some things grow by adding the same amount each step; others grow by multiplying by the same factor. The second kind, exponential growth, starts slowly and then outruns any straight line. Its mirror image, decay, shrinks by a factor each step. Both are everywhere in machine learning: learning rates decay over training, and the exponential function turns a model's raw scores into positive numbers it can compare. You'll also rebuild the number e from scratch, with a loop that pays interest more and more often, and see why exponential growth always overtakes a straight line in the end.
- Recognize exponential growth and decay: the same factor each step
- Compare exponential growth with a straight line, and see where e = 2.718… comes from
- Compute exponentials in Python with ** and math.exp, and with a loop that multiplies each step
- Build a decaying learning-rate schedule, and see how e^score makes any score positive
The math: the same factor each step
Purpose. Describe things that multiply by the same amount each step.
Linear growth adds the same amount each step: 100, 150, 200, 250. Exponential growth multiplies by the same factor each step: 100, 200, 400, 800. In symbols:
In plain words: start at a, and multiply by r once for each of t steps. With r = 2 it doubles every step.
When r is above 1 the value grows; when r is between 0 and 1 it shrinks, which is decay. Halving each step, r = 0.5, takes 80 to 40, 20, 10. It never quite reaches zero: there's always half of something left.