Derivatives

Computing it in code

You can't make $h$ shrink forever on a computer. But a tiny $h$ gets you very close. A better trick is to take a small step each side of the point and measure across both. This is the central difference:

In plain words: look a little to the right and a little to the left. Take the change in output between those two points. Divide by the distance between them, which is $2h$.

Worked at $x = 3$ with $h = 1$: the outputs at 4 and 2 are 16 and 4. The change is 12. The distance is 2. So the estimate is 12 ÷ 2 = 6, the true answer.

Here are both methods side by side, with smaller and smaller steps:

def f(x):
    return x ** 2

x = 3
for h in [1, 0.1, 0.01, 0.0001]:
    one_sided = (f(x + h) - f(x)) / h
    central = (f(x + h) - f(x - h)) / (2 * h)
    print(f"h={h:<7} one-sided={one_sided:.6f}  central={central:.6f}")

The one-sided column creeps toward 6: 7, then 6.1, then 6.01, then 6.0001. The central column shows 6 on every row. That's why the central difference is the one to use.

Engineers use numerical derivatives like this to check their gradient code. They're too slow for training, though. Each weight needs at least one extra run of the function, and big models have billions of weights. The next lessons give exact formulas instead.

def miles(t):
    return 50 * t + 5 * t ** 2

t, h = 2, 0.5
print(miles(t + h), miles(t - h))
print((miles(t + h) - miles(t - h)) / (2 * h))

f = lambda x: x ** 2
print((f(4.1) - f(3.9)) / 0.2)

Reading a derivative

Back to the hike. The derivative tells you what the ground is like under your feet, if you face toward bigger $x$:

At the very top of a hill and the very bottom of a valley, the ground is flat. The derivative is zero there. That's why "find where the derivative is zero" is the classic way to find the best value of something.

It's also why a tiny gradient during training can mean two different things. (A gradient is the derivative when there are many weights; you'll meet it soon.) Training may have reached the bottom of the valley. Or it may be stuck on a flat plateau.

Marginal returns

In the functions lesson you met a curve for ad conversions. As spend grows, conversions keep rising, but more and more slowly. Here it is again. The letter $s$ is the spend in dollars. Then $c(s)$ is the number of conversions that spend buys:

In plain words: conversions can never pass 900, the ceiling. The 4000 sets how fast you approach it: spending $4,000 gets you halfway, to 450.

Worked example: spend $1,000 and you get 900 × 1000 ÷ 5000 = 180 conversions.

The derivative $c'(s)$ is the marginal return: the extra conversions from one more dollar at spend $s$. Each conversion earns a $60 margin. So one more dollar of spend brings back $60 \cdot c'(s)$ dollars. That extra dollar pays for itself only while this is more than 1:

In plain words: keep spending while the next dollar earns back more than a dollar.

The code estimates $c'(s)$ with the central difference at five budgets:

import numpy as np

def conversions(s):
    return 900 * s / (s + 4000)

def derivative(f, x, h=1e-3):
    return (f(x + h) - f(x - h)) / (2 * h)

for spend in [1000, 5000, 10000, 15000, 20000]:
    marginal = derivative(conversions, spend)
    label = f"${spend:,}"
    print(f"at {label:>7}: {marginal * 1000:6.2f} conversions per extra $1,000, worth ${60 * marginal:.2f} per $1")

Read one row. At $10,000 the next $1,000 buys about 18.37 conversions. That's about 0.018 per dollar, worth $1.10 at $60 each. Still worth it. At $15,000 the next dollar brings back only $0.60.

So somewhere between $10,000 and $15,000 the next dollar starts returning less than a dollar. That crossover is the profit peak you found by grid search in the functions lesson. This time you found it by reading slopes.