Partial derivatives

Hold $y$ fixed at 2. Then $g$ becomes $x^2 + 6x + 4$. That's an ordinary function of one input. Its slope is $2x + 6$, and the 6 there is 3 times $y$. Keep $y$ as a letter instead of 2, and you get the general answer. Holding $x$ fixed instead gives the other partial:

In plain words: for $\partial g / \partial x$, treat $y$ as a fixed number. The $x^2$ term gives $2x$. The $3xy$ term is "$3y$ times $x$", so it gives $3y$. The $y^2$ term doesn't move when only $x$ moves, so it gives 0. For $\partial g / \partial y$, swap the roles.

Worked with small numbers, at $x = 1$ and $y = 2$:

• $\partial g / \partial x = 2 \times 1 + 3 \times 2 = 8$;
• $\partial g / \partial y = 3 \times 1 + 2 \times 2 = 7$.

At this point, nudging $x$ moves $g$ a little more than nudging $y$ does. You'll confirm both numbers with Python further down.

When inputs team up

Look again at $\partial g / \partial x = 2x + 3y$. It contains $y$. So how much $x$ matters depends on where $y$ is. That's called an interaction.

You know interactions from everyday life. An extra bag of chips is worth more when there's also salsa on the table. In marketing terms, the return on search spend depends on how much you spend on social.

Compare a function with no teamwork, such as $k(x, y) = 4x + 7y$. Here $\partial k / \partial x = 4$, whatever $y$ is. Each input does its own job, and neither changes the other's effect.

Slices of the surface

Go back to the conversions landscape. Fix social spend $m$ at one value, and $f$ becomes an ordinary curve in $s$. It's like slicing a loaf of bread: each slice shows the shape at one social budget. The partial derivative $\partial f / \partial s$ is that curve's slope.

Different values of $m$ give different slices, with different slopes. The code below draws three slices: social at $1,000, $5,000 and $15,000. Then it measures the search slope at $s = 10$ for two of them. It uses the central difference from the derivatives lesson: nudge up and down by a tiny $h$ of 0.00001, written 1e-5 in code.

import numpy as np
import matplotlib.pyplot as plt

def f(s, m):
    return 40 * np.sqrt(s) + 25 * np.sqrt(m) + 3 * np.sqrt(s * m)

s = np.linspace(0.1, 20, 200)
for m in [1, 5, 15]:
    plt.plot(s, f(s, m), label=f"social = ${m}k")
plt.xlabel("search spend ($ thousands)")
plt.ylabel("conversions")
plt.legend()

h = 1e-5
for m in [1, 15]:
    slope = (f(10 + h, m) - f(10 - h, m)) / (2 * h)
    print(f"∂f/∂s at s=10, social={m}k: {slope:.2f} conversions per extra $1k of search")

With $1,000 of social, the search slope is 6.80: roughly 6.8 extra conversions per extra $1,000 of search. With $15,000 of social, it's 8.16. More social spend makes each search dollar a bit more productive, so the curves are steeper. That's the bonus term at work.

Estimating partials with nudges

The nudge recipe from the derivatives lesson works with many inputs too. Number the inputs. Then $x_i$ means input number $i$. The dots in the formula stand for every other input, left unchanged.

In plain words: move input number $i$ up a little, by $h$. Then move it down by $h$. Leave every other input where it is. Take the change in output and divide by the distance between the two points, which is $2h$.

Worked by hand with $g$ at $x = 1$, $y = 2$, using a big nudge of $h = 1$:

• For $x$: $g(2, 2) = 20$ and $g(0, 2) = 4$. So $(20 - 4) / 2 = 8$.
• For $y$: $g(1, 3) = 19$ and $g(1, 1) = 5$. So $(19 - 5) / 2 = 7$.

Both match the rule. For $g$, this recipe happens to be exact for any $h$. Most functions need a tiny $h$ to get close.

The code does the same with a tiny $h$. The partial function makes two copies of the point, up and down. It nudges input i in each copy and returns the slope. The copies matter: if you nudged the original point, you'd change it for any code that uses it next. Python counts from 0, so i = 0 is $x$ and i = 1 is $y$.

import numpy as np

def g(p):
    x, y = p
    return x ** 2 + 3 * x * y + y ** 2

def partial(f, point, i, h=1e-5):
    up = np.array(point, dtype=float)
    down = np.array(point, dtype=float)
    up[i] += h
    down[i] -= h
    return (f(up) - f(down)) / (2 * h)

p = [1.0, 2.0]
print("∂g/∂x:", round(partial(g, p, 0), 6), " rule: 2x + 3y =", 2 * 1 + 3 * 2)
print("∂g/∂y:", round(partial(g, p, 1), 6), " rule: 3x + 2y =", 3 * 1 + 2 * 2)

The nudges give 8.0 and 7.0, the same as the rule.

The cost of nudging

Each partial needs two runs of the function. A model with $n$ weights needs $2n$ runs of the loss to get every partial. With 1,000 weights, that's 2,000 runs. That's fine for checking code on a small model. For a billion weights it's impossible.

Training uses a method called backpropagation instead, which you'll meet later. It gets all $n$ partials in one backward pass through the model.

import math

def f(s, m):
    return 40 * math.sqrt(s) + 25 * math.sqrt(m) + 3 * math.sqrt(s * m)

h = 1e-5
s = 9
for m in [4, 5]:
    by_search = (f(s + h, m) - f(s - h, m)) / (2 * h)   # social held at m
    by_social = (f(s, m + h) - f(s, m - h)) / (2 * h)   # search held at 9
    print(m, round(by_search, 2), round(by_social, 2))