Phase 2 · Math, when you need it · Calculus

Derivative rules

Power, sum and product rules, so you rarely compute from scratch.

About 25 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

You'll rarely work out a slope by hand. PyTorch and JAX, the libraries that train models, do it for you. But a handful of rules explain what those libraries compute. They also explain why some of the small functions inside a network train better than others. Sigmoid's slope almost vanishes for big inputs; ReLU's doesn't. And they give you a way to check a slope you wrote yourself. Ten minutes with these rules pays off every time you read a training formula.

What you'll be able to do

Apply the power, constant, sum and product rules
Know the derivatives of eˣ, ln x, sigmoid and ReLU
Differentiate a polynomial from its list of coefficients
Check derivatives symbolically with SymPy and numerically with a small step

Shortcuts for slopes

In the last lesson you found a slope by nudging $x$ a tiny bit and seeing how far the output moved. That works, but it's slow, and it only gets close to the true answer. A few shortcut rules give the exact slope in one step. It's like knowing your times tables instead of counting on your fingers.

A quick reminder of the notation. $f'(x)$ , said "f prime of x", is the derivative of $f$ : its slope at $x$ . The symbol $\frac{d}{dx}$ means the same thing, written in front of a formula. Read it as "the slope of what follows, as $x$ changes".

The power rule

$x^n$ means $x$ multiplied by itself $n$ times, so $x^3 = x \cdot x \cdot x$ . The small raised number $n$ is called the power. To find the slope, bring the power down in front and lower it by one:

\frac{d}{dx} x^n = n\,x^{n-1} \qquad \frac{d}{dx} x^3 = 3x^2 \qquad \frac{d}{dx} x = 1 \qquad \frac{d}{dx} 7 = 0

In plain words: move the power to the front, then take one off it. So $x^3$ becomes $3x^2$ . A plain $x$ has slope 1. A lone number such as 7 never changes, so its slope is 0.

Here's a worked example. How steep is $x^3$ at $x = 2$ ? The rule says $3 \cdot 2^2 = 3 \cdot 4 = 12$ . The nudge method from the last lesson agrees:

def f(x):
    return x ** 3

x = 2
rule = 3 * x ** 2                               # power rule: 3x²
nudge = (f(x + 0.001) - f(x - 0.001)) / 0.002   # central difference
print(rule, round(nudge, 4))

It runs right here in your browser, with nothing to install.

It prints 12 12.0. The shortcut and the slow way give the same slope.

Go deeper Why the power rule works

Take $x^2$ and nudge $x$ by a small amount $h$ . The output becomes $(x + h)^2 = x^2 + 2xh + h^2$ . So the output grew by $2xh + h^2$ . Divide by the nudge $h$ to get the slope: $2x + h$ . As $h$ shrinks toward 0, that becomes $2x$ . That's the power rule's answer.

Higher powers work the same way. After you divide by $h$ , every term but one still contains $h$ . Those terms fade away as $h$ shrinks, and $n\,x^{n-1}$ is the one left.

Quick check

Using the power rule, what is the slope of x⁴ at x = 1?

Show the answer

The slope of x⁴ is 4x³. At x = 1 that's 4 · 1 = 4.

Numbers in front, and sums

Two more rules let you handle a long formula one piece at a time.

A number in front stays in front. The slope of $5x^2$ is 5 times the slope of $x^2$ . That's $5 \cdot 2x = 10x$ .
The slope of a sum is the sum of the slopes. Work out each piece, then add them up.

Think of a shopping receipt. If the prices of a few items change, the total changes by the sum of those changes. Slopes add up the same way:

\frac{d}{dx}\big(3x^2 + 5x - 2\big) = 6x + 5

In plain words: go term by term. $3x^2$ gives $3 \cdot 2x = 6x$ . Then $5x$ gives 5. The lone $-2$ gives 0. Add them up to get $6x + 5$ .

At $x = 2$ , the slope is $6 \cdot 2 + 5 = 17$ . Nudge $x$ up a tiny bit there, and the output rises about 17 times as much. You'll check this formula with Python later in the lesson.

Quick check

What is the derivative of 4x³ − 2x + 9?

12x³ − 2
4x² − 2
12x² − 2
12x² − 2x + 9

Show the answer

12x² − 2

Use the power rule on each term: 4 · 3x² gives 12x², −2x gives −2, and the lone 9 gives 0.

The product rule

Sometimes a formula is two changing things multiplied together. Revenue is a good example. It's the price times the number of units sold. Raise the price and each sale earns more, but you sell fewer.

Picture a rectangle whose width and height are both growing. The area grows for two reasons. The width grows while the height stays put. And the height grows while the width stays put. The product rule adds those two effects:

\frac{d}{dx}\big(f(x)\,g(x)\big) = f'(x)\,g(x) + f(x)\,g'(x)

Here $f(x)$ and $g(x)$ are the two things being multiplied. The prime marks on $f'(x)$ and $g'(x)$ mean their slopes.

In plain words: each part takes a turn changing while the other stays fixed. Then you add the two results. You do not multiply the two slopes together.

Now a worked example. In the functions lesson, a shop sold $1000 - 8p$ units a month at a price of $p$ dollars. Call that demand $D(p)$ . Revenue is price times demand: $R(p) = p \cdot D(p)$ . It has two parts:

the price $p$ has slope 1;
demand, $1000 - 8p$ , has slope $-8$ . Each extra dollar loses 8 sales.

The product rule gives $R'(p) = 1 \cdot (1000 - 8p) + p \cdot (-8) = 1000 - 16p$ .

Revenue peaks where its slope is zero, because the top of a hill is flat. Solving $1000 - 16p = 0$ gives $p = 62.5$ . Your grid search in the functions lesson hunted for that peak by trying price after price. Here it takes one line.

Watch out: multiplying the two slopes would give $1 \cdot (-8) = -8$ . That says revenue always falls as the price rises. It's wrong: at low prices, raising the price earns more.

Go deeper Why the product rule works

Give the rectangle width $f$ and height $g$ . Nudge $x$ by a small amount $h$ . The width grows by about $f'h$ , and the height grows by about $g'h$ .

The new area adds three pieces. There's a strip down the side, $f'h \cdot g$ . There's a strip along the top, $f \cdot g'h$ . And there's a tiny corner, $f'g'h^2$ . Divide by $h$ to get the slope: $f'g + fg' + f'g'h$ . As $h$ shrinks, the corner term fades away. What's left is the product rule.

Quick check

A student finds the slope of x · (10 − x) by multiplying the two slopes: 1 × (−1) = −1. What is the correct slope?

10 − 2x
−1
−2x
10 − x

Show the answer

10 − 2x

Use the product rule: 1 · (10 − x) + x · (−1) = 10 − 2x. Multiplying the slopes is a common mistake.

A table worth knowing

AI code uses a handful of functions again and again. Their slopes are worth knowing by sight. A few of the names may be new, so here they are first:

$e$ is a fixed number, about 2.718. The curve $e^x$ is special: its slope always equals its own height. It shows up wherever things grow in proportion to their size, like compound interest.
$\ln x$ , the natural logarithm, runs $e^x$ backwards. It answers "e to what power gives x?" You met both in the exponentials lesson.
$\sigma(x)$ , said "sigma of x", is the sigmoid from the functions lesson. It squashes any number into the range 0 to 1.
ReLU keeps positive numbers and turns negative ones into 0.
$\tanh x$ , said "tanch x", is an S-curve like the sigmoid. It runs from −1 to 1 instead.

$f(x)$	$f'(x)$	Note
$x^n$	$n x^{n-1}$	power rule
$e^x$	$e^x$	its own slope
$\ln x$	$1 / x$	slows down as x grows
$\sigma(x) = \frac{1}{1 + e^{-x}}$	$\sigma(x)\,(1 - \sigma(x))$	at most 0.25, nearly 0 far from 0
$\text{ReLU}(x) = \max(0, x)$	1 if $x > 0$ , else 0	constant slope where active
$\tanh x$	$1 - \tanh^2 x$	at most 1

In plain words: for the sigmoid, take its output and call it s. Its slope is s times (1 − s). You never need anything beyond the output itself.

Try it with small numbers. At $x = 0$ , the sigmoid outputs 0.5. The slope is 0.5 × 0.5 = 0.25, the steepest it ever gets. At $x = 5$ , the output is about 0.9933. The slope is about 0.9933 × 0.0067, roughly 0.0066. That's nearly flat.

Quick check

Where is the sigmoid at its steepest?

At an input of 0
At very large inputs
It's equally steep everywhere
At very negative inputs

Show the answer

At an input of 0

Its slope is s × (1 − s), which is largest when s is 0.5. Far from 0 the curve flattens, and that's why sigmoids can stall training.

ReLU has a sharp corner at exactly 0, so it has no single slope there. Code uses 0 by convention.

Why ReLU replaced sigmoid

That sigmoid row explains a famous problem. In a deep network, the slopes of many layers get multiplied together. (That's the chain rule, the next lesson.) The sigmoid's slope is never more than 0.25. Ten layers of 0.25 multiply to about 0.000001. So the learning signal for the early layers all but disappears. People call this the vanishing gradient problem. A gradient is the set of slopes that training uses to adjust a model.

ReLU's slope is exactly 1 wherever it's active, and 1 × 1 × 1 stays 1. That's a big reason it replaced the sigmoid inside networks.

Try it now

Drag along the sigmoid. The slope is 0.25 at x = 0 and almost flat beyond ±5: a nearly vanishing gradient.

Quick check

A sigmoid outputs 0.9 at some point. What does this print?

s = 0.9
print(round(s * (1 - s), 2))

0.9
0.09
0.81
0.1

Show the answer

0.09

The sigmoid's slope is s × (1 − s) = 0.9 × 0.1 = 0.09. Far from the middle, the curve is already flattening out.

Checking with SymPy

SymPy is a Python library that does algebra with letters instead of numbers. So it can find slopes exactly, the way you would on paper. sp.symbols creates the letters. Then sp.diff(formula, letter) finds the slope.

import sympy as sp

x, p = sp.symbols("x p")
print(sp.diff(3 * x**2 + 5 * x - 2, x))
print(sp.diff(p * (1000 - 8 * p), p))
sigmoid = 1 / (1 + sp.exp(-x))
print(sp.simplify(sp.diff(sigmoid, x) - sigmoid * (1 - sigmoid)))   # 0: the table is right

It prints 6*x + 5 and 1000 - 16*p. Those match the sum rule and the product rule above. The last line subtracts the table's sigmoid formula from SymPy's own answer. sp.simplify tidies up the result, and 0 means the two agree.

Checking with numbers

You can also test a rule with the nudge method from the last lesson. Nudge each point up and down by 0.00001 (written 1e-5 in code), then compare. This is exactly how engineers test the code that computes gradients.

import numpy as np

sigmoid = lambda x: 1 / (1 + np.exp(-x))
rule = lambda x: sigmoid(x) * (1 - sigmoid(x))
xs = np.array([-6.0, -1.0, 0.0, 2.0, 6.0])
numeric = (sigmoid(xs + 1e-5) - sigmoid(xs - 1e-5)) / 2e-5
print(np.round(rule(xs), 5))
print(np.allclose(rule(xs), numeric))

The first line shows the rule's slope at five points. It peaks at 0.25 in the middle and shrinks to about 0.0025 at −6 and 6. np.allclose asks "are these all nearly equal?" The answer True means the rule and the nudges agree.

Worked example

Worked example: pricing cookies

A bakery sells cookies. At a price of p dollars, it sells 40 − 4p cookies a day. So revenue is p × (40 − 4p). Use the product rule to find the slope at three prices. Check each one with nudges, then find the best price.

def revenue(p):
    return p * (40 - 4 * p)          # price times cookies sold

def rule(p):
    return (40 - 4 * p) + p * (-4)   # product rule: 40 - 8p

for p in [3, 5, 7]:
    nudge = (revenue(p + 0.001) - revenue(p - 0.001)) / 0.002
    print(p, rule(p), round(nudge, 4))

In the lesson you can change this and run it.

The product rule gives 1 × (40 − 4p) + p × (−4), which is 40 − 8p. At $3 the slope is 16. So raising the price by 10 cents adds about $1.60 a day. At $5 the slope is 0, so revenue peaks there: 5 × 20 cookies is $100 a day. At $7 the slope is −16, which means the price is too high. The nudges match the rule at every price.

Practice what you just read

4 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
5 review cards for spaced repetition

Create a free account

Try a quiz question

What is the derivative of eˣ?

eˣ
x·eˣ⁻¹
1/x
0

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Derivatives