Computation graphs

Chaining them

The chain rule from Phase 2 says rates along a chain multiply. Each dollar buys 2 clicks, and each click is worth $3, so each dollar is worth $2 \times 3 = \$6$. The graph is a chain too. $w$ feeds $p$, which feeds $q$, then $r$, then $L$. So the effect of $w$ on $L$ is the product of the local derivatives along that path:

In plain words: start at the loss and walk back to $w$. At each node, multiply by that node's local derivative. The square gives $2r$. The subtract and the add each give 1. The multiply gives $x$, the other input.

Now put in the numbers. Here $r = -3$, so $2r = -6$. Then $-6 \times 1 \times 1 \times 3 = -18$. For $b$, the path is shorter: from $L$ to $r$ to $q$, then to $b$. That gives $-6 \times 1 \times 1 = -6$.

The code below does both passes. The forward lines fill in the nodes. The backward lines multiply local derivatives, from the loss back toward the inputs. A name like dL_dq holds the rate of $L$ with respect to that node.

w, x, b, y = 2.0, 3.0, 1.0, 10.0

# forward: evaluate the graph node by node
p = w * x
q = p + b
r = q - y
L = r ** 2

# backward: multiply local derivatives along each path
dL_dr = 2 * r
dL_dq = dL_dr * 1
dL_dp = dL_dq * 1
dL_dw = dL_dp * x          # multiply node: the slope for w is the other input, x
dL_db = dL_dq * 1
print(f"L = {L}   dL/dw = {dL_dw}   dL/db = {dL_db}")

# check against finite differences: nudge w and b a tiny bit
h = 1e-6
f = lambda w, b: (w * x + b - y) ** 2
print(f"numerical dL/dw ≈ {(f(w + h, b) - f(w - h, b)) / (2 * h):.4f}   dL/db ≈ {(f(w, b + h) - f(w, b - h)) / (2 * h):.4f}")

The first line prints L = 9.0 dL/dw = -18.0 dL/db = -6.0. The second line checks those answers another way. It nudges $w$ and $b$ a tiny amount and measures how far $L$ moves. It gets about −18.0000 and −6.0000, a match.

Both rates are negative. That means increasing $w$ or $b$ would lower the loss. This makes sense. The prediction is 7 and the target is 10, so the model should aim higher.

When a value is used twice, gradients add

Picture a café. The price of milk goes into both the latte and the cappuccino. If milk costs 10 cents more, your costs go up through both drinks. The total change is the two changes added together.

A graph works the same way. If a value feeds into several later nodes, it affects the output along several paths. Its total rate is the sum of what each path contributes. These rates are often called gradients.

Take $f = a \cdot a + 3a$, with $a = 2$. Split it into nodes: $u = a \cdot a$, then $v = 3a$, then $f = u + v$. The value $a$ reaches $f$ along three paths:

1. As the first input to the multiply node $u$. Its local slope is the other input, $a = 2$.
2. As the second input to $u$. Again, the slope is the other input: 2.
3. Through $v = 3a$. Raising $a$ by 1 raises $v$ by 3, so the slope is 3.

The add node $f = u + v$ passes each path through with a slope of 1. Add the three paths: $2 + 2 + 3 = 7$.

In plain words: follow every road from the input to the output. Work out each road's rate. Then add the roads together.

a = 2.0
u = a * a          # a is used twice here...
v = 3 * a          # ...and once more here
f = u + v

df_du, df_dv = 1.0, 1.0
df_da = df_du * a + df_du * a + df_dv * 3     # three paths from a to f
print(f"f = {f}, df/da = {df_da}   (calculus says 2a + 3 = {2 * a + 3})")

It prints f = 10.0, df/da = 7.0. Calculus agrees. The slope of $a^2 + 3a$ is $2a + 3$, which is 7 when $a = 2$.

Watch out: when you write the backward pass by hand, add each new path to the running total (+=). Replacing the total (=) keeps only the last path. Forgetting to add is the classic bug in hand-written backpropagation.

grad = 0.0
for contribution in [4.0, -1.0]:
    grad += contribution
print(grad)

w, x, b = 0.5, 4.0, -1.0

# forward
p = w * x          # multiply
z = p + b          # add
out = max(0.0, z)  # ReLU

# backward: start at 1 and multiply local slopes
d_out = 1.0
d_z = d_out * (1.0 if z > 0 else 0.0)   # ReLU slope
d_p = d_z * 1.0                         # add: slope 1
d_b = d_z * 1.0
d_w = d_p * x                           # multiply: the other input
print(out, d_w, d_b)

Why frameworks think in graphs

• PyTorch records the graph as your code runs. Each operation on a tensor that needs gradients adds a node. (A tensor is PyTorch's name for an array of numbers.) When you call loss.backward(), it walks the graph backward and fills in every rate. It never writes out a formula for the whole thing, the way you might on paper. It only knows each small step's rule.
• Memory. The backward pass needs values from the forward pass. The multiply rule, for example, needs the other input. So the framework stores those in-between values, called activations. It's like keeping your receipts on a trip, because you'll need them on the way back. Storing activations is why training uses far more memory than running a model. Some setups save memory by throwing some activations away and working them out again when needed. That trick is called activation checkpointing.
• Scale. One forward pass of a large language model (LLM) is a graph of millions of these small operations on large arrays. Yet every one of them has a local derivative as simple as the ones in the table above.