Activation functions

The common activations

Each activation takes one number in and gives one number out. Call the number going in $z$. It's the neuron's weighted sum plus bias, from the last lesson.

• ReLU keeps positive numbers and turns negative ones into 0. Think of a bank account that can't go below zero.
• Leaky ReLU keeps positive numbers too. Negative ones shrink to 1% of their size instead of vanishing, so −4 becomes −0.04.
• Sigmoid squeezes any number into the range 0 to 1. You used it in logistic regression.
• tanh (short for "hyperbolic tangent") has the same S-shape as sigmoid, but its range is −1 to 1.
• GELU is a smooth version of ReLU. Big positive numbers pass almost unchanged. Big negative ones fade to nearly 0. In between it curves gently, and dips a little below 0.

Here they are side by side. The letter $e$ is the number about 2.718 from the exponentials lesson. The symbol $\Phi$ in GELU's row is explained just below the table.

In plain words: ReLU is "the number or 0, whichever is bigger". Leaky ReLU is "the number, or 1% of it if it's negative". Sigmoid and tanh are S-shaped squashers. GELU is "the number, times a share between 0 and 1 that grows with the number".

In GELU, $\Phi$ (capital phi, said "fie") is the normal distribution's cumulative probability. It's the chance that a value from the standard bell curve (average 0, spread 1) lands below $z$, so it's always between 0 and 1. Many recent LLMs use SwiGLU instead. It's a gated cousin of these: it multiplies two linear projections, one of them passed through a smooth activation.

A hidden layer is any layer before the last one. An MLP is a plain stack of layers (the next lesson). CNNs are networks for images, and RNNs are older networks for sequences. You'll meet both later.

import math

print("z, ReLU, leaky ReLU, sigmoid, tanh, GELU")
for z in [-1.0, 0.0, 2.0]:
    relu = max(0.0, z)
    leaky = z if z > 0 else 0.01 * z
    sigmoid = 1 / (1 + math.exp(-z))
    tanh = math.tanh(z)
    gelu = z * 0.5 * (1 + math.erf(z / math.sqrt(2)))   # z times Phi(z)
    print(z, relu, leaky, round(sigmoid, 3), round(tanh, 3), round(gelu, 3))

z = -4.0
relu = max(0.0, z)
leaky = z if z > 0 else 0.01 * z
print(relu, leaky)

What the slopes do to learning

An activation's slope says how much its output moves when its input moves a little. You met slopes in the calculus lessons.

Slopes matter because of how training works. Training sends an error signal backward through the network. At each layer, the chain rule multiplies that signal by the activation's slope. A slope of 1 lets the whole signal through. A slope of 0.1 lets a tenth through. A slope of 0 blocks it. Think of a row of taps on a pipe: each one can open fully, open a little, or shut.

So the slope decides how much learning signal gets through. For sigmoid, written $\sigma$ (the Greek letter sigma), the slope has a neat form. The mark $\sigma'$ means "the slope of sigma":

In plain words: the sigmoid's slope is its output times "one minus its output".

Try it with small numbers. At $z = 0$, the sigmoid gives 0.5, so the slope is $0.5 \times 0.5 = 0.25$. That's the biggest it ever gets. At $z = 2$, the sigmoid gives about 0.881, so the slope is about $0.881 \times 0.119 \approx 0.105$. The other slopes are simpler. ReLU's is 1 for positive inputs and 0 otherwise. Leaky ReLU's is 1 or 0.01. The code below works out all four at five points (tanh's slope is 1 - np.tanh(z) ** 2):

import numpy as np

z = np.array([-6.0, -2.0, 0.0, 2.0, 6.0])
sig = 1 / (1 + np.exp(-z))
slopes = {
    "sigmoid": sig * (1 - sig),
    "tanh": 1 - np.tanh(z) ** 2,
    "ReLU": (z > 0).astype(float),
    "leaky ReLU": np.where(z > 0, 1.0, 0.01),
}
print("z:          ", z)
for name, s in slopes.items():
    print(f"{name:<12}", np.round(s, 4))

Sigmoid's slope peaks at 0.25 and falls to 0.0025 at ±6. Tanh's peaks at 1 but is about 0 at ±6. ReLU's is 0 or 1, nothing in between.

• Saturation: sigmoid and tanh go flat at both ends. Picture a dimmer switch already turned all the way up: turning it further changes nothing. A neuron pushed to a large positive or negative input passes almost no signal back. Even at best, the sigmoid's slope is only 0.25. Multiply ten of those together and you get less than one millionth. Through deep layers the signal vanishes. A later lesson covers this.
• ReLU passes the signal through unchanged for any positive input. That's why deep networks became trainable once it caught on, around 2010–2012. It's also cheap to compute.
• Dead ReLUs: some ReLU neurons get a negative input for every example. Their slope is then 0 every time, so their weights never change again: no gradient reaches those weights, so they stop changing. Leaky ReLU and GELU keep a small slope to avoid this.

Choosing

Pick hidden-layer activations and output activations separately. They answer different questions.

• Hidden layers: ReLU for MLPs and CNNs. GELU or SwiGLU in transformers, the design behind LLMs. Don't use sigmoid there: it saturates.
• Output layer: this depends on the task, not on the hidden layers. Use sigmoid for one probability, such as "will this email get opened?". Use softmax for one-of-many classes (Phase 2). Use none at all (a linear output) to predict a number, such as next month's sales.