GELU activation function, explained

A smooth cousin of ReLU used in most transformers. It lets small negative values through a little instead of cutting them to 0.

What it means

GELU stands for Gaussian Error Linear Unit. It's an activation function, introduced by Dan Hendrycks and Kevin Gimpel in 2016 and used in models such as BERT and GPT-2. The formula is GELU(x) = x × Φ(x), where Φ(x) is the chance that a standard normal value falls below x. You can read that as a soft switch: the bigger x is, the more of it gets through.

GELU next to ReLU

import math

def relu(x):
    return max(0.0, x)

def gelu(x):
    # x times the chance that a standard normal value is below x
    return x * 0.5 * (1 + math.erf(x / math.sqrt(2)))

print(" input   ReLU   GELU")
for x in [-3, -2, -1, -0.5, 0, 0.5, 1, 2, 3]:
    print(f"{x:>6} {relu(x):>6.3f} {gelu(x):>6.3f}")
print("lowest GELU value:", round(min(gelu(i / 100) for i in range(-400, 0)), 3))

What it prints

 input   ReLU   GELU
    -3  0.000 -0.004
    -2  0.000 -0.046
    -1  0.000 -0.159
  -0.5  0.000 -0.154
     0  0.000  0.000
   0.5  0.500  0.346
     1  1.000  0.841
     2  2.000  1.954
     3  3.000  2.996
lowest GELU value: -0.17

For large positive inputs GELU acts like ReLU and passes the value through. For large negative inputs it goes to zero. The difference is in the middle: ReLU has a sharp corner at 0, while GELU is a smooth curve that dips slightly below zero, reaching about −0.17.

In practice

In PyTorch it's nn.GELU(). ReLU is simpler and cheaper, and it's still a fine default for smaller networks. GELU became popular in transformers, and newer language models often use relatives such as SwiGLU. To see one in a working model, read the transformer block entry.

GELU

What it means

GELU next to ReLU

In practice

Try it now

Learn it properly

Other terms from the same lesson

Never trained an AI?