Glossary · Phase 4 · Deep learning
Activation function
The simple function a neuron applies after its weighted sum. Without it, stacking layers would add nothing.
Try it now
Drag along ReLU and watch its slope: exactly 1 for positive inputs and 0 for negative ones. Sigmoid and tanh flatten out at both ends.
Learn it properly
Activation functions · ReLU, sigmoid, GELU, and why nonlinearity matters. About 25 minutes.
Other terms from the same lesson
- GELU: A smooth cousin of ReLU used in most transformers. It lets small negative values through a little instead of cutting them to 0.
- ReLU: The most common activation function: keep positive numbers, turn negative ones into 0.
- tanh: An S-shaped activation like sigmoid, but its outputs run from -1 to 1 instead of 0 to 1.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.