Glossary · Phase 4 · Deep learning
Saturated
When a sigmoid or tanh neuron's input is so large that its output is stuck near the flat end, so its slope (and learning) is almost zero.
Try it now
Good starting weights keep each layer's scaling near 1, so the signal survives.
Learn it properly
Weight initialization · Start weights at the right scale. About 25 minutes.
Other terms from the same lesson
- fan-in: How many inputs feed into one neuron. Good starting weights shrink as fan-in grows.
- He initialization: Starting weights sized for ReLU layers: random, with a spread of about √(2 / fan-in), so signals keep a steady size through many layers.
- symmetry breaking: Starting weights at different random values so neurons in a layer learn different things instead of staying identical copies.
- weight initialization: Choosing a network's starting weights. Small random numbers of the right size let signals and gradients flow from the first step.
- Xavier initialization: Starting weights sized for tanh or sigmoid layers, using the average of how many inputs and outputs a layer has.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.