Glossary · Phase 4 · Deep learning
Symmetry breaking
Starting weights at different random values so neurons in a layer learn different things instead of staying identical copies.
Try it now
Good starting weights keep each layer's scaling near 1, so the signal survives.
Learn it properly
Weight initialization · Start weights at the right scale. About 25 minutes.
Other terms from the same lesson
- fan-in: How many inputs feed into one neuron. Good starting weights shrink as fan-in grows.
- He initialization: Starting weights sized for ReLU layers: random, with a spread of about √(2 / fan-in), so signals keep a steady size through many layers.
- saturated: When a sigmoid or tanh neuron's input is so large that its output is stuck near the flat end, so its slope (and learning) is almost zero.
- weight initialization: Choosing a network's starting weights. Small random numbers of the right size let signals and gradients flow from the first step.
- Xavier initialization: Starting weights sized for tanh or sigmoid layers, using the average of how many inputs and outputs a layer has.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.