Glossary · Phase 4 · Deep learning
Vanishing gradients
When the error signal shrinks to almost nothing (or blows up) as it passes back through many layers, so learning stalls.
Try it now
Sigmoid's slope is at most 0.25, so with 16 layers the blame shrinks to almost nothing. Try 1, then 1.2.
Learn it properly
Vanishing and exploding gradients · Why deep networks were hard to train. About 30 minutes.
Other terms from the same lesson
- gradient clipping: Capping how big the gradient can get before a step, so one bad batch can't blow up training.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.