Glossary · Phase 4 · Deep learning
Gradient clipping
Capping how big the gradient can get before a step, so one bad batch can't blow up training.
Fixing the network's design helps. The standard safety net on top is gradient clipping. Think of a speed limiter on a car. You can still steer anywhere, but you can't go faster than the limit.
Try it now
Sigmoid's slope is at most 0.25, so with 16 layers the blame shrinks to almost nothing. Try 1, then 1.2.
Learn it properly
Vanishing and exploding gradients · Why deep networks were hard to train. About 30 minutes.
Other terms from the same lesson
- vanishing gradients: When the error signal shrinks to almost nothing (or blows up) as it passes back through many layers, so learning stalls.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.