Glossary · Phase 4 · Deep learning
Residual stream
The running total that flows through a transformer: each layer reads it and adds its result back in, instead of replacing it.
Try it now
Thirty plain layers lose almost all the gradient. With skip connections, it arrives intact.
Learn it properly
Residual connections · Skip connections that let gradients flow. About 30 minutes.
Other terms from the same lesson
- residual block: A layer that adds its result to its own input (x + f(x)), so the input always has a clean path through.
- residual connection: A shortcut that adds a layer's input to its output, so information and gradients flow easily through deep networks.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.