Glossary · Phase 5 · Transformers
Online softmax
Computing a softmax one chunk at a time with a running maximum and a running total, so the full row of scores never has to be stored.
Try it now
Ten times the tokens means a hundred times the scores. That's why FlashAttention avoids storing them, and sliding windows avoid computing them.
Learn it properly
Long context · Efficient attention and million-token windows. About 35 minutes.
Other terms from the same lesson
- position interpolation: Stretching a model's context by squeezing the new, longer positions into the range it was trained on, then fine-tuning it briefly.
- sliding-window attention: Attention where each token only looks at a fixed number of recent tokens, so the cost grows in step with the length instead of with its square.
- state-space model: An alternative to attention that carries a fixed-size summary forward as it reads, so long texts cost no more per token. Mamba is the best known.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.