Glossary · Phase 5 · Transformers
Multi-head attention
Running several attention patterns side by side, so the model can track different relationships at once.
Try it now
Switch heads and pick a word. Each head spends its own attention on a different kind of pattern. These scores are illustrative.
Learn it properly
Multi-head attention · Several attention patterns in parallel. About 35 minutes.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.