Glossary · Phase 5 · Transformers
Attention
How a transformer lets each token look at the other tokens and pull in the information it needs.
Try it now
Pick "bank": it leans on "river", which is how it knows this isn't about money. These scores are illustrative, but the percentages are a real softmax.
Learn it properly
Attention: the intuition · Each token decides which other tokens matter to it. About 30 minutes.
Other terms from the same lesson
- attention weights: The shares of focus a token gives to each earlier token. They are all positive and add up to 1.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.