Attention in transformers, explained interactively
Pick a word and see how it shares its attention across a sentence, then see what multiple heads and the causal mask add.
A word's meaning depends on the words around it. Attention is how a transformer handles that: each token gives every token it can see a share of its attention, the shares add up to 100%, and it blends in information according to those shares. In the three pieces below the scores are written by hand so the pattern is easy to read, and the percentages are a real softmax of those scores. To see attention measured from a trained model, try the transformer explainer.
Where the shares come from
Each token produces three vectors: a query (what it's looking for), a key (what it offers) and a value (what it passes along). A token's score for another token is its query times that token's key, divided by the square root of the vector size. Softmax turns the scores into shares that add up to 100%, and the token then takes that weighted average of the values.
Why more than one head
One set of shares can only follow one kind of pattern. Multi-head attention runs several side by side, so one head can follow meaning while another follows grammar.
Why a model can't look ahead
While a language model trains, it's predicting each next word, so it must not be able to see that word. The causal mask blocks every token from looking to its right.
Learn how this works, properly
- Attention: the intuition
- Queries, keys and values
- Scaled dot-product attention
- Multi-head attention
- Causal masking
A free account opens all 244 lessons, with Python that runs in your browser, saved progress and certificates.