Glossary · Phase 5 · Transformers
Scaled dot-product attention
The standard attention recipe: compare queries with keys by dot product, divide by √(size), softmax, then mix the values.
Try it now
With big scores one word takes everything. Dividing by √d_k keeps attention soft enough to learn.
Learn it properly
Scaled dot-product attention · The attention formula, line by line. About 30 minutes.
Other terms from the same lesson
- FlashAttention: A faster way to compute exactly the same attention on a GPU, by working in small tiles so it rarely touches slow memory.
- head size: How many numbers each query and key holds in one attention head. Scores are divided by its square root.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.