Glossary · Phase 5 · Transformers
Head size
How many numbers each query and key holds in one attention head. Scores are divided by its square root.
Try it now
With big scores one word takes everything. Dividing by √d_k keeps attention soft enough to learn.
Learn it properly
Scaled dot-product attention · The attention formula, line by line. About 30 minutes.
Other terms from the same lesson
- FlashAttention: A faster way to compute exactly the same attention on a GPU, by working in small tiles so it rarely touches slow memory.
- scaled dot-product attention: The standard attention recipe: compare queries with keys by dot product, divide by √(size), softmax, then mix the values.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.