Glossary · Phase 5 · Transformers
Positional encoding
Extra numbers that tell the model where each token sits in the sequence.
The simplest fix works like seat numbers in a theater. Give every position its own vector, called a position embedding. Then add it to the token's embedding, number by number. The token's embedding says what the word is. The position vector says where it is. The sum carries both.
Try it now
Low-numbered waves spin fast and tell near neighbors apart. High-numbered ones turn slowly and mark long distances, like the hands of a clock.
Learn it properly
Positional encodings · Tell the model where each token sits. About 30 minutes.
Other terms from the same lesson
- RoPE: Rotary position embeddings: marking word order by turning each query and key by an angle that grows with position, so attention scores depend on how far apart two tokens are. Most modern open models use it.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.