Glossary · Phase 5 · Transformers

RoPE

Also written: rotary position embedding, rotary position embeddings, rotary embeddings

Rotary position embeddings: marking word order by turning each query and key by an angle that grows with position, so attention scores depend on how far apart two tokens are. Most modern open models use it.

Most modern LLMs, such as Llama, Mistral and Qwen, use rotary position embeddings, or RoPE. RoPE adds nothing to the embeddings. Instead it turns each query and key by an amount that depends on the position.
From the lesson Positional encodings

Try it now

Low-numbered waves spin fast and tell near neighbors apart. High-numbered ones turn slowly and mark long distances, like the hands of a clock.

Learn it properly

Positional encodings · Tell the model where each token sits. About 30 minutes.

Create a free account See what's in the lesson

Other terms from the same lesson

positional encoding: Extra numbers that tell the model where each token sits in the sequence.

Never trained an AI?

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.

Teach it to write