Phase 5 · Transformers · The transformer

Positional encodings

Tell the model where each token sits.

About 30 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

"The dog bit the man" and "the man bit the dog" use the same five words. Only the order tells you who got hurt. Attention, as you've built it so far, can't see order at all. It treats the words like tiles in a bag. So every transformer adds a signal that says where each token sits. The way it does this matters. The method (learned, sine waves or rotary) is one of the main things that decides how long a text a model can handle.

What you'll be able to do

Show that attention on its own ignores word order
Add learned or sinusoidal position embeddings to token embeddings
Explain rotary position embeddings (RoPE) and why they encode relative distance
Connect position methods to how far a model's context can stretch

Attention ignores order

Picture a bag of Scrabble tiles. You can tell which letters are inside. You can't tell what word they spelled before someone tipped them in. Attention sees its input the same way.

Recall how it works. Each token makes a query, a key and a value. Its query is compared with every key to get scores. Softmax turns the scores into weights, and the weights blend the values. Nothing in those steps asks where a token sits. A token's output depends on which tokens are present, not on their order.

You can test this. Make five random token vectors and run attention on them. Then shuffle the tokens and run it again. If order doesn't matter, the new output is the old output, shuffled the same way.

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(5, 8))                       # 5 tokens
W_q, W_k, W_v = (rng.normal(size=(8, 8)) for _ in range(3))

def attention(X):
    Q, K, V = X @ W_q, X @ W_k, X @ W_v
    s = Q @ K.T / np.sqrt(8)
    w = np.exp(s - s.max(axis=1, keepdims=True))
    return (w / w.sum(axis=1, keepdims=True)) @ V

order = [3, 0, 4, 1, 2]
print("shuffled input gives the shuffled output:", np.allclose(attention(X[order]), attention(X)[order]))

It runs right here in your browser, with nothing to install.

The code makes five tokens with eight numbers each. order is the shuffle: the new first token is old token 3, then token 0, and so on. X[order] shuffles the input before attention. attention(X)[order] shuffles the output after attention. np.allclose checks that two arrays match, allowing for tiny rounding differences.

It prints True. Shuffling the input only shuffles the output. So "dog" gets the same output vector whether it comes first or last, and the model can't tell who bit whom.

Note: This test uses no causal mask. A causal mask, from the earlier lesson, does leak a little order, since each token sees a different number of earlier tokens. That hint is far too weak to rely on.

Quick check

Why does a transformer need positional information at all?

Self-attention treats its input as an unordered set, so without positions word order is invisible
To make the vocabulary larger
To speed up training
Because embeddings are random

Show the answer

Self-attention treats its input as an unordered set, so without positions word order is invisible

Shuffling the inputs only shuffles the outputs. Nothing in attention itself depends on order.

Fix one: a number for every seat

The simplest fix works like seat numbers in a theater. Give every position its own vector, called a position embedding. Then add it to the token's embedding, number by number. The token's embedding says what the word is. The position vector says where it is. The sum carries both.

Here's a tiny example with two numbers per vector. The numbers are made up.

The word "dog" has the embedding [0.3, 0.7].
Position 1 has the vector [0.1, −0.2].
Position 4 has the vector [−0.5, 0.0].

Add them up, number by number:

"dog" at position 1: [0.3 + 0.1, 0.7 − 0.2] = [0.4, 0.5]
"dog" at position 4: [0.3 − 0.5, 0.7 + 0.0] = [−0.2, 0.7]

Same word, two different vectors. Now attention can tell the two places apart. In a real model, each vector holds hundreds of numbers, but the adding works the same way.

GPT-2 learns its position vectors. It keeps one vector for each position, up to its limit of 1,024 tokens. Training adjusts them like any other weights. In PyTorch that's an nn.Embedding(1024, d_model). Here d_model is the number of values in each embedding.

There's a catch. GPT-2 has no vector for a 1,025th position. Learned position vectors only cover the lengths the model was built for.

Quick check

GPT-2 has learned position vectors for 1,024 positions. You want it to read 1,500 tokens at once. What's the problem?

It has no position vectors past 1,024, so the text must be cut down to fit
Nothing: it reuses position 0 for the extra tokens
It learns new position vectors on the spot
Nothing: it doesn't use positions

Show the answer

It has no position vectors past 1,024, so the text must be cut down to fit

Learned absolute positions stop at the length the model was built for. GPT-2 has no vector for the 1,025th token, so longer inputs have to be trimmed.

Fix two: waves at different speeds

The original transformer used fixed position vectors instead of learned ones. Nothing is trained. The vectors come from a formula built on two wave functions, sine and cosine (written sin and cos).

Each one takes an angle and gives back a number between −1 and 1. As the angle grows, both swing smoothly up and down and repeat, like a point on a turning wheel. Python has them as math.sin and math.cos. They measure angles in radians: a full turn is about 6.283 radians.

Now think of a clock. It shows the time with hands that turn at different speeds. The second hand turns fast, so it tells apart moments one second apart. The hour hand turns slowly, so it tells morning from evening. Read all the hands together and every moment of the day has its own pattern.

Sine-wave positions work the same way. Each pair of values in the vector is one hand. The first pairs turn fast. Later pairs turn slowly. Here are the symbols the formula uses:

$p$ is the position: 0, 1, 2 and so on.
$d$ is the number of values in each position vector. (Each value is also called a dimension.)
$i$ says which pair you're filling: 0 for the first pair, 1 for the second, and so on.

Pair number $i$ fills two columns, numbered $2i$ and $2i+1$ . PE stands for "position encoding".

PE(p, 2i) = \sin\!\left(\frac{p}{10000^{2i/d}}\right) \qquad PE(p, 2i+1) = \cos\!\left(\frac{p}{10000^{2i/d}}\right)

In plain words: each pair of columns holds the sine and the cosine of one angle. The angle is the position divided by some number. That number is 1 for the first pair and grows toward 10,000 for the last pair. So the first pair turns fast and the last pairs turn very slowly.

Try it now

Low-numbered waves spin fast and tell near neighbors apart. High-numbered ones turn slowly and mark long distances, like the hands of a clock.

Worked example

Worked example: four values per position

Take vectors of four values, so $d = 4$ and there are two pairs. For pair 0, the angle is divided by $10000^{0} = 1$ . For pair 1, it's divided by $10000^{2/4} = 100$ . So position 1 gets sin(1), cos(1), sin(0.01) and cos(0.01). Work out the first three positions.

import math

d = 4                                   # numbers per position
for p in [0, 1, 2]:                     # the first three positions
    row = []
    for i in range(d // 2):             # one pair of columns for each i
        angle = p / 10000 ** (2 * i / d)
        row += [math.sin(angle), math.cos(angle)]
    print(p, [round(v, 3) for v in row])

In the lesson you can change this and run it.

Position 0 is [0, 1, 0, 1], because sin 0 is 0 and cos 0 is 1. Position 1 is [0.841, 0.54, 0.01, 1.0]. Position 2 is [0.909, −0.416, 0.02, 1.0].

The first pair changed a lot from one position to the next. That's the fast hand. The second pair barely moved. It's the slow hand, and it takes dozens of positions to change much. Read together, the pairs still give every position its own pattern.

Quick check

At position 0, every angle in the formula is 0. Predict what this prints for one pair.

import math
print(round(math.sin(0), 1), round(math.cos(0), 1))

1.0 0.0
0.0 1.0
1.0 1.0
0.0 0.0

Show the answer

0.0 1.0

sin 0 is 0.0 and cos 0 is 1.0. Every pair at position 0 has an angle of 0, so position 0 is always [0, 1, 0, 1, ...].

The full table

Now build the whole table: 64 positions with 32 values each. Then compare position 20 with every other position.

import numpy as np
import matplotlib.pyplot as plt

def sinusoidal(T, d):
    pos = np.arange(T)[:, None]
    freq = 1 / 10000 ** (np.arange(0, d, 2) / d)
    pe = np.zeros((T, d))
    pe[:, 0::2] = np.sin(pos * freq)
    pe[:, 1::2] = np.cos(pos * freq)
    return pe

pe = sinusoidal(64, 32)
sim = pe @ pe[20]                                  # how similar is each position to position 20?
print("most similar positions to 20:", np.argsort(-sim)[:5].tolist())
plt.imshow(pe.T, aspect="auto", cmap="coolwarm")
plt.xlabel("position")
plt.ylabel("dimension")

Here's what each piece does.

pos is a column of positions, 0 to 63.
freq holds one speed for each pair. It's 1 divided by the number from the formula.
pe[:, 0::2] means every even column, and pe[:, 1::2] means every odd column.
pe @ pe[20] takes the dot product of every position's vector with position 20's. A bigger dot product means more alike.
np.argsort(-sim)[:5] lists the five positions with the biggest dot products.

The list starts with 20, since position 20 matches itself best. Next come its nearest neighbors, 19 and 21, then 18 and 22. (Each pair of neighbors ties exactly, so the two can swap places in the printout.) Nearby positions get similar vectors, which gives the model a sense of distance.

The picture has one column per position and one row per value. The top rows stripe quickly: those are the fast hands. Lower down, the stripes stretch out as the hands slow.

Learned vectors and sine waves are both absolute schemes. Each vector says "I'm at seat 20". Their weak spot is length. The sine formula can make a vector for any position. But positions past the longest training text are still unfamiliar to the model.

Go deeper Why sine and cosine come in pairs

Take one pair, with its own speed. Move forward a fixed number of positions, say 3. The pair's (sin, cos) values turn by the same angle, whatever position you started from. So "3 positions later" always looks like the same small turn of each pair. The original authors hoped this would make relative positions easy for the model to learn. The idea of position as a turn is the seed of RoPE, next.

One more detail: each pair's sin and cos, squared and added, always make 1. So every position's dot product with itself is 16 here, one for each of the 16 pairs.

Quick check

Why are learned vectors and sine waves both called absolute schemes?

They only use positive numbers
Each vector says which seat a token is in, not how far it is from another token
They never change during training
They work only for short texts

Show the answer

Each vector says which seat a token is in, not how far it is from another token

Attention often cares about distance, and absolute schemes make the model work that out for itself. RoPE, up next, builds it in.

Fix three: turn instead of add (RoPE)

Most modern LLMs, such as Llama, Mistral and Qwen, use rotary position embeddings, or RoPE. RoPE adds nothing to the embeddings. Instead it turns each query and key by an amount that depends on the position.

Here's the picture. Split a query's numbers into pairs. Each pair is a little arrow on a flat page. The first number says how far right it points, the second how far up. RoPE turns each arrow around its tail. A token at position 1 gets turned a little. Position 2 gets turned twice as much, and so on. Each pair has its own speed, the same speeds as the sine waves.

Call the pair's two numbers $(x, y)$ . To turn the pair by an angle $\theta$ (theta), use this rule:

(x, y) \;\to\; (x\cos\theta - y\sin\theta,\; x\sin\theta + y\cos\theta)

In plain words: to turn an arrow, mix its two numbers using the cosine and sine of the angle. The arrow keeps its length. Only its direction changes.

Try it on the arrow (2, 1) with a quarter turn. A quarter turn is about 1.571 radians. Its cosine is 0 and its sine is 1. So the new arrow is (2 × 0 − 1 × 1, 2 × 1 + 1 × 0) = (−1, 2).

import math
x, y = 2.0, 1.0
angle = math.pi / 2                     # a quarter turn
new_x = x * math.cos(angle) - y * math.sin(angle)
new_y = x * math.sin(angle) + y * math.cos(angle)
print(round(new_x, 3), round(new_y, 3))
print(round(math.hypot(x, y), 3), round(math.hypot(new_x, new_y), 3))

It prints -1.0 2.0, then 2.236 2.236. math.hypot gives an arrow's length, and the length didn't change.

Why turning helps

Go back to the clock. Look at two of its hands and notice the angle between them. Now spin the whole clock. Both hands move, but the angle between them stays the same.

An attention score is a dot product. A dot product depends only on the two arrows' lengths and the angle between them. RoPE turns the query by its position and the key by its position. So the angle between them shifts by an amount set by the gap between the two positions. A query at 5 with a key at 2 gets the same shift as a query at 105 with a key at 102.

So scores depend on the relative distance between tokens. That's usually what matters: "the word two before me", not "the word at seat 4,017". The same pattern works anywhere in the text. The code below checks it with 16 values, so 8 pairs.

import numpy as np

def rotate(x, pos, base=10000.0):
    d = len(x)
    theta = pos / base ** (np.arange(0, d, 2) / d)          # one angle per pair of dimensions
    x1, x2 = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = x1 * np.cos(theta) - x2 * np.sin(theta)    # rotate each pair by its angle
    out[1::2] = x1 * np.sin(theta) + x2 * np.cos(theta)
    return out

rng = np.random.default_rng(1)
q, k = rng.normal(size=16), rng.normal(size=16)
for m, n in [(5, 2), (105, 102), (1005, 1002), (7, 2)]:
    print(f"query at {m:>4}, key at {n:>4} (distance {m - n}): score {rotate(q, m) @ rotate(k, n):.4f}")

rotate turns every pair of a vector at once, using the formula above. theta holds one angle per pair: the position divided by that pair's number. The loop scores the same query and key at four placements. The first three all have distance 3, and they print the same score. The last has distance 5 and prints a different one. Same distance, same score, wherever the pair sits.

Go deeper Why the score only sees the gap

For one pair, the dot product of two arrows equals their two lengths multiplied together, times the cosine of the angle between them. Turning doesn't change lengths. Say the query sits at position $m$ . Its pair turns by $m$ steps of that pair's angle.

Say the key sits at position $n$ . Its pair turns by $n$ steps. The angle between them changes by $m - n$ steps, and nothing else changes. So that pair's share of the score depends only on $m - n$ . The full score adds up the shares of all the pairs. Each share depends only on the gap, so the total does too.

Quick check

A RoPE model scores a query at position 40 against a key at position 37. Using the same query and key vectors, which placement gives exactly the same score?

Query at 37, key at 40
Query at 80, key at 74
Query at 40, key at 40
Query at 1,003, key at 1,000

Show the answer

Query at 1,003, key at 1,000

Only the gap counts. 40 − 37 = 3, and 1,003 − 1,000 = 3. The other placements have gaps of 0, −3 and 6.

Stretching the context

Because RoPE works with gaps, it's easier to stretch. A RoPE model can often be extended to longer texts than it was trained on. Tricks such as position interpolation or YaRN rescale the turning angles, and some extra training finishes the job.

Other schemes exist too. ALiBi, for example, uses no position vectors at all. It adds a penalty to each attention score that grows with the distance between the two tokens.

method	what it does	used by
Learned	one trained vector per position, added at the input	GPT-2
Sine waves	fixed sine and cosine vectors, added at the input	the original transformer
RoPE	queries and keys turned by position, inside attention	Llama, Mistral, Qwen

Where it goes in the code

x = token_embedding(idx) + position_embedding(torch.arange(T))   # GPT-2 style: added once at the input
# RoPE style: no addition at the input; q and k are rotated inside every attention layer

In GPT-2 style, the position vectors are added once, at the input. idx holds the token ids, and torch.arange(T) gives the positions 0 to T − 1 for a text of T tokens. In RoPE style, nothing is added at the input. Instead, every attention layer turns the queries and keys right before it scores them.

Practice what you just read

6 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
4 review cards for spaced repetition

Create a free account

Try a quiz question

How does GPT-2 tell the model where each token is?

It adds a learned position embedding (one vector per position, up to 1,024) to each token embedding
It rotates each token's queries and keys by an angle set by the token's position in the text
It sorts the tokens by position first, so that attention always reads them in the right order
It doesn't use positions at all, since attention can work out the order from the words alone

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Multi-head attention