The tokenizer left us with three integers, 104, 324 and 782. An integer can't be reasoned with: 324 isn't "more" than 104, and the model can't multiply "cat" by a weight. So the first layer of every transformer swaps each integer for a row of numbers, its embedding, and then stamps each row with where in the sentence it sits. The result is the matrix X that the attention, feed-forward and LM-head lessons of this track start from.
This lesson follows the three IDs into that matrix, one number at a time, with a model small enough to print: 8 numbers per token instead of GPT-2's 768.
What you'll be able to do
Look up a token's embedding as one row of the V × d_model matrix, and give the shape of the result
Compute a sine-cosine positional encoding for any position and column from its formula
Add the positional encoding to the embeddings and explain what the sum carries into the model
Count the parameters of an embedding layer, and say why its rows change during training
Each ID picks one row of the embedding matrix, a positional wave is added, and the boards show the three rows of E, the positional matrix and the sum for the " cat" row. Transcript
Quick check
An embedding lookup is NumPy fancy indexing: rows picked by number. A guess: what does this print?
import numpy as np
E = np.arange(20).reshape(5, 4) # a tiny table: 5 tokens, 4 numbers eachprint(E[[3, 1]])
[[3 3 3 3]
[1 1 1 1]]
[[12 13 14 15]
[ 4 5 6 7]]
[[ 4 5 6 7]
[12 13 14 15]]
[3 1]
Show the answer
[[12 13 14 15]
[ 4 5 6 7]]
A list of row numbers picks those rows, in the order given: row 3 is 12 to 15, row 1 is 4 to 7. That is all an embedding lookup does, with V rows instead of 5.
The idea: an ID becomes a row, and a position is stamped on it
The model keeps a table called the embedding matrix, E, with one row per vocabulary entry and dmodel numbers in each row. Token ID i is simply row i: the lookup xi=E[i]∈Rdmodel. For a whole prompt of T tokens, stacking the rows gives a matrix:
Xemb=E[x]∈RT×dmodel,E∈RV×dmodel
Our small model has V=1000 and dmodel=8, so E is 1,000 rows of 8 numbers (8,000 parameters) and "The cat sat" becomes a 3 × 8 matrix. GPT-2 has V=50,257 and dmodel=768: a table of 38,597,376 numbers, about 31% of the whole model.
Those numbers are learned. At the start of training they're random; training nudges row 324 until it's a useful summary of how " cat" behaves, which is why similar words end up with similar rows. Nothing about the lookup itself is clever. The row number is the only thing the token contributes.
Worked example
Look up the three rows
A seeded random matrix stands in for a trained one. The same code, with the same seed, prints the same numbers in every lesson of this section.
It runs right here in your browser, with nothing to install.
Three row numbers in, a (3, 8) matrix out: the first row is E[104], the second E[324], the third E[782]. Row 324 starts 0.23, 0.36, -0.80. If " cat" appeared twice in a prompt, both of its rows would be this same vector: at this point the model can't yet tell the two apart.
Quick check
A prompt uses the token " cat" twice, at positions 2 and 6. Right after the embedding lookup, how do its two rows compare?
The second row is all zeros until attention runs
They differ, because the lookup also reads the neighbors
They're identical: both are row 324 of E
The second is scaled by 6/2 = 3
Show the answer
They're identical: both are row 324 of E
The lookup only knows the ID. Same ID, same row. Telling the two apart is the job of the position signal added next.
Try it now
The embeddings of "The", " cat" and " sat" (top), the positional matrix (middle), and their sum (bottom): the same 3 × 8 shape before and after. Columns 6 and 7 barely change, because their waves are thousands of positions long.
Positions: the same word in two places
Here is the problem the position signal solves. The attention layer you'll meet in Inside the attention matrix mixes rows with weighted sums, and a weighted sum doesn't care about order: "the cat sat on the mat" and "the mat sat on the cat" would give the same rows the same treatment. So the model needs each row to carry its position as well as its token.
The original transformer did this with fixed waves. For position pos (0, 1, 2, …) and column pair i (0, 1, 2, …), the matrix PE∈RT×dmodel is:
Read it as: even columns hold a sine, odd columns the matching cosine, and each pair has its own speed. Pair 0 divides by 100000=1, so it's a wave with wavelength 2π≈6.3 positions. Pair 1 divides by 100002/8=10: wavelength 62.8. Pair 2: 628.3. Pair 3: 6,283.2. Fast waves on the left tell nearby positions apart; slow waves on the right separate positions that are far apart. Every position gets a pattern no other position has, and the same formula works for any T without new parameters.
Worked example
The positional matrix for three positions
The formula, with NumPy doing all four pairs at once. pos is a column of positions and i a row of even column indices, so angle is 3 × 4.
import numpy as np
np.set_printoptions(precision=2, suppress=True)
T, d_model = 3, 8
pos = np.arange(T)[:, None] # (3, 1): the positions 0, 1, 2
i = np.arange(0, d_model, 2)[None, :] # (1, 4): the even columns 0, 2, 4, 6
angle = pos / 10000 ** (i / d_model) # (3, 4): one angle per position and pair
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle) # even columns
PE[:, 1::2] = np.cos(angle) # odd columnsprint(PE)
print("wavelength of each pair:", (2 * np.pi * 10000 ** (i / d_model)).round(1))
In the lesson you can change this and run it.
Position 0 is 0, 1, 0, 1, …: sin 0 and cos 0 in every pair. Position 1, pair 0 is sin 1 = 0.84 and cos 1 = 0.54; pair 1 is sin 0.1 = 0.10 and cos 0.1 = 1.00. By pair 3 the angle is 0.001 and the columns read 0.00 and 1.00 for all three positions: that wave needs thousands of positions to turn.
Quick check
Two entries of the matrix above, computed by hand. What does this print?
sin(1/10) is position 1's entry in column 2 (pair 1, exponent 2/8 = 0.25), and cos(2) is position 2's entry in column 1 (pair 0). Both appear in the printed PE: 0.1 and -0.42.
In your own words
What does the embedding matrix know about a token, and what does it not know?
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: the input to the first block
The formula for position is a drill away from code. Write it, draw it, add it to the rows, then see the same lookup as a PyTorch layer and read it on the trained tiny GPT.
Drill
Write the positional encoding
Write positional_encoding(T, d_model) that returns the T × d_model matrix from the formula: sines in the even columns, cosines in the odd ones, the pair i dividing the position by 10000 ** (2i / d_model).
In the lesson, your code is checked as you write it.
Worked example
See the waves
The same formula for 64 positions and 32 dimensions, drawn as a heatmap with dimensions down the side. Red is +1, blue is −1.
import numpy as np
import matplotlib.pyplot as plt
T, d_model = 64, 32
pos = np.arange(T)[:, None]
i = np.arange(0, d_model, 2)[None, :]
angle = pos / 10000 ** (i / d_model)
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle)
PE[:, 1::2] = np.cos(angle)
fig, ax = plt.subplots(figsize=(6.5, 3.6))
# one row per dimension, positions left to right
im = ax.imshow(PE.T, aspect="auto", cmap="RdBu")
ax.set_xlabel("position")
ax.set_ylabel("dimension")
ax.set_title("Positional encoding: fast waves at the top, slow waves at the bottom")
fig.colorbar(im, ax=ax)
In the lesson you can change this and run it.
The top rows flip between red and blue every few positions; the bottom rows change so slowly that 64 positions don't finish a single swing. Read down any one column and you get that position's signature. Read across and you see why positions 10 and 11 look alike while 10 and 50 don't: the fast rows have moved on, the slow rows have barely changed.
Add, don't append
The two matrices have the same shape, so they're added:
X=Xemb+PE∈RT×dmodel
Adding keeps dmodel fixed, so every matrix after this point keeps the shape it was designed for. It also works: the fast columns on the left carry mostly position, the slow columns on the right are nearly constant and carry mostly token, and the layers in between learn to read both out of the sum. Row 2 of X, for " cat", is E[324] plus the wave values for position 1: 0.23+0.84=1.07, 0.36+0.54=0.90, −0.80+0.10=−0.70, and so on.
Not every model uses the sine-cosine waves. GPT-2 learns its position rows instead (a second table, 1,024 positions by 768), and most recent models rotate the queries and keys by position inside the attention layer (RoPE). The purpose is the same every time: give each row a usable sense of where it is. X is the matrix that enters the first transformer block, the X in the Q=XWQ of Inside the attention matrix.
Quick check
Why is the positional signal added to the embedding rather than appended as extra columns?
Appended columns would make the sines negative and break the wave pattern
Addition is the only operation NumPy supports on matrices of that size
Adding keeps d_model fixed, and the layers learn to read both from the sum
Appending would lose the position information held inside the embedding
Show the answer
Adding keeps d_model fixed, and the layers learn to read both from the sum
Appending would work in principle but changes the width that every later weight matrix depends on. Adding costs nothing in shape, and the fast and slow columns carry position and token in a way the layers learn to separate.
Worked example
The input to the first block
Lookup, positional matrix, sum. The printed X is the starting point of the attention lesson.
import numpy as np
np.set_printoptions(precision=2, suppress=True)
rng = np.random.default_rng(42)
V, d_model, T = 1000, 8, 3
E = rng.normal(0.0, 0.5, size=(V, d_model)).round(2)
ids = [104, 324, 782]
pos = np.arange(T)[:, None]
i = np.arange(0, d_model, 2)[None, :]
angle = pos / 10000 ** (i / d_model)
PE = np.zeros((T, d_model))
PE[:, 0::2] = np.sin(angle)
PE[:, 1::2] = np.cos(angle)
X = E[ids] + PE
print(X)
print(X.shape)
In the lesson you can change this and run it.
Row 0 is "The" at position 0: its embedding plus 0, 1, 0, 1, …, so -0.56 stays and -0.19 becomes 0.81. Row 1 is " cat" at position 1: 1.07, 0.90, -0.70, …. The shape is still (3, 8): three tokens, eight numbers each, now with their positions inside the numbers.
Drill
Build the model's input
Write model_input(E, ids): look up the rows of E for ids, add positional_encoding(len(ids), E.shape[1]) (given), and return the sum.
import numpy as np
defpositional_encoding(T, d_model):
pos = np.arange(T)[:, None]
i = np.arange(0, d_model, 2)[None, :]
angle = pos / 10000 ** (i / d_model)
pe = np.zeros((T, d_model))
pe[:, 0::2] = np.sin(angle)
pe[:, 1::2] = np.cos(angle)
return pe
defmodel_input(E, ids):
return E[:len(ids)]
E = np.arange(40, dtype=float).reshape(10, 4) / 10print(model_input(E, [7, 2]).round(2))
In the lesson, your code is checked as you write it.
Worked example
The same lookup in PyTorch
nn.Embedding is the matrix E with the lookup built in. This runs on the course's teaching version of PyTorch: the same code as the real library, on the CPU and slower, and its random starting values differ from a real install's.
import torch
torch.manual_seed(0)
emb = torch.nn.Embedding(1000, 8) # V = 1000 rows, d_model = 8
ids = torch.tensor([104, 324, 782])
x = emb(ids)
print(tuple(emb.weight.shape), "->", tuple(x.shape))
print(sum(p.numel() for p in emb.parameters()), "parameters")
print(torch.equal(x[1], emb.weight[324])) # the lookup really is "row 324"
In the lesson you can change this and run it.
The weight is the (1000, 8) table and the output is (3, 8): 8,000 parameters, all of them trained. The last line confirms that the layer does nothing but pick row 324. GPT-2's version of this table has 50,257 × 768 = 38,597,376 parameters, learned the same way.
Quick check
You double the vocabulary of a model from 1,000 tokens to 2,000 and keep d_model = 8. What happens to the embedding layer?
Nothing changes, because the vocabulary size isn't part of this layer
The output for a 3-token prompt becomes (6, 8), two rows per token
It gains 1,000 rows: 16,000 parameters, and the output is still (3, 8)
Each row doubles in length to 16 numbers, and the output becomes (3, 16)
Show the answer
It gains 1,000 rows: 16,000 parameters, and the output is still (3, 8)
The table has one row per vocabulary entry, so V sets the number of rows and d_model the length of each. The output shape depends only on T and d_model.
Quick check
GPT-2's tokenizer has V = 50,257 tokens and its d_model is 768. How many numbers are in its token embedding matrix?
50,257 + 768 = 51,025
1,024 × 768, one row per position
768, one per dimension
50,257 × 768 = 38,597,376
Show the answer
50,257 × 768 = 38,597,376
One row per vocabulary entry, 768 numbers in each: 50,257 × 768 = 38,597,376. That's about 31% of GPT-2's 124 million parameters, spent on the lookup table alone.
The trained tiny GPT's tables
The tiny GPT has the same two tables, with its own sizes: model.tok holds one row of 32 numbers for each of its 195 tokens, and model.pos one row for each of its 64 positions. Its position rows are learned, not waves, as in GPT-2. The cell adds them for "Describe the Lark headlamp ." and confirms that the lookup is row selection.
Worked example
Token rows plus position rows on the trained model
The model's own tok and pos layers do the lookup. The sum is the matrix that enters block 0.
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
ids = tok.encode("Describe the Lark headlamp .")
idx = torch.tensor([ids])
print(tuple(model.tok.weight.shape), tuple(model.pos.weight.shape))
e = model.tok(idx)
p = model.pos(torch.arange(len(ids)))
x = e + p
print(tuple(e.shape), tuple(p.shape), tuple(x.shape))
print(torch.equal(e[0, 1], model.tok.weight[tok.token_id("the")]))
share = model.tok.weight.numel() / sum(q.numel() for q in model.parameters())
print(round(100 * share), "% of the parameters are the token table")
In the lesson you can change this and run it.
The token table is (195, 32) and the position table (64, 32). Five ids give e and p of shape (5, 32) and a sum x of the same shape: five rows, one per token. The check confirms that "the" is row 175 of the table. The token table holds 6,240 of the model's 40,195 parameters, 16%. GPT-2 spends 31% on the same table.
In your own words
Describe the matrix X that leaves this layer: its shape, what each row holds, and which two things are mixed into every row.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: the tables in GPT-2, and where this bites
GPT-2 keeps the same two tables, named wte for tokens and wpe for positions. The map shows where this step sits: after the ids, before any block.
Where this shows up in an engineer's week
On the job: adding tokens to a model, such as new special tokens for a chat format, means growing the embedding table with resize_token_embeddings. The new rows start random and mean nothing until fine-tuning trains them. A model also cannot read a longer prompt than its position table has rows for, unless it uses a scheme such as RoPE that has no table.
Embedding tables are also why memory is tight with a big vocabulary: they cost V times d_model numbers before any block runs.
03A bigram modelThe simplest language model, built from counts.35 min
04Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
05Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
14The GPT architectureA stack of decoder blocks that predicts the next token.40 min
15Build a tiny GPTWrite and train a small GPT from scratch.45 min
16From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
17Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
27The block in today's open modelsOptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
28Long contextEfficient attention and million-token windows.35 min
29Mixture of expertsRoute each token to a few specialist sub-networks.35 min
30Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
31Jacobians: derivatives with shapesOptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
32Backprop through a linear layerOptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
33The softmax Jacobian and p − yOptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
34Backprop through attentionOptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
36Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
37Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
38Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
39Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
40Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
41Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
42Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
43Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
44Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
45Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
46Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
47The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write