A language model reads words, but it computes with vectors. Its first step turns each word's id into a list of numbers, and its last step turns a list of numbers back into a score for every word in its vocabulary. Both steps are matrix products you already know, with one table at the heart of them: the embedding table. This lesson builds that table, picks rows from it two ways, and scores a next word.
What you'll be able to do
Write the one-hot vector for an id, counting places from 0
Show that a one-hot vector times a table picks out one row, so eᵢE = E[i]
Look up a batch of ids in an embedding table with E[ids], and name every shape on the way
Compute a score for every word in one product, h @ E.T, and read off the best next word
Quick check
Have a guess before you start. The row (0, 1, 0) times the 3 × 2 matrix [[1, 2], [3, 4], [5, 6]]: what comes out?
(3, 4)
(0, 0)
(9, 12)
(1, 2, 3)
Show the answer
(3, 4)
Each output is the row dotted with a column: 0 × 1 + 1 × 3 + 0 × 5 = 3, and 0 × 2 + 1 × 4 + 0 × 6 = 4. The single 1 picks out the matrix's second row, (3, 4), and the 0s wipe out the rest.
The math: a one-hot row picks a row
Purpose. Turn a word's id into a vector with a product, and see why reading a row is the same thing.
Lesson 28 gave every word an id: its place in a sorted vocabulary, counted from 0. But an id is a label, not an amount. "tent" being id 3 and "trail" id 4 doesn't make a trail one more than a tent. So a model never feeds the id in as a number. It first writes it as a one-hot vector: as many places as there are words, a 1 at place i and 0 everywhere else. In a vocabulary of 5 words, id 3 is
e3=(0,0,0,1,0)
In plain words: one switch per word, and only word 3's switch is on.
Quick check
From an earlier lesson: a row of 5 numbers times a 5 × 3 matrix. How many numbers come out?
3
8
15
5
Show the answer
3
(1 × 5) @ (5 × 3) gives (1 × 3): the inner 5s match and drop out, and the outer sizes are left. One number per column of the matrix.
The picture. Here is a small embedding table E: one row of 3 numbers for each of 5 words, fire, lake, rain, tent and trail. Watch e₃ times E. The flow first writes the one-hot vector, and the picture stands it beside the table as a column of ×0s and one ×1. Then the lit row slides down the table: each row is multiplied by its 0 or 1 and added to the running total, the row under the table. Watch what the total does.
Try it now
Then set id to 0 and press Use these numbers. The 1 moves to the top, and the answer is fire's row.
Symbols and formula. A vocabulary of V words, each given d numbers, makes a table E of shape V × d. Then for every id i:
eiE=Eiscores=hET
In plain words: a one-hot vector times the table is the table's row i. And a vector h times the table turned on its side gives one score per word: h dotted with each word's row.
The first rule runs words into numbers: the row E_i is word i's embedding, its vector. The second runs the other way. Say a model has worked out a vector h = (1, 0, 1) and wants to know which word fits it best. Eᵀ is 3 × 5, so h Eᵀ has 5 numbers, one per word. Fire's is h · (0.9, 0.1, −0.5) = 0.9 + 0 − 0.5 = 0.4, and trail's is 0.5 + 0 + 0.7 = 1.2. A big score means h points the same way as that word's row.
Quick check
A vocabulary has 4 words. Which is the one-hot vector for id 2?
(2, 2, 2, 2)
(0, 1, 0, 0)
(0, 0, 2, 0)
(0, 0, 1, 0)
Show the answer
(0, 0, 1, 0)
Places count from 0, so id 2 is the third place: (0, 0, 1, 0). Every other place is 0.
Quick check
h has 3 numbers and the table E is 5 × 3. What is h Eᵀ?
A 5 × 3 table
One number
5 numbers, one score per word
3 numbers, one per column
Show the answer
5 numbers, one score per word
Eᵀ is 3 × 5, and (3) @ (3 × 5) gives 5 numbers. Each one is h dotted with a word's row: a score for every word.
In your own words
In your own words: why does a one-hot vector times a table give exactly one of the table's rows?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: index the row, skip the product
Purpose. Check eᵢE = E[i] in NumPy, then look up a whole sentence at once.
np.eye(5) is the 5 × 5 identity from lesson 57. Each of its rows is a one-hot vector, so np.eye(5)[3] is e₃:
import numpy as np
E = np.array([[0.9, 0.1, -0.5],
[-0.2, 0.8, 0.4],
[-0.6, 0.9, 0.1],
[0.3, -0.4, 0.8],
[0.5, 0.2, 0.7]])
onehot = np.eye(5)[3]
print(onehot)
print(onehot @ E, E[3])
print(np.array_equal(onehot @ E, E[3]))
The product and the row are the same three numbers, and np.array_equal confirms it with True.
Code flow. Here is what onehot @ E does inside, written as a loop. Step through it and watch the boxes of out: each pass adds one row of E times its one-hot number, the flow's running total.
A whole sentence at once. Give NumPy a list or array of ids inside the brackets and it reads every one of those rows, in that order, into a new matrix. E[ids] is the same as stacking one-hot rows into a matrix and multiplying, np.eye(5)[ids] @ E:
Four ids give a 4 × 3 matrix, one row per id, and an id may repeat. The one-hot route builds a 4 × 5 matrix first: (4 × 5) @ (5 × 3) is 4 × 3, the same answer.
Quick check
What does this print?
import numpy as np
E = np.array([[1, 2], [3, 4], [5, 6]])
ids = np.array([2, 0, 2])
print(E[ids].shape, E[ids][0].tolist())
(3, 3) [5, 6]
(2, 3) [5, 6]
(3, 2) [1, 2]
(3, 2) [5, 6]
Show the answer
(3, 2) [5, 6]
Three ids bring back three rows of 2 numbers: shape (3, 2). The first id is 2, so the first row is E[2], [5, 6].
Worked example
Worked example: a score for every word
The math part scored h = (1, 0, 1) against two words by hand. Score all five at once, and name the best.
import numpy as np
E = np.array([[0.9, 0.1, -0.5], [-0.2, 0.8, 0.4], [-0.6, 0.9, 0.1], [0.3, -0.4, 0.8], [0.5, 0.2, 0.7]])
words = ["fire", "lake", "rain", "tent", "trail"]
h = np.array([1.0, 0.0, 1.0])
scores = h @ E.T
print(scores.shape, scores.round(2).tolist())
best = int(np.argmax(scores))
print(best, words[best])
In the lesson you can change this and run it.
E.T turns the 5 × 3 table on its side, 3 × 5, so h @ E.T is 5 scores: 0.4 for fire and 1.2 for trail, as worked by hand, then 0.2, −0.5 and 1.1 for lake, rain and tent. np.argmax finds the biggest at place 4, and words[4] is trail. The same table that turned ids into vectors now turns a vector into a score per word.
Why index, when the product works? The product multiplies every number in the table, almost all of them by 0. The index just copies one row. This cell does each 200 times on a table of 5,000 words with 64 numbers each, and times them with time.perf_counter(), a clock in seconds:
import time
import numpy as np
V, d = 5000, 64
E = np.ones((V, d))
onehot = np.zeros(V)
onehot[4321] = 1
t0 = time.perf_counter()
for r inrange(200):
a = onehot @ E
t1 = time.perf_counter()
for r inrange(200):
b = E[4321]
t2 = time.perf_counter()
print(V * d, d, np.array_equal(a, b))
print(round(t1 - t0, 4), round(t2 - t1, 4), t1 - t0 > t2 - t1)
Each product does 320,000 multiplications; each index copies 64 numbers. Your times depend on your machine, but the index wins by a wide margin, and True says so. Real models don't build the one-hot vector: they index. The one-hot product is how to think about the lookup, and it's what lets training treat the table like any other weight matrix.
A named mistake: an id past the end. A table of 12 rows has ids 0 to 11. Asking for row 12 stops with an error:
import numpy as np
E = np.zeros((12, 4))
print(E[11].shape, E[-1].shape)
print(E[12])
The last row, 11, prints its shape (4,); row 12 raises an IndexError. It usually means the ids were counted from 1, or a word was added to the vocabulary after the table was built. Watch the other end too: Python reads E[-1] as the last row, so a negative id slips through with no error at all, as the first print shows.
Quick check
A table E has 6 rows. Which of these lookups stops with an error?
E[5]
E[-1]
E[0]
E[6]
Show the answer
E[6]
Six rows have ids 0 to 5, so 6 is past the end: an IndexError.
Drill
One-hot vectors and safe lookups
Fix one_hot(i, V) so it returns the one-hot vector for id i in a vocabulary of V, and lookup(E, ids) so it returns the rows for a list of ids, or None if any id is outside 0 to len(E) - 1. There's one slip in each.
In the lesson, your code is checked as you write it.
In your own words
If E[i] gives the same answer, why learn the one-hot product at all?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: a language model's first and last step
Purpose. Follow the camp log through the table a language model starts and ends with.
The camp log's words as vectors. The trail club's camp log uses 12 words. Sorted, they get the ids 0 to 11, and each gets a row of 4 numbers in a 12 × 4 table. The numbers here are made up to have a meaning you can read: roughly, "a place or thing outdoors", "wet", "warm" and "a small joining word". A real model starts its table at random and learns the numbers in training. Here is one line of the log, looked up:
"rain is on the tent" becomes the ids 6, 2, 5, 7 and 8, and E[ids] turns them into a 5 × 4 matrix: one row per word, rain's first, (0.5, 0.9, −0.4, 0.1). Every layer after this works on that matrix, never on the words.
Quick check
Why doesn't the model feed in the id itself, say 8 for "tent", as its input number?
The id is too big a number for a model
Ids change every time the model runs
A single number can't be multiplied by a weight
An id is a label: 8 isn't more than 7 in any useful way
Show the answer
An id is a label: 8 isn't more than 7 in any useful way
The ids come from alphabetical order, so their sizes mean nothing: "tent" (8) isn't more of anything than "the" (7). A one-hot vector, and so a row of the table, gives every word its own numbers to learn.
Scores for the next word. After reading "the fire is", a model's layers hand back one vector, h. To guess the next word, it scores h against every word's row: h @ E.T, 12 numbers. Many models use the very same table at both ends, which is called tying the output layer to the embedding table. One table then turns words into vectors and vectors back into words:
"lit" scores highest, 1.1, just ahead of "fire" at 1.01: h is warm and dry, and so are their rows. "lake", "rain" and "wet" score lowest, −0.96, −0.96 and −0.94. These raw scores are the logits that softmax, in lesson 104, turns into chances.
In PyTorch. The same table, as the real tool holds it. nn.Embedding(12, 4) is a table of 12 rows of 4; it starts random, so the cell loads ours with copy_ under torch.no_grad(), as in lesson 50. emb(ids) is the lookup, emb.weight is the table itself, and F.one_hot(ids, 12) builds the one-hot rows, which .float() turns into decimals so they can multiply the table:
The same shapes as NumPy, 12 × 4 and 5 × 4, the one-hot product equal to the lookup, and the same next word, "lit". h @ emb.weight.T is the tied output layer: one table at both ends of the model.
Quick check
What does this print?
import torch
import torch.nn as nn
emb = nn.Embedding(1000, 32)
ids = torch.tensor([[4, 8, 15], [16, 23, 42]])
print(emb.weight.shape, emb(ids).shape)
torch.Size([1000, 32]) torch.Size([2, 3, 32])
torch.Size([32, 1000]) torch.Size([2, 3, 32])
torch.Size([1000, 32]) torch.Size([2, 32])
torch.Size([1000, 32]) torch.Size([6, 32])
Show the answer
torch.Size([1000, 32]) torch.Size([2, 3, 32])
The table is 1,000 rows of 32. The ids come as a batch of 2 sentences of 3 words, and every id is swapped for its row of 32, so (2, 3) becomes (2, 3, 32).
Drill
From words to a next word
Finish the camp-log pipeline. ids should hold the ids of "the fire is", X their rows, scores one score per vocabulary word for the vector h, and next_word the best-scoring word. The starter has three slips.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write