One-hot vectors and lookup tables: embeddings
Why this matters
A language model reads words, but it computes with vectors. Its first step turns each word's id into a list of numbers, and its last step turns a list of numbers back into a score for every word in its vocabulary. Both steps are matrix products you already know, with one table at the heart of them: the embedding table. This lesson builds that table, picks rows from it two ways, and scores a next word.
- Write the one-hot vector for an id, counting places from 0
- Show that a one-hot vector times a table picks out one row, so eᵢE = E[i]
- Look up a batch of ids in an embedding table with E[ids], and name every shape on the way
- Compute a score for every word in one product, h @ E.T, and read off the best next word
The math: a one-hot row picks a row
Purpose. Turn a word's id into a vector with a product, and see why reading a row is the same thing.
Lesson 28 gave every word an id: its place in a sorted vocabulary, counted from 0. But an id is a label, not an amount. "tent" being id 3 and "trail" id 4 doesn't make a trail one more than a tent. So a model never feeds the id in as a number. It first writes it as a one-hot vector: as many places as there are words, a 1 at place i and 0 everywhere else. In a vocabulary of 5 words, id 3 is
In plain words: one switch per word, and only word 3's switch is on.