Index Ai Academy
Foundation · Foundation mathVectors and matrices

One-hot vectors and lookup tables: embeddings

About 35 minLooking ahead: some prerequisites aren't done yet

Why this matters

A language model reads words, but it computes with vectors. Its first step turns each word's id into a list of numbers, and its last step turns a list of numbers back into a score for every word in its vocabulary. Both steps are matrix products you already know, with one table at the heart of them: the embedding table. This lesson builds that table, picks rows from it two ways, and scores a next word.

  • Write the one-hot vector for an id, counting places from 0
  • Show that a one-hot vector times a table picks out one row, so eᵢE = E[i]
  • Look up a batch of ids in an embedding table with E[ids], and name every shape on the way
  • Compute a score for every word in one product, h @ E.T, and read off the best next word
Quick check
Have a guess before you start. The row (0, 1, 0) times the 3 × 2 matrix [[1, 2], [3, 4], [5, 6]]: what comes out?

The math: a one-hot row picks a row

Purpose. Turn a word's id into a vector with a product, and see why reading a row is the same thing.

Lesson 28 gave every word an id: its place in a sorted vocabulary, counted from 0. But an id is a label, not an amount. "tent" being id 3 and "trail" id 4 doesn't make a trail one more than a tent. So a model never feeds the id in as a number. It first writes it as a one-hot vector: as many places as there are words, a 1 at place i and 0 everywhere else. In a vocabulary of 5 words, id 3 is

e3=(0,0,0,1,0)\mathbf{e}_3 = (0, 0, 0, 1, 0)

In plain words: one switch per word, and only word 3's switch is on.