Index Ai Academy
How transformers think · 0:57

How transformers think

One prompt, “The cat sat”, through the whole model: tokens, embeddings, the attention matrix, the feed-forward layer, and the probability over the next word.

Transcript

One prompt, 'The cat sat', through a whole language model, every number shown. First, tokens: the text becomes three integers, 104, 324, 782. Each ID picks a row of eight learned numbers, its embedding; a positional wave marks its place. Inside each block, attention lets the rows read from each other: 'sat' takes 0.4 of its value from 'cat'. A feed-forward layer widens each row to 32 and back, 552 parameters, and the block repeats, layer after layer. Finally the LM head scores every token, and softmax turns scores into probabilities: 'on' gets 0.81. The chosen token joins the prompt, and the model runs again on four tokens. Start free at indexaiacademy.com.

Start the course

Keep watching

More videos.

All 38 videos, by what they're for.

Video 0:39

Tokenization: text to integer IDs

“The cat sat” is cut into three tokens and looked up as 104, 324 and 782: a tensor of shape (3,) with T = 3 and V = 1,000.

Video 0:42

Embeddings and positional waves

Each ID picks one row of the embedding matrix, a positional wave is added, and the result is the 3 × 8 matrix that enters the first block.

Video 0:41

Inside the attention matrix

Queries, keys and values from the same rows, then the 3 × 3 matrix of weights: “ sat” reads 0.40 of its value from “ cat”.

Video 0:38

The feed-forward layer

Each row is widened from 8 numbers to 32, bent by GELU and squeezed back: 552 parameters here, 4,722,432 per layer in GPT-2.

Video 0:37

The LM head and sampling

The last row becomes a score for every token, softmax makes them probabilities, temperature reshapes them, and the chosen token joins the prompt.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write