Index Ai Academy
How transformers think · 0:38

The feed-forward layer

Each row is widened from 8 numbers to 32, bent by GELU and squeezed back: 552 parameters here, 4,722,432 per layer in GPT-2.

Transcript

After attention, each row passes through a small two-layer network alone: widened from 8 numbers to 32, bent by GELU, projected back to 8. 552 parameters here, 4.7 million per layer in GPT-2. GELU keeps a little of the negatives: at −1 it gives −0.16, where ReLU gives 0. Each hidden unit is a memory slot: for 'sat', three of the 32 make 93% of the output. Start free at indexaiacademy.com.

Open the lesson

Keep watching

More videos.

All 38 videos, by what they're for.

Video 0:57

How transformers think

One prompt, “The cat sat”, through the whole model: tokens, embeddings, the attention matrix, the feed-forward layer, and the probability over the next word.

Video 0:39

Tokenization: text to integer IDs

“The cat sat” is cut into three tokens and looked up as 104, 324 and 782: a tensor of shape (3,) with T = 3 and V = 1,000.

Video 0:42

Embeddings and positional waves

Each ID picks one row of the embedding matrix, a positional wave is added, and the result is the 3 × 8 matrix that enters the first block.

Video 0:41

Inside the attention matrix

Queries, keys and values from the same rows, then the 3 × 3 matrix of weights: “ sat” reads 0.40 of its value from “ cat”.

Video 0:37

The LM head and sampling

The last row becomes a score for every token, softmax makes them probabilities, temperature reshapes them, and the chosen token joins the prompt.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write