
How transformers think
One prompt, “The cat sat”, through the whole model: tokens, embeddings, the attention matrix, the feed-forward layer, and the probability over the next word.
Each row is widened from 8 numbers to 32, bent by GELU and squeezed back: 552 parameters here, 4,722,432 per layer in GPT-2.
After attention, each row passes through a small two-layer network alone: widened from 8 numbers to 32, bent by GELU, projected back to 8. 552 parameters here, 4.7 million per layer in GPT-2. GELU keeps a little of the negatives: at −1 it gives −0.16, where ReLU gives 0. Each hidden unit is a memory slot: for 'sat', three of the 32 make 93% of the output. Start free at indexaiacademy.com.
All 38 videos, by what they're for.

One prompt, “The cat sat”, through the whole model: tokens, embeddings, the attention matrix, the feed-forward layer, and the probability over the next word.

“The cat sat” is cut into three tokens and looked up as 104, 324 and 782: a tensor of shape (3,) with T = 3 and V = 1,000.

Each ID picks one row of the embedding matrix, a positional wave is added, and the result is the 3 × 8 matrix that enters the first block.

Queries, keys and values from the same rows, then the 3 × 3 matrix of weights: “ sat” reads 0.40 of its value from “ cat”.

The last row becomes a score for every token, softmax makes them probabilities, temperature reshapes them, and the chosen token joins the prompt.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write