How a transformer works, with a real GPT you can run

This is a complete GPT, the same design behind the big language models, small enough to run on this page. It has 40,195 parameters, where the big models have billions, and it was trained on one small product catalog. So it knows only 195 tokens and writes about jackets and mugs. Nothing here is canned: every number is computed from its trained weights, for the words in the box.

What to try

1. Watch it use attention. With Our Cove camp mug is made from, it splits its bet between enamel and titanium. Open block 2, head 4: the last row is the word from, and it puts about 94% of its attention on mug. Now change mug to jacket in the box. The prediction moves to recycled.
2. Let it write. Press Add the likeliest word a few times, then start over and press Sample a word instead.
3. Move the temperature. At 0.1 nearly all the probability piles onto one word. At 2 it spreads out across words that fit less well.
4. Confuse it. Type a word it has never seen. It reads that as unknown, which is what a model's vocabulary limit looks like from the inside.

What happens between your words and the prediction

1. Tokens become vectors. Each token looks up a learned list of 32 numbers, and another list for its position is added to it.
2. Attention lets tokens read each other. Each token compares itself with the tokens before it, decides how to share its attention, and blends in what it finds. Four heads do this side by side, each with its own pattern. The table's empty upper right is the causal mask: a token can't look at what comes after it.
3. An MLP processes the result. Each token, on its own, runs what it gathered through a small two-layer network.
4. Blocks repeat. Attention and the MLP together make one transformer block. This model has two, and big models stack dozens.
5. The last token's vector becomes a prediction. It's turned into one score for every token in the vocabulary. Softmax turns the scores into the probabilities you see, and temperature divides the scores first.

A big language model does these same steps with far more parameters, far more text and a far larger vocabulary. That's why it's worth seeing them at this size, where you can read every number.