Glossary · Phase 5 · Transformers
Weight tying
Reusing the input embedding table as the output layer, so the model needs fewer numbers to learn.
Try it now
Set it to 50,257 / 1,024 / 768 / 12 and you get GPT-2 small's roughly 124 million. Width costs the most, because it's squared.
Learn it properly
The GPT architecture · A stack of decoder blocks that predicts the next token. About 35 minutes.
Other terms from the same lesson
- GPT: Generative pre-trained transformer: a stack of transformer blocks trained to predict the next token.
- language-model head: The last layer of a language model. It turns the final vector at each position into one score per token in the vocabulary.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.