Glossary · Phase 5 · Transformers
SwiGLU
A feed-forward layer design used in Llama-style models, where one set of numbers acts as a gate that turns another set up or down.
Try it now
Communicate, then compute, with a skip connection around each. Stack dozens of these and you have a GPT.
Learn it properly
The transformer block · Attention, MLP, residuals and norms together. About 35 minutes.
Other terms from the same lesson
- decoder: A transformer that reads left to right and predicts the next token; each token can only see the ones before it. GPT-style chat models are decoders.
- feed-forward network: The small neural network inside each transformer block. It works on each token on its own, after attention has mixed in information from the other tokens.
- model card: The fact sheet published with a model: its size, what it was trained on, what it's for and its known limits.
- transformer: The design behind today's language models: layers of attention plus small neural networks, repeated many times.
- transformer block: One repeating unit of a transformer: attention, then a small neural network, each with a shortcut and normalization.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.