Glossary · Phase 5 · Transformers
Feed-forward network
The small neural network inside each transformer block. It works on each token on its own, after attention has mixed in information from the other tokens.
Try it now
Communicate, then compute, with a skip connection around each. Stack dozens of these and you have a GPT.
Learn it properly
The transformer block · Attention, MLP, residuals and norms together. About 35 minutes.
Other terms from the same lesson
- decoder: A transformer that reads left to right and predicts the next token; each token can only see the ones before it. GPT-style chat models are decoders.
- model card: The fact sheet published with a model: its size, what it was trained on, what it's for and its known limits.
- SwiGLU: A feed-forward layer design used in Llama-style models, where one set of numbers acts as a gate that turns another set up or down.
- transformer: The design behind today's language models: layers of attention plus small neural networks, repeated many times.
- transformer block: One repeating unit of a transformer: attention, then a small neural network, each with a shortcut and normalization.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.