Glossary · Phase 6 · Modern LLM engineering
Ablation
Switching off one part of a model, such as an attention head, to see what changes. A big change means that part mattered.
Think of a band playing a song. To hear what the bass adds, you mute the bass and listen again. The standard test inside a model is the same idea, and it's called an ablation: switch one part off and see what breaks.
Try it now
A picture of attention is only a hypothesis. Switching parts off is what tests it.
Learn it properly
Interpretability · Looking inside models to see what they compute. About 40 minutes.
Other terms from the same lesson
- interpretability: Research into what is actually happening inside a model's numbers.
- logit lens: Reading a model's in-progress guess at each layer by passing the hidden state straight to the output layer.
- steering: Changing a model's behavior by adding a direction to its hidden states while it runs, without retraining it.
- superposition: When a model packs more features than it has neurons by sharing each neuron across several of them, which makes single neurons hard to read.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.