Interpretability

Step 2: test it by switching heads off

Think of a band playing a song. To hear what the bass adds, you mute the bass and listen again. The standard test inside a model is the same idea, and it's called an ablation: switch one part off and see what breaks.

The code below runs the model's forward pass by hand. (The forward pass is the trip from the input words to the next-word chances.) There's one change. Each head's output gets multiplied by 1 to keep it, or by 0 to switch it off. The test uses four products and three setups:

• normal: every head on;
• layer-1 heads 1-3 off: the three heads that stared at the product;
• all layer-0 heads off: a comparison that silences the other layer.

If the three layer-1 heads carry "which product is this?", switching them off should leave the model guessing materials blindly. For each run the code prints the top three next words and their chances:

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")

def next_token_probs(idx, ablate=()):
    h = model.tok(idx) + model.pos(torch.arange(idx.shape[1]))
    for layer, blk in enumerate(model.blocks):
        a = blk.ln1(h)
        B, T, d = a.shape
        q, k, v = blk.qkv(a).split(d, dim=-1)
        q, k, v = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k, v))
        scores = (q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)).masked_fill(~torch.tril(torch.ones(T, T)).bool(), float("-inf"))
        heads = torch.softmax(scores, dim=-1) @ v                       # (B, heads, T, head size)
        keep = torch.tensor([0.0 if (layer, hd) in ablate else 1.0 for hd in range(blk.h)])
        heads = heads * keep.view(1, -1, 1, 1)                          # switch chosen heads off
        h = h + blk.proj(heads.transpose(1, 2).reshape(B, T, d))
        h = h + blk.mlp(blk.ln2(h))
    return torch.softmax(model.head(model.ln_f(h))[0, -1], dim=-1)

experiments = {"normal": (), "layer-1 heads 1-3 off": ((1, 1), (1, 2), (1, 3)),
               "all layer-0 heads off": ((0, 0), (0, 1), (0, 2), (0, 3))}
for product in ["tent", "camp mug", "headlamp", "fleece"]:
    idx = torch.tensor([tok.encode(f"Our Kestrel {product} is made from")])
    for name, heads in experiments.items():
        with torch.no_grad():
            top = next_token_probs(idx, heads).topk(3)
        print(f"{product:<9} {name:<22} " + ", ".join(f"{tok.vocab[i]} {p:.2f}" for p, i in zip(top.values.tolist(), top.indices.tolist())))

Normally each product gets its own materials. The tent gets silnylon, and the mug gets enamel or titanium. The headlamp gets "a", as in "a recycled aluminum body".

With the three layer-1 heads off, all four products get nearly the same generic guess: "responsibly", as in "responsibly sourced down". The model no longer knows what it's describing.

Switching off all of layer 0's heads shifts the chances, and for the tent it's enough to put "recycled" (0.62) just ahead of "silnylon" (0.35). But every product still gets its own materials near the top, with no sign of the generic "responsibly". So the hypothesis is supported: those three layer-1 heads move the product's identity to the place where the material is predicted. (torch here is the course's in-browser PyTorch subset.)

with_head = 0.92
without_head = 0.91
print(round(with_head - without_head, 2))

normal = 0.64                      # chance of "silnylon" with every head on
heads = {                          # head: (attention on "tent", chance with that head off)
    "head A": (0.95, 0.61),
    "head B": (0.30, 0.12),
    "head C": (0.90, 0.20),
}
for name, (attention, chance_off) in heads.items():
    drop = normal - chance_off
    print(f"{name}: looks at tent {attention:.2f}, drop when off {drop:.2f}")

Step 3: watch the prediction form

Picture a friend reading a mystery novel. You stop them after each chapter and ask, "Who did it?" Their answers show how their thinking built up. The logit lens does this to a model.

Recall two ideas from Phase 5. Every position carries a running list of numbers called the residual stream. Each block reads that list and adds its own update to it. At the very end, the model's final layer norm and output layer turn the list into a score for every word in the vocabulary. Those scores are the logits, and a softmax turns them into chances.

The logit lens applies that same final layer norm and output layer partway through. It asks, "What would the model predict if it stopped here?" The code reads the prediction three times: from the embeddings alone, after block 0, and after block 1. It prints the top four words each time:

import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
idx = torch.tensor([tok.encode("Our Kestrel tent is made from")])
with torch.no_grad():
    h = model.tok(idx) + model.pos(torch.arange(idx.shape[1]))
    stages = [("embeddings only", h)]
    for i, blk in enumerate(model.blocks):
        h = blk(h)
        stages.append((f"after block {i}", h))
    for name, hidden in stages:
        p = torch.softmax(model.head(model.ln_f(hidden))[0, -1], dim=-1)
        top = p.topk(4)
        print(f"{name:<16} " + ", ".join(f"{tok.vocab[i]} {v:.2f}" for v, i in zip(top.values.tolist(), top.indices.tolist())))

From the embedding of "from" alone, the best guess is "a", a common word after "from". After the first block, it makes the generic guess you saw in the ablation: "responsibly" (0.46), then "recycled" (0.38). After the second, it predicts materials for tents. The picture matches the ablation: the product-specific information arrives in layer 1.

Interpretability at scale

Large models are much harder. They have billions of parameters. Worse, a single neuron usually responds to many unrelated things.

The reason is that models pack in more concepts than they have neurons. Think of an old phone keypad: the 2 key stands for A, B and C at once. Each key does several jobs, and you can only tell which letter is meant from context. Neurons share concepts in a similar way. This is called superposition.

Researchers have several tools for this:

• Probes: train a small classifier on a layer's hidden numbers. If it can read off a topic, a sentiment, or "is this statement true?", that information is present at that layer.
• Sparse autoencoders (SAEs): learn a dictionary of directions in the hidden numbers, where each direction switches on only rarely. These directions turn out to be far easier to read than single neurons. Anthropic's 2024 "Scaling Monosemanticity" work pulled out millions of these features from Claude 3 Sonnet. ("Monosemantic" means "one meaning each".) They ranged from the Golden Gate Bridge to code bugs to sycophantic praise. Turning up the Golden Gate Bridge feature gave a model that steered every conversation toward the bridge.
• Circuit tracing: follow how features connect across layers to explain one behavior. One example is how a model plans a rhyme before it writes the line. Another is how it does arithmetic along several paths at once.

What people use these tools for:

• debugging failures;
• detecting hidden goals or deception in safety audits;
• steering: adding or removing a feature direction to change behavior, with no retraining.

Be honest about the limits. Today's methods explain small pieces of big models. Their explanations can be incomplete or misleading. Interpretability is more a research field than a product feature, though its tools show up more and more in model audits.

For marketers: interpretability is why "the model has a hidden bias toward X" can sometimes be checked and fixed directly. Without it, you could only guess from the outputs. Expect it to show up in model safety reports and audits.