Index Ai Academy
Phase 6 · Editing models

Directions: steer the run or edit the weights

Find a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.

About 40 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

The best-known weight edit so far removed one direction from a model. The editor found that a looping model's fault lived along a single line in its residual stream, and changed the weights so that nothing could write along that line again. One direction, 0.016% of the weights, and a large change in behavior.

This lesson builds that edit from its parts on the tiny GPT, with a harmless behavior: which material the model names for a product. You will find the direction, push the model along it while it runs, then cut it out of the weights for good. You will also meet the catch that the headlines skip: the same push moves different inputs by very different amounts, and some of them the wrong way.

What you'll be able to do

  • Find a behavior's direction in the residual stream as the difference between two averages
  • Steer a model at run time by adding that direction, and measure the effect on every input
  • Remove a direction from the weights with a projection, and test it against a random direction
  • Explain why the effect varies from input to input, and when to steer and when to edit
Quick check
You add the same small vector to a model's residual stream on 360 different prompts, hoping to make one word more likely. A guess: what happens across the prompts?
  1. Nothing changes: a model ignores numbers added mid-run
  2. The model stops producing readable text on every single one of the 360 prompts
  3. The word gets more likely on most prompts, by very different amounts, and less likely on a few
  4. It gets more likely by the same amount on every prompt
Show the answer

The word gets more likely on most prompts, by very different amounts, and less likely on a few

The push is identical, but what the model does with it depends on everything else in the stream. You will measure the spread: a large effect on some products, almost none on others, and a few that move the wrong way.

The idea: a behavior as a direction

Purpose. Find a direction as a difference of two averages, and see steering and projection as two uses of it.

This lesson rests on Angle, cosine similarity and projection, where a dot product measures how far a vector reaches along another, and on Reading weights without running the model.

The residual stream at each position is a list of numbers, 32 of them in the tiny GPT. A list of 32 numbers is a point in a space with 32 axes, and a direction in that space is a line through it. A surprising amount of what a model "is about to do" turns out to be readable as how far the stream sits along some line.

Here is a behavior to work with. Ask the base model what a product is made from and it often says "recycled": recycled nylon, recycled polyester, recycled down. For some products that is its top guess. For others it never is. That is harmless, easy to measure, and a fair stand-in for any tendency you might want more or less of.

Quick check
A direction is a difference of two averages, scaled to length 1. What does this print?
import numpy as np

yes = np.array([[4.0, 1.0], [6.0, 1.0]])
no = np.array([[1.0, 1.0], [3.0, 1.0]])
difference = yes.mean(axis=0) - no.mean(axis=0)
print(difference, difference / np.linalg.norm(difference))
  1. [3. 0.] [1. 0.]
  2. [2. 1.] [1. 0.]
  3. [3. 0.] [3. 0.]
  4. [5. 1.] [1. 0.]
Show the answer

[3. 0.] [1. 0.]

The "yes" group averages to [5, 1] and the "no" group to [2, 1]. Their difference is [3, 0]: the groups differ only along the first axis. Dividing by its length, 3, gives the direction [1, 0].

Editing: take the direction out of the weights

Steering adds a vector on every run. An edit changes the weights so the model can no longer move its stream along a direction at all. The arithmetic is a projection. To remove a direction d (of length 1) from a vector v, subtract the part of v that lies along d:

v′=v−(v⋅d) dv' = v - (v \cdot d) \, d

Read it as: v⋅dv \cdot d is how far vv reaches along the direction. Multiply the direction by that amount and subtract it. What is left has nothing along dd.

A model's stream is the sum of what its writers put there: the embeddings, and each block's attention output and feed-forward output. Apply that projection to every vector a writer can produce, and no sum of them can have any part along d. wsp.project_out(state, d) does exactly this to all ten writer tensors.

Try it now

The dot product with a unit direction is how far a reaches along it. Subtracting that part is the projection.

In your own words
In your own words: how do you find a direction for a behavior without any training?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: find it, steer with it, project it out

Purpose. Find the direction in the tiny GPT's stream, push along it while the model runs, then remove it from the weights and test against a control.

Worked example

Find the direction

Build 360 prompts, one for every product name and kind. Split them by whether "recycled" is the model's likely next word. Take the stream after the first block in each group, and subtract the averages.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
model, tok = wsp.build(base, info)
facts = wsp.facts()
prompts = [f"The {name} {p['type']} is made from" for name in facts["names"]
           for p in facts["products"]]
recycled = tok.token_id("recycled")

p0 = np.array([wsp.next_probs(model, tok, t)[recycled] for t in prompts])
leans = p0 > 0.5
print(f"{len(prompts)} prompts; average chance of 'recycled' {p0.mean():.2f};",
      f"it is the likely word on {leans.sum()}")

# 360 points in 32 dimensions
H = np.stack([wsp.stream(model, tok, t, after=0) for t in prompts])
difference = H[leans].mean(axis=0) - H[~leans].mean(axis=0)
# length 1: a pure direction
direction = difference / np.linalg.norm(difference)
# how far each prompt sits along it
along = H @ direction
typical = np.linalg.norm(H, axis=1).mean()
print(f"length of the difference {np.linalg.norm(difference):.2f};",
      f"a typical stream is {typical:.2f} long")
print(f"position along the direction: {along[leans].mean():.2f} when it leans",
      f"'recycled', {along[~leans].mean():.2f} when it doesn't")

import matplotlib.pyplot as plt

plt.hist(along[~leans], bins=20, alpha=0.7, label="does not lean recycled")
plt.hist(along[leans], bins=20, alpha=0.7, label="leans recycled")
plt.xlabel("position along the direction")
plt.ylabel("prompts")
plt.legend()
plt.show()
It runs right here in your browser, with nothing to install.

On 107 of the 360 prompts "recycled" is the likely next word, and its average chance over all of them is 0.28. The difference between the two groups' average streams is 1.25 long, small beside a typical stream of 9.92. So the two groups differ by a modest shift along one line, not by being in different places altogether. Along that line the leaning prompts sit at 1.61 on average and the others at 0.37. You now have a direction. Whether the model uses it is the next question, and the way to ask is to move the stream and watch.

Drill

A direction from two groups

Write direction_between(yes, no). Each argument is a 2-D NumPy array with one row per example. Return the difference of the two groups' averages (yes minus no), scaled to length 1.

import numpy as np

def direction_between(yes, no):
    return np.zeros(yes.shape[1])

yes = np.array([[4.0, 1.0, 0.0], [6.0, 1.0, 0.0]])
no = np.array([[1.0, 1.0, 4.0], [3.0, 1.0, 4.0]])
print(direction_between(yes, no))
In the lesson, your code is checked as you write it.

Steering: push the stream while the model runs

Steering leaves the weights alone. While the model processes a prompt, you add a multiple of the direction to its residual stream and let the remaining layers carry on. The multiple is the strength. A positive strength should make "recycled" more likely, and a negative one less.

wsp.next_probs(model, tok, prompt, add=vector, after=0) runs the model with vector added to the stream at the last position, after the first block.

To compare chances fairly, the next cell uses log-odds: the logarithm of a chance divided by one minus that chance. A move from 0.5 to 0.9 and a move from 0.9 to 0.99 are both about 2.2 in log-odds, where plain differences would call the second one tiny.

In plain words: log-odds stretch out the ends of the 0-to-1 scale, so that "twice as sure" counts the same wherever you start.

Worked example

The same push on 360 prompts

Add the direction at strength +4 and at strength −4 to every prompt. For each, report the average change in the log-odds of "recycled", how much that change varies, and the share of prompts that moved the wrong way. Then the average chance per kind of product at +4.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
model, tok = wsp.build(base, info)
facts = wsp.facts()
kinds = [p["type"] for p in facts["products"]]
prompts = [(kind, f"The {name} {kind} is made from") for name in facts["names"]
           for kind in kinds]
recycled = tok.token_id("recycled")

p0 = np.array([wsp.next_probs(model, tok, t)[recycled] for _, t in prompts])
H = np.stack([wsp.stream(model, tok, t, after=0) for _, t in prompts])
difference = H[p0 > 0.5].mean(axis=0) - H[p0 <= 0.5].mean(axis=0)
direction = difference / np.linalg.norm(difference)

log_odds = lambda p: np.log(p + 1e-6) - np.log(1 - p + 1e-6)
steered, changes = {}, {}
for strength in (4, -4):
    p1 = np.array([wsp.next_probs(model, tok, t, add=strength * direction,
                                  after=0)[recycled] for _, t in prompts])
    change = log_odds(p1) - log_odds(p0)
    wrong = (np.sign(change) == -np.sign(strength)).mean()
    print(f"strength {strength:+d}: log-odds change {change.mean():+.2f} on average,",
          f"spread {change.std():.2f}, wrong way on {wrong:.0%}")
    steered[strength], changes[strength] = p1, change

import matplotlib.pyplot as plt

change, start = changes[-4], log_odds(p0)
wrong = change > 0
plt.scatter(start[~wrong], change[~wrong], s=10, label="moved away, as intended")
plt.scatter(start[wrong], change[wrong], s=10, color="red", label="moved toward")
plt.axhline(0, color="black")
plt.xlabel("log-odds of recycled before")
plt.ylabel("change at strength -4")
plt.legend()
plt.show()

print("chance of 'recycled' before -> after a push of +4:")
for kind in kinds:
    rows = [i for i, (k, _) in enumerate(prompts) if k == kind]
    print(f"  {kind:13s} {p0[rows].mean():.2f} -> {steered[4][rows].mean():.2f}")
In the lesson you can change this and run it.

Pushing at +4 raises the log-odds of "recycled" by 1.99 on average, so the direction is one the model really uses. But the spread is 1.21, more than half the average, and 4% of prompts moved the wrong way. The table shows why. Rain jackets go from 0.67 to 0.97 and sun hoodies from 0.30 to 0.87, while water bottles stay at 0.00. Nothing in the catalog makes a water bottle from a recycled anything, and one push doesn't change that. Pushing the other way is worse behaved: an average change of −1.39 with a spread of 1.79, and 28% of prompts moved toward the word you were trying to suppress.

Quick check
At strength −4 the average prompt moved away from "recycled", but 28% of prompts moved toward it. What is the right conclusion?
  1. Steering only works with positive strengths
  2. The average hides a wide spread: steering has to be checked input by input, not by its mean
  3. The model is broken
  4. The direction was computed with the wrong sign, so it points away from the word
Show the answer

The average hides a wide spread: steering has to be checked input by input, not by its mean

The direction is right on average, which is how it was built: from two group averages. What a later layer does with a push depends on the rest of the stream, so individual inputs can respond weakly or backwards. A published study of steering vectors on larger models found the same thing.

That last result is not a quirk of a tiny model. A study presented at a major machine learning conference in 2024, "Analyzing the Generalization and Reliability of Steering Vectors," tested steering vectors for over 40 behaviors on two chat models. It found that the effect varied widely from input to input, that for several behaviors close to half the inputs moved opposite to the intended direction, and that some vectors were partly tracking surface features of the prompts they were built from. Steering works on average. An average is not a guarantee about any one input.

Steering has one great virtue: it is temporary. Stop adding the vector and the model is exactly as it was. That makes it the right tool for exploring. The next section makes the change permanent.

Quick check
From the interpretability lesson: why is changing something inside a model and watching the output stronger evidence than only observing it?
  1. It isn't stronger: watching and changing show the same thing, and watching is cheaper
  2. Changing the model makes it faster to run
  3. A change tests cause: if moving the stream along a direction moves the output, the model uses that direction
  4. Observation is always wrong
Show the answer

A change tests cause: if moving the stream along a direction moves the output, the model uses that direction

Seeing that two groups sit in different places along a line is a correlation. Pushing the stream along that line and watching the prediction follow shows the later layers act on it. That is why the steering run came before any edit.

This time the direction is measured at the end of the stream, where the output layer reads it, and the edit covers every writer. An honest edit also needs a control: the same operation with a random direction. If a random direction does as much, yours wasn't special.

Code flow. The edit's one line is v − (v · d) d. Step through it on a single stream.

Worked example

Project the direction out, with a control

Find the direction at the end of the stream, remove it from every writer, and measure three things on the edited model: the chance of "recycled" on the prompts that leaned toward it, the loss on ordinary catalog text, and how many weights changed. Then the same edit with three random directions.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
model, tok = wsp.build(base, info)
facts = wsp.facts()
prompts = [f"The {name} {p['type']} is made from" for name in facts["names"]
           for p in facts["products"]]
recycled = tok.token_id("recycled")
docs = wsp.catalog()

p0 = np.array([wsp.next_probs(model, tok, t)[recycled] for t in prompts])
leans = p0 > 0.5
# the end of the stream this time
H = np.stack([wsp.stream(model, tok, t) for t in prompts])
direction = H[leans].mean(axis=0) - H[~leans].mean(axis=0)

def measure(label, state):
    m, _ = wsp.build(state, info)
    p = np.array([wsp.next_probs(m, tok, t)[recycled]
                  for t in np.array(prompts)[leans]])
    print(f"{label:17s} chance of 'recycled' {p.mean():.2f}, still the likely word",
          f"on {(p > 0.5).sum():3d} of {leans.sum()}; catalog loss",
          f"{wsp.text_loss(m, tok, docs):.2f}")

measure("before", base)
edited = wsp.project_out(base, direction)
measure("direction removed", edited)
print("weights changed:", wsp.make_patch(base, edited)["changed"], "of",
      wsp.count(base))

rng = np.random.default_rng(0)
for i in range(3):
    measure(f"random direction {i + 1}", wsp.project_out(base, rng.normal(size=32)))
In the lesson you can change this and run it.

Before the edit, the leaning prompts gave "recycled" a chance of 0.70. After it, 0.24, and the word is still the likely one on only 6 of 107. The three random directions leave it at 0.69, 0.76 and 0.45: mostly unchanged, with one that happened to hit something. So the effect belongs to this direction. The edit is not free. It changed 18,656 weights, every writer in the model, and the loss on ordinary catalog text rose from 0.47 to 0.86. Random projections raise it too, to about 0.61 to 0.67, so some of that cost comes from removing any direction from a model with only 32 of them. In a model thousands of dimensions wide, one direction is a far smaller share, which is part of why the published edit could be so cheap.

Quick check
Two random directions left the chance of "recycled" near 0.70, but the third cut it to 0.45. What should you take from that?
  1. Random directions are a better way to edit a model
  2. The third run must have had a bug
  3. The edit doesn't work, because removing a random direction did the same thing to the model
  4. One control isn't enough: in 32 dimensions a random direction can overlap the real one by chance, so run several
Show the answer

One control isn't enough: in 32 dimensions a random direction can overlap the real one by chance, so run several

With only 32 dimensions, a random line is fairly likely to lean partly along the one you care about. Two of three controls left the behavior alone, and none came near 0.24. Several controls show the range that chance alone produces, which a single one can't.

Worked example

What the edited model writes

Let both models finish the same two sentences, always taking the most likely next word.

import numpy as np
import torch
import wsp

base, info = wsp.load_state("tiny-gpt")
model, tok = wsp.build(base, info)
facts = wsp.facts()
prompts = [f"The {name} {p['type']} is made from" for name in facts["names"]
           for p in facts["products"]]
p0 = np.array([wsp.next_probs(model, tok, t)[tok.token_id("recycled")]
               for t in prompts])
H = np.stack([wsp.stream(model, tok, t) for t in prompts])
direction = H[p0 > 0.5].mean(axis=0) - H[p0 <= 0.5].mean(axis=0)
edited_model, _ = wsp.build(wsp.project_out(base, direction), info)

for start in ["The River rain jacket is made from", "The Summit fleece is made from"]:
    for label, m in [("before", model), ("after ", edited_model)]:
        ids = torch.tensor([tok.encode(start)])
        print(label, "|", tok.decode(m.generate(ids, 14, 0)[0].tolist()))
In the lesson you can change this and run it.

The base model makes the rain jacket from recycled nylon and the fleece from recycled polyester fleece. The edited model picks each product's other material from the catalog: a waterproof shell fabric, and soft brushed fleece. It didn't go silent or invent a material. It fell back to the next thing it knew. You can also see the cost of the edit in the rain jacket's second sentence, which comes out a little wrong. That is the higher catalog loss, made visible.

Quick check
What does this print?
import numpy as np

d = np.array([0.0, 1.0])
v = np.array([3.0, 4.0])
after = v - (v @ d) * d
print(after, after @ d)
  1. [3. 0.] 0.0
  2. [3. 0.] 3.0
  3. [3. 4.] 4.0
  4. [0. 4.] 4.0
Show the answer

[3. 0.] 0.0

v reaches 4 along d. Subtracting 4 times d leaves [3, 0], which has nothing along d: the last number printed, its dot product with d, is 0.

Drill

Remove a direction from a writer

Write remove_direction(W, d). W is a writer's weight matrix of shape (stream width, inputs): each column is a vector it can add to the stream. d is a direction, which may not have length 1. Return a new matrix in which every column has had its part along d removed.

import numpy as np

def remove_direction(W, d):
    return W

W = np.array([[1.0, 2.0, 0.0], [3.0, 4.0, 5.0]])
print(remove_direction(W, np.array([0.0, 2.0])))
In the lesson, your code is checked as you write it.
In your own words
In your own words: why does the projection have to be applied to every writer, and not just to one layer?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: steer or edit?

Purpose. Choose between a run-time push and a permanent edit, and read a layer with a hook.

The two tools answer different needs. Steering is a probe: nothing permanent happens, you can try ten strengths in a minute, and you learn whether a direction matters. But it needs code wrapped around the model on every run, and whoever runs the model can leave it out.

An edit travels with the weights. Anyone who downloads the file gets the changed behavior, with no extra code. That is its strength and its danger. The same projection that removes a harmless word preference has been used by others to remove a safety-trained model's tendency to refuse, which Safety, licenses and provenance returns to. It is why this course's labs stay on harmless behavior, and why an edit should always ship with its control and its measured side effects.

Where this shows up in an engineer's week

You use steering to ask a question quickly: does this direction matter? You try ten strengths on a few hundred prompts and report the spread and the share that moved the wrong way, not only the average. Only when the answer is yes do you consider an edit, and then it ships with a random-direction control and the loss on ordinary text.

Goes deeper in

In your own words
In your own words: when would you steer at run time, and when would you edit the weights?

Write a sentence or two, then compare it with a good answer in the lesson.

Quick check
A behavior is common in some prompts. Which two things do you need to find its direction in the stream?
  1. The weights of the very last layer only
  2. The average stream where it happens and where it doesn't
  3. A full list of the model's tokens
  4. A second model trained on the behavior itself
Show the answer

The average stream where it happens and where it doesn't

The direction is the difference of the two group averages, pointing from "doesn't" toward "does". No training is needed.

The standard way to find a direction for a behavior needs no training. Collect the stream on prompts where the behavior happens and on prompts where it doesn't. Average each group. Subtract. The difference of the two averages points from "doesn't" toward "does."

wsp.next_probs gives the model's probability for each next token after a prompt. wsp.stream gives the residual stream at the prompt's last position: after the first block when you pass after=0, and at the very end when you leave that out.

Quick check
A direction is a difference of two averages, scaled to length 1. What does this print?
import numpy as np

yes = np.array([[4.0, 1.0], [6.0, 1.0]])
no = np.array([[1.0, 1.0], [3.0, 1.0]])
difference = yes.mean(axis=0) - no.mean(axis=0)
print(difference, difference / np.linalg.norm(difference))
  1. [3. 0.] [3. 0.]
  2. [5. 1.] [1. 0.]
  3. [3. 0.] [1. 0.]
  4. [2. 1.] [1. 0.]
Show the answer

[3. 0.] [1. 0.]

The "yes" group averages to [5, 1] and the "no" group to [2, 1]. Their difference is [3, 0]: the groups differ only along the first axis. Dividing by its length, 3, gives the direction [1, 0].

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

How is a behavior's direction usually found in the residual stream?

  1. Average the stream on inputs with the behavior and on inputs without it, and subtract
  2. Train a new network to predict the behavior
  3. Pick the one stream dimension with the largest value on those inputs
  4. Read it from the model's configuration file

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Go deeper, if you like

Optional lessons that build on this one. They never block a phase or a certificate.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 17 lessons

Creating Custom LLMs: Editing Open Weights Using Weight-Space Programming

Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write