Index Ai Academy
Phase 6 · Editing models

Low-rank edits

Cut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.

About 35 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

A fine-tune changes every weight, so its delta is as big as the model. Yet people share "adapters" for huge models that are a hundredth of the size and still carry the behavior. The reason is a fact about the shape of the change: most of it lies along a few directions.

In the LoRA lesson you trained a low-rank update on purpose. Here you go the other way. You start from a finished fine-tune, find the few directions that matter, throw the rest away, and measure when the skill survives. It is how adapters get extracted from full fine-tunes, and it is a second way to make an edit small enough to read.

What you'll be able to do

  • Cut a matrix down to its largest directions with the SVD, and measure what was lost
  • Find how few directions a fine-tuned behavior needs, and count the numbers each choice stores
  • Turn a fine-tune's delta into LoRA's two small factors and apply them to the base
  • Explain why the saving is modest in a narrow model and large in a wide one
Quick check
A fine-tune changed all 40,195 weights of a tiny GPT and taught it to answer 72 of 72 test questions. A guess: how much of that change can you throw away and still get all 72?
  1. None: every changed weight is needed
  2. About two thirds of it, if you keep each matrix's largest directions
  3. Exactly half, whichever half you pick
  4. Almost all of it: a single number per matrix is enough to keep
Show the answer

About two thirds of it, if you keep each matrix's largest directions

Keeping the 8 largest directions of each matrix stores about a third of the numbers and still answers all 72. You will find that point by measuring, and see where it stops working.

The idea: a change with a few big directions

Purpose. See why a fine-tune's change can be stored as two small matrices, and count the numbers each choice stores.

This lesson rests on LoRA, where you trained a low-rank update on purpose, and on Edits are deltas: task vectors, patches and hashes.

Reading weights without running the model used singular values to describe a layer: how much it stretches along each of its main directions. The same tool describes a change to a layer. Take one matrix of the fine-tune, subtract the same matrix of the base, and you have that matrix's delta. Its singular values say how the change is spread: across many directions evenly, or mostly along a few.

The SVD does more than describe. It splits a matrix into a sum of simple pieces, one per singular value. Each piece is a single column times a single row, scaled by its singular value. Keep the pieces with the largest values, drop the rest, and you have the closest matrix of that rank to the original. Rank is the number of pieces. No other choice of that many pieces comes closer.

In plain words: a rank-1 matrix is one pattern, repeated across every row at different strengths. A rank-8 matrix is eight such patterns added together. Cutting a matrix to rank 8 keeps its eight strongest patterns.

The next cell asks how lopsided the fine-tune's change is, tensor by tensor.

Quick check
A matrix made from one column times one row has rank 1. What does this print?
import numpy as np

M = np.outer([1.0, 2.0, 3.0], [4.0, 5.0])
print(M.shape, np.linalg.matrix_rank(M))
  1. (3, 2) 6
  2. (3, 2) 2
  3. (2, 3) 1
  4. (3, 2) 1
Show the answer

(3, 2) 1

The outer product of a 3-entry column and a 2-entry row is a 3 by 2 matrix. Every row is the same pattern, [4, 5], at a different strength, so its rank is 1.

Try it now

The same five tensors, before and after the fine-tune. Choose "What changed": the bars at the bottom are then the singular values of the change itself.

In your own words
In your own words: what does it mean to cut a matrix to rank 8?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: cut the delta and score the skill

Purpose. Cut a real fine-tune's change to a few directions, count the numbers stored, and extract the factors as an adapter.

Worked example

How the change is spread

For five of the fine-tune's matrices: the share of the change's energy in its single largest direction, and how many directions hold 90% of it.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
tuned, _ = wsp.load_state("tiny-gpt-uses")
delta = wsp.delta(tuned, base)

matrices = [name for name, t in delta.items() if t.ndim == 2]
inside = sum(delta[n].size for n in matrices)
print(len(matrices), "matrices hold", inside, "of the delta's", wsp.count(delta),
      "numbers")
for name in ["head.weight", "tok.weight", "blocks.0.proj.weight",
             "blocks.0.mlp.0.weight", "blocks.1.mlp.2.weight"]:
    s = np.linalg.svd(delta[name], compute_uv=False)
    energy = np.cumsum(s ** 2) / np.sum(s ** 2)
    need = int(np.searchsorted(energy, 0.9) + 1)
    print(f"{name:22s} largest direction {energy[0]:.0%} of the energy;",
          f"90% needs {need:2d} of {len(s)}")
It runs right here in your browser, with nothing to install.

Eleven matrices hold 39,104 of the delta's 40,195 numbers; the rest are biases and LayerNorm settings. The change to the output layer, head.weight, is very lopsided: one direction holds 63% of its energy, and 10 of 32 hold 90%. The others are less extreme, needing 10 to 17 directions for 90%. None is spread evenly over all 32. So in every matrix, a minority of directions holds most of the change. Whether that minority holds the skill is a separate question, and the next section measures it.

Drill

Cut a matrix to rank r

Write truncate(M, r): the best rank-r version of the matrix M, built from its r largest singular values and their directions. Use np.linalg.svd(M, full_matrices=False), which returns U, s and Vt.

import numpy as np

def truncate(M, r):
    return M

M = np.array([[4.0, 0.0, 0.0], [0.0, 3.0, 0.0], [0.0, 0.0, 1.0]])
print(truncate(M, 2))
In the lesson, your code is checked as you write it.

How few directions does the skill need?

Energy is a property of the numbers. The skill is a property of the model's answers, and the only way to find what it needs is to cut and measure. So: cut every matrix of the delta to rank r, leave the small bias and LayerNorm changes as they are, add the result to the base, and count good answers.

Each rank also has a cost. A full matrix with m rows and n columns stores m × n numbers. Its rank-r version can be stored as r columns of length m and r rows of length n: r × (m + n) numbers.

Worked example

The skill at five ranks

Cut the whole delta to ranks 1, 2, 4, 8 and 16. For each: good answers out of 72, and the numbers stored as a share of the full delta.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
tuned, _ = wsp.load_state("tiny-gpt-uses")
delta = wsp.delta(tuned, base)
full = wsp.count(delta)

def cut(delta, r):
    out, stored = {}, 0
    for name, d in delta.items():
        if d.ndim == 2:
            U, s, Vt = np.linalg.svd(d, full_matrices=False)
            out[name] = (U[:, :r] * s[:r]) @ Vt[:r]
            stored += r * (d.shape[0] + d.shape[1])
        else:
            out[name] = d                      # biases and LayerNorms: kept whole
            stored += d.size
    return out, stored

ranks, answers, shares = (1, 2, 4, 8, 16), [], []
for r in ranks:
    small, stored = cut(delta, r)
    model, tok = wsp.build(wsp.add(base, small), info)
    answers.append(wsp.score(model, tok, "uses"))
    shares.append(stored / full)
    print(f"rank {r:2d}: {answers[-1]:2d} of 72   stores {stored:6,} numbers,",
          f"{shares[-1]:.0%} of the full delta")

import matplotlib.pyplot as plt

fig, (left, right) = plt.subplots(1, 2, figsize=(8, 3))
left.plot(ranks, answers, "o-")
left.set_xlabel("rank kept")
left.set_ylabel("answers out of 72")
right.plot(ranks, shares, "o-")
right.set_xlabel("rank kept")
right.set_ylabel("share of the full delta stored")
plt.show()
In the lesson you can change this and run it.

One direction per matrix answers nothing. Two answer 45 of 72, four answer 63, and at rank 8 all 72 are back, with 13,683 numbers stored: 34% of the full delta. Rank 16 adds storage and no answers. So this skill needs about eight directions per matrix. The other two thirds of the fine-tune's change was real, in the sense that training put it there, and unnecessary, in the sense that the behavior you test doesn't depend on it.

Quick check
At rank 1 every matrix kept its single largest direction. For the output layer that direction held 63% of the change's energy. The model answered 0 of 72. What does that show?
  1. The SVD picked the wrong direction
  2. Keeping most of the energy isn't the same as keeping the skill
  3. The output layer isn't involved in the skill at all
  4. Rank 1 always destroys a model
Show the answer

Keeping most of the energy isn't the same as keeping the skill

Energy counts how much of the numbers survive. The skill needed several directions working together, across every matrix, and one per matrix wasn't enough. The only way to learn what a cut keeps is to score the model after it.

Quick check
A 195 by 32 matrix is stored at rank 8, as 8 columns of 195 and 8 rows of 32. What does this print?
rows, cols, rank = 195, 32, 8
print(rank * (rows + cols), rows * cols)
  1. 1816 227
  2. 1816 6240
  3. 49920 6240
  4. 6240 1816
Show the answer

1816 6240

8 × (195 + 32) = 1,816 numbers for the rank-8 version, against 195 × 32 = 6,240 for the full matrix: 29% of the size.

Code flow. The cut in the next cells is the SVD split below, on a 3 by 3 change that happens to have rank 1. Step through it: B and A multiply back to the whole change.

From a delta to an adapter

A truncated delta is still written as a full-size matrix with most of its content removed. The compact form keeps the two factors apart. Split the rank-r piece into a tall matrix B, with r columns, and a wide matrix A, with r rows, so that:

Δ≈B A\Delta \approx B \, A

Read it as: AA reads the layer's input and boils it down to rr numbers. BB turns those rr numbers into a change to the layer's output. Their product is the change to the weight matrix.

Those are LoRA's two matrices, the same names and shapes. A set of them, one pair per matrix, is an adapter: a small file that sits beside an untouched base model and is added in when the model loads. Pulling an adapter out of a finished fine-tune is a standard trick. It turns a download the size of the model into a far smaller one, and it lets one base serve many behaviors.

Worked example

Extract an adapter, apply it, and compare with a cruder cut

Split each matrix of the delta into B and A at rank 8, rebuild the model from the base and the factors, and score it. Then try the obvious alternative: keep the largest half of the delta's entries, wherever they are.

import numpy as np
import wsp

base, info = wsp.load_state("tiny-gpt")
tuned, _ = wsp.load_state("tiny-gpt-uses")
delta = wsp.delta(tuned, base)
R = 8

adapter, rest = {}, {}
for name, d in delta.items():
    if d.ndim == 2:
        U, s, Vt = np.linalg.svd(d, full_matrices=False)
        adapter[name] = (U[:, :R] * s[:R], Vt[:R])    # B: (rows, R)  A: (R, columns)
    else:
        rest[name] = d
B, A = adapter["head.weight"]
print("head.weight: B", B.shape, "A", A.shape, "- product", (B @ A).shape)

def score(change):
    model, tok = wsp.build(wsp.add(base, change), info)
    return wsp.score(model, tok, "uses")

zero = {name: np.zeros_like(d) for name, d in delta.items()}
factors = {name: B @ A for name, (B, A) in adapter.items()}
print("adapter and the small tensors:", score({**zero, **factors, **rest}), "of 72")
print("adapter alone:                ", score({**zero, **factors}), "of 72")

flat = np.abs(wsp.flatten(delta))
for keep in (0.5, 0.34):
    threshold = np.quantile(flat, 1 - keep)
    sparse = {name: np.where(np.abs(d) >= threshold, d, 0) for name, d in delta.items()}
    print(f"largest {keep:.0%} of entries kept:    ", score(sparse), "of 72")
In the lesson you can change this and run it.

For the output layer, B is 195 by 8 and A is 8 by 32, and their product is the full 195 by 32. The adapter with the small tensors answers all 72. The second line drops the bias and LayerNorm changes, as most real adapters do, and here that costs nothing: still 72. The last two lines keep entries by size where the adapter keeps directions. The largest half of the entries answers 70. At the adapter's own budget, a third of the entries, only 27 survive. For this edit, "which directions" compresses better than "which weights."

Drill

Split a change into B and A

Write to_adapter(D, r): return a tuple (B, A) where B has shape (rows, r), A has shape (r, columns), and B @ A is the best rank-r version of D. Put the singular values into B.

import numpy as np

def to_adapter(D, r):
    return D, np.eye(D.shape[1])

rng = np.random.default_rng(1)
D = rng.normal(size=(6, 4))
B, A = to_adapter(D, 2)
print(B.shape, A.shape)
In the lesson, your code is checked as you write it.
Quick check
An adapter stores B and A for a matrix where a full delta would store the whole matrix. What do you give up?
  1. Nothing: B @ A is always exactly the original change
  2. The ability to remove the edit later
  3. The base model, which the adapter replaces
  4. Everything in the change outside its r largest directions
Show the answer

Everything in the change outside its r largest directions

B @ A is the closest rank-r matrix to the change, not the change itself. What lies outside those directions is gone. Whether that matters is something you measure: here rank 8 kept every answer and rank 4 lost nine.

In your own words
In your own words: what is an adapter, and how is extracting one from a fine-tune different from training one?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: adapters at real width

Purpose. Why the saving is small in a narrow model and large in a wide one, and the calls that extract and merge an adapter.

Thirty-four percent is a saving, but not the hundredfold one adapters are famous for. The reason is the model's width. Rank 8 in a model 32 wide is a quarter of everything a matrix can hold. Real models are thousands wide, and the same eight directions are then a sliver. Try it:

Quick check
From the LoRA lesson: what does LoRA train, and what does it leave alone?
  1. It trains every weight, then deletes the small ones
  2. It trains only the biases
  3. It trains two small matrices whose product is added to a weight matrix, and leaves the original weights frozen
  4. It trains a second, smaller model that reads the first one's answers and corrects them
Show the answer

It trains two small matrices whose product is added to a weight matrix, and leaves the original weights frozen

LoRA adds a low-rank update, the product of two small matrices, beside each frozen weight matrix. It bets in advance on what you just measured after the fact: that the useful change has low rank.

Where this shows up in an engineer's week

You get a full fine-tune that is the size of the model and need to ship it as a small file. Extract an adapter at a few ranks, score each on your held-out questions and keep the smallest that still passes. Re-score after merging the adapter back in, since the merged weights are what ships.

Goes deeper in

  • LoRA: training the two factors directly.
  • Merging models: combining adapters and full fine-tunes.
In your own words
In your own words: why did rank 8 save only two thirds here, when adapters for large models are a hundredth of the size?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 3 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

What does cutting a matrix to rank r with the SVD give you?

  1. The closest matrix to the original that can be built from r patterns
  2. The r largest entries of the matrix, with the rest set to 0
  3. The first r rows of the matrix
  4. A matrix r times smaller in each direction

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 17 lessons

Creating Custom LLMs: Editing Open Weights Using Weight-Space Programming

Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write