Read the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.
About 45 minutes · Optional lesson: deeper material that no phase or certificate requires · A free account opens the whole lesson. Already have an account? Sign in to open this lesson
Everything so far ran on models of 40,000 weights that fit in a browser tab. The edit this course began with was made to an open model of 870 million weights, and its authors published it: the changed entries, and a hash for each tensor before and after.
That is enough to check their work yourself. In this lesson you read the published edit, redo its key steps on the tiny model where every step can be seen, and find out what happens to an edit when a model is stored in fewer bits. A notebook then takes you through the same steps on the real model, on your own machine.
What you'll be able to do
Read the manifest of a published weight edit and say what it changes in a real open model
Apply an edit stored as positions and new values, and check it by hash on both sides
Explain why some weights an edit touches don't change once they are stored in 16 bits
Measure whether an edit survives rounding to 8, 4 and 3 bits, and say when to measure again
Quick check
A fix changes 1,024 of a model's weights, each by a small amount. The model is then rounded to 4 bits per weight so that it runs on a phone. A guess: what happens to the fix?
None of it is still there, because changes that small can't be stored in four bits
All of it is still there, because rounding only affects weights that weren't edited
Most of it is still there, but less than before, so the model has to be measured again
It gets stronger, because rounding pushes every edited weight further the same way
Show the answer
Most of it is still there, but less than before, so the model has to be measured again
On the tiny model in this lesson, over half of the edited weights become identical to the faulty model's after rounding to 4 bits, and the fix still restores 63 of 72 answers, down from 68. At 3 bits it restores 38. An edit and a quantizer are two changes to the same numbers, and only a measurement says what is left.
The idea: an edit is positions, new values and two hashes
The post's eighth experiment edited Qwen/Qwen3.5-0.8B, an open model under the Apache 2.0 license. Its authors released the edit in the public logs of the experiment. It is two small files:
a manifest: the tensors that changed, their shape and number format, how many entries changed, and a SHA-256 hash of each tensor before and after;
the entries: for each tensor, the flat positions that changed and the new value at each one.
That is the patch format you have used since Edits are deltas: task vectors, patches and hashes: positions, new values, and a hash on both sides. The practice API holds a copy of the manifest, which is data under a CC BY 4.0 license, so you can read the real numbers here. The model itself is 1.75 GB and isn't served by the academy.
Worked example
Read the real manifest
Load the published edit's manifest and work out, from its own numbers, how many weights it touches and what share of the language model that is.
import wsp
edit = wsp.published("qwen-loop-edit")
model = edit["model"]
name, weights = model["name"], model["weights"]
print(name, "|", model["license"], "|", f"{weights:,}", "weights in the file,",
f'{model["language_model_weights"]:,}', "in the language model")
touched = 0for t in edit["manifest"]["tensors"]:
rows, columns = t["shape"]
touched += t["changed_columns"] * rows
short = t["name"].replace("model.language_model.", "")
print(short, t["shape"], t["dtype"], "|", f'{t["changed_entries"]:,}',
"entries in", t["changed_columns"], "columns")
changed = edit["manifest"]["changed_entries_total"]
print(f"touched: {touched:,} | changed: {changed:,} |",
f"touched and left unchanged: {touched - changed}")
share = 100 * touched / model["language_model_weights"]
print(f"share of the language model touched: {share:.4f}%")
print("source:", edit["source"]["logs"], "|", edit["source"]["license"])
It runs right here in your browser, with nothing to install.
Two tensors, the output matrices of the feed-forward layers in blocks 15 and 19. In each, 64 of 3,584 columns were edited: 64 hidden units, 1,024 numbers each. That is 131,072 weights touched, 0.0174% of the language model. The post says 0.016%, counting against a rounder 0.8 billion.
It is the edit you made by hand on the looping model, at another scale: one direction, taken out of the columns that write it, in two feed-forward output matrices. And one number doesn't add up yet. Of the 131,072 weights touched, 884 did not change.
Quick check
Each tensor in the manifest has the shape [1024, 3584], and 64 of its columns were edited. There are two such tensors. What does this print: the number of weights the edit touches?
Each edited column holds one number per row, 1,024 of them. Sixty-four columns in each of two tensors is 2 × 64 × 1,024 = 131,072 weights. The second number in the shape, 3,584, is how many columns the matrix has in all, and most of them are untouched.
Apply it, with a hash on both sides
Applying the published edit is one line of arithmetic per tensor: write the new values at the listed positions. The notebook does it on the real model in a dozen lines.
What makes it safe to apply is the pair of hashes. Before: if your tensor's hash isn't the manifest's, you are holding a different model or a different revision of it, and the edit's positions mean nothing. After: if the rebuilt tensor's hash isn't the manifest's, something went wrong in between. wsp.apply_patch has done both checks for you all along. Here you do them yourself, in the published form.
The drill uses a small tensor. The real ones differ only in size.
Drill
Apply an edit, checked before and after
Write apply_edit(tensor, entry, index, value). tensor is a NumPy array, entry is its line of the manifest with "base" and "edited" hashes, index holds flat positions and value the new numbers. Raise ValueError if the tensor's hash isn't entry["base"]. Return a new array with the new values written in, and raise ValueError if its hash isn't entry["edited"]. The starter gives you digest, the hash of an array.
In the lesson, your code is checked as you write it.
Try it now
In your own words
In your own words: the edit is two small files, and anyone can apply them. What do the two hashes in the manifest add?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: sixteen bits, then fewer
Sixteen bits, and the 884 weights that didn't change
The manifest says the tensors are bfloat16. Until now a weight has been a 32-bit float, good for about seven significant digits. Most open models are stored in 16 bits to halve the file. A bfloat16, the format from GPUs and compute, keeps the sign and the whole exponent of a 32-bit float and only 7 of its 23 fraction bits. That is two or three significant digits: near 1.0, the values it can hold are 0.0078 apart.
So an edit computed in full precision is rounded when it is saved. If the change to a weight is less than half the gap to its neighbor, the new value rounds back to the old one. The weight was touched and didn't change. That is the 884.
wsp.to_bfloat16 rounds numbers the way the file format does and hands them back as ordinary floats.
Quick check
Six weights are rounded to bfloat16. Two of them, 1.003 and 1.004, start out 0.001 apart. What does this print?
import numpy as np
import wsp
w = np.array([1.0, 1.003, 1.004, 0.1], dtype=np.float32)
print([float(v) for v in wsp.to_bfloat16(w)])
[1.0, 1.0, 1.0, 0.0]
[1.0, 1.0078125, 1.0078125, 0.10009765625]
[1.0, 1.0, 1.0078125, 0.10009765625]
[1.0, 1.003, 1.004, 0.1]
Show the answer
[1.0, 1.0, 1.0078125, 0.10009765625]
Near 1.0 a bfloat16 can hold 1.0 and 1.0078125 and nothing between. The halfway point is about 1.0039, so 1.003 rounds down to 1.0 and 1.004 rounds up. An edit that moved a weight from 1.0 to 1.003 would vanish when saved. Small numbers keep their fine steps: 0.1 lands within 0.0001 of itself.
Worked example
The tiny model, stored in 16 bits
Round the looping model and the patched model to bfloat16. Count how many weights the rounding moves, how many of the patch's 1,024 entries end up equal to the faulty model's, and whether the fix still works.
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
tok = wsp.build(loop, info)[1]
patch = wsp.load_patch("loop-fix")
fixed = wsp.apply_patch(loop, patch)
loop16, fixed16 = wsp.to_bfloat16(loop), wsp.to_bfloat16(fixed)
moved = sum(int((loop16[name] != loop[name]).sum()) for name in loop)
print(f"rounding moved {moved:,} of {wsp.count(loop):,} weights of the looping model")
defequal_after(a, b):
returnsum(int((a[name].ravel()[t["index"]] == b[name].ravel()[t["index"]]).sum())
for name, t in patch["tensors"].items())
same = equal_after(fixed16, loop16)
print(f"the patch touches {patch['changed']:,} weights; after rounding,",
f"{same} of them equal the faulty model's value")
versions = [("looping, 32-bit", loop), ("looping, bfloat16", loop16),
("patched, 32-bit", fixed), ("patched, bfloat16", fixed16)]
for label, state in versions:
model, _ = wsp.build(state, info)
right = wsp.score(model, tok, 'uses')
ends = sum(wsp.endings(model, tok, 'uses'))
print(f"{label:18s} held-out right {right:2d} of 72 | replies that end {ends}")
In the lesson you can change this and run it.
Rounding to 16 bits moves almost every weight, 40,193 of 40,195, each by less than half a percent. It changes nothing you can measure: the looping model still scores 0 and the patched one 68. And 12 of the patch's 1,024 entries round back to the faulty model's value, the tiny model's version of the real edit's 884 of 131,072. Sixteen bits is a safe place to store an edit of this size. The next step down is not.
In your own words
In your own words: the published edit "touched" 131,072 weights and "changed" 130,188. Which number would you put on a model card, and why?
Write a sentence or two, then compare it with a good answer in the lesson.
Quantize after editing, then measure again
To run on a laptop or a phone, a model is usually quantized further: 8 bits per weight, or 4. You did this in Quantization with one scale per row: divide by the scale, round to a whole number of levels, multiply back. wsp.quantize(state, bits) does that to every weight matrix of a model.
Four bits means 15 levels in a row of weights. An edit that moves a weight by less than half a level disappears from that weight. But your fix isn't one weight. It is a direction spread over 1,024 of them, and some of it can survive when many of the single changes don't.
So there are two questions, and they have different answers. How many of the edited weights can still be told apart from the faulty model's after rounding? And does the model still behave as fixed?
Worked example
The fix at 8, 6, 5, 4 and 3 bits
Quantize the patched model and the faulty model the same way. For each precision: how many of the patch's 1,024 entries are now identical in the two, and what the patched model scores.
import matplotlib.pyplot as plt
import wsp
loop, info = wsp.load_state("tiny-gpt-loop")
tok = wsp.build(loop, info)[1]
patch = wsp.load_patch("loop-fix")
fixed = wsp.apply_patch(loop, patch)
defsame_entries(a, b):
returnsum(int((a[name].ravel()[t["index"]] == b[name].ravel()[t["index"]]).sum())
for name, t in patch["tensors"].items())
scores = []
print(f"{'bits':>4s} {'edited weights lost':>20s} {'held-out right':>15s}",
f"{'replies that end':>17s}")
for bits in (8, 6, 5, 4, 3):
fixed_q, loop_q = wsp.quantize(fixed, bits), wsp.quantize(loop, bits)
lost = same_entries(fixed_q, loop_q)
model, _ = wsp.build(fixed_q, info)
right = wsp.score(model, tok, 'uses')
ends = sum(wsp.endings(model, tok, 'uses'))
print(f"{bits:4d} {lost:15d}/1024 {right:12d}/72 {ends:14d}/72")
scores.append(right)
plt.plot([8, 6, 5, 4, 3], scores, marker="o")
plt.xlabel("bits per weight")
plt.ylabel("held-out questions right of 72")
plt.title("The fix, quantized")
plt.show()
In the lesson you can change this and run it.
At 8 bits, 75 of the edited weights are lost and the fix is whole: 68 of 72. At 4 bits more than half are lost, 538 of 1,024, and the fix still restores 63, with 68 replies ending. At 3 bits, 743 are lost and the score is 38.
So the count of surviving weights is a poor guide to behavior. The fix degrades slowly, then quickly, and the only way to know where a given precision falls is to score the quantized model. The rule for a release: the model you measure is the model you ship. If it ships at 4 bits, the verdict is run at 4 bits.
Quick check
You measured a fix at full precision: 68 of 72. The model will be released at 4 bits. Which score belongs in the release record?
Neither: a quantized model can't be scored, because its weights are no longer exact
The score of the 4-bit model, 63 of 72, measured after quantizing; the other can sit beside it
The full-precision score, 68 of 72, since that is the score of the edit as it was designed
The average of the two, 65.5, since the released model is a mix of both versions
Show the answer
The score of the 4-bit model, 63 of 72, measured after quantizing; the other can sit beside it
People will run the 4-bit file, so the record has to describe the 4-bit file. The full-precision score is worth keeping next to it, because the gap between the two is the cost of quantizing. Reporting only the higher number describes a model nobody downloads.
Drill
What is left of an edit after rounding?
Write survivors(base, edited, rounder). base and edited are arrays of the same shape, and rounder is a function that rounds an array (for storage in fewer bits). Count the entries the edit changed, and how many of those still differ once both arrays are rounded. Return the pair (changed, still_different).
In the lesson, your code is checked as you write it.
In your own words
In your own words: at 4 bits more than half of the edited weights were lost, and the fix mostly held. What does that say about counting changed weights?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: run it for real, and what you can conclude
Run it for real
The practice API can't serve a 1.75 GB model, and the academy's Python runs in your browser. So the last step is a notebook for your own machine. It needs Python, about 4 GB of disk and no GPU.
It downloads the base model from Hugging Face and the edit from the post's logs, checks the base hashes, applies the edit with the one line you wrote in the drill, and checks the result's hashes. Then it compares your rebuilt tensors, bit for bit, with the edited model the authors published. Last, it asks the base and the edited model the same small math problems, and asks again after rounding the weights to 8 and to 4 bits.
The notebook was run once for this lesson, on November 10, 2026, on a MacBook, and its small test was repeated with a longer limit. What they found is stored with the manifest, so the next cell can print it.
Worked example
What one run of the notebook found
Print the checks from the notebook's run, as stored with the published edit, and then its small test of the two models.
import wsp
edit = wsp.published("qwen-loop-edit")
checked = edit["checked"]
print("checked on", checked["date"], "on", checked["on"])
KEYS = ("base_hashes_match", "rebuilt_hashes_match",
"published_model_holds_the_same_bits", "other_tensors_that_differ",
"touched_entries_unchanged_after_rounding",
"share_of_the_change_in_one_direction")
for key in KEYS:
print(f" {key.replace('_', ' ')}: {checked[key]}")
rows = edit["run"]["rows"]
print()
print(f"{'model':22s} {'limit':>6s} {'ran to the limit':>17s}",
f"{'finished and right':>19s}")
for row in rows:
n = row['problems']
print(f"{row['model']:22s} {row['limit']:6,d} {row['hit_limit']:14d}/{n}",
f"{row['finished_right']:16d}/{n}")
for row in rows:
if"changed_entries_still_different"in row:
still = row['changed_entries_still_different']
total = edit['manifest']['changed_entries_total']
print(f"{row['model']}: {still:,} of {total:,} changed entries still differ",
"from the base model rounded the same way")
per_problem = {(row["model"], row["limit"]): row["per_problem"] for row in rows}
for limit in (2048, 8192):
pair = [per_problem[(m, limit)] for m in ("base model", "edited model")]
v = wsp.verdict(*pair)
print(f"base against edited at {limit:,} tokens: {v['better']} better,",
f"{v['worse']} worse, p = {v['p']:.2f}, {v['verdict']}")
In the lesson you can change this and run it.
The files check out completely. The base tensors hash to the manifest's values and so do the rebuilt ones. The model its authors published holds exactly those bits, no other tensor differs, and all of the change lies along one direction.
The small test is another matter. It was eight arithmetic problems with one sampled answer each, at two limits on answer length. At both limits the edited model finished one more problem than the base model, the same problem each time. One better and none worse gives p = 1.00: no evidence. With plain rounding to 8 bits the edited model finished one problem, and at 4 bits none, its text visibly damaged, though 99,806 of the 130,188 changed entries were still there. Here too the count of surviving weights told you little.
Quick check
The post reports 48.1% against 7.9% on 500 problems, four samples each, at up to 32,768 tokens. The notebook's test found 1 problem better and 0 worse out of 8, at up to 8,192 tokens. Do the two results conflict?
Yes: the small test proves that the edit has no effect, since its p-value came out as 1.00
No: a test of eight problems can neither confirm nor contradict the post's measurement
Yes: the small test proves the post right, since the edited model came out one problem ahead
No: the two results are the same result, since both of them show the edited model ahead
Show the answer
No: a test of eight problems can neither confirm nor contradict the post's measurement
A p-value of 1.00 means the small test saw nothing it could tell from chance. That is what a test this size returns whether the true effect is large, small or zero. It also used a quarter of the post's token limit, and the post's base model mostly failed by running past a much longer one. To check the claim you would need the post's evaluation, or one of similar size.
Quick check
From Safety, licenses and provenance: the base model is under Apache 2.0. You publish your quantized copy of the edited model. What does the license ask you to include?
The license and its notices, and a statement of what you changed
Nothing, because a quantized model counts as a new work of your own
A fee to the model's authors, for each copy that is downloaded
The training data of the base model, since the weights were made from it
Show the answer
The license and its notices, and a statement of what you changed
Apache 2.0 allows changing and redistributing, including for sale, on two conditions you met in that lesson: keep the license and notices with the files, and say what you changed. Here that is two changes: the published edit, with its source, and the quantization, with its precision.
What you can and can't conclude
Checked here, by you: the published edit's size and shape from its manifest, how an edit of that form is applied and verified, what 16-bit storage does to an edit, and how a fix on the tiny model fares from 8 bits down to 3.
Checked by one run of the notebook: that the public files do what the post says they do. The hashes match, the edit is one direction removed from 64 columns in each of two matrices, and the edited model on Hugging Face is that edit and nothing else.
Not checked by anyone here: the post's headline numbers. MATH-500 at 48.1% against 7.9% was measured by the authors on 500 problems, four samples each, at up to 32,768 tokens an answer. The notebook's test is eight problems. It can show you what a looping answer looks like. It can't confirm or contradict a benchmark.
That is the normal position for a reader of a new result. You can verify the artifact cheaply. Verifying the claim costs what the original experiment cost.
Where it shows up in your week
When you quantize a model that carries an edit, you rerun the edit's own test at the precision you ship, and you put that score in the release record. A count of changed weights is not a score.
In your own words
In your own words: you verified that the published files match the post's description. Why isn't that the same as verifying the post's result?
Write a sentence or two, then compare it with a good answer in the lesson.
A published edit comes as a manifest with two hashes per tensor and a file of positions and new values. Your copy of the model fails the first hash. What should you do?
Stop: you hold a different model or revision, and the edit's positions don't apply to it
Apply the edit anyway, since the second hash will show whether it worked out
Edit the manifest so that its first hash is the hash of the model you have
Apply the edit twice, since a second pass corrects a mismatch in the first
Answer it at the end of the lesson to see if you're right, and why.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Read a model's weights, change them by hand and judge an edit someone else made: deltas, merging, low-rank edits, directions and fact edits on a tiny GPT in your browser. Then replay recorded sessions of an LLM reading, fixing and writing weights, check the published edit of a real open model, and ship a model of your own with its patch, verdict and card. Every number is printed by the lesson's own code, and the method's limits are taught with it.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write