Free prompt engineering course
Learn to write prompts that work reliably, then test them like an engineer: clear instructions, examples, structured output, chains, evals and defenses against prompt injection.
Change one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.
Most of an engineer's work on an LLM feature is not the first prompt. It is the tenth: change a sentence, rerun the eval, see what moved. Done by feel, that loop drifts: you fix the review you were looking at and break three you were not, and the score you celebrate was measured on the cases you tuned for. This lesson does the loop on the practice reviews with rules that stop the drift: one change at a time, a held-back set you do not tune on, a look at every case that flipped, and an honest estimate of how much noise a small set carries.
Which cases flipped, and whether any label got worse
A higher total can hide a label that got worse, and 3 more right out of 20 can be noise. Read the cases that flipped and every label's recall before you decide.
This lesson rests on Evals, which gave you a labeled set, accuracy and recall for each label, and on Samples, the law of large numbers and the central limit theorem, which says how far a score measured on a small sample can be trusted.
Improving a prompt is a loop, and the loop has rules.
A win that breaks a label is not a win. A prompt that says "use mixed when the review has praise and a complaint" fixes the mixed reviews and may drag mild-complaint positives down with them.
truth = ["mixed", "mixed", "positive", "mixed"] got = ["positive", "mixed", "positive", "mixed"] found = sum(1 for t, g in zip(truth, got) if t == "mixed" and g == "mixed") print(found, truth.count("mixed"), round(found / truth.count("mixed"), 2))
2 3 0.67
Three reviews really are mixed, and the prompt got two of them. Recall for mixed is 2 of 3. Accuracy would be 3 of 4 and would hide that one mixed review was missed.
The picture. Three prompt versions were run over the 20 practice reviews. Each row is a review, each column a version, and a cell is dark when the version got that review right. The first twelve reviews are the development set and the last eight are held back. The outlined cells in the middle column are the cases version 2 changed against version 1: five fixed, two broken.
Reviews 1 to 12 are the development set; 13 to 20 are held back. The outlined cells are what version 2 changed.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Score each version on both sets, list what a change fixed and broke, and see where the held-back score stops rising.
The lesson's setup loads the 20 practice reviews, whose labels are the answers, and three stand-in runs written by hand to show the loop. They are not model output: on your own machine you would score three real runs of three real prompts the same way. First, accuracy on the development set, the held-back set and all 20.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import requests reviews = requests.get("https://api.gradient.test/v1/reviews").json()["data"] WORDS = {"P": "positive", "N": "negative", "M": "mixed"} # Stand-ins for three recorded runs of three prompt versions: one label per review. RUNS = { "v1": "PNPPNPPPNPNPNPPPPPNP", "v2": "PNPMNMPMNPNPNMPMMPNM", "v3": "PNPMNPPMNPNPNMPPMPNP", } truth = [r["label"] for r in reviews] runs = {name: [WORDS[c] for c in code] for name, code in RUNS.items()} DEV, HELD = range(0, 12), range(12, 20)
def accuracy(got, want, idx): return sum(got[i] == want[i] for i in idx) / len(idx) for name, got in runs.items(): dev = accuracy(got, truth, DEV) held = accuracy(got, truth, HELD) total = accuracy(got, truth, range(20)) print(name, round(dev, 3), round(held, 3), round(total, 3))
Version 1 scores 0.833 on development and 0.625 held back; version 2 scores 0.917 and 0.875; version 3 scores 1.0 and 0.875. Version 3 looks perfect on the cases it was tuned on and no better than version 2 on the held-back ones. That flat step is the warning.
The extra gains were tuned to the development cases
A rise on the cases you studied that does not carry over to cases you did not is the sign of tuning to the set. The held-back score is the honest one.
Accuracy is one number. Recall for each label shows what a change did underneath it.
Uses the setup above, 13 lines.
def recall(got, want, label): hits = [g == label for g, w in zip(got, want) if w == label] return sum(hits) / len(hits) for name, got in runs.items(): scores = {lab: round(recall(got, truth, lab), 2) for lab in WORDS.values()} print(name, scores)
Version 1 never finds a mixed review (recall 0.0). Version 2 finds all of them (1.0) but its recall for positive falls to 0.78: it now calls two positive reviews mixed. Version 3 restores positive to 1.0 and gives back one mixed review (0.8). Each change fixed something and cost something, which the total alone would not show.
The same comparison on the real versions lists exactly which reviews moved, and puts a number on how far the gap can be trusted.
Uses the setup above, 13 lines.
import math def flips(a, b, want): cases = range(len(want)) fixed = [i for i in cases if a[i] != want[i] and b[i] == want[i]] broke = [i for i in cases if a[i] == want[i] and b[i] != want[i]] return fixed, broke fixed, broke = flips(runs["v1"], runs["v2"], truth) print("fixed", [reviews[i]["id"] for i in fixed]) print("broke", [reviews[i]["id"] for i in broke]) for i in broke: print(" ", reviews[i]["text"], "| v2 said", runs["v2"][i]) b, c, n = len(fixed), len(broke), len(truth) gap = (b - c) / n se = math.sqrt(b + c - (b - c) ** 2 / n) / n print(round(gap, 3), round(se, 3), round(gap / se, 2))
Version 2 fixed five reviews (4, 8, 14, 17 and 20) and broke two (6 and 16), both positives with a mild complaint ("slightly smaller", "runs a little large"). The gap is 0.15 and its standard error is 0.128, so the gap is only 1.17 standard errors from zero. With 20 reviews you cannot yet say version 2 is better than chance; paired evals shows how to test it, and more cases shrink the error.
0.7
9 + 5 = 14 right out of 20, and 14 ÷ 20 = 0.7. Averaging 0.75 and 0.625 would give 0.6875, because the sets have different sizes.
Promising, but within noise: collect more cases
A gap about one standard error from zero can come from chance alone. The fix is more cases, not a stronger claim.
Write scores(got, want, split): split is the number of reviews in the development set at the start. Return a tuple of the accuracy on the first split cases and the accuracy on the rest, each rounded to 3 places.
Uses the setup above, 13 lines.
def scores(got, want, split): right = sum(g == w for g, w in zip(got, want)) return right / len(want), right / len(want) print(scores(["a", "b", "a", "a"], ["a", "b", "b", "a"], 2))
A total can hide a label that got worse. Write recall_by_label(got, want): return a dictionary from each label that appears in want to its recall, the share of that label's true cases that got labeled correctly, rounded to 2 places, with labels in alphabetical order.
Uses the setup above, 13 lines.
def recall_by_label(got, want): right = sum(g == w for g, w in zip(got, want)) return {"all": round(right / len(want), 2)} print(recall_by_label(["a", "b", "b", "a"], ["a", "b", "a", "a"]))
Last, draw the score by round for both sets, so the stall is visible.
Uses the setup above, 13 lines.
import matplotlib.pyplot as plt def accuracy(got, want, idx): return sum(got[i] == want[i] for i in idx) / len(idx) names = list(runs) dev = [accuracy(runs[n], truth, DEV) for n in names] held = [accuracy(runs[n], truth, HELD) for n in names] plt.plot(names, dev, marker="o", label="development") plt.plot(names, held, marker="s", label="held back") plt.ylabel("accuracy") plt.legend() plt.show()
The development line keeps rising to 1.0. The held-back line rises from 0.625 to 0.875 and then goes flat between versions 2 and 3. When the two lines part, stop tuning on the development cases: they have started to teach you the set, not the task.
old = ["P", "M", "N", "M"] new = ["P", "P", "N", "M"] want = ["P", "M", "N", "M"] fixed = sum(1 for o, n, w in zip(old, new, want) if o != w and n == w) broke = sum(1 for o, n, w in zip(old, new, want) if o == w and n != w) print(fixed, broke)
0 1
At position 1 the old run was right (M) and the new run says P, so that case broke. No case went from wrong to right, so fixed is 0.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Put the loop where the team can see it: prompts in version control, an eval in CI, and a pull request that shows the evidence.
In a real repository the prompt is a file, and a change to it is a pull request like any code change. What belongs in that pull request:
An eval that runs on every pull request, with a pass mark, turns that into a gate. The card shows one as a GitHub Actions workflow.
Write a sentence or two, then compare it with a good answer in the lesson.
Why keep a held-back set you never tune on?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Learn to write prompts that work reliably, then test them like an engineer: clear instructions, examples, structured output, chains, evals and defenses against prompt injection.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write