Index Ai Academy
Phase 1 · Prompting

Improving a prompt against an eval

Change one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.

About 30 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Most of an engineer's work on an LLM feature is not the first prompt. It is the tenth: change a sentence, rerun the eval, see what moved. Done by feel, that loop drifts: you fix the review you were looking at and break three you were not, and the score you celebrate was measured on the cases you tuned for. This lesson does the loop on the practice reviews with rules that stop the drift: one change at a time, a held-back set you do not tune on, a look at every case that flipped, and an honest estimate of how much noise a small set carries.

What you'll be able to do

  • Run the prompt-improvement loop: one change, rerun, compare, read what flipped, keep or revert
  • Score each prompt version on a development set and a held-back set, and see when the held-back score stops rising
  • Find the cases a change fixed and the cases it broke, and read recall for every label, not only accuracy
  • Say how far a gap on 20 cases can be trusted, using the standard error of the gap
Quick check
Your new prompt takes accuracy on 20 test reviews from 0.75 to 0.90. What do you check before you ship it?
  1. Which cases flipped, and whether any label got worse
  2. Only whether the new prompt is shorter
  3. Whether the reviews were easy
  4. Nothing: 0.90 is simply higher than 0.75
Show the answer

Which cases flipped, and whether any label got worse

A higher total can hide a label that got worse, and 3 more right out of 20 can be noise. Read the cases that flipped and every label's recall before you decide.

The idea: change one thing, then compare

This lesson rests on Evals, which gave you a labeled set, accuracy and recall for each label, and on Samples, the law of large numbers and the central limit theorem, which says how far a score measured on a small sample can be trusted.

Improving a prompt is a loop, and the loop has rules.

  1. Split your cases. Keep a development set you may study and tune on, and a held-back set you only score on. If you tune against every case, your score measures how well the prompt memorized them.
  2. Change one thing per round. A new sentence, a new example, a different format. If you change three things and the score moves, you do not know which did it.
  3. Rerun every case and compare. Look at the total, then at recall for each label, then at the cases that flipped: wrong to right (fixed) and right to wrong (broke).
  4. Read the failures. The cases still wrong tell you what to change next.
  5. Keep or revert. Keep a change only if it helps and breaks no label. Otherwise revert it and try another.

A win that breaks a label is not a win. A prompt that says "use mixed when the review has praise and a complaint" fixes the mixed reviews and may drag mild-complaint positives down with them.

Quick check
From the lesson on evals: recall for a label is the share of its true examples that were found. For the label mixed, what does this print?
truth = ["mixed", "mixed", "positive", "mixed"]
got = ["positive", "mixed", "positive", "mixed"]
found = sum(1 for t, g in zip(truth, got) if t == "mixed" and g == "mixed")
print(found, truth.count("mixed"), round(found / truth.count("mixed"), 2))
  1. 2 4 0.5
  2. 2 3 0.67
  3. 3 3 1.0
  4. 2 2 1.0
Show the answer

2 3 0.67

Three reviews really are mixed, and the prompt got two of them. Recall for mixed is 2 of 3. Accuracy would be 3 of 4 and would hide that one mixed review was missed.

The picture. Three prompt versions were run over the 20 practice reviews. Each row is a review, each column a version, and a cell is dark when the version got that review right. The first twelve reviews are the development set and the last eight are held back. The outlined cells in the middle column are the cases version 2 changed against version 1: five fixed, two broken.

Try it now

Reviews 1 to 12 are the development set; 13 to 20 are held back. The outlined cells are what version 2 changed.

In your own words
In your own words: why keep a held-back set, and why look at the cases that flipped?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: score the versions and read what flipped

Purpose. Score each version on both sets, list what a change fixed and broke, and see where the held-back score stops rising.

The lesson's setup loads the 20 practice reviews, whose labels are the answers, and three stand-in runs written by hand to show the loop. They are not model output: on your own machine you would score three real runs of three real prompts the same way. First, accuracy on the development set, the held-back set and all 20.

Setup (runs before every cell) · 13 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import requests

reviews = requests.get("https://api.gradient.test/v1/reviews").json()["data"]
WORDS = {"P": "positive", "N": "negative", "M": "mixed"}
# Stand-ins for three recorded runs of three prompt versions: one label per review.
RUNS = {
    "v1": "PNPPNPPPNPNPNPPPPPNP",
    "v2": "PNPMNMPMNPNPNMPMMPNM",
    "v3": "PNPMNPPMNPNPNMPPMPNP",
}
truth = [r["label"] for r in reviews]
runs = {name: [WORDS[c] for c in code] for name, code in RUNS.items()}
DEV, HELD = range(0, 12), range(12, 20)
def accuracy(got, want, idx):
    return sum(got[i] == want[i] for i in idx) / len(idx)

for name, got in runs.items():
    dev = accuracy(got, truth, DEV)
    held = accuracy(got, truth, HELD)
    total = accuracy(got, truth, range(20))
    print(name, round(dev, 3), round(held, 3), round(total, 3))

Version 1 scores 0.833 on development and 0.625 held back; version 2 scores 0.917 and 0.875; version 3 scores 1.0 and 0.875. Version 3 looks perfect on the cases it was tuned on and no better than version 2 on the held-back ones. That flat step is the warning.

Quick check
From the table: version 3's development accuracy is 1.0 and its held-back accuracy is 0.875, the same as version 2's. What does that suggest?
  1. The extra gains were tuned to the development cases
  2. Version 3 is worse than version 2 held back
  3. The held-back set is wrong
  4. Version 3 is perfect
Show the answer

The extra gains were tuned to the development cases

A rise on the cases you studied that does not carry over to cases you did not is the sign of tuning to the set. The held-back score is the honest one.

Accuracy is one number. Recall for each label shows what a change did underneath it.

Uses the setup above, 13 lines.

def recall(got, want, label):
    hits = [g == label for g, w in zip(got, want) if w == label]
    return sum(hits) / len(hits)

for name, got in runs.items():
    scores = {lab: round(recall(got, truth, lab), 2) for lab in WORDS.values()}
    print(name, scores)

Version 1 never finds a mixed review (recall 0.0). Version 2 finds all of them (1.0) but its recall for positive falls to 0.78: it now calls two positive reviews mixed. Version 3 restores positive to 1.0 and gives back one mixed review (0.8). Each change fixed something and cost something, which the total alone would not show.

Worked example

Worked example: what version 2 fixed and broke

The same comparison on the real versions lists exactly which reviews moved, and puts a number on how far the gap can be trusted.

Uses the setup above, 13 lines.

import math

def flips(a, b, want):
    cases = range(len(want))
    fixed = [i for i in cases if a[i] != want[i] and b[i] == want[i]]
    broke = [i for i in cases if a[i] == want[i] and b[i] != want[i]]
    return fixed, broke

fixed, broke = flips(runs["v1"], runs["v2"], truth)
print("fixed", [reviews[i]["id"] for i in fixed])
print("broke", [reviews[i]["id"] for i in broke])
for i in broke:
    print(" ", reviews[i]["text"], "| v2 said", runs["v2"][i])
b, c, n = len(fixed), len(broke), len(truth)
gap = (b - c) / n
se = math.sqrt(b + c - (b - c) ** 2 / n) / n
print(round(gap, 3), round(se, 3), round(gap / se, 2))
In the lesson you can change this and run it.

Version 2 fixed five reviews (4, 8, 14, 17 and 20) and broke two (6 and 16), both positives with a mild complaint ("slightly smaller", "runs a little large"). The gap is 0.15 and its standard error is 0.128, so the gap is only 1.17 standard errors from zero. With 20 reviews you cannot yet say version 2 is better than chance; paired evals shows how to test it, and more cases shrink the error.

Quick check
A run gets 9 of the 12 development reviews and 5 of the 8 held-back reviews right. What is its accuracy on all 20?
  1. 0.14
  2. 0.75
  3. 0.7
  4. 0.5
Show the answer

0.7

9 + 5 = 14 right out of 20, and 14 ÷ 20 = 0.7. Averaging 0.75 and 0.625 would give 0.6875, because the sets have different sizes.

Quick check
The gap between two versions is 0.15 and its standard error is 0.128. What is the fair reading?
  1. Version 2 is certainly worse
  2. Version 2 is certainly better, so ship it now
  3. Promising, but within noise: collect more cases
  4. The standard error is the gap
Show the answer

Promising, but within noise: collect more cases

A gap about one standard error from zero can come from chance alone. The fix is more cases, not a stronger claim.

Drill

Score on both sets

Write scores(got, want, split): split is the number of reviews in the development set at the start. Return a tuple of the accuracy on the first split cases and the accuracy on the rest, each rounded to 3 places.

Uses the setup above, 13 lines.

def scores(got, want, split):
    right = sum(g == w for g, w in zip(got, want))
    return right / len(want), right / len(want)


print(scores(["a", "b", "a", "a"], ["a", "b", "b", "a"], 2))
In the lesson, your code is checked as you write it.
Drill

Recall for every label

A total can hide a label that got worse. Write recall_by_label(got, want): return a dictionary from each label that appears in want to its recall, the share of that label's true cases that got labeled correctly, rounded to 2 places, with labels in alphabetical order.

Uses the setup above, 13 lines.

def recall_by_label(got, want):
    right = sum(g == w for g, w in zip(got, want))
    return {"all": round(right / len(want), 2)}


print(recall_by_label(["a", "b", "b", "a"], ["a", "b", "a", "a"]))
In the lesson, your code is checked as you write it.

Last, draw the score by round for both sets, so the stall is visible.

Uses the setup above, 13 lines.

import matplotlib.pyplot as plt

def accuracy(got, want, idx):
    return sum(got[i] == want[i] for i in idx) / len(idx)

names = list(runs)
dev = [accuracy(runs[n], truth, DEV) for n in names]
held = [accuracy(runs[n], truth, HELD) for n in names]
plt.plot(names, dev, marker="o", label="development")
plt.plot(names, held, marker="s", label="held back")
plt.ylabel("accuracy")
plt.legend()
plt.show()

The development line keeps rising to 1.0. The held-back line rises from 0.625 to 0.875 and then goes flat between versions 2 and 3. When the two lines part, stop tuning on the development cases: they have started to teach you the set, not the task.

Quick check
What does this print?
old = ["P", "M", "N", "M"]
new = ["P", "P", "N", "M"]
want = ["P", "M", "N", "M"]
fixed = sum(1 for o, n, w in zip(old, new, want) if o != w and n == w)
broke = sum(1 for o, n, w in zip(old, new, want) if o == w and n != w)
print(fixed, broke)
  1. 1 0
  2. 1 1
  3. 0 0
  4. 0 1
Show the answer

0 1

At position 1 the old run was right (M) and the new run says P, so that case broke. No case went from wrong to right, so fixed is 0.

In your own words
Version 2 raises accuracy and drags positive recall down. What do you do next, and why not just ship it?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: the loop in a pull request

Purpose. Put the loop where the team can see it: prompts in version control, an eval in CI, and a pull request that shows the evidence.

In a real repository the prompt is a file, and a change to it is a pull request like any code change. What belongs in that pull request:

  • The table. Accuracy on the development and held-back sets for the old and new prompt, and recall for each label.
  • Five failures. Cases the new prompt still gets wrong, copied in, so the reviewer sees what kind.
  • What flipped. The cases fixed and the cases broken.
  • The cost per run. Tokens and dollars for one pass over the set, so a bigger prompt shows its price.

An eval that runs on every pull request, with a pass mark, turns that into a gate. The card shows one as a GitHub Actions workflow.

In your own words
What goes in the pull request when a teammate changes a prompt?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why keep a held-back set you never tune on?

  1. To test a prompt on cases it was not shaped around
  2. To make the development set bigger
  3. To tune the prompt twice as fast
  4. To make the prompt longer

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 1 · about 23 hours of study

Building with LLMs

Prompting, retrieval, tools, agents and evals 35 lessons 1 free
  1. What an LLM actually does FreeNext-token prediction, and why that produces useful behavior.25 min
  2. Tokens and the context windowHow text becomes tokens, and what fits in one request.25 min
  3. Prompt fundamentalsClear instructions, context and constraints: the 80% of prompting.35 min
  4. System prompts and rolesSet behavior once, separately from the conversation.30 min
  5. Examples in promptsShow the model what good looks like (few-shot prompting).30 min
  6. Structured outputGet reliable JSON back that your code can use.38 min
  7. EvalsMeasure output quality with test sets instead of vibes.35 min
  8. Improving a prompt against an evalChange one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.30 min
  9. Prompt chainingBreak big tasks into a pipeline of smaller model calls.35 min
  10. Choosing a modelTrade capability, speed and cost across model families.25 min
  11. Reasoning models and effortWhen models think before answering, and how to control it.35 min
  12. Streaming responsesShow output as it's generated instead of waiting.35 min
  13. Conversations and memoryAPIs are stateless: how chat history and memory really work.36 min
  14. Images and documentsSend screenshots, PDFs and charts as input.30 min
  15. EmbeddingsMeaning as a list of numbers, so text can be compared.35 min
  16. Similarity searchFind the most relevant text by comparing embeddings.35 min
  17. Chunking documentsSplit long documents so the right piece can be retrieved.30 min
  18. Vector databasesStore and search millions of embeddings quickly.40 min
  19. Retrieval-augmented generationAnswer from your own documents instead of the model's memory.30 min
  20. Measuring retrievalA labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.30 min
  21. Making RAG goodHybrid search, reranking and citations.30 min
  22. Hallucinations and groundingWhy models make things up, and how to design against it.30 min
  23. Tool useLet the model call your functions (function calling).30 min
  24. The agent loopModel → tool → result → model, until the task is done.30 min
  25. Model Context ProtocolA standard way to plug tools and data into any AI app.30 min
  26. Agent design patternsRouting, orchestrator-workers and when not to build an agent.35 min
  27. LLMs in automationsPut models inside scripts and workflow tools that run on their own.35 min
  28. Prompt injection and securityHow untrusted text hijacks models, and how to limit the damage.35 min
  29. LLM-as-judgeGrade open-ended output with rubrics and a second model.35 min
  30. BLEU, ROUGE and BERTScoreWhat the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.35 min
  31. Cost and latencyPrompt caching, batching and routing to cheaper models.35 min
  32. Logging and monitoringSee what your LLM app actually did in production.30 min
  33. Shipping an LLM appDeploy, secure keys, rate-limit and handle failures.35 min
  34. One app, several model providersTwo API formats side by side, adapters to one internal format, a gateway with fallbacks, and evals before switching.40 min
  35. Capstone: research agentA marketing research agent with retrieval, tools and evals.45 min
Part of a free course 15 lessons

Free prompt engineering course

Learn to write prompts that work reliably, then test them like an engineer: clear instructions, examples, structured output, chains, evals and defenses against prompt injection.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write