Evals

A first eval: review sentiment

The practice API has 20 product reviews. A person has labeled each one positive, negative or mixed. That's a ready-made dataset:

import requests

reviews = requests.get("https://api.gradient.test/v1/reviews").json()["data"]
labels = [r["label"] for r in reviews]
print(len(reviews), "reviews")
for label in ["positive", "negative", "mixed"]:
    print(label, labels.count(label))
print(reviews[3]["text"], "->", reviews[3]["label"])

Before you test any model, work out the baseline. That's the score of a classifier that always gives the most common label. Here the most common label is "positive". It's right 9 times out of 20, so the baseline is 9 ÷ 20 = 0.45.

Why bother? Say a prompt scores 0.50. That sounds fine, until you notice it barely beats one line of code that never even reads the review.

predicted = ["pos", "neg", "pos", "pos"]
actual = ["pos", "neg", "neg", "pos"]
right = sum(p == a for p, a in zip(predicted, actual))
print(right / len(actual))

Accuracy hides things

Accuracy is the share of answers that are correct. It's a good headline number, but it hides where the mistakes are.

Suppose a classifier only ever says positive or negative. It never says mixed. Its accuracy can still look respectable while it gets every mixed review wrong.

Recall by label shows this straight away. For each label, it's the share of that label's examples the classifier got right:

labels      = ["positive", "positive", "negative", "mixed", "mixed", "negative"]
predictions = ["positive", "positive", "negative", "positive", "negative", "negative"]

correct = sum(p == l for p, l in zip(predictions, labels))
print("accuracy:", round(correct / len(labels), 2))
for label in ["positive", "negative", "mixed"]:
    pairs = [(p, l) for p, l in zip(predictions, labels) if l == label]
    print(label, "recall:", sum(p == l for p, l in pairs) / len(pairs))

Read the output. Accuracy is 0.67, which looks okay. But recall for mixed is 0.0: the classifier never got a single mixed review right. Positive and negative are both 1.0.

import json
import requests

SCHEMA = {
    "type": "object",
    "properties": {"label": {"type": "string", "enum": ["positive", "negative", "mixed"]}},
    "required": ["label"],
    "additionalProperties": False,
}

def classify(text):
    response = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-haiku-5-5", "max_tokens": 50,
              "system": "Classify the sentiment of a product review. Use mixed when it has clear praise and a clear complaint.",
              "messages": [{"role": "user", "content": text}],
              "output_config": {"format": {"type": "json_schema", "schema": SCHEMA}}},
        timeout=60)
    response.raise_for_status()
    reply = "".join(b["text"] for b in response.json()["content"] if b["type"] == "text")
    return json.loads(reply)["label"]

reviews = requests.get("https://api.gradient.test/v1/reviews").json()["data"][:5]
for r in reviews:
    got = classify(r["text"])
    print("ok " if got == r["label"] else "BAD", r["id"], "expected", r["label"], "got", got)

Using evals to make decisions

Once your eval runs in a few seconds, use it for every decision:

• Prompt changes. Run it before and after. Keep the change only if the score goes up and no label gets worse.
• Model changes. Try the cheaper model. If it matches the expensive one on your eval, you've cut costs with evidence instead of hope.
• Catching breakage. Run the eval automatically before each release. If the score drops below a set level, stop the release.

Two habits keep evals honest.

Look at the failures, not only the score. Reading ten wrong answers usually tells you exactly what to fix.

Don't tune on your whole dataset. If you keep editing the prompt until every example passes, you've memorized the test. Keep a few examples aside. Check them only at the end.

For marketers: this is a holdout group. You wouldn't judge a campaign on the same audience you used to pick the creative.

Don't tune until everything passes

If you edit a prompt until every example passes, you've only learned to pass those examples. Keep some back, so you can check on cases the prompt has never seen.