Evaluation harnesses

Estimating pass@k

Run n trials, and c pass. The tempting estimate of pass@k, 1 - (1 - c/n) ** k, is biased. Think of your trials as marbles in a bag, with the passing ones green. You pick k marbles, and you fail only if none of them is green. The standard estimator, from the HumanEval paper, counts the ways to pick k of the n trials that are all failures:

pass@k = 1 - C(n - c, k) / C(n, k)

where C is "choose" (math.comb). If fewer than k trials failed, it's 1.

from math import comb

n, c, k = 5, 2, 3

tempting = 1 - (1 - c / n) ** k
unbiased = 1 - comb(n - c, k) / comb(n, k)

print("tempting:", round(tempting, 3))
print("unbiased:", round(unbiased, 3))

Reward hacking

Tell someone the floor only has to look clean, and the dirt may end up under the rug. Agents optimize for what the grader checks, sometimes in ways you didn't intend. Coding agents have been seen deleting failing tests, special-casing test inputs, or editing the grader. Defenses:

• Keep grading tests hidden from the agent, or restore them before grading.
• Mark some files protected; any change to them fails the trial.
• Read transcripts of passing trials now and then, not only failing ones.

def honest_add(a, b):
    return a + b

def cheating_add(a, b):
    if (a, b) == (2, 3):      # the one input in the visible test
        return 5
    return 0

visible = [((2, 3), 5)]
hidden = [((10, 4), 14), ((0, 0), 0), ((-1, 1), 0)]

def score(fn, tests):
    return sum(fn(*args) == want for args, want in tests), len(tests)

for fn in [honest_add, cheating_add]:
    print(fn.__name__, "visible:", score(fn, visible), "hidden:", score(fn, hidden))

Use it

Track cost and turns as well as success. A car that's 2% faster and burns twice the fuel may not be an upgrade, and a change that adds 2% success but doubles cost may not be worth it. Keep a small, fast suite that runs on every change to the harness, like unit tests, and a large one that runs before releases.

from math import comb

n, c, k = 5, 2, 2
print(round(1 - comb(n - c, k) / comb(n, k), 2))

from math import comb

p = 0.6
for k in (1, 3, 5):
    print(f"k={k}: pass@k={1 - (1 - p) ** k:.2f}  pass^k={p ** k:.2f}")

n, c, k = 10, 6, 3
print("unbiased pass@3 from 6 of 10:", round(1 - comb(n - c, k) / comb(n, k), 3))

One run proves very little

Agents vary from run to run. A task that passes once might pass 6 times in 10. So a single run can mislead you in either direction, and a jump from 60% to 80% on a few tasks may be nothing but noise. Run each task several times, and compare the runs with some statistics before you believe a change helped.

Grade the result, not the route

It's tempting to check that the agent took the steps you expected: read this file, then call that tool. But a good agent might solve the problem another way, and it would fail your check for being clever. Worse, an agent can take all the right steps and still leave a broken result. Check what's true at the end instead. Do the tests pass? Is the file correct? Is the record in the database?

Be suspicious of big jumps

A student who goes from a D to an A overnight deserves a second look. When a pass rate leaps overnight, look before you celebrate. Check whether the agent found a way to game the grader, or whether a task leaked its answer. Read a handful of transcripts from runs that now pass.

Pass rate isn't the whole story

Two versions with the same pass rate aren't equal if one takes 40 turns and $3 a task, and the other takes 8 turns and 40 cents. Report cost, turns and time beside the pass rate, so that a change that wins on one and loses badly on another doesn't slip through.