Glossary · Phase 10 · Agent harnesses
SWE-bench
A benchmark of real GitHub issues in real repositories, graded by the projects' own tests.
Try it now
When a person can retry, pass@k is what matters. An unattended agent needs pass^k, which falls fast.
Learn it properly
Evaluation harnesses · Run agents on tasks in fresh environments and grade what they leave behind. About 40 minutes.
Other terms from the same lesson
- evaluation harness: Code that runs a model or agent across a fixed set of tasks, in clean environments, and scores the results.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.