Headline metric
The one number you agree in advance to judge a model by, so you can't pick whichever number looks best afterwards.
What it means
A headline metric is the one number a team agrees on in advance to judge a change. In business reports the phrase means much the same thing: the number that leads the report, such as revenue.
Agreeing on it beforehand is the point. If you choose after seeing the results, some number will always have improved, and you'll be tempted to lead with it.
An example: a fine-tuned model
before = {"violations per answer": 1.57, "names the wrong product": 0.42}
after = {"violations per answer": 1.11, "names the wrong product": 0.51}
for metric in before:
print(f"{metric:<24} {before[metric]:.2f} -> {after[metric]:.2f}")
headline_better = after["violations per answer"] <= 0.8 * before["violations per answer"]
nothing_worse = after["names the wrong product"] - before["names the wrong product"] <= 0.05
print("headline metric improved:", headline_better)
print("safe to ship:", headline_better and nothing_worse)
violations per answer 1.57 -> 1.11 names the wrong product 0.42 -> 0.51 headline metric improved: True safe to ship: False
Violations per answer fell by 29%, so the headline metric says this is a success. But the same model now names the wrong product in 51% of answers, up from 42%. A team that watched only the headline would have shipped something worse.
Headline metrics and guardrail metrics
That's why a headline metric rarely travels alone. Guardrail metrics are the numbers that must not get worse, and they're set in advance too. You can follow this example in full, with real training runs, in fine-tuning an LLM and deciding whether to ship it.
Try it now
A release decision is a test you set up in advance. That's what makes it honest.
Learn it properly
Capstone: a tuned, evaluated assistant · Fine-tune, evaluate and deploy a specialist assistant. About 45 minutes.
Other terms from the same lesson
- release gate: A test a model must pass before it ships, with the pass mark written down in advance.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.