LLM-as-judge
Grade open-ended output with rubrics and a second model.
Create a free account Try part of it here
Why it matters
Code can check that a reply is valid JSON or under 60 characters. It can't check whether a reply sounds like your brand, answers the customer's actual question, or sticks to the policy it quotes. For those, you can ask a second model to grade the first one against a checklist. Done carelessly, a judge adds confident noise. Done well, it lets you check open-ended writing, such as ad copy, support replies and summaries, on hundreds of examples in minutes.
What you'll be able to do
- Decide which quality checks need a model judge and which code can do
- Write a judge prompt with specific pass/fail criteria and structured output
- Compare two outputs fairly by judging in both orders
- Measure how often the judge agrees with human labels before trusting it
What it covers
- Code first, judge second
- Comparing two versions: judge both orders
- Agreement and the swap test
- Worked example: a brand-voice judge
- Why a 1-to-10 score misleads
- Judge both orders
Code first, judge second
Think of a newspaper. A spell-checker catches typos instantly and for free. An editor decides whether the story is fair and clear. You wouldn't pay the editor to count commas.
Take the shop's brand voice guide. Some of its rules are mechanical, and code checks them for free with no judgment calls:
import re HYPE = ["revolutionary", "game-changing", "epic", "must-have"] copy = "The Epic Rain Jacket! Stay dry on every commute. Waterproof, packs into its own pocket!" problems = [] if copy.count("!") > 1: problems.append(f"{copy.count('!')} exclamation marks (max 1)") problems += [f"hype word: {w}" for w in HYPE if re.search(rf"\b{w}\b", copy, re.IGNORECASE)] print(problems)
It prints ['2 exclamation marks (max 1)', 'hype word: epic']. The pattern \b{w}\b matches the word on its own, not inside a longer word, and re.IGNORECASE ignores capitals.
Other rules need judgment: "lead with what the product does for the customer", or "warm and plainspoken, never pushy". That's where a model judge earns its cost. Use the cheapest grader that can decide each rule, and send only the rest to the judge.
Writing a judge that works
1. Specific rules, each pass or fail. "Rate the quality from 1 to 10" gives numbers that drift between runs and mean nothing. "Does the first sentence say what the product does for the customer? (pass or fail)" is a question two people would answer the same way. Split a vague goal into several questions like that. The list of questions is called a rubric.
2. Give it the reference material. To judge "follows our voice guide" or "matches the return policy", the judge needs the guide or the policy in its prompt, just like a RAG answer does.
3. Reasons before verdicts. Ask for a short reason for each rule along with the pass or fail. Or turn on adaptive thinking. Judges that decide first and explain afterward are less accurate.
4. Structured output. Your code needs to count passes, so make the judge return JSON:
JUDGE_SCHEMA = {
"type": "object",
"properties": {
"criteria": {"type": "array", "items": {
"type": "object",
"properties": {"name": {"type": "string"}, "reason": {"type": "string"},
"verdict": {"type": "string", "enum": ["pass", "fail"]}},
"required": ["name", "reason", "verdict"], "additionalProperties": False}},
},
"required": ["criteria"], "additionalProperties": False,
}
5. A capable model. The judge should be at least as strong as the model it's grading. Grading is often harder than writing.
A judge is a model too, so it needs its own eval.
Show the answer
"Does the reply state the 60-day return window from the policy? Answer pass or fail."
Specific, checkable and pass-or-fail, with the source of truth named.
Comparing two versions: judge both orders
To pick between two prompts, show the judge both outputs and ask which one better meets the rubric. Judges have known biases:
- Position bias: preferring whichever answer is shown first (or second).
- Length bias: preferring longer answers, even when they're worse.
- Self-preference: preferring text written by the same model.
The usual fix for position bias is cheap. Judge twice, swapping the order. Count a win only when the judge picks the same output both times. If its choice flips with the order, call it a tie. To counter length bias, say so in the rubric: "don't reward length; shorter is better when equally complete".
Trust, but check
A judge is a measuring tool, so check it against a known standard before you rely on it:
- Label 30 to 50 outputs yourself, pass or fail on the rubric.
- Run the judge on the same outputs.
- Work out agreement: the share of cases where the judge matches your label.
If agreement is around 0.9, the judge is useful. If it's 0.7, read the disagreements. Usually a rule is vague, or the judge is missing information it needs. Fix the rubric and measure again. Check again whenever you change the judge's model or prompt.
For marketers: this is how you scale a copy review. A judge applies your brand guide to 500 generated product descriptions overnight. You read only the ones it fails, plus a random sample of passes to make sure it's still right.
judge = ["pass", "fail", "pass", "pass"] human = ["pass", "fail", "fail", "pass"] agree = sum(j == h for j, h in zip(judge, human)) print(agree)
Show the answer
3
The two lists match at positions 0, 1 and 3, and differ at position 2. j == h gives True for each match, and sum counts each True as 1, so it prints 3. Measure how often the judge agrees with people before you trust it.
Agreement and the swap test
You labeled 10 replies, and the judge labeled the same 10. Count where they match. Then a comparison: in the first order the judge picked A, and with the order swapped it picked B. It picked whichever was shown first both times.
human = ["pass", "fail", "pass", "pass", "fail", "pass", "fail", "pass", "pass", "pass"] judge = ["pass", "fail", "pass", "fail", "fail", "pass", "pass", "pass", "pass", "pass"] agree = sum(h == j for h, j in zip(human, judge)) print(agree, "of", len(human), "->", agree / len(human)) picks = ["A", "B"] print("A wins" if picks == ["A", "A"] else "B wins" if picks == ["B", "B"] else "tie")
It prints 8 of 10 -> 0.8: the judge disagreed with you on 2 replies, so read those two before trusting it. Then tie: the judge's choice flipped when the order flipped, which is position bias, not a real preference.
Show the answer
A tie: the choice followed the position, not the content
When the pick flips with the order, the judge was choosing a position. Only a pick that holds in both orders counts as a win.
Worked example: a brand-voice judge
The judge gets the guide, the rubric and the copy, and returns one verdict per rule. Without an API key, the simulator returns a placeholder that follows the schema, so the code path is the same.
import json import requests guide = requests.get("https://api.gradient.test/v1/docs/doc_voice").json()["text"] RUBRIC = ["The first sentence says what the product does for the customer", "The tone is warm and plainspoken, not pushy", "Any sustainability claim is specific and modest"] JUDGE_SCHEMA = {"type": "object", "properties": {"criteria": {"type": "array", "items": { "type": "object", "properties": {"name": {"type": "string"}, "reason": {"type": "string"}, "verdict": {"type": "string", "enum": ["pass", "fail"]}}, "required": ["name", "reason", "verdict"], "additionalProperties": False}}}, "required": ["criteria"], "additionalProperties": False} copy = "Stay dry on your bike commute. This shell is eco-friendly and the best jacket ever made, so buy now." rubric = "\n".join(f"{i}. {c}" for i, c in enumerate(RUBRIC, start=1)) response = requests.post("https://api.anthropic.com/v1/messages", headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"}, json={"model": "claude-opus-5-5", "max_tokens": 1000, "system": "You review marketing copy against a brand voice guide. Judge each criterion separately; give a one-sentence reason, then pass or fail.", "messages": [{"role": "user", "content": f"<guide>\n{guide}\n</guide>\n\n<rubric>\n{rubric}\n</rubric>\n\n<copy>\n{copy}\n</copy>"}], "output_config": {"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}}}, timeout=60) response.raise_for_status() verdict = json.loads("".join(b["text"] for b in response.json()["content"] if b["type"] == "text")) print(verdict)
Without an API key, the simulator sends back an empty list, {'criteria': []}. With a real model you'd expect the first rule to pass and the other two to fail. "Buy now" and "best jacket ever made" are pushy. And "eco-friendly" is exactly the vague claim the guide forbids.
Show the answer
Read the disagreements, sharpen the vague rules or give the judge missing information, then measure again
Low agreement usually means a vague rule or missing context. Fix those and re-measure before relying on the judge.
Why a 1-to-10 score misleads
A judge that scores 7 today may score 8 tomorrow on the same text, so a small difference between two versions usually means nothing. Ask a yes-or-no question for each thing you care about instead, and count the yeses over many examples. And never send a model what code can check. Length and banned words cost nothing to test exactly.
Judge both orders
Readers tend to favor whichever option they see first, and models do too. So compare each pair in both orders, and count a win only when the verdict stays the same.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Which brand-guide rule should NOT go to an LLM judge?