BLEU, ROUGE and BERTScore
What the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.
What it covers9
Why it matters
When a model writes a sentence and you have a reference to compare it with, three scores appear in almost every paper and leaderboard: BLEU, ROUGE and BERTScore. They are cheap, automatic and widely misread. This lesson builds each one in a few lines, scores the same five candidates against one reference, and shows exactly what each rewards: shared n-grams, shared subsequences, or similar token embeddings.
You will see a correct paraphrase score zero, a sentence that says the opposite score 0.735, and a shuffled sentence get a perfect ROUGE-1. You will see what BERTScore fixes and what it cannot when its embeddings are weak. Then the practical question: when an overlap metric is the right tool, when a task grader is, and when the judge from the LLM-as-judge lesson is.
What you'll be able to do
- Compute BLEU (clipped n-gram precision with a brevity penalty) and ROUGE-1, ROUGE-2 and ROUGE-L in code
- Show three ways word overlap misleads: a paraphrase, a negation and a shuffled sentence
- Compute a miniature BERTScore from token embeddings and say what it fixes and what it inherits
- Choose between an overlap metric, a task grader and an LLM judge for a given evaluation
Show the answer
The candidate's n-grams that also appear in the reference, each count capped at how often the reference has it
BLEU is a precision: of the candidate's n-grams, how many are in the reference, with clipping so that repeating a reference word cannot inflate the score.
The idea: three ways to compare two sentences
This lesson rests on Logarithms and Sequences and Σ: BLEU averages four natural logs, and a sum of terms is a Σ. Scores below use F1, the F-measure: from a precision and a recall it is , an average that stays close to the smaller of the two.
Counting shared n-grams: BLEU
BLEU scores a candidate against a reference by precision: of the candidate's n-grams, how many also appear in the reference? It does that for n = 1, 2, 3 and 4 and combines the four with a geometric mean, so a candidate needs shared words and shared word order to score. Two details make it fair. Clipping: a candidate n-gram counts at most as often as the reference contains it, so "the the the the" cannot score four hits on a reference with one "the". A brevity penalty: precision alone rewards saying little, so a candidate shorter than the reference is multiplied by , where and are the reference and candidate lengths.
import math mean_log = (math.log(2) + math.log(8)) / 2 print(round(math.exp(mean_log), 3))
Show the answer
4.0
The mean of the two logs is the log of the geometric mean, so exp undoes it: the square root of 2 × 8 is 4. The arithmetic mean would be 5.
Here is the clipped precision for n-grams of length n, and is the natural logarithm (see logarithms). Averaging the four logs and raising to the result is the geometric mean: multiply the four precisions and take the fourth root.
Because the mean is geometric, a single of zero makes the whole score zero: a sentence with no shared 4-gram scores 0 however many words it shares. That is the harsh original; reported BLEU is usually computed over a whole test set, or smoothed.
Part 2 scores five candidates against one reference: an exact copy, a correct paraphrase, a negation, the reference shuffled, and a correct but short version.
Two more ways: ROUGE and BERTScore
ROUGE asks the opposite question to BLEU: did the candidate cover the reference? ROUGE-N is recall of the reference's n-grams, and ROUGE-L uses the longest common subsequence, the longest run of words that appear in both sentences in order, not necessarily side by side. BERTScore replaces exact word matches with the cosine similarity of token embeddings: each token takes its best match in the other sentence, so "light" can match "lightweight". The picture shows its matrix.
The outlined cell is each row's best match. Their average is the precision.
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: five candidates, three metrics
Purpose. Compute each score from its definition and read what it rewards.
Start with BLEU on five candidates for one reference. The cell prints the BLEU of each and two diagnostics.
Five candidates, one reference, under BLEU
Clipped n-gram precision for n = 1 to 4, geometric mean, brevity penalty. Then two diagnostics: the shuffled candidate's unigram precision and the short candidate's brevity penalty.
import math from collections import Counter def ngrams(tokens, n): return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)) def bleu(candidate, reference, max_n=4): c, r = candidate.split(), reference.split() logs = [] for n in range(1, max_n + 1): cn, rn = ngrams(c, n), ngrams(r, n) # a hit counts at most as often as the reference has it clipped = sum(min(count, rn[g]) for g, count in cn.items()) if clipped == 0: return 0.0 logs.append(math.log(clipped / sum(cn.values()))) bp = 1.0 if len(c) >= len(r) else math.exp(1 - len(r) / len(c)) return bp * math.exp(sum(logs) / max_n) ref = "The Kestrel tent is made from silnylon and packs down small" cands = { "copy": "The Kestrel tent is made from silnylon and packs down small", "paraphrase": "Silnylon makes the Kestrel tent light and it packs into a tiny bag", "negation": "The Kestrel tent is not made from silnylon and packs down small", "shuffled": "small down packs and silnylon from made is tent Kestrel The", "short": "The Kestrel tent is made from silnylon", } for name, cand in cands.items(): print(f"{name:11s} BLEU {bleu(cand, ref):.3f}") c = cands["shuffled"].split() print("unigram precision of shuffled:", sum(min(n, ngrams(ref.split(), 1)[g]) for g, n in ngrams(c, 1).items()) / len(c)) print("brevity penalty of short:", round(math.exp(1 - len(ref.split()) / len(cands["short"].split())), 3))
The copy scores 1.000 and the correct paraphrase scores 0.000: it shares "Kestrel tent" and "the Kestrel tent" with the reference but no 4-gram (the reference's "The" is capitalized, so even "the Kestrel tent is" misses), and one zero precision zeroes the geometric mean. The negation, which says the opposite, scores 0.735: one inserted word only breaks the n-grams that span it. The shuffled sentence has a unigram precision of 1.0, every word is in the reference, yet scores 0.000 because no two words stay in order. The short version has perfect precision at every n and is multiplied by the brevity penalty e^(1 − 11/7) = 0.565.
from collections import Counter ref = "The Kestrel tent is made from silnylon and packs down small".split() cand = "small down packs and silnylon from made is tent Kestrel The".split() rc = Counter(ref) print(sum(min(n, rc[w]) for w, n in Counter(cand).items()) / len(cand))
Show the answer
1.0
Every word of the shuffled sentence appears in the reference exactly once, so each of the 11 words is a clipped hit: 11/11. Unigram precision cannot see order.
Show the answer
e^(1 − 11/7) ≈ 0.565: perfect precision scaled by the brevity penalty
The brevity penalty multiplies the geometric mean (here 1) by e^(1 − r/c) when the candidate is shorter: e^(1 − 11/7) = 0.565.
Clipped n-gram precision
Write clipped_precision(candidate, reference, n): the share of the candidate's n-grams (as word tuples) that appear in the reference, each candidate n-gram counting at most as many times as the reference contains it. Return 0.0 when the candidate has no n-grams.
from collections import Counter def ngrams(tokens, n): return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)) def clipped_precision(candidate, reference, n): return 0.0 print(clipped_precision("the the the the", "the cat sat", 1))
Plot the five BLEU scores so the gap between the copy and the paraphrase is visible at a glance.
import matplotlib.pyplot as plt names = ["copy", "paraphrase", "negation", "shuffled", "short"] scores = [1.000, 0.000, 0.735, 0.000, 0.565] plt.bar(names, scores) plt.ylabel("BLEU") plt.show()
These are the numbers the BLEU cell printed. The negation sits at 0.735, far above the correct paraphrase at 0.000: the metric rewards shared words, not meaning.
Recall and the longest common subsequence: ROUGE
ROUGE comes from summarization, where the worry is the opposite of BLEU's: did the summary cover the reference? ROUGE-N is recall: of the reference's n-grams, how many appear in the candidate. ROUGE-L uses the longest common subsequence, the longest run of words that appear in both sentences in the same order, not necessarily next to each other. Divide that length by the candidate's length for a precision, , and by the reference's length for a recall, ; ROUGE-L reports their F-measure, also called F1, : an average of the two that stays close to the smaller one.
The same five candidates show the difference. The shuffled sentence has every reference word, so ROUGE-1 is 1.000, but its longest common subsequence with the reference is a single word, so ROUGE-L is 0.091. The negation keeps every reference word in order and only adds one: ROUGE-1 1.000, ROUGE-2 0.900, ROUGE-L 0.957. The opposite meaning scores almost perfectly on all three metrics so far. The paraphrase, which is right, gets 0.364, 0.100 and 0.333.
The same five, under ROUGE
ROUGE-1 and ROUGE-2 as recall of reference n-grams; ROUGE-L as the F-measure of the longest common subsequence.
from collections import Counter def ngrams(tokens, n): return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)) def rouge_n(candidate, reference, n): cn, rn = ngrams(candidate.split(), n), ngrams(reference.split(), n) return sum(min(count, cn[g]) for g, count in rn.items()) / max(1, sum(rn.values())) def lcs(a, b): table = [[0] * (len(b) + 1) for _ in range(len(a) + 1)] for i in range(len(a)): for j in range(len(b)): table[i + 1][j + 1] = (table[i][j] + 1 if a[i] == b[j] else max(table[i][j + 1], table[i + 1][j])) return table[-1][-1] def rouge_l(candidate, reference): c, r = candidate.split(), reference.split() l = lcs(c, r) if l == 0: return 0.0 p, rec = l / len(c), l / len(r) return 2 * p * rec / (p + rec) ref = "The Kestrel tent is made from silnylon and packs down small" cands = { "copy": "The Kestrel tent is made from silnylon and packs down small", "paraphrase": "Silnylon makes the Kestrel tent light and it packs into a tiny bag", "negation": "The Kestrel tent is not made from silnylon and packs down small", "shuffled": "small down packs and silnylon from made is tent Kestrel The", "short": "The Kestrel tent is made from silnylon", } for name, cand in cands.items(): print(f"{name:11s} ROUGE-1 {rouge_n(cand, ref, 1):.3f} ROUGE-2 " f"{rouge_n(cand, ref, 2):.3f} ROUGE-L {rouge_l(cand, ref):.3f}")
ROUGE-1 is recall of single words: the shuffled sentence and the negation both contain every reference word, so both score 1.000, while the short version covers 7 of 11 words, 0.636. ROUGE-2 asks for word pairs: shuffling destroys them all (0.000), the negation keeps 9 of the reference's 10 (0.900). ROUGE-L follows order without requiring adjacency: the short version's seven words are all in sequence (0.778), the negation's twelve words contain all eleven of the reference in order (0.957), the shuffled sentence manages one word in sequence (0.091). The paraphrase is low on all three.
ROUGE-N recall
Write rouge_n(candidate, reference, n): of the reference's n-grams (word tuples, with repeats counted), the share that the candidate contains, each reference n-gram matched at most as many times as the candidate has it. Return 0.0 when the reference has no n-grams.
from collections import Counter def ngrams(tokens, n): return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)) def rouge_n(candidate, reference, n): return 0.0 print(round(rouge_n("a b c", "a b d", 1), 3))
from collections import Counter def rouge_1(candidate, reference): cn, rn = Counter(candidate.split()), Counter(reference.split()) return sum(min(count, cn[w]) for w, count in rn.items()) / sum(rn.values()) print(round(rouge_1("a b c", "a b d"), 3))
Show the answer
0.667
The reference has three words; the candidate contains a and b but not d: 2/3 = 0.667.
Meaning, not words: BERTScore
BLEU and ROUGE cannot tell "light" from "lightweight". BERTScore replaces exact word matches with similarity between token embeddings, the vectors you met in the context-and-retrieval lessons. Build a matrix with one row per candidate token and one column per reference token, each entry the cosine similarity of the two embeddings. Each candidate token takes its best match in the reference; the average of those bests is the precision. Each reference token takes its best match in the candidate; that average is the recall. F1 combines them, as in ROUGE-L.
The real metric uses contextual embeddings from a BERT-style model, so the same word gets a different vector in different sentences, and weights rare words more. The mechanism is the matching, and the next cell shows it with a hand-made table of three-number embeddings, where "shelter" sits near "tent" and "lightweight" near "light". The cell after runs the same recipe with real learned embeddings: the token table of the course's tiny GPT, a model you will build in Phase 5, which has a vector for each of its 195 words.
BERTScore with a hand-made embedding table
Nine words with three-number embeddings. Precision, recall and F1 by greedy matching, next to one-word BLEU.
import math import numpy as np from collections import Counter E = {"tent": [1, 0, 0], "shelter": [0.9, 0.1, 0], "mug": [0, 1, 0], "cup": [0, 0.9, 0.2], "light": [0, 0, 1], "lightweight": [0.1, 0, 1], "is": [0.3, 0.3, 0.3], "a": [0.3, 0.3, 0.3], "the": [0.3, 0.3, 0.3]} def cos(a, b): a, b = np.array(a, float), np.array(b, float) return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))) def bertscore(cand, ref, E): c, r = cand.split(), ref.split() # rows: candidate tokens, columns: reference tokens S = np.array([[cos(E[x], E[y]) for y in r] for x in c]) P, R = S.max(axis=1).mean(), S.max(axis=0).mean() return P, R, 2 * P * R / (P + R) def bleu1(cand, ref): c, r = cand.split(), ref.split() rc = Counter(r) return (sum(min(n, rc[w]) for w, n in Counter(c).items()) / len(c) * (1.0 if len(c) >= len(r) else math.exp(1 - len(r) / len(c)))) for cand in ["the tent is light", "the shelter is lightweight", "the mug is light", "a cup"]: P, R, F = bertscore(cand, "the tent is light", E) print(f"{cand!r:28s} P {P:.3f} R {R:.3f} F1 {F:.3f} BLEU-1 " f"{bleu1(cand, 'the tent is light'):.3f}")
"the shelter is lightweight" shares two words with "the tent is light" and gets BLEU-1 0.500, but its embeddings sit next to the reference's: F1 0.997. "the mug is light" has three identical words, BLEU-1 0.750, yet BERTScore drops to 0.894 because "mug" is far from "tent", the one word that matters. "a cup" shares nothing, BLEU-1 0.000, and still scores 0.816, mostly from "cup" near "mug"... which is the wrong product: similarity is not correctness. With good embeddings the metric rewards meaning; with any embeddings it rewards whatever they encode.
The same recipe with learned embeddings
The tiny GPT's token embeddings (195 words, 32 numbers each) on five candidates for "Made for cold nights .", with the plain word overlap beside each, and two cosines from its table. The model loads with PyTorch, which PyTorch tensors teaches; here you need only two of its pieces. model.tok.weight.detach() is the table of token embeddings, one row per word, ready to read like a NumPy array, and .norm() gives a vector's length.
import numpy as np import torch from tinygpt import load_pretrained model, tok = load_pretrained("tiny-gpt") E = model.tok.weight.detach() # one 32-number embedding per token def cos(a, b): return float((a @ b) / (a.norm() * b.norm())) def vec(word): return E[tok.token_id(word)] def bertscore(cand, ref): c, r = cand.split(), ref.split() S = np.array([[cos(vec(x), vec(y)) for y in r] for x in c]) P, R = S.max(axis=1).mean(), S.max(axis=0).mean() return P, R, 2 * P * R / (P + R) ref = "Made for cold nights ." for cand in ["Made for cold nights .", "Made for chilly evenings .", "Made for warm days .", "Built for cold nights .", "Perfect for rocky trails ."]: P, R, F = bertscore(cand, ref) overlap = len(set(cand.split()) & set(ref.split())) / len(cand.split()) print(f"{cand!r:30s} P {P:.3f} R {R:.3f} F1 {F:.3f} word overlap {overlap:.2f}") print("cos(cold, chilly):", f"{cos(vec('cold'), vec('chilly')):+.2f}", " cos(cold, warm):", f"{cos(vec('cold'), vec('warm')):+.2f}")
Here the metric fails, and the reason is instructive. "Made for chilly evenings" means the same as the reference and "Made for warm days" means the opposite; both score F1 0.655, because in this model's table "cold" is no closer to "chilly" (cosine +0.05) than to "warm" (−0.03). A 40,000-parameter model trained on 300 product descriptions learned which words appear near which, not which words mean the same, so its embeddings cannot carry the metric. Real BERTScore rests on a model trained on billions of words, and even then it inherits that model's blind spots: negation, numbers, names. The metric is only as good as the embeddings under it.
Show the answer
For each candidate token, its best cosine match in the reference, averaged over the candidate's tokens
Precision looks from the candidate's side: is each thing it said close to something in the reference? Recall looks from the reference's side.
Write a sentence or two, then compare it with a good answer in the lesson.
In production: which metric, and with which settings
Purpose. Pick a metric only where it fits, and report it so another team can reproduce it.
Overlap numbers depend on three settings: how the text is split into tokens, how many references there are, and the implementation. Two libraries do this the standard way. sacrebleu prints a signature line you copy into your report, and rouge-score gives precision, recall and F1 for each ROUGE variant.
Which metric, when
Overlap metrics earn their place where wording is constrained and references are plentiful: translation scored against several references, extractive summaries, outputs that should match a template. They are cheap, stable and comparable across runs of the same setup. They mislead for open-ended text, where many right answers share few words, and all three are blind to negation: "not made from silnylon" scored 0.735, 0.957 and high on BERTScore too.
Two other tools from this track usually fit better. When correctness is checkable, write a task grader, as in the evals lesson: does the JSON parse, is the number right, do the tests pass, does the answer contain the required fact. When quality is a judgment, tone, faithfulness to a source, helpfulness, use an LLM judge with a specific rubric, calibrated against human labels as the LLM-as-judge lesson described, and keep code checks for everything mechanical.
Whatever you report, report it with examples: a metric and five failures it missed tell the reader more than the metric alone. And never compare a BLEU from one paper with a BLEU from another unless the tokenization, the references and the implementation match; the number depends on all three.
The Phase 1 capstone grades a research agent with task graders: code checks its citations and its key facts, and no overlap metric is needed.
Show the answer
Judge both orders and count a win only when the judge agrees with itself; otherwise call it a tie
Position bias is handled by symmetry: run A-then-B and B-then-A and trust only consistent verdicts.
Show the answer
An LLM judge with the article in its prompt and a specific pass/fail rubric, checked against a few dozen human labels
Faithfulness is a judgment about meaning with the source in hand: a judge with the source and a rubric, calibrated. Overlap with the article rewards copying, not faithfulness.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- LLM-as-judge: when quality is a judgment, not a match.
- Evals: task graders for answers you can check.
- The Phase 1 capstone: grading an agent with code checks and no overlap metric.
Practice what you just read
- 5 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 9 review cards for spaced repetition
Try a quiz question
A correct paraphrase of the reference scores BLEU 0.000. Why?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 1 · about 23 hours of study
Building with LLMs
Prompting, retrieval, tools, agents and evals
35 lessons 1 free
- What an LLM actually does FreeNext-token prediction, and why that produces useful behavior.25 min
- Tokens and the context windowHow text becomes tokens, and what fits in one request.25 min
- Prompt fundamentalsClear instructions, context and constraints: the 80% of prompting.35 min
- System prompts and rolesSet behavior once, separately from the conversation.30 min
- Examples in promptsShow the model what good looks like (few-shot prompting).30 min
- Structured outputGet reliable JSON back that your code can use.38 min
- EvalsMeasure output quality with test sets instead of vibes.35 min
- Improving a prompt against an evalChange one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.30 min
- Prompt chainingBreak big tasks into a pipeline of smaller model calls.35 min
- Choosing a modelTrade capability, speed and cost across model families.25 min
- Reasoning models and effortWhen models think before answering, and how to control it.35 min
- Streaming responsesShow output as it's generated instead of waiting.35 min
- Conversations and memoryAPIs are stateless: how chat history and memory really work.36 min
- Images and documentsSend screenshots, PDFs and charts as input.30 min
- EmbeddingsMeaning as a list of numbers, so text can be compared.35 min
- Similarity searchFind the most relevant text by comparing embeddings.35 min
- Chunking documentsSplit long documents so the right piece can be retrieved.30 min
- Vector databasesStore and search millions of embeddings quickly.40 min
- Retrieval-augmented generationAnswer from your own documents instead of the model's memory.30 min
- Measuring retrievalA labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.30 min
- Making RAG goodHybrid search, reranking and citations.30 min
- Hallucinations and groundingWhy models make things up, and how to design against it.30 min
- Tool useLet the model call your functions (function calling).30 min
- The agent loopModel → tool → result → model, until the task is done.30 min
- Model Context ProtocolA standard way to plug tools and data into any AI app.30 min
- Agent design patternsRouting, orchestrator-workers and when not to build an agent.35 min
- LLMs in automationsPut models inside scripts and workflow tools that run on their own.35 min
- Prompt injection and securityHow untrusted text hijacks models, and how to limit the damage.35 min
- LLM-as-judgeGrade open-ended output with rubrics and a second model.35 min
- BLEU, ROUGE and BERTScoreWhat the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.35 min
- Cost and latencyPrompt caching, batching and routing to cheaper models.35 min
- Logging and monitoringSee what your LLM app actually did in production.30 min
- Shipping an LLM appDeploy, secure keys, rate-limit and handle failures.35 min
- One app, several model providersTwo API formats side by side, adapters to one internal format, a gateway with fallbacks, and evals before switching.40 min
- Capstone: research agentA marketing research agent with retrieval, tools and evals.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write