Index Ai Academy
Phase 1 · Quality & production

BLEU, ROUGE and BERTScore

What the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.

About 35 minutes · Versions checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

When a model writes a sentence and you have a reference to compare it with, three scores appear in almost every paper and leaderboard: BLEU, ROUGE and BERTScore. They are cheap, automatic and widely misread. This lesson builds each one in a few lines, scores the same five candidates against one reference, and shows exactly what each rewards: shared n-grams, shared subsequences, or similar token embeddings.

You will see a correct paraphrase score zero, a sentence that says the opposite score 0.735, and a shuffled sentence get a perfect ROUGE-1. You will see what BERTScore fixes and what it cannot when its embeddings are weak. Then the practical question: when an overlap metric is the right tool, when a task grader is, and when the judge from the LLM-as-judge lesson is.

What you'll be able to do

  • Compute BLEU (clipped n-gram precision with a brevity penalty) and ROUGE-1, ROUGE-2 and ROUGE-L in code
  • Show three ways word overlap misleads: a paraphrase, a negation and a shuffled sentence
  • Compute a miniature BERTScore from token embeddings and say what it fixes and what it inherits
  • Choose between an overlap metric, a task grader and an LLM judge for a given evaluation
Quick check
BLEU compares a candidate sentence with a reference by counting:
  1. The reference's words that the candidate left out, as a share of the reference's length
  2. The candidate's n-grams that also appear in the reference, each count capped at how often the reference has it
  3. The characters the two sentences share, in order, divided by the longer length
  4. The number of edits (insertions, deletions, swaps) that turn the candidate into the reference
Show the answer

The candidate's n-grams that also appear in the reference, each count capped at how often the reference has it

BLEU is a precision: of the candidate's n-grams, how many are in the reference, with clipping so that repeating a reference word cannot inflate the score.

The idea: three ways to compare two sentences

This lesson rests on Logarithms and Sequences and Σ: BLEU averages four natural logs, and a sum of terms is a Σ. Scores below use F1, the F-measure: from a precision PP and a recall RR it is 2PR/(P+R)2PR / (P + R), an average that stays close to the smaller of the two.

Counting shared n-grams: BLEU

BLEU scores a candidate against a reference by precision: of the candidate's n-grams, how many also appear in the reference? It does that for n = 1, 2, 3 and 4 and combines the four with a geometric mean, so a candidate needs shared words and shared word order to score. Two details make it fair. Clipping: a candidate n-gram counts at most as often as the reference contains it, so "the the the the" cannot score four hits on a reference with one "the". A brevity penalty: precision alone rewards saying little, so a candidate shorter than the reference is multiplied by e1−r/ce^{1 - r/c}, where rr and cc are the reference and candidate lengths.

BLEU=BP⋅exp⁡(14∑n=14ln⁡pn),BP=min⁡(1,e1−r/c).\text{BLEU} = BP \cdot \exp\Big(\tfrac{1}{4} \sum_{n=1}^{4} \ln p_n\Big), \qquad BP = \min(1, e^{1 - r/c}).
Quick check
From the lesson on logarithms: BLEU averages logs, then raises e to the result. What does this print for the geometric mean of 2 and 8?
import math

mean_log = (math.log(2) + math.log(8)) / 2
print(round(math.exp(mean_log), 3))
  1. 5.0
  2. 4.0
  3. 2.0
  4. 16.0
Show the answer

4.0

The mean of the two logs is the log of the geometric mean, so exp undoes it: the square root of 2 × 8 is 4. The arithmetic mean would be 5.

Here pnp_n is the clipped precision for n-grams of length n, and ln⁡\ln is the natural logarithm (see logarithms). Averaging the four logs and raising ee to the result is the geometric mean: multiply the four precisions and take the fourth root.

Because the mean is geometric, a single pnp_n of zero makes the whole score zero: a sentence with no shared 4-gram scores 0 however many words it shares. That is the harsh original; reported BLEU is usually computed over a whole test set, or smoothed.

Part 2 scores five candidates against one reference: an exact copy, a correct paraphrase, a negation, the reference shuffled, and a correct but short version.

Two more ways: ROUGE and BERTScore

ROUGE asks the opposite question to BLEU: did the candidate cover the reference? ROUGE-N is recall of the reference's n-grams, and ROUGE-L uses the longest common subsequence, the longest run of words that appear in both sentences in order, not necessarily side by side. BERTScore replaces exact word matches with the cosine similarity of token embeddings: each token takes its best match in the other sentence, so "light" can match "lightweight". The picture shows its matrix.

Try it now

The outlined cell is each row's best match. Their average is the precision.

In your own words
What does each metric count, and what can none of the three see?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: five candidates, three metrics

Purpose. Compute each score from its definition and read what it rewards.

Start with BLEU on five candidates for one reference. The cell prints the BLEU of each and two diagnostics.

Worked example

Five candidates, one reference, under BLEU

Clipped n-gram precision for n = 1 to 4, geometric mean, brevity penalty. Then two diagnostics: the shuffled candidate's unigram precision and the short candidate's brevity penalty.

import math
from collections import Counter

def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def bleu(candidate, reference, max_n=4):
    c, r = candidate.split(), reference.split()
    logs = []
    for n in range(1, max_n + 1):
        cn, rn = ngrams(c, n), ngrams(r, n)
        # a hit counts at most as often as the reference has it
        clipped = sum(min(count, rn[g]) for g, count in cn.items())
        if clipped == 0:
            return 0.0
        logs.append(math.log(clipped / sum(cn.values())))
    bp = 1.0 if len(c) >= len(r) else math.exp(1 - len(r) / len(c))
    return bp * math.exp(sum(logs) / max_n)

ref = "The Kestrel tent is made from silnylon and packs down small"
cands = {
    "copy": "The Kestrel tent is made from silnylon and packs down small",
    "paraphrase": "Silnylon makes the Kestrel tent light and it packs into a tiny bag",
    "negation": "The Kestrel tent is not made from silnylon and packs down small",
    "shuffled": "small down packs and silnylon from made is tent Kestrel The",
    "short": "The Kestrel tent is made from silnylon",
}
for name, cand in cands.items():
    print(f"{name:11s} BLEU {bleu(cand, ref):.3f}")
c = cands["shuffled"].split()
print("unigram precision of shuffled:",
      sum(min(n, ngrams(ref.split(), 1)[g]) for g, n in ngrams(c, 1).items()) / len(c))
print("brevity penalty of short:",
      round(math.exp(1 - len(ref.split()) / len(cands["short"].split())), 3))
It runs right here in your browser, with nothing to install.

The copy scores 1.000 and the correct paraphrase scores 0.000: it shares "Kestrel tent" and "the Kestrel tent" with the reference but no 4-gram (the reference's "The" is capitalized, so even "the Kestrel tent is" misses), and one zero precision zeroes the geometric mean. The negation, which says the opposite, scores 0.735: one inserted word only breaks the n-grams that span it. The shuffled sentence has a unigram precision of 1.0, every word is in the reference, yet scores 0.000 because no two words stay in order. The short version has perfect precision at every n and is multiplied by the brevity penalty e^(1 − 11/7) = 0.565.

Quick check
The shuffled candidate's unigram precision. What does this print?
from collections import Counter
ref = "The Kestrel tent is made from silnylon and packs down small".split()
cand = "small down packs and silnylon from made is tent Kestrel The".split()
rc = Counter(ref)
print(sum(min(n, rc[w]) for w, n in Counter(cand).items()) / len(cand))
  1. 0.5
  2. 0.091
  3. 1.0
  4. 0.0
Show the answer

1.0

Every word of the shuffled sentence appears in the reference exactly once, so each of the 11 words is a clipped hit: 11/11. Unigram precision cannot see order.

Quick check
A candidate has perfect clipped precision at every n but is 7 words long against an 11-word reference. Its BLEU is:
  1. 7/11 ≈ 0.636: the share of the reference it covers
  2. 1.0: precision is all BLEU measures
  3. 0.0: a candidate shorter than the reference cannot score
  4. e^(1 − 11/7) ≈ 0.565: perfect precision scaled by the brevity penalty
Show the answer

e^(1 − 11/7) ≈ 0.565: perfect precision scaled by the brevity penalty

The brevity penalty multiplies the geometric mean (here 1) by e^(1 − r/c) when the candidate is shorter: e^(1 − 11/7) = 0.565.

Drill

Clipped n-gram precision

Write clipped_precision(candidate, reference, n): the share of the candidate's n-grams (as word tuples) that appear in the reference, each candidate n-gram counting at most as many times as the reference contains it. Return 0.0 when the candidate has no n-grams.

from collections import Counter

def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def clipped_precision(candidate, reference, n):
    return 0.0

print(clipped_precision("the the the the", "the cat sat", 1))
In the lesson, your code is checked as you write it.

Plot the five BLEU scores so the gap between the copy and the paraphrase is visible at a glance.

import matplotlib.pyplot as plt

names = ["copy", "paraphrase", "negation", "shuffled", "short"]
scores = [1.000, 0.000, 0.735, 0.000, 0.565]
plt.bar(names, scores)
plt.ylabel("BLEU")
plt.show()

These are the numbers the BLEU cell printed. The negation sits at 0.735, far above the correct paraphrase at 0.000: the metric rewards shared words, not meaning.

Recall and the longest common subsequence: ROUGE

ROUGE comes from summarization, where the worry is the opposite of BLEU's: did the summary cover the reference? ROUGE-N is recall: of the reference's n-grams, how many appear in the candidate. ROUGE-L uses the longest common subsequence, the longest run of words that appear in both sentences in the same order, not necessarily next to each other. Divide that length by the candidate's length for a precision, PP, and by the reference's length for a recall, RR; ROUGE-L reports their F-measure, also called F1, 2PR/(P+R)2PR / (P + R): an average of the two that stays close to the smaller one.

The same five candidates show the difference. The shuffled sentence has every reference word, so ROUGE-1 is 1.000, but its longest common subsequence with the reference is a single word, so ROUGE-L is 0.091. The negation keeps every reference word in order and only adds one: ROUGE-1 1.000, ROUGE-2 0.900, ROUGE-L 0.957. The opposite meaning scores almost perfectly on all three metrics so far. The paraphrase, which is right, gets 0.364, 0.100 and 0.333.

Worked example

The same five, under ROUGE

ROUGE-1 and ROUGE-2 as recall of reference n-grams; ROUGE-L as the F-measure of the longest common subsequence.

from collections import Counter

def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def rouge_n(candidate, reference, n):
    cn, rn = ngrams(candidate.split(), n), ngrams(reference.split(), n)
    return sum(min(count, cn[g]) for g, count in rn.items()) / max(1, sum(rn.values()))

def lcs(a, b):
    table = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i in range(len(a)):
        for j in range(len(b)):
            table[i + 1][j + 1] = (table[i][j] + 1 if a[i] == b[j]
                                   else max(table[i][j + 1], table[i + 1][j]))
    return table[-1][-1]

def rouge_l(candidate, reference):
    c, r = candidate.split(), reference.split()
    l = lcs(c, r)
    if l == 0:
        return 0.0
    p, rec = l / len(c), l / len(r)
    return 2 * p * rec / (p + rec)

ref = "The Kestrel tent is made from silnylon and packs down small"
cands = {
    "copy": "The Kestrel tent is made from silnylon and packs down small",
    "paraphrase": "Silnylon makes the Kestrel tent light and it packs into a tiny bag",
    "negation": "The Kestrel tent is not made from silnylon and packs down small",
    "shuffled": "small down packs and silnylon from made is tent Kestrel The",
    "short": "The Kestrel tent is made from silnylon",
}
for name, cand in cands.items():
    print(f"{name:11s} ROUGE-1 {rouge_n(cand, ref, 1):.3f}  ROUGE-2 "
          f"{rouge_n(cand, ref, 2):.3f}  ROUGE-L {rouge_l(cand, ref):.3f}")
In the lesson you can change this and run it.

ROUGE-1 is recall of single words: the shuffled sentence and the negation both contain every reference word, so both score 1.000, while the short version covers 7 of 11 words, 0.636. ROUGE-2 asks for word pairs: shuffling destroys them all (0.000), the negation keeps 9 of the reference's 10 (0.900). ROUGE-L follows order without requiring adjacency: the short version's seven words are all in sequence (0.778), the negation's twelve words contain all eleven of the reference in order (0.957), the shuffled sentence manages one word in sequence (0.091). The paraphrase is low on all three.

Drill

ROUGE-N recall

Write rouge_n(candidate, reference, n): of the reference's n-grams (word tuples, with repeats counted), the share that the candidate contains, each reference n-gram matched at most as many times as the candidate has it. Return 0.0 when the reference has no n-grams.

from collections import Counter

def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def rouge_n(candidate, reference, n):
    return 0.0

print(round(rouge_n("a b c", "a b d", 1), 3))
In the lesson, your code is checked as you write it.
Quick check
ROUGE-1 of a candidate that shares two of the reference's three words. What does this print?
from collections import Counter
def rouge_1(candidate, reference):
    cn, rn = Counter(candidate.split()), Counter(reference.split())
    return sum(min(count, cn[w]) for w, count in rn.items()) / sum(rn.values())
print(round(rouge_1("a b c", "a b d"), 3))
  1. 0.667
  2. 0.333
  3. 0.5
  4. 1.0
Show the answer

0.667

The reference has three words; the candidate contains a and b but not d: 2/3 = 0.667.

Meaning, not words: BERTScore

BLEU and ROUGE cannot tell "light" from "lightweight". BERTScore replaces exact word matches with similarity between token embeddings, the vectors you met in the context-and-retrieval lessons. Build a matrix with one row per candidate token and one column per reference token, each entry the cosine similarity of the two embeddings. Each candidate token takes its best match in the reference; the average of those bests is the precision. Each reference token takes its best match in the candidate; that average is the recall. F1 combines them, as in ROUGE-L.

The real metric uses contextual embeddings from a BERT-style model, so the same word gets a different vector in different sentences, and weights rare words more. The mechanism is the matching, and the next cell shows it with a hand-made table of three-number embeddings, where "shelter" sits near "tent" and "lightweight" near "light". The cell after runs the same recipe with real learned embeddings: the token table of the course's tiny GPT, a model you will build in Phase 5, which has a vector for each of its 195 words.

Worked example

BERTScore with a hand-made embedding table

Nine words with three-number embeddings. Precision, recall and F1 by greedy matching, next to one-word BLEU.

import math
import numpy as np
from collections import Counter

E = {"tent": [1, 0, 0], "shelter": [0.9, 0.1, 0], "mug": [0, 1, 0],
     "cup": [0, 0.9, 0.2], "light": [0, 0, 1],
     "lightweight": [0.1, 0, 1], "is": [0.3, 0.3, 0.3], "a": [0.3, 0.3, 0.3],
     "the": [0.3, 0.3, 0.3]}

def cos(a, b):
    a, b = np.array(a, float), np.array(b, float)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def bertscore(cand, ref, E):
    c, r = cand.split(), ref.split()
    # rows: candidate tokens, columns: reference tokens
    S = np.array([[cos(E[x], E[y]) for y in r] for x in c])
    P, R = S.max(axis=1).mean(), S.max(axis=0).mean()
    return P, R, 2 * P * R / (P + R)

def bleu1(cand, ref):
    c, r = cand.split(), ref.split()
    rc = Counter(r)
    return (sum(min(n, rc[w]) for w, n in Counter(c).items()) / len(c)
            * (1.0 if len(c) >= len(r) else math.exp(1 - len(r) / len(c))))

for cand in ["the tent is light", "the shelter is lightweight",
             "the mug is light", "a cup"]:
    P, R, F = bertscore(cand, "the tent is light", E)
    print(f"{cand!r:28s} P {P:.3f} R {R:.3f} F1 {F:.3f}   BLEU-1 "
          f"{bleu1(cand, 'the tent is light'):.3f}")
In the lesson you can change this and run it.

"the shelter is lightweight" shares two words with "the tent is light" and gets BLEU-1 0.500, but its embeddings sit next to the reference's: F1 0.997. "the mug is light" has three identical words, BLEU-1 0.750, yet BERTScore drops to 0.894 because "mug" is far from "tent", the one word that matters. "a cup" shares nothing, BLEU-1 0.000, and still scores 0.816, mostly from "cup" near "mug"... which is the wrong product: similarity is not correctness. With good embeddings the metric rewards meaning; with any embeddings it rewards whatever they encode.

Worked example

The same recipe with learned embeddings

The tiny GPT's token embeddings (195 words, 32 numbers each) on five candidates for "Made for cold nights .", with the plain word overlap beside each, and two cosines from its table. The model loads with PyTorch, which PyTorch tensors teaches; here you need only two of its pieces. model.tok.weight.detach() is the table of token embeddings, one row per word, ready to read like a NumPy array, and .norm() gives a vector's length.

import numpy as np
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
E = model.tok.weight.detach()                    # one 32-number embedding per token

def cos(a, b):
    return float((a @ b) / (a.norm() * b.norm()))

def vec(word):
    return E[tok.token_id(word)]

def bertscore(cand, ref):
    c, r = cand.split(), ref.split()
    S = np.array([[cos(vec(x), vec(y)) for y in r] for x in c])
    P, R = S.max(axis=1).mean(), S.max(axis=0).mean()
    return P, R, 2 * P * R / (P + R)

ref = "Made for cold nights ."
for cand in ["Made for cold nights .", "Made for chilly evenings .",
             "Made for warm days .", "Built for cold nights .",
             "Perfect for rocky trails ."]:
    P, R, F = bertscore(cand, ref)
    overlap = len(set(cand.split()) & set(ref.split())) / len(cand.split())
    print(f"{cand!r:30s} P {P:.3f} R {R:.3f} F1 {F:.3f}   word overlap {overlap:.2f}")
print("cos(cold, chilly):", f"{cos(vec('cold'), vec('chilly')):+.2f}",
      "  cos(cold, warm):", f"{cos(vec('cold'), vec('warm')):+.2f}")
In the lesson you can change this and run it.

Here the metric fails, and the reason is instructive. "Made for chilly evenings" means the same as the reference and "Made for warm days" means the opposite; both score F1 0.655, because in this model's table "cold" is no closer to "chilly" (cosine +0.05) than to "warm" (−0.03). A 40,000-parameter model trained on 300 product descriptions learned which words appear near which, not which words mean the same, so its embeddings cannot carry the metric. Real BERTScore rests on a model trained on billions of words, and even then it inherits that model's blind spots: negation, numbers, names. The metric is only as good as the embeddings under it.

Quick check
In BERTScore, what does the precision measure?
  1. For each reference token, its best cosine match in the candidate, averaged over the reference's tokens
  2. For each candidate token, its best cosine match in the reference, averaged over the candidate's tokens
  3. The cosine between the average of all candidate embeddings and the average of all reference embeddings
  4. The share of candidate tokens whose embedding is identical to some reference token's embedding
Show the answer

For each candidate token, its best cosine match in the reference, averaged over the candidate's tokens

Precision looks from the candidate's side: is each thing it said close to something in the reference? Recall looks from the reference's side.

In your own words
The tiny GPT's embeddings gave the right paraphrase and the wrong opposite the same BERTScore. What would you check before trusting a BERTScore from any embedding model on your own task?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: which metric, and with which settings

Purpose. Pick a metric only where it fits, and report it so another team can reproduce it.

Overlap numbers depend on three settings: how the text is split into tokens, how many references there are, and the implementation. Two libraries do this the standard way. sacrebleu prints a signature line you copy into your report, and rouge-score gives precision, recall and F1 for each ROUGE variant.

Which metric, when

Overlap metrics earn their place where wording is constrained and references are plentiful: translation scored against several references, extractive summaries, outputs that should match a template. They are cheap, stable and comparable across runs of the same setup. They mislead for open-ended text, where many right answers share few words, and all three are blind to negation: "not made from silnylon" scored 0.735, 0.957 and high on BERTScore too.

Two other tools from this track usually fit better. When correctness is checkable, write a task grader, as in the evals lesson: does the JSON parse, is the number right, do the tests pass, does the answer contain the required fact. When quality is a judgment, tone, faithfulness to a source, helpfulness, use an LLM judge with a specific rubric, calibrated against human labels as the LLM-as-judge lesson described, and keep code checks for everything mechanical.

Whatever you report, report it with examples: a metric and five failures it missed tell the reader more than the metric alone. And never compare a BLEU from one paper with a BLEU from another unless the tokenization, the references and the implementation match; the number depends on all three.

The Phase 1 capstone grades a research agent with task graders: code checks its citations and its key facts, and no overlap metric is needed.

Quick check
From the LLM-as-judge lesson: a judge comparing two answers tends to prefer whichever comes first. The fix is:
  1. Use a longer rubric, which makes the judge read both answers more carefully
  2. Always put the model under test second, so the bias works against it
  3. Ask the judge to ignore the order in the prompt
  4. Judge both orders and count a win only when the judge agrees with itself; otherwise call it a tie
Show the answer

Judge both orders and count a win only when the judge agrees with itself; otherwise call it a tie

Position bias is handled by symmetry: run A-then-B and B-then-A and trust only consistent verdicts.

Quick check
You are evaluating a support assistant's replies for whether each one is faithful to the knowledge base article it cites. Which tool fits?
  1. BLEU against one reference reply per ticket, since faithful replies will share its wording
  2. ROUGE-L against the article itself, since a faithful reply reuses the article's words in order
  3. BERTScore against the article, since similar embeddings mean the reply says the same thing
  4. An LLM judge with the article in its prompt and a specific pass/fail rubric, checked against a few dozen human labels
Show the answer

An LLM judge with the article in its prompt and a specific pass/fail rubric, checked against a few dozen human labels

Faithfulness is a judgment about meaning with the source in hand: a judge with the source and a rubric, calibrated. Overlap with the article rewards copying, not faithfulness.

In your own words
A team reports that their new model raised BLEU from 31 to 34 on a summarization set. Name two questions you would ask before believing the model is better.

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A correct paraphrase of the reference scores BLEU 0.000. Why?

  1. It shares no 4-gram with the reference, and one zero precision makes the geometric mean zero
  2. BLEU subtracts a point for every word that is not in the reference
  3. The brevity penalty is zero when the candidate uses different words
  4. BLEU requires the candidate to start with the same word as the reference

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 1 · about 23 hours of study

Building with LLMs

Prompting, retrieval, tools, agents and evals 35 lessons 1 free
  1. What an LLM actually does FreeNext-token prediction, and why that produces useful behavior.25 min
  2. Tokens and the context windowHow text becomes tokens, and what fits in one request.25 min
  3. Prompt fundamentalsClear instructions, context and constraints: the 80% of prompting.35 min
  4. System prompts and rolesSet behavior once, separately from the conversation.30 min
  5. Examples in promptsShow the model what good looks like (few-shot prompting).30 min
  6. Structured outputGet reliable JSON back that your code can use.38 min
  7. EvalsMeasure output quality with test sets instead of vibes.35 min
  8. Improving a prompt against an evalChange one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.30 min
  9. Prompt chainingBreak big tasks into a pipeline of smaller model calls.35 min
  10. Choosing a modelTrade capability, speed and cost across model families.25 min
  11. Reasoning models and effortWhen models think before answering, and how to control it.35 min
  12. Streaming responsesShow output as it's generated instead of waiting.35 min
  13. Conversations and memoryAPIs are stateless: how chat history and memory really work.36 min
  14. Images and documentsSend screenshots, PDFs and charts as input.30 min
  15. EmbeddingsMeaning as a list of numbers, so text can be compared.35 min
  16. Similarity searchFind the most relevant text by comparing embeddings.35 min
  17. Chunking documentsSplit long documents so the right piece can be retrieved.30 min
  18. Vector databasesStore and search millions of embeddings quickly.40 min
  19. Retrieval-augmented generationAnswer from your own documents instead of the model's memory.30 min
  20. Measuring retrievalA labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.30 min
  21. Making RAG goodHybrid search, reranking and citations.30 min
  22. Hallucinations and groundingWhy models make things up, and how to design against it.30 min
  23. Tool useLet the model call your functions (function calling).30 min
  24. The agent loopModel → tool → result → model, until the task is done.30 min
  25. Model Context ProtocolA standard way to plug tools and data into any AI app.30 min
  26. Agent design patternsRouting, orchestrator-workers and when not to build an agent.35 min
  27. LLMs in automationsPut models inside scripts and workflow tools that run on their own.35 min
  28. Prompt injection and securityHow untrusted text hijacks models, and how to limit the damage.35 min
  29. LLM-as-judgeGrade open-ended output with rubrics and a second model.35 min
  30. BLEU, ROUGE and BERTScoreWhat the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.35 min
  31. Cost and latencyPrompt caching, batching and routing to cheaper models.35 min
  32. Logging and monitoringSee what your LLM app actually did in production.30 min
  33. Shipping an LLM appDeploy, secure keys, rate-limit and handle failures.35 min
  34. One app, several model providersTwo API formats side by side, adapters to one internal format, a gateway with fallbacks, and evals before switching.40 min
  35. Capstone: research agentA marketing research agent with retrieval, tools and evals.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write