Index Ai Academy
Phase 1 · Context & retrieval

Measuring retrieval

A labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.

About 30 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

When a RAG answer is wrong, the cause is usually upstream: the right passage never reached the model. If you only read answers, you cannot tell a bad search from a bad prompt. So measure the search on its own. A labeled test set, a handful of metrics and a comparison between two searches turn "retrieval feels off" into numbers you can move. This lesson builds the five metrics, scores a vector search against a keyword search on the practice help center, and shows how teams keep watching retrieval once the app is live.

What you'll be able to do

  • Label a test set of questions with every document that answers each one
  • Compute hit rate, reciprocal rank, precision at k, recall at k and nDCG for a ranked list, by hand and in code
  • Compare two searches on the same questions and find the questions one wins and the other loses
  • Say which half of a RAG system failed: the search or the answer
Quick check
A support bot answers "Do gift cards expire?" with the wrong policy. Which do you check first?
  1. Whether the model needs a longer system prompt for refunds
  2. Whether the gift-card article was in the passages the model was given
  3. Whether the answer is too short
  4. Whether the user typed the question correctly
Show the answer

Whether the gift-card article was in the passages the model was given

If the right passage never reached the model, no prompt can fix the answer. Check the search first, then the answer.

The idea: score the search before the answer

This lesson rests on Retrieval-augmented generation (RAG), where a search hands passages to the model, and on Logarithms: the power question, because one of the metrics divides by a log.

To score a search you need a labeled test set: a list of real questions, and for each one the ids of every document that answers it. A person writes the labels once. Some questions have one right document and some have several ("Can I return an online order at a store?" needs both the returns page and the stores page). A search returns a ranked list, best first. Every metric below reads that list against the labels.

  • Hit rate at k. Did any right document appear in the top k? It is 1 or 0 for a question, and the hit rate is the share of questions with a 1.
  • Reciprocal rank. One divided by the rank of the first right document: 1 for first place, 1/2 for second, 1/3 for third, 0 if none appears. Its average over questions is the mean reciprocal rank, MRR.
Quick check
The first right document is at rank 4. What is the reciprocal rank for that question?
  1. 4
  2. 0.4
  3. 0.25
  4. 1
Show the answer

0.25

Reciprocal rank is 1 divided by the rank of the first right document: 1 ÷ 4 = 0.25. Fourth place earns a quarter of the credit of first place.

  • Precision at k. Of the k documents returned, what share are right?
  • Recall at k. Of the right documents, what share appear in the top k? When a question has several right documents, this is the one that matters.
  • nDCG at k. A right document lower down counts for less: its gain is 1 ÷ log₂(rank + 1), and the sum is divided by the best sum possible. It is 1 for a perfect list.

In plain words: hit rate asks "did I find any?", reciprocal rank asks "how soon?", precision asks "how much of it was useful?", recall asks "how much of the useful stuff did I get?", and nDCG asks "were the useful ones near the top?".

Quick check
From the lesson on logarithms: nDCG discounts rank r by 1 ÷ log₂(r + 1). What does this print?
import math

print(math.log2(8), round(1 / math.log2(3), 3))
  1. 2.0 0.631
  2. 8.0 0.333
  3. 3.0 0.333
  4. 3.0 0.631
Show the answer

3.0 0.631

2 raised to 3 is 8, so log₂(8) is 3. The discount for rank 2 is 1 ÷ log₂(3) = 1 ÷ 1.585 = 0.631: a right answer in second place counts about 63% as much as one in first.

The picture. Take one question with two right documents, and a search that returns five. The first picture marks where the right documents landed: ranks 2 and 4. The second shows how much each rank counts toward nDCG.

Try it now

The dark bars are the ranks that held a right document.

In your own words
In your own words: when does recall at k tell you more than hit rate?

Write a sentence or two, then compare it with a good answer in the lesson.

In Python: five metrics, two searches

Purpose. Score a vector search and a keyword search on the same 12 questions, and find where each one wins.

The lesson's setup holds the labeled questions, two searches over the practice help center, and the five metrics as short functions (you write them yourself in the drills and the exercise). The vector search is the practice embedding search; the keyword search counts words shared with each article. First, the five numbers for the worked list: right documents at ranks 2 and 4.

Setup (runs before every cell) · 61 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import re
import requests

API = "https://api.gradient.test/v1"
# A labeled test set: a question and the ids of every article that answers it.
QUESTIONS = [
    ("Can I send a gift card to a friend on their birthday?", {"doc_giftcards"}),
    ("What if my package gets lost on the way?", {"doc_shipping"}),
    ("Are your factories checked?", {"doc_sustainability"}),
    ("Is there a place I can try things on?", {"doc_stores"}),
    ("Can I get cash back for a gift card?", {"doc_giftcards", "doc_returns"}),
    ("How do I start a repair and how long does it take?",
     {"doc_warranty", "doc_sustainability"}),
    ("Can I return an online order at a store?", {"doc_stores", "doc_returns"}),
    ("How do I join the rewards club?", {"doc_loyalty"}),
    ("Do you ship to Mexico?", {"doc_shipping"}),
    ("prepaid label", {"doc_returns", "doc_warranty"}),
    ("import duties Canada", {"doc_shipping"}),
    ("spam folder", {"doc_giftcards"}),
]
docs = requests.get(f"{API}/docs").json()["data"]
STOP = set("a an the i my to of in on at is it do does can you your for and or me be "
           "how what when are if should".split())

def words(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

def vector_search(question):
    hits = requests.get(f"{API}/docs", params={"q": question, "limit": 11}).json()
    return [h["id"] for h in hits["data"]]

def keyword_search(question):
    wanted = set(words(question))
    scored = []
    for d in docs:
        shared = sum(w in wanted for w in words(d["title"] + " " + d["text"]))
        scored.append((shared, d["id"]))
    return [doc_id for _, doc_id in sorted(scored, key=lambda s: (-s[0], s[1]))]

def hit_at(ranked, relevant, k):
    return 1.0 if any(d in relevant for d in ranked[:k]) else 0.0

def reciprocal_rank(ranked, relevant):
    for place, doc_id in enumerate(ranked, start=1):
        if doc_id in relevant:
            return 1 / place
    return 0.0

def precision_at(ranked, relevant, k):
    return sum(d in relevant for d in ranked[:k]) / k

def recall_at(ranked, relevant, k):
    return sum(d in relevant for d in ranked[:k]) / len(relevant)

def ndcg_at(ranked, relevant, k):
    gain = sum(1 / math.log2(place + 1)
               for place, d in enumerate(ranked[:k], start=1) if d in relevant)
    best = range(1, min(k, len(relevant)) + 1)
    ideal = sum(1 / math.log2(place + 1) for place in best)
    return gain / ideal
ranked = ["d1", "d2", "d3", "d4", "d5"]
relevant = {"d2", "d4"}
print(hit_at(ranked, relevant, 3), reciprocal_rank(ranked, relevant))
print(precision_at(ranked, relevant, 5), recall_at(ranked, relevant, 3))
print(round(ndcg_at(ranked, relevant, 5), 3))

It prints 1.0 and 0.5, then 0.4 and 0.5, then 0.651: the numbers from the ladder.

Worked example

Worked example: both searches on all 12 questions

Score each search at k = 3 on every question and average. Then print where each search put the first right document, for the questions where the two disagree most.

Uses the setup above, 61 lines.

def evaluate(search, k=3):
    rows = [(search(q), rel) for q, rel in QUESTIONS]
    return {
        "hit": sum(hit_at(r, rel, k) for r, rel in rows) / len(rows),
        "mrr": sum(reciprocal_rank(r, rel) for r, rel in rows) / len(rows),
        "precision": sum(precision_at(r, rel, k) for r, rel in rows) / len(rows),
        "recall": sum(recall_at(r, rel, k) for r, rel in rows) / len(rows),
        "ndcg": sum(ndcg_at(r, rel, k) for r, rel in rows) / len(rows),
    }

for name, search in [("vector", vector_search), ("keyword", keyword_search)]:
    print(name, {key: round(value, 2) for key, value in evaluate(search).items()})
In the lesson you can change this and run it.

Each search is scored on the same 12 questions with the same five functions, so the numbers are comparable. The averages are close, so the totals alone say "about the same". The next cell shows why: the two searches fail on different questions.

Quick check
Both searches have a hit rate of 0.75 at 3, yet their recall at 3 is 0.71 and 0.75. How can two searches tie on hit rate and differ on recall?
  1. The two metrics are computed in different ways by mistake
  2. Recall can never differ from hit rate
  3. Hit rate counts documents past rank 3
  4. Hit rate needs only one right document, but recall counts all of them
Show the answer

Hit rate needs only one right document, but recall counts all of them

Hit rate is satisfied by any one right document. Recall counts every right document, so a search that finds the first and misses the second keeps a high hit rate and loses recall.

Averages hide the questions. List the ones where one search is clearly better, by nDCG at 3.

Uses the setup above, 61 lines.

for question, relevant in QUESTIONS:
    v = ndcg_at(vector_search(question), relevant, 3)
    k = ndcg_at(keyword_search(question), relevant, 3)
    if abs(v - k) > 0.3:
        winner = "vector" if v > k else "keyword"
        print(f"{winner:8s} {v:.2f} vs {k:.2f}  {question}")

The vector search wins the paraphrases ("What if my package gets lost on the way?", "Are your factories checked?"): the right article shares no words with the question, but its meaning is close. The keyword search wins the questions that name a term in the text ("prepaid label", "import duties Canada", "spam folder"), which the embedding search ranks well down. The two have opposite failures, which is the case for hybrid search in Making RAG good.

Now draw recall at k for k = 1 to 10 for both searches, to see how deep you must look to find every right document.

Uses the setup above, 61 lines.

import matplotlib.pyplot as plt

ks = range(1, 11)
for name, search in [("vector", vector_search), ("keyword", keyword_search)]:
    rows = [(search(q), rel) for q, rel in QUESTIONS]
    curve = [sum(recall_at(r, rel, k) for r, rel in rows) / len(rows) for k in ks]
    print(name, [round(x, 2) for x in curve])
    plt.plot(ks, curve, marker="o", label=name)
plt.xlabel("k")
plt.ylabel("recall at k")
plt.legend()
plt.show()

Both curves rise as k grows and cannot fall. They stay close: the keyword search is a little ahead at small k and reaches 1.0 at k = 8, the vector search at k = 10, because each has a question the other ranks well ("Is there a place I can try things on?" sits at rank 10 for the vector search and rank 7 for the keyword search). A bigger k always raises recall, and costs more context for the model to read.

Quick check
What does this print?
ranked = ["a", "b", "c", "d"]
relevant = {"c", "d"}
first = next(i for i, d in enumerate(ranked, start=1) if d in relevant)
found = sum(d in relevant for d in ranked[:3])
print(round(1 / first, 2), found / len(relevant))
  1. 0.33 1.0
  2. 0.25 1.0
  3. 0.5 0.5
  4. 0.33 0.5
Show the answer

0.33 0.5

The first right document, c, is at rank 3, so reciprocal rank is 1 ÷ 3 = 0.33. Of the two right documents, only c is in the top 3, so recall at 3 is 1 ÷ 2 = 0.5.

Drill

Reciprocal rank

Write reciprocal_rank(ranked, relevant): ranked is a list of document ids, best first, and relevant is a set of right ids. Return 1 divided by the rank (counting from 1) of the first right document, or 0.0 if none appears.

Uses the setup above, 61 lines.

def reciprocal_rank(ranked, relevant):
    return 0.0


print(reciprocal_rank(["a", "b", "c"], {"b", "c"}))
In the lesson, your code is checked as you write it.
Drill

Recall at k

Write recall_at(ranked, relevant, k): the share of the right documents that appear in the first k entries of ranked.

Uses the setup above, 61 lines.

def recall_at(ranked, relevant, k):
    return 0.0


print(recall_at(["a", "b", "c", "d"], {"b", "d"}, 2))
In the lesson, your code is checked as you write it.
In your own words
The vector search and the keyword search have similar averages but fail on different questions. What does that tell you to do?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: watch retrieval, and grade the two halves apart

Purpose. Keep measuring the search after launch, and tell a search failure from an answer failure.

A test set you wrote once goes stale as documents and questions change, so teams keep two habits. First, log the rank of the cited chunk for real questions: when the answer cites a passage, record where the search ranked it. A rising share of citations from rank 4 or lower means the search is slipping, long before users complain. Second, grade the two halves apart. The search half is graded with the metrics above. The answer half is graded on whether each claim in the reply is supported by the retrieved text, which is a job for the checks in Hallucinations and grounding and the judge in LLM-as-judge. A wrong answer with the right passage retrieved is a prompt problem; a wrong answer without it is a search problem.

Libraries compute the same metrics. The card uses scikit-learn's ndcg_score on the worked list.

Quick check
In production, the share of answers that cite a chunk the search ranked 4th or lower rises from 5% to 20% in a month. What does that suggest?
  1. The test set is too big
  2. The model got smarter
  3. The right passages are slipping down the ranking
  4. Nothing: ranks do not matter after launch
Show the answer

The right passages are slipping down the ranking

If answers lean on lower-ranked chunks, the best passages are no longer first. That can come from new documents, drifted questions or a changed index, and it shows up here before users complain.

In your own words
Why log the rank of the cited chunk in production?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A question has three right documents, and the search finds two of them in its top 5. Which metric says 2 of 3?

  1. Precision at 5
  2. Recall at 5
  3. Reciprocal rank
  4. Hit rate at 5

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 1 · about 23 hours of study

Building with LLMs

Prompting, retrieval, tools, agents and evals 35 lessons 1 free
  1. What an LLM actually does FreeNext-token prediction, and why that produces useful behavior.25 min
  2. Tokens and the context windowHow text becomes tokens, and what fits in one request.25 min
  3. Prompt fundamentalsClear instructions, context and constraints: the 80% of prompting.35 min
  4. System prompts and rolesSet behavior once, separately from the conversation.30 min
  5. Examples in promptsShow the model what good looks like (few-shot prompting).30 min
  6. Structured outputGet reliable JSON back that your code can use.38 min
  7. EvalsMeasure output quality with test sets instead of vibes.35 min
  8. Improving a prompt against an evalChange one thing, rerun, compare, read what flipped, keep or revert: the loop that makes a prompt better, with a held-back set and the error on the gap.30 min
  9. Prompt chainingBreak big tasks into a pipeline of smaller model calls.35 min
  10. Choosing a modelTrade capability, speed and cost across model families.25 min
  11. Reasoning models and effortWhen models think before answering, and how to control it.35 min
  12. Streaming responsesShow output as it's generated instead of waiting.35 min
  13. Conversations and memoryAPIs are stateless: how chat history and memory really work.36 min
  14. Images and documentsSend screenshots, PDFs and charts as input.30 min
  15. EmbeddingsMeaning as a list of numbers, so text can be compared.35 min
  16. Similarity searchFind the most relevant text by comparing embeddings.35 min
  17. Chunking documentsSplit long documents so the right piece can be retrieved.30 min
  18. Vector databasesStore and search millions of embeddings quickly.40 min
  19. Retrieval-augmented generationAnswer from your own documents instead of the model's memory.30 min
  20. Measuring retrievalA labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.30 min
  21. Making RAG goodHybrid search, reranking and citations.30 min
  22. Hallucinations and groundingWhy models make things up, and how to design against it.30 min
  23. Tool useLet the model call your functions (function calling).30 min
  24. The agent loopModel → tool → result → model, until the task is done.30 min
  25. Model Context ProtocolA standard way to plug tools and data into any AI app.30 min
  26. Agent design patternsRouting, orchestrator-workers and when not to build an agent.35 min
  27. LLMs in automationsPut models inside scripts and workflow tools that run on their own.35 min
  28. Prompt injection and securityHow untrusted text hijacks models, and how to limit the damage.35 min
  29. LLM-as-judgeGrade open-ended output with rubrics and a second model.35 min
  30. BLEU, ROUGE and BERTScoreWhat the three text-similarity scores reward, how each one misleads, and when a grader or a judge is the better tool.35 min
  31. Cost and latencyPrompt caching, batching and routing to cheaper models.35 min
  32. Logging and monitoringSee what your LLM app actually did in production.30 min
  33. Shipping an LLM appDeploy, secure keys, rate-limit and handle failures.35 min
  34. One app, several model providersTwo API formats side by side, adapters to one internal format, a gateway with fallbacks, and evals before switching.40 min
  35. Capstone: research agentA marketing research agent with retrieval, tools and evals.45 min
Part of a free course 14 lessons

Free RAG course

Build retrieval-augmented generation step by step: embeddings, chunking, vector search, grounded answers with sources, and the measurements that tell you whether it works.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write