Free RAG course
Build retrieval-augmented generation step by step: embeddings, chunking, vector search, grounded answers with sources, and the measurements that tell you whether it works.
A labeled test set, hit rate, reciprocal rank, precision and recall at k and nDCG, and which half of a RAG system failed.
When a RAG answer is wrong, the cause is usually upstream: the right passage never reached the model. If you only read answers, you cannot tell a bad search from a bad prompt. So measure the search on its own. A labeled test set, a handful of metrics and a comparison between two searches turn "retrieval feels off" into numbers you can move. This lesson builds the five metrics, scores a vector search against a keyword search on the practice help center, and shows how teams keep watching retrieval once the app is live.
Whether the gift-card article was in the passages the model was given
If the right passage never reached the model, no prompt can fix the answer. Check the search first, then the answer.
This lesson rests on Retrieval-augmented generation (RAG), where a search hands passages to the model, and on Logarithms: the power question, because one of the metrics divides by a log.
To score a search you need a labeled test set: a list of real questions, and for each one the ids of every document that answers it. A person writes the labels once. Some questions have one right document and some have several ("Can I return an online order at a store?" needs both the returns page and the stores page). A search returns a ranked list, best first. Every metric below reads that list against the labels.
0.25
Reciprocal rank is 1 divided by the rank of the first right document: 1 ÷ 4 = 0.25. Fourth place earns a quarter of the credit of first place.
In plain words: hit rate asks "did I find any?", reciprocal rank asks "how soon?", precision asks "how much of it was useful?", recall asks "how much of the useful stuff did I get?", and nDCG asks "were the useful ones near the top?".
import math print(math.log2(8), round(1 / math.log2(3), 3))
3.0 0.631
2 raised to 3 is 8, so log₂(8) is 3. The discount for rank 2 is 1 ÷ log₂(3) = 1 ÷ 1.585 = 0.631: a right answer in second place counts about 63% as much as one in first.
The picture. Take one question with two right documents, and a search that returns five. The first picture marks where the right documents landed: ranks 2 and 4. The second shows how much each rank counts toward nDCG.
The dark bars are the ranks that held a right document.
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Score a vector search and a keyword search on the same 12 questions, and find where each one wins.
The lesson's setup holds the labeled questions, two searches over the practice help center, and the five metrics as short functions (you write them yourself in the drills and the exercise). The vector search is the practice embedding search; the keyword search counts words shared with each article. First, the five numbers for the worked list: right documents at ranks 2 and 4.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math import re import requests API = "https://api.gradient.test/v1" # A labeled test set: a question and the ids of every article that answers it. QUESTIONS = [ ("Can I send a gift card to a friend on their birthday?", {"doc_giftcards"}), ("What if my package gets lost on the way?", {"doc_shipping"}), ("Are your factories checked?", {"doc_sustainability"}), ("Is there a place I can try things on?", {"doc_stores"}), ("Can I get cash back for a gift card?", {"doc_giftcards", "doc_returns"}), ("How do I start a repair and how long does it take?", {"doc_warranty", "doc_sustainability"}), ("Can I return an online order at a store?", {"doc_stores", "doc_returns"}), ("How do I join the rewards club?", {"doc_loyalty"}), ("Do you ship to Mexico?", {"doc_shipping"}), ("prepaid label", {"doc_returns", "doc_warranty"}), ("import duties Canada", {"doc_shipping"}), ("spam folder", {"doc_giftcards"}), ] docs = requests.get(f"{API}/docs").json()["data"] STOP = set("a an the i my to of in on at is it do does can you your for and or me be " "how what when are if should".split()) def words(text): return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP] def vector_search(question): hits = requests.get(f"{API}/docs", params={"q": question, "limit": 11}).json() return [h["id"] for h in hits["data"]] def keyword_search(question): wanted = set(words(question)) scored = [] for d in docs: shared = sum(w in wanted for w in words(d["title"] + " " + d["text"])) scored.append((shared, d["id"])) return [doc_id for _, doc_id in sorted(scored, key=lambda s: (-s[0], s[1]))] def hit_at(ranked, relevant, k): return 1.0 if any(d in relevant for d in ranked[:k]) else 0.0 def reciprocal_rank(ranked, relevant): for place, doc_id in enumerate(ranked, start=1): if doc_id in relevant: return 1 / place return 0.0 def precision_at(ranked, relevant, k): return sum(d in relevant for d in ranked[:k]) / k def recall_at(ranked, relevant, k): return sum(d in relevant for d in ranked[:k]) / len(relevant) def ndcg_at(ranked, relevant, k): gain = sum(1 / math.log2(place + 1) for place, d in enumerate(ranked[:k], start=1) if d in relevant) best = range(1, min(k, len(relevant)) + 1) ideal = sum(1 / math.log2(place + 1) for place in best) return gain / ideal
ranked = ["d1", "d2", "d3", "d4", "d5"] relevant = {"d2", "d4"} print(hit_at(ranked, relevant, 3), reciprocal_rank(ranked, relevant)) print(precision_at(ranked, relevant, 5), recall_at(ranked, relevant, 3)) print(round(ndcg_at(ranked, relevant, 5), 3))
It prints 1.0 and 0.5, then 0.4 and 0.5, then 0.651: the numbers from the ladder.
Score each search at k = 3 on every question and average. Then print where each search put the first right document, for the questions where the two disagree most.
Uses the setup above, 61 lines.
def evaluate(search, k=3): rows = [(search(q), rel) for q, rel in QUESTIONS] return { "hit": sum(hit_at(r, rel, k) for r, rel in rows) / len(rows), "mrr": sum(reciprocal_rank(r, rel) for r, rel in rows) / len(rows), "precision": sum(precision_at(r, rel, k) for r, rel in rows) / len(rows), "recall": sum(recall_at(r, rel, k) for r, rel in rows) / len(rows), "ndcg": sum(ndcg_at(r, rel, k) for r, rel in rows) / len(rows), } for name, search in [("vector", vector_search), ("keyword", keyword_search)]: print(name, {key: round(value, 2) for key, value in evaluate(search).items()})
Each search is scored on the same 12 questions with the same five functions, so the numbers are comparable. The averages are close, so the totals alone say "about the same". The next cell shows why: the two searches fail on different questions.
Hit rate needs only one right document, but recall counts all of them
Hit rate is satisfied by any one right document. Recall counts every right document, so a search that finds the first and misses the second keeps a high hit rate and loses recall.
Averages hide the questions. List the ones where one search is clearly better, by nDCG at 3.
Uses the setup above, 61 lines.
for question, relevant in QUESTIONS: v = ndcg_at(vector_search(question), relevant, 3) k = ndcg_at(keyword_search(question), relevant, 3) if abs(v - k) > 0.3: winner = "vector" if v > k else "keyword" print(f"{winner:8s} {v:.2f} vs {k:.2f} {question}")
The vector search wins the paraphrases ("What if my package gets lost on the way?", "Are your factories checked?"): the right article shares no words with the question, but its meaning is close. The keyword search wins the questions that name a term in the text ("prepaid label", "import duties Canada", "spam folder"), which the embedding search ranks well down. The two have opposite failures, which is the case for hybrid search in Making RAG good.
Now draw recall at k for k = 1 to 10 for both searches, to see how deep you must look to find every right document.
Uses the setup above, 61 lines.
import matplotlib.pyplot as plt ks = range(1, 11) for name, search in [("vector", vector_search), ("keyword", keyword_search)]: rows = [(search(q), rel) for q, rel in QUESTIONS] curve = [sum(recall_at(r, rel, k) for r, rel in rows) / len(rows) for k in ks] print(name, [round(x, 2) for x in curve]) plt.plot(ks, curve, marker="o", label=name) plt.xlabel("k") plt.ylabel("recall at k") plt.legend() plt.show()
Both curves rise as k grows and cannot fall. They stay close: the keyword search is a little ahead at small k and reaches 1.0 at k = 8, the vector search at k = 10, because each has a question the other ranks well ("Is there a place I can try things on?" sits at rank 10 for the vector search and rank 7 for the keyword search). A bigger k always raises recall, and costs more context for the model to read.
ranked = ["a", "b", "c", "d"] relevant = {"c", "d"} first = next(i for i, d in enumerate(ranked, start=1) if d in relevant) found = sum(d in relevant for d in ranked[:3]) print(round(1 / first, 2), found / len(relevant))
0.33 0.5
The first right document, c, is at rank 3, so reciprocal rank is 1 ÷ 3 = 0.33. Of the two right documents, only c is in the top 3, so recall at 3 is 1 ÷ 2 = 0.5.
Write reciprocal_rank(ranked, relevant): ranked is a list of document ids, best first, and relevant is a set of right ids. Return 1 divided by the rank (counting from 1) of the first right document, or 0.0 if none appears.
Uses the setup above, 61 lines.
def reciprocal_rank(ranked, relevant): return 0.0 print(reciprocal_rank(["a", "b", "c"], {"b", "c"}))
Write recall_at(ranked, relevant, k): the share of the right documents that appear in the first k entries of ranked.
Uses the setup above, 61 lines.
def recall_at(ranked, relevant, k): return 0.0 print(recall_at(["a", "b", "c", "d"], {"b", "d"}, 2))
Write a sentence or two, then compare it with a good answer in the lesson.
Purpose. Keep measuring the search after launch, and tell a search failure from an answer failure.
A test set you wrote once goes stale as documents and questions change, so teams keep two habits. First, log the rank of the cited chunk for real questions: when the answer cites a passage, record where the search ranked it. A rising share of citations from rank 4 or lower means the search is slipping, long before users complain. Second, grade the two halves apart. The search half is graded with the metrics above. The answer half is graded on whether each claim in the reply is supported by the retrieved text, which is a job for the checks in Hallucinations and grounding and the judge in LLM-as-judge. A wrong answer with the right passage retrieved is a prompt problem; a wrong answer without it is a search problem.
Libraries compute the same metrics. The card uses scikit-learn's ndcg_score on the worked list.
The right passages are slipping down the ranking
If answers lean on lower-ranked chunks, the best passages are no longer first. That can come from new documents, drifted questions or a changed index, and it shows up here before users complain.
Write a sentence or two, then compare it with a good answer in the lesson.
A question has three right documents, and the search finds two of them in its top 5. Which metric says 2 of 3?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Build retrieval-augmented generation step by step: embeddings, chunking, vector search, grounded answers with sources, and the measurements that tell you whether it works.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write