Making RAG good

Two kinds of search

Embeddings match meaning. That's why a question about a "parcel" finds the shipping page. But embeddings are weak on exact terms. Product names, SKUs (stock codes such as "WX-220"), error codes and rare words like "exclamation" often slip past them.

Classic keyword search is the opposite. It works like the Find box in a document. It only matches words that actually appear, and it's excellent at them.

The standard keyword method is called BM25. The cell below uses a simpler version that counts how many of the question's words each document contains. First it drops stopwords. These are common words like "the", "how" and "can" that appear everywhere and tell you little.

import re
import requests

STOPWORDS = {"a", "an", "the", "and", "or", "of", "to", "in", "on", "for", "with", "is", "are",
             "can", "do", "does", "we", "you", "i", "my", "how", "what", "many", "have", "it"}

def terms(text):
    return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOPWORDS}

docs = requests.get("https://api.gradient.test/v1/docs").json()["data"]
query = "How many exclamation marks can an email have?"
scores = {d["id"]: len(terms(query) & terms(d["text"])) for d in docs}
print(sorted(scores.items(), key=lambda item: -item[1])[:3])

Here's what happened. terms lowercases the text, splits it into words and drops the stopwords. The & keeps only the words that two sets share, and len counts them. So each document scores one point per shared word. The last line prints the top three.

The voice guide wins with two matching words: "exclamation" and "email". (The guide says "exclamation mark", so "marks" doesn't match. Plain counting treats them as different words.) The runners-up share only the word "email".

That exposes the weakness of plain counting. "Email" appears in four articles and "exclamation" in only one. Yet both count the same. A rare word tells you much more about which document you want. BM25 gives rare words more weight, which is a big part of why it beats this simple version.

Hybrid search: meaning plus exact words

Hybrid search runs both searches and merges the two rankings. Picture two friends who each give you a ranked list of restaurants. A place near the top of both lists is a safe bet. A place only one friend mentioned can still be worth a try, if they ranked it highly.

The simplest merge that works well is called reciprocal rank fusion, or RRF. ("Fusion" means merging.) Every list a document appears in gives it some points: 1 ÷ (60 + its rank in that list). Then you add up each document's points and sort by the total.

In plain words: RRF score = for every list the document appears in, add 1 ÷ (60 + its rank in that list).

print(round(1 / (60 + 1), 4))

A document near the top of both lists wins. A document that only one method found can still rank well. The 60 is a conventional constant that stops first place from dominating.

Here's the worked example for "Can we call a jacket carbon neutral?". Vector search put the sustainability page first and the voice guide second. Keyword search put the voice guide first, because it contains "jacket", "carbon" and "neutral". The sustainability page came third.

Vector search alone put the wrong document first. The keyword evidence tips it, and the voice guide wins by a hair. You'll check this in Python in the worked example below. In the exercise, you'll measure how often this happens across a whole test set.

You rarely need to build this yourself. Most vector databases (pgvector with a text index, Weaviate, Qdrant, Elasticsearch) can run hybrid search for you.

# Made-up results: where the correct document landed for four test questions
vector_ranks = [1, 3, 1, 2]
hybrid_ranks = [1, 1, 1, 3]

def hit_rate_at_1(ranks):
    return sum(r == 1 for r in ranks) / len(ranks)

def mrr(ranks):
    return sum(1 / r for r in ranks) / len(ranks)

print("vector:", hit_rate_at_1(vector_ranks), round(mrr(vector_ranks), 2))
print("hybrid:", hit_rate_at_1(hybrid_ranks), round(mrr(hybrid_ranks), 2))

# RRF for "Can we call a jacket carbon neutral?"
voice = 1 / (60 + 2) + 1 / (60 + 1)
sustainability = 1 / (60 + 1) + 1 / (60 + 3)
print("doc_voice:", round(voice, 4))
print("doc_sustainability:", round(sustainability, 4))

Reranking

Think about hiring. A recruiter skims 200 job applications quickly and picks 20 that look close. Then the hiring manager reads those 20 carefully against the job. Careful reading is too slow for all 200, but it's worth it for a short list.

Search works the same way. Retrieval has to be fast. So it compares the question with each passage separately, using vectors worked out ahead of time. A reranker reads the question and each candidate passage together. Then it scores how well that passage answers that question.

A reranker is slower and more accurate. So you use it on a short list. Retrieve 20 to 50 candidates cheaply, rerank them, and keep the best 3 to 5.

Dedicated reranking models (from Cohere, Voyage AI and open-source cross-encoders) are fast and cheap. You can also ask an LLM to pick the most relevant passages:

def rerank(question, candidates, keep=3):
    listing = "\n\n".join(f'[{d["id"]}] {d["text"][:400]}' for d in candidates)
    reply = ask("You judge which passages answer a question. Reply with document ids only, best first, one per line.",
                f"Question: {question}\n\nPassages:\n{listing}\n\nList the {keep} most relevant ids.")
    ranked = [line.strip(" []") for line in reply.splitlines() if line.strip()]
    known = {d["id"]: d for d in candidates}
    return [known[i] for i in ranked if i in known][:keep]

Here ask is a small helper like the one in earlier lessons. It sends a system prompt and a prompt, and returns the reply's text. The function lists each candidate as its id in brackets plus its first 400 characters. It asks for the best ids, one per line. Then it strips spaces and brackets from each line of the reply.

Note the last two lines. The code keeps only ids it actually sent. A model can return an id that doesn't exist. Never trust a model's output to be a valid id without checking.

known = {"doc_returns": 1, "doc_sizing": 2}
ranked = ["doc_sizing", "doc_refunds", "doc_returns"]
print([i for i in ranked if i in known][:2])

Citations you can check

A citation is a tag in the answer that names the source of a fact, such as [doc_shipping]. Picture a student's essay with a footnote pointing to a book the library has never owned. That footnote is made up, however sure it sounds.

Asking for [doc_id] citations is only useful if you verify them. A citation to a document you didn't retrieve means the model invented a source:

import re

reply = "Express takes 1 to 2 business days [doc_shipping]. Members ship free in their birthday month [doc_loyalty]."
retrieved = ["doc_shipping", "doc_returns"]

cited = re.findall(r"\[(doc_[a-z_]+)\]", reply)
unknown = [c for c in cited if c not in retrieved]
print("cited:", cited)
print("not retrieved:", unknown)

re.findall returns every match of a pattern as a list. This pattern looks for [doc_, then lowercase letters or underscores, then ]. It keeps only the id inside the brackets. The reply cites two documents, but only doc_shipping was retrieved. So doc_loyalty is flagged. The birthday fact might even be true, but the model didn't get it from the passages you sent.

Claude's API can also do this for you. Send each document as a document content block with "citations": {"enabled": True}. The reply's text blocks then come back with a citations list. It points at the exact passages used.

{"type": "document",
 "source": {"type": "text", "media_type": "text/plain", "data": doc["text"]},
 "title": doc["title"],
 "citations": {"enabled": True}}

cited = ["doc_returns", "doc_care", "doc_returns"]
retrieved = ["doc_returns", "doc_sizing"]
print([c for c in cited if c not in retrieved])

A checklist of other fixes

• Rewrite the query. A chatty message ("hi, ordered last week, still nothing, help??") makes a poor search query. Ask a model to turn it into one, such as "order delivery time", before searching.
• Add context to chunks. Before embedding a chunk, put its document title or a one-line summary in front of it. Then "It takes 2 to 4 weeks" is embedded as part of "Lifetime repair warranty".
• Filter by metadata before ranking. Metadata is information about a document, such as its product line, region or date.
• Tune k, the number of chunks you put in the prompt. Too few chunks miss facts. Too many bury them. Measure to find the right number.