Retrieval-augmented generation

Step 2: retrieve

The practice API has already done step 1 for you. Its help center is indexed. The endpoint GET /v1/docs?q=... runs the embedding search: it embeds the query and returns the closest documents. That's the same thing your search function did in the similarity search lesson.

import requests

API = "https://api.gradient.test/v1"
question = "Can I return a gift card?"
hits = requests.get(f"{API}/docs", params={"q": question, "limit": 2}).json()["data"]
for hit in hits:
    print(hit["score"], hit["id"], hit["title"])

Each hit has three parts. The score says how close the document is to the question, and higher means closer. The id is the document's short name, and the title is its heading. The hits come back best first, and "limit": 2 keeps the top two.

Search results carry ids and scores, but not the full text. Fetch the full text of each hit before building the prompt:

import requests

API = "https://api.gradient.test/v1"
doc = requests.get(f"{API}/docs/doc_giftcards").json()
print(doc["title"])
print(doc["text"][:120], "...")

The [:120] keeps the printout short by showing the first 120 characters. Your prompt will use the whole text.

Step 3: the grounded prompt

A grounded prompt is one that ties the model to the documents you hand it. How you present those documents decides whether the model uses them. Four habits work well:

1. Wrap each document in tags with its id and title. Then the model can tell where one ends and cite it: <document id="doc_returns" title="Returns and refunds">...</document>.
2. Put the documents first and the question last. Claude answers long-context questions better when the material comes before the question.
3. Tell it to answer only from the documents. If they don't contain the answer, it should say so plainly. Without that rule, a model fills gaps from its general knowledge. That's how a support bot invents a 90-day return window.
4. Ask for citations by document id. A citation points at the source of a claim, so you (and your customers) can check the answer.

system:  You answer customer questions for an outdoor clothing shop. Use only the documents
         provided. If they don't contain the answer, say you don't know and suggest contacting
         support. Cite the ids of the documents you used in square brackets, like [doc_returns].

user:    <documents>
         <document id="doc_giftcards" title="Gift cards">Digital gift cards are ...</document>
         <document id="doc_returns" title="Returns and refunds">You can return ...</document>
         </documents>

         Question: Can I return a gift card?

The rules go in the system prompt because they apply to every question. The documents and the question change each time, so they go in the user message.

Watch out: retrieved text is data, not instructions. If a document says "ignore your rules," the model might follow it. You'll learn how to limit that in the prompt injection lesson.

Try it below. The two documents are shortened copies of the articles this question retrieves.

passages = ["Refunds take 5 days.", "Shipping is free."]
context = "\n\n".join(passages)
prompt = f"Answer only from this:\n{context}"
print("Refunds" in prompt)

import requests

API = "https://api.gradient.test/v1"
SYSTEM = ("You answer customer questions for an outdoor clothing shop. Use only the documents provided. "
          "If they don't contain the answer, say you don't know and suggest contacting support. "
          "Cite the ids of the documents you used in square brackets, like [doc_returns].")

def ask(system, prompt, max_tokens=400):
    response = requests.post("https://api.anthropic.com/v1/messages",
        headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"},
        json={"model": "claude-opus-5-5", "max_tokens": max_tokens, "system": system,
              "messages": [{"role": "user", "content": prompt}]}, timeout=60)
    response.raise_for_status()
    return "".join(b["text"] for b in response.json()["content"] if b["type"] == "text")

def answer(question, k=2):
    hits = requests.get(f"{API}/docs", params={"q": question, "limit": k}).json()["data"]
    docs = [requests.get(f"{API}/docs/{h['id']}").json() for h in hits]
    context = "\n".join(f'<document id="{d["id"]}" title="{d["title"]}">{d["text"]}</document>' for d in docs)
    prompt = f"<documents>\n{context}\n</documents>\n\nQuestion: {question}"
    return ask(SYSTEM, prompt), [d["id"] for d in docs]

reply, sources = answer("How long does express shipping take?")
print("sources:", sources)
print(reply)

When RAG is the right tool

RAG is not always the answer. A small shop with a one-page price list doesn't need a librarian. It can hand the model the whole page every time.

The context window is the most text a model can read in one request. A useful rule follows from it. Say your whole knowledge base is under about 100,000 tokens and doesn't change often. Then try putting all of it in the prompt before you build retrieval.

Pair that with prompt caching, which is a feature that lets the API reuse a long, unchanging start of a prompt across requests. Repeat requests then cost less and come back faster. You'll meet it in the cost and latency lesson. RAG earns its extra moving parts when the material is large or changes daily.