Hallucinations and grounding

Prompt techniques that help

1. Ground it. Give the model the facts, and tell it to use only those. The facts can come from retrieval (RAG), a pasted product sheet or data from your own systems. This is called grounding, and it's the biggest single lever you have.

2. Give it permission to say "I don't know". Write "If the documents don't contain the answer, say you don't know." It sounds obvious. It measurably cuts invented answers. Make "I don't know" an acceptable outcome, not a failure.

3. Quotes first, then the answer. For questions about long documents, ask the model to copy the exact sentences it will rely on first. Then it answers using only those quotes. Quotes are easy to check, and the extra step keeps the answer close to the text:

Answer the question using the documents.
First, inside <quotes> tags, copy word for word the sentences from the documents that answer it.
If no sentence answers it, write <quotes>none</quotes> and say you don't know.
Then give your answer inside <answer> tags, using only the quoted sentences.

4. Limit the possible answers. Sometimes the answer must be one of a known set: a product id, a category, yes or no. Use structured output with a fixed list of allowed values (an enum, from the structured output lesson). A model can't invent a product id the list doesn't allow.

5. Keep a person on high-stakes claims. Prices, legal, medical and environmental claims, and anything quoting a real person: have someone approve them before they go out.

For marketers: treat model-written claims like a new copywriter's draft. Every number and every "proven" needs a source before it ships.

Check answers with code

Some hallucinations can be caught by a simple program. Here's the idea. If an answer contains a number that appears in none of the source documents, something is off. At best the model calculated it. At worst it invented it. Either way, a person should look.

The code below fetches the practice shop's returns page. Then it pulls every number out of an answer and lists the ones the page never mentions:

import re
import requests

source = requests.get("https://api.gradient.test/v1/docs/doc_returns").json()["text"]
answer = "You can return unworn items within 90 days, and refunds arrive within 5 business days."

def numbers(text):
    return re.findall(r"\d+(?:\.\d+)?", text)

source_numbers = set(numbers(source))
print("numbers in the answer:", numbers(answer))
print("not in the source:", [n for n in numbers(answer) if n not in source_numbers])

How it works:

• re.findall finds every piece of text that matches a pattern. The pattern \d+ means "one or more digits", and the optional part catches decimals such as 4.5.
• set(...) makes a collection you can check membership in quickly.
• The last line keeps only the answer's numbers that aren't in the source.

The same idea works for quotes. If you asked the model to quote first, check that every quote really appears in the documents. Compare after lowercasing and squeezing all spaces and line breaks into single spaces, so a line break doesn't count as a mismatch:

def normalize(text):
    return " ".join(text.lower().split())

source = "Gift cards never expire and can be used on any product,\nincluding sale items."
quotes = ["Gift cards never expire", "Gift cards can be refunded for cash"]
for q in quotes:
    print(normalize(q) in normalize(source), "|", q)

The first quote is in the source. The second one isn't, so it prints False: the model invented it.

These checks are cheap, so you can run them on every answer. They turn "the bot sometimes makes things up" into a short list of specific answers for a person to review. A second model asked "is every claim in this answer backed by these documents?" catches subtler cases. You'll build that in the LLM-as-judge lesson.

source = "The warranty lasts 2 years."
quote = "lasts 5 years"
print(quote in source)

import re
source = "Returns are free within 60 days. Refunds take 5 to 10 business days."
answer = "Return within 90 days and get your refund in 5 days."
found = re.findall(r"\d+", answer)
print(found)
print([n for n in found if n not in re.findall(r"\d+", source)])