RAG interview questions and answers
Questions on retrieval-augmented generation: embeddings, chunking, vector databases, hybrid search and keeping answers grounded.
1. A colleague says "RAG means the model learns our documents." Correct them in two or three sentences.
RAG doesn't change the model at all. For each question, our code searches the documents, pastes the most relevant passages into the prompt, and the model answers from what it was given, so if retrieval misses the right passage, the answer will be wrong or missing.
2. Explain to a colleague why searching with embeddings can find results that keyword search misses.
Keyword search only finds documents that contain the words you typed. Embeddings turn text into vectors that capture meaning, so 'my parcel hasn't come' lands near a 'Shipping and delivery' article even though they share no words.
3. Explain the trade-off between small and large chunks.
Small chunks give sharp embeddings, so search finds precise passages, but each passage carries little context and facts can be split. Large chunks keep more context together but their embeddings average many topics, so matches get vague. You pick a size by testing real questions, and use overlap to avoid splitting facts.
4. Explain why hybrid search often beats embedding search alone, with an example.
Embeddings find passages with similar meaning, so 'parcel' matches the shipping page, but they're weak on exact terms like a product code 'WX-220' or a rare word. Keyword search is the reverse. Merging the two rankings with reciprocal rank fusion keeps documents either method found and promotes ones both agree on.
5. Describe the two phases of similarity search and what happens in each.
First you index: embed every document once and keep the vectors. Then for each search you embed only the query, score it against all the stored vectors with a dot product, and return the top few that clear a minimum score.
6. When would you move from an in-memory index to a vector database? Name two signals.
I'd move when the index gets too big for exact search to stay fast, or when several services need to share an index that's updated and deleted from frequently and has to survive restarts.
7. Describe the "quotes first" technique and why it reduces hallucination.
You ask the model to first copy, word for word, the sentences from the documents that answer the question, then to answer using only those quotes, or say it doesn't know if there are none. It keeps the answer anchored to the actual text, and because quotes must match the source exactly, your code can verify them.
8. A stakeholder says: 'Let's fine-tune a model on our 300-page product manual so it knows our products.' What would you recommend instead, and why?
I'd build retrieval over the manual first: split it into sections, find the relevant ones for each question, and have the model answer from them with references. That gives accurate answers that update the moment the manual changes, with sources we can check. Fine-tuning on the manual would teach the model to sound like it knows our products, but it learns facts unreliably, makes confident mistakes, and would need retraining after every manual update. If we later want answers in a particular format or tone, a small fine-tune on top can help, but only after an evaluation shows retrieval and prompting aren't enough.
9. A colleague says: 'The new model has a 1-million-token window, so we can drop our retrieval system and paste in everything.' Respond.
A million-token window is great for tasks that need whole documents at once, and it can simplify our pipeline when the material fits. But we'd pay for every one of those tokens on every request and wait longer for each answer, unless the same prefix repeats and can be cached. Models also don't use huge contexts perfectly: facts in the middle get missed more often and combining many scattered facts is harder than finding one. And our full knowledge base is larger than a million tokens anyway. I'd test both on our real questions, and likely keep retrieval but give it a bigger budget, retrieving more passages into the long context.
10. Explain how attention resembles the retrieval step in RAG, and one way it differs.
Both start with a query, compare it against keys, and return content from the best matches, just like searching a vector index for relevant documents. The differences: attention doesn't pick the top few, it blends all the values weighted by how well they matched; the query, key and value vectors are learned projections tuned by training rather than fixed embeddings; and it happens inside the model, for every token in every layer, over the text already in context rather than an external database.