RAG Interview Questions and Answers (2026)

Retrieval-Augmented Generation. Instead of hoping the model memorised a fact during training, you fetch relevant text at question time and hand it over as context — then it answers grounded in what you retrieved. Open-book exam versus closed-book. The model’s job shifts from “recall” to “read these passages and answer.”

Two lanes. Offline: chunk your documents, embed each chunk into a vector, and store the vectors — plus the original text — in a vector database. Online: embed the user’s question with the same model, run a similarity search for the top-k chunks, drop them into the prompt with “answer using only the context below,” and generate. The answer that scores points adds the two things beginners skip — a re-ranker between retrieval and generation, and an “I don’t know” path when nothing relevant comes back.

When the knowledge is small and static enough to just sit in the prompt. When you need the model to reason over the whole corpus rather than a few passages — RAG only ever sees top-k. Or when the extra retrieval hop isn’t worth the latency and cost. And if the task is about behaviour or style rather than facts — “always reply in this format” — that’s a fine-tuning job, not a retrieval one.

Look at what got retrieved before you look at the answer. Pull the top-k chunks for a failing question. If the right passage isn’t in there, it’s retrieval — fix chunking, embeddings, or the query. If the right passage is there and the model still answered wrong, it’s generation — fix the prompt, the model, or add a re-ranker so the good chunk isn’t buried at position eight. Most people blame the model; most bugs are retrieval.

Chunk too big and each vector becomes a blurry average of several ideas, so search gets vague. Chunk too small and you shred the context a sentence needs to make sense. I start around a few hundred tokens with a little overlap so ideas don’t get cut mid-thought — but the honest answer is to chunk along the document’s natural structure: headings, sections, list items, not blind character counts. I’ve watched retrieval accuracy jump from the 60s to the 90s just by fixing chunking, before touching the model at all.

Roughly in order of effort: fix chunking first; try a stronger embedding model; add a re-ranker so the best of the top-k floats up; go hybrid — combine dense vector search with keyword/BM25 search, because dense retrieval famously whiffs on exact terms like error codes, SKUs, or names; and rewrite the query (expand acronyms, split multi-part questions). If the answer is spread across many documents, that’s where a graph or an agentic retrieval step earns its keep.

Split it in two. Retrieval: build a small golden set of question → correct-chunk pairs and measure hit rate / recall@k — is the right passage even showing up? Generation: measure faithfulness (did the answer stick to the retrieved context, or wander off?) and answer relevance (did it actually answer the question?). Tools like RAGAS automate this, often with an LLM as judge. The point: you can now change a chunking rule or a model and see a number move instead of trading opinions.

You never fully stop it — you shrink it. Instruct the model to answer only from the provided context and to say “I don’t have that information” when the context doesn’t cover it. Keep retrieval quality high so it isn’t forced to improvise. Add a faithfulness check that flags answers the sources don’t support. And show citations so a human can verify. The honest framing in an interview is “reduce and detect,” not “eliminate.”

That’s actually RAG’s superpower over fine-tuning — you update the index, not the weights. Run an ingestion pipeline that re-chunks and re-embeds changed documents (incrementally, not a full rebuild), handle deletes so stale facts don’t linger, and version your embeddings so you can re-embed cleanly when you switch models. No retraining, no downtime.

Cache embeddings and frequent queries. Retrieve fewer, better chunks — a re-ranker lets you send k=3 instead of k=10. Use a smaller, cheaper model for easy questions and escalate only when needed. Batch or stream where you can. And always measure cost-per-query — a RAG system that’s accurate but negative-margin doesn’t ship.