Concepts

RAG vs Fine-Tuning: When to Use Which (2026)

The Skillumen team ·Updated Aug 28, 2026 ·10 min read

The RAG vs fine-tuning question comes up constantly — in interviews, in design docs, in that Slack thread where someone says "let's just fine-tune it." And most of the time it gets answered backwards, because people treat it like a performance question ("which one is better?") when it's really a what-kind-of-problem-is-this question.

Here's the thing I wish someone had told me early: they're not competitors. They solve different problems, and once you can name which problem you actually have, the choice usually makes itself. So this post isn't a benchmark shootout. It's a decision guide — one mental model, a few clear rules, and a checklist you can run before you burn a week on the wrong approach.

The mental model that makes it click

Forget the architecture diagrams for a second. Here's the whole post in one analogy:

RAG is an open-book exam. The model doesn't need to have memorised anything. At question time, you go fetch the relevant pages, hand them over, and say "answer using these." The knowledge lives outside the model, in an index you control. If the facts change, you swap the pages — the model is untouched.

Fine-tuning is a closed-book exam. You study beforehand until the knowledge — or more usefully, the skill and style — is baked into the weights. At question time there's no lookup; the model just answers from what it internalised during training.

Open book means you can update the facts without touching the model. Closed book means the model behaves a certain way without needing any lookup. That single distinction decides almost every case.

Notice what that implies. If your problem is "the model needs to know something," that's an open-book problem — reach for retrieval. If your problem is "the model needs to act a certain way," that's a closed-book problem — reach for fine-tuning. Knowledge versus behaviour. Hold onto that; the rest of the post is just consequences of it.

When RAG wins

RAG is the default I reach for first, and here's when it's clearly right:

The facts change. Product docs, policies, prices, a support knowledge base, this quarter's numbers. Anything that would be stale next month belongs in an index you can re-embed, not in frozen weights.
You need citations or an audit trail. If a wrong answer is expensive — legal, medical, financial — you want to point at the source passage the answer came from. RAG gives you that for free; fine-tuning can't tell you why it said something.
The corpus is large. Thousands of documents won't fit in a prompt and aren't worth baking into weights. Retrieval pulls the handful of relevant chunks per question and ignores the rest.
You want to start cheap and move fast. No training run, no labelled dataset, no GPU bill. You can stand up a working RAG prototype in an afternoon and iterate on chunking and retrieval, which is usually where the real wins are anyway.

If you're new to the retrieval side of this, my what is RAG explainer walks through the pipeline end to end. The short version: RAG shines whenever the answer depends on current, specific, verifiable information.

When fine-tuning wins

Fine-tuning gets over-recommended, but there's a real set of problems where it's the right tool — and RAG can't touch them:

Consistent format, tone, or behaviour. You always want valid JSON in a specific shape. You want a house voice. You want the model to follow a rigid step-by-step procedure every time. That's a skill, not a fact — you teach it by example, and it belongs in the weights.
A domain style the base model doesn't have. Legal drafting cadence, medical shorthand, a company's particular way of writing tickets. You can nudge this with prompts, but if you need it reliably, fine-tuning bakes it in far more consistently.
Latency matters and there's no room for a retrieval hop. Every RAG query pays for an embedding lookup and a bigger prompt. If you're latency-sensitive and the "knowledge" is really just how to respond, fine-tuning skips that hop entirely.
You want a smaller, cheaper model to punch above its weight. Fine-tuning a small model on your task can match a much larger general model, which drops your per-call cost at scale.

The tell for fine-tuning: you find yourself writing a longer and longer system prompt full of "always do X, never do Y, respond like this" — and it still drifts. That drift is the signal that the behaviour wants to live in the weights, not the prompt.

The trade-offs side by side

Same distinction, viewed through the levers you'll actually be judged on:

                 RAG (open book)        Fine-tuning (closed book)
Freshness        update the index       stale until you retrain
Citations        yes, points to source  no, answer is opaque
Upfront effort   low, no dataset        high, need labelled data
Cost to start    cheap                  training + data prep
Latency          extra retrieval hop    none, one forward pass
Best at          knowledge / facts      behaviour / format / tone
Hallucination    grounded in retrieved  reduced drift, but still guesses
                 text; can still cite    beyond training data

A couple of these deserve a word. On hallucination control, RAG helps by grounding the answer in retrieved text you can verify — but only if retrieval actually found the right passage; bad retrieval gives you confident, well-cited nonsense. Fine-tuning reduces off-style drift but doesn't stop the model from inventing facts outside its training. Neither is a hallucination cure. On effort, the honest gap is data: RAG needs documents you probably already have, while fine-tuning needs a curated set of input-output examples, and building that dataset is usually the hard, unglamorous part everyone underestimates.

You can use both — and often should

The framing that trips people up is treating this as either/or. The mature answer, the one that lands well in an interview, is that the two compose:

Fine-tune for behaviour, RAG for knowledge. Fine-tune a model so it always responds in your format, your tone, your procedure — and then wire that model into a RAG pipeline so it answers over live, up-to-date facts it retrieves at question time. The weights handle how it responds; the index handles what it knows.

A support assistant is the classic example. You fine-tune it to sound like your brand and follow your escalation rules, and you give it RAG over the current help centre so it never quotes last year's refund policy. Neither approach alone gets you there. Together they do.

A quick decision checklist

When you're actually staring at a problem, run it through this. It resolves most cases in about thirty seconds:

Does the answer depend on facts that change? Yes → RAG.
Do you need to cite where the answer came from? Yes → RAG.
Is it really about format, tone, or a fixed procedure? Yes → fine-tune.
Is your system prompt ballooning and the model still won't behave? That drift → fine-tune.
Both a live-knowledge need and a strict-behaviour need? → do both.
Genuinely unsure? Start with RAG. It's cheaper, faster to stand up, and you'll learn a lot about your problem before committing to a training run you might not need.

And the rule of thumb to carry out of here, the one that survives most follow-up questions: knowledge that changes goes in RAG; behaviour and format go in fine-tuning. If someone asks you to defend a choice, start there and reason back to the specifics.

This exact decision shows up in interviews all the time, usually as a curveball after the RAG questions — you'll find it woven through the RAG interview questions too. Being able to say "this is a behaviour problem, so I'd fine-tune" without hedging is a strong signal.

FAQ

Is RAG or fine-tuning better?

Neither is better in general — they solve different problems. RAG adds knowledge the model looks up at question time (open book), while fine-tuning bakes behaviour and style into the weights (closed book). Ask what your problem actually is: knowledge that changes points to RAG, consistent behaviour or format points to fine-tuning.

Can you use RAG and fine-tuning together?

Yes, and it's often the strongest setup. Fine-tune a model so it responds in the right tone, format, and procedure, then put RAG in front of it so it answers over current, retrievable facts. The weights control how it behaves; the index controls what it knows.

Which is cheaper to start with?

RAG, almost always. It needs no labelled training dataset and no training run — you can stand up a working prototype in an afternoon using documents you already have. Fine-tuning needs curated input-output examples plus compute, and building that dataset is usually the expensive, time-consuming part.

Does fine-tuning stop hallucinations?

No. Fine-tuning reduces off-style drift and makes behaviour more consistent, but the model can still invent facts beyond its training data. RAG helps more with factual grounding because answers come from retrieved text you can cite and verify — but only when retrieval actually surfaces the right passage.

What's the quick rule of thumb for RAG vs fine-tuning?

Knowledge that changes goes in RAG; behaviour and format go in fine-tuning. If the answer depends on current or citable facts, retrieve it. If the problem is really about how the model responds, bake it into the weights. When in doubt, start with RAG.

Open-source companion: Awesome AI Engineer Interview Questions — 105 curated questions on GitHub, free.