RAG vs. Fine-Tuning: A Decision Framework for Enterprise Teams

EXECUTIVE SUMMARY - Teams burn quarters debating RAG versus fine-tuning as if it were a technology preference. It is not. It is a property of your use case, determined by two questions: how fast does the knowledge change (volatility) and how far from stock behavior must the model act (specificity). Plot those two answers on the Volatility–Specificity Grid and the architecture chooses itself - including the two quadrants where the honest answer is "both" or "neither." This report gives you the grid, the cost asymmetry that most comparisons omit, and the escalation path that avoids buying tomorrow's problem today.

Two questions, not twenty

Question 1 - Volatility: if the model's knowledge were frozen today, when would answers become wrong? Product docs change weekly: high volatility. Regulatory interpretations shift quarterly: medium. Company writing style: nearly static. Volatile knowledge belongs in retrieval, where updating means re-indexing a document, not retraining a model.

Question 2 - Specificity: does the stock model, well-prompted, produce acceptable behavior - or do you need consistent specialized judgment, format, or voice that prompting can't reliably hold? Tone compliance across 10,000 outputs, rigid structured formats, domain-specific classification: that is weight-level behavior, and it is what fine-tuning actually buys.

The one-line separation that resolves most debates: RAG changes what the model knows; fine-tuning changes how the model behaves. Teams reach for fine-tuning to fix knowledge problems constantly - it is the most expensive possible way to build a database that is stale on arrival.

The quadrants

Prompting is enough (low/low). The majority of enterprise use cases on first honest inspection. System prompts, few-shot examples, and structured outputs - ship it, measure it, escalate only on evidence. Skipping this quadrant is how six-week projects become six-month programs.

RAG (high volatility, low specificity). Knowledge assistants, support deflection, policy Q&A. The work is unglamorous and decisive: document hygiene, chunking, retrieval evaluation, citation display. RAG quality is 80% retrieval quality - a mediocre model over excellent retrieval beats the reverse.

Fine-tune (low volatility, high specificity). Stable-domain classification, house-style generation, rigid extraction formats. Requirements people underestimate: hundreds-to-thousands of quality examples, an evaluation harness before training, and a retraining plan for every base-model deprecation - because your tuned model now lives on someone else's release calendar.

Both (high/high). Real - a fine-tuned model for behavior, retrieval for facts - and the most operationally expensive square on the board: two pipelines, two failure modes, compound debugging. Enter only after a single-technique version has demonstrably hit its ceiling, with metrics proving which dimension failed.

The asymmetry that decides ties

When a use case sits near a boundary, decide on cost of being wrong. Choosing RAG when you needed fine-tuning costs you a few weeks and some prompt frustration; the retrieval pipeline you built remains useful. Choosing fine-tuning when you needed RAG costs training runs, eval-set construction, and a model that confidently serves stale facts - and the sunk cost makes teams defend it for quarters. Boundary cases default to the reversible option. Default order: prompt → RAG → fine-tune → both, promoting only on measured failure.

FAQ

Doesn't long context replace RAG? Longer context windows shrink the chunking problem, not the freshness or access-control problems. You still need to decide what the model may see and when it was last true - that is retrieval governance, whatever the window size.

Is fine-tuning cheaper now? Training cost has fallen; lifecycle cost hasn't. Eval harnesses, example curation, and deprecation-driven retraining dominate, and they are headcount, not compute.

Where does MCP fit? Tool use via protocols like MCP is the third leg - for acting, not knowing or behaving. Many "RAG projects" are quietly better served by a tool call to the system of record. See MCP and Agentic Workflows.

Action checklist

• [ ] Score your use case 1–5 on volatility and specificity; plot it
• [ ] If either score ≤ 2: run the prompting baseline for two weeks first
• [ ] For RAG: budget more effort for retrieval evaluation than for the model
• [ ] For fine-tuning: build the eval harness before the training set
• [ ] Write the promotion criteria between ladder steps now, not after failure
• [ ] Re-plot annually - volatility drifts as businesses change

Related: The Analytics Stack for Growth-Stage SaaS · The Four Gates

Facing this decision on a live system? I run this as a one-day architecture decision workshop - grid placement, cost model, and the promotion plan. → Book a consultation