RAG vs Fine-Tuning: How to Choose What Your AI App Actually Needs

If an assistant keeps quoting last month's return policy, I would fix how it gets current information before training a model. If it has the correct policy in front of it but repeatedly applies your support labels incorrectly, I would investigate the instructions, examples and model behavior instead. Calling both problems “hallucinations” hides the distinction that should determine your next engineering task.

RAG and fine-tuning can be useful in the same application, but they are poor substitutes for diagnosing a failure. My starting preference is to keep changing business information in a system that can be inspected and updated, then consider training when a stable, measurable behavior still needs improvement.

What each approach changes

Retrieval-augmented generation, or RAG, supplies relevant information when a request runs. The application searches a source, adds selected evidence to the model's input, and asks it to answer using that evidence. Microsoft's RAG overview describes keyword, vector and hybrid search as possible retrieval methods, so choosing RAG does not automatically mean buying a vector database.

Supervised fine-tuning changes a model using examples of inputs and desired outputs. OpenAI's fine-tuning guide describes applications such as classification, translation and instruction following, and recommends establishing evaluations before investing in training.

For a first decision, ask whether the missing ingredient is evidence the model needs for this particular request, or a repeatable behavior you can demonstrate with representative examples. That distinction will not settle every case, but it gives you a hypothesis you can test rather than an architecture chosen from a diagram.

What is failing?	What I would try first	What would justify more work?
Answers depend on changing policies or product documentation	Retrieve the applicable version and preserve its source	Evidence still fails to reach the answer, or the model misuses correct evidence
An exact order status or account value is needed	An authorized lookup against the system of record	The query itself requires a more capable interface
A recurring classification or writing behavior is inconsistent	Clear instructions and a small set of examples	The same failure persists on unseen cases and good training examples exist
Output keys or types are inconsistent	A supported structured-output interface and validation	Correctly shaped outputs still fail the actual task
The baseline already meets the product requirement	Keep the simpler system	New traffic reveals a specific failure worth addressing

The table is a decision aid, not a claim that one method always wins. A database lookup may be enough for order status, while a long policy question may need retrieval across several documents. Adding training to either workflow should have a reason beyond making the architecture look more sophisticated.

Locate the failure before changing the model

Take a manageable sample of failed requests and reconstruct what the application actually supplied to the model. Do this in an environment approved for the data, with customer identifiers and secrets removed from shared debugging examples.

For each case, record the question, the authoritative answer, the source version, the retrieved material and the final response. Then assign the earliest failure you can establish:

• The evidence did not exist: the source system lacks an answer, so the product needs clarification or escalation rather than a confident guess.
• The evidence existed but was not retrieved: investigate ingestion, filters, query construction or ranking before changing generation.
• The right evidence was retrieved but superseded: check effective dates and version selection.
• The right evidence was present but misused: inspect the prompt, task difficulty and model behavior.
• The answer was correct but the application rejected it: inspect parsing, schema and business-rule validation.
• The answer exposed information the requester should not see: treat access control as a release blocker, even if the answer was accurate.

These labels become useful when they change ownership. An indexing problem belongs in the retrieval backlog; an outdated source needs a content owner; a policy-application failure needs a controlled generation test. A larger training set is unlikely to be the most direct response to all three.

Run a controlled evidence test

Before building a training pipeline, try the same held-out question in three conditions:

1. Current baseline: use the existing prompt and inputs exactly as the application would.
2. Manually selected evidence: supply the small, authoritative excerpt a reviewer believes is sufficient, without adding the expected answer as an instruction.
3. Actual retrieval: let the proposed retriever select the evidence from the same approved document collection.

Keep the model, output contract and scoring rules fixed for this comparison. If the answer improves with manually selected evidence but fails with actual retrieval, the next experiment should focus on getting the right evidence into the request. If both evidence conditions fail, investigate how the model uses that evidence rather than assuming more search will help.

This test narrows the problem; it does not prove causality from one successful answer. Repeat representative cases and inspect failures, especially when the manually selected excerpt was unusually clean compared with real documents. If an excerpt was chosen with knowledge of the expected answer, label that condition as a diagnostic aid rather than a production-quality result.

Only after the behavior remains deficient with adequate evidence would I compare a tuned model against the strongest practical prompt-only baseline. Include a different base model if that is an available, lower-effort option, and keep the training set separate from the cases used to judge the result.

Use a record that makes the evidence visible

Here is a synthetic evaluation record for a fictional returns assistant. It defines the expected behavior but leaves the actual result empty, so it cannot be mistaken for a completed benchmark. The made-up policy allows returns within 21 days, and the question concerns day 25.

{
  "case_id": "returns-example-001",
  "split": "held_out",
  "question": "I bought this 25 days ago. Can I return it?",
  "requester_scope": "public_policy_only",
  "source_snapshot": "example-policy-v3",
  "permitted_source_ids": ["returns-v3"],
  "source_excerpt": "Returns are accepted within 21 days of purchase.",
  "expected_behavior": {
    "must_state": ["The request is outside the 21-day window"],
    "must_not_state": ["A return is guaranteed", "An exception is approved"],
    "requires_source": true
  },
  "run": {
    "variant": null,
    "model_id": null,
    "prompt_version": null,
    "retrieved_source_ids": [],
    "answer": null,
    "latency_ms": null,
    "billable_cost_usd": null
  },
  "review": {
    "evidence_sufficient": null,
    "answer_supported": null,
    "permission_check_passed": null,
    "accepted": null,
    "failure_reason": null
  }
}

The JSON was parsed locally to check its syntax; no model was run on this case. In your implementation, store enough source content or a controlled snapshot reference to reproduce the judgment later. An unexplained source ID is not very useful after the underlying document has changed.

Keep permission scope separate from the question text, because the application should decide what the requester can access. OpenAI's retrieval documentation supports attribute filters before semantic search, but a filtering feature only helps if your application supplies the correct attributes and trusted criteria. Never let a user-provided tenant name become the sole authorization check.

Score the outcome without hiding the failure

OpenAI's evaluation guidance recommends task-specific tests, representative cases and human calibration of automated scoring. You can apply those principles in your own test harness; this guide does not require OpenAI's hosted Evals service, which is being retired.

For a document-answering workflow, I would keep the following judgments separate rather than collapse everything into one impressive-looking score:

Check	Question the reviewer must answer
Retrieval sufficiency	Did the permitted material contain enough evidence to answer?
Answer support	Does the response follow that evidence without inventing conditions?
Source attribution	Can the reader inspect the source that supports the relevant claim?
Abstention	Does the assistant ask or escalate when the evidence is insufficient?
Authorization	Were only permitted records exposed?
Completion	Is the result actually usable for the task?

Decide which failures are unacceptable before looking at results. For example, I would not trade a cross-customer disclosure for a higher average answer score. That is a product judgment, not a threshold that a model vendor can choose for you.

Build the held-out set from more than easy questions. Include ambiguous requests, old and new policy versions, missing facts and attempts to retrieve another customer's information. Keep closely related records together when splitting training and test data, so a paraphrased copy of a training example does not make the evaluation look stronger than it is.

If an automated judge scores answers, review its disagreements with people and check that it is not rewarding verbosity or merely spotting the expected keywords. The synthetic record's must_state field is a review criterion, not a recommendation to accept every answer containing the phrase “21 days.”

Check the operating cost and current availability

For RAG, budget for preparing and updating the index, retrieving evidence and sending additional context to the model. For tuning, budget for preparing examples, training, evaluating and maintaining the resulting deployment. Compare cost per accepted task and latency under the same acceptance rules, including retries and human correction rather than looking only at the model's token price.

Availability can eliminate an otherwise attractive option. As checked on November 6, 2026, OpenAI's fine-tuning availability notice says its platform is closed to new users and is winding down for existing users. Do not treat an older tutorial's working training command as proof that a new account can use it. Existing users should check the current job-creation and base-model retirement deadlines before depending on a tuned deployment.

That restriction concerns a particular hosted platform, not fine-tuning as a technique. If you consider another provider or a self-managed model, verify its current training access, model license, deployment requirements and data handling before investing in dataset preparation.

My preference is to solve the smallest demonstrated problem first: connect reliable evidence when knowledge is missing, improve instructions when the task is underspecified, and train only when you have a stable failure that examples can plausibly address. If retrieval and tuning together win a fair comparison, use both, but make the measured improvement justify the extra system you will have to maintain.

I'm Paras, a technical writer working with AI and developer-tool teams.

I help turn engineering decisions into clear technical guides and product stories that developers can use.

Book a 30-minute call.