RAG vs Fine-Tuning: Which One, and When to Skip Both

Check availability before you compare anything else. As of August 2026, OpenAI's docs say its fine-tuning platform is closed to new users, the Gemini Developer API pricing page publishes no tuning price for any model, and Anthropic sells no first-party fine-tuning. For many teams on the three biggest APIs, that branch is already shut. The real choice is retrieval, better prompting, or the whole corpus in the context window.

Key takeaways

• Branch one is availability, not cost — confirm the model you intend to ship on can be tuned at all before reading another comparison
• Retrieval edits the prompt; fine-tuning edits the weights. OpenAI files missing, outdated, and proprietary knowledge under context optimization, not fine-tuning
• Fine-tuned inference on OpenAI is priced above the base model — gpt-4o costs $2.50/$10.00 per 1M tokens, its fine-tuned variant $3.75/$15.00
• On Amazon Bedrock a customized model requires Provisioned Throughput, which bills hourly until you delete it, traffic or no traffic
• Under about 200,000 tokens, Anthropic says skip retrieval entirely and cache the whole knowledge base in the prompt
• If any answerable fact changes faster than you will run and evaluate a training job, weights cannot be your knowledge store

What each one actually changes#

Retrieval edits the prompt. Fine-tuning edits the weights. Almost every practical difference follows from that one sentence.

OpenAI's guide to optimizing accuracy frames the problem as two axes rather than one spectrum. Context optimization covers the model that lacks knowledge because it was not in the training set, whose knowledge is out of date, or that needs proprietary information; that axis, the guide says, maximizes response accuracy. Model optimization covers inconsistent results, incorrect formatting, wrong tone, and reasoning that is not followed consistently; that axis maximizes consistency of behavior.

Fine-tuning sits on the second axis. The same guide describes it as continuing training on a smaller, domain-specific dataset — showing the model many examples of a task being performed correctly. It also notes that retrieval only solves the in-context learning issue, and that with retrieval you have to tune the retrieval as well as model behavior. Two axes, two tools, and the failure mode is confusing them.

Match the symptom to the axis before you pick a technique

What you actually observe	Which axis is failing	The tool that fits
The model does not know your refund policy exists	Missing context	Retrieval, or the whole corpus in the prompt
It knows the policy but answers in prose when you need structured fields	Behavior	Prompt and output format first, fine-tuning last
Answers are correct and the tone is wrong on every third reply	Behavior	Prompt engineering, then preference tuning only if it survives eval
It confidently names a product you discontinued last week	Missing context plus stale data	Retrieval — a retraining cycle cannot keep up
Users cannot tell where an answer came from	Neither — this is a citation problem	Retrieval with citations; weights cannot cite

Branch one is availability, not cost#

Capability and cost are the usual first questions. Availability now settles the matter before either one gets asked.

OpenAI. The fine-tuning guide leads with a wind-down notice. It states the platform is "no longer accessible to new users," and that existing users will be able to create training jobs for the coming months. No end date is published for that window. Models you already fine-tuned remain available for inference until their base models are deprecated, so nothing breaks today — but the road ahead is closed to newcomers.

Google. The Gemini Developer API pricing page publishes no tuning price for any model. Most models carry no tuning row at all; Gemini 2.0 Flash and Gemini 2.0 Flash-Lite carry one, and both read "Not available" on the free tier and the paid tier alike. Vertex AI is a different product and does document tuning: supervised fine-tuning, reinforcement learning fine-tuning, preference tuning, and continuous tuning, with LoRA and QLoRA named as efficiency techniques. If you built against the Developer API, moving to Vertex is a platform migration, not a config flag.

Anthropic. There is no fine-tuning price anywhere on Anthropic's pricing page, because Anthropic does not sell first-party fine-tuning. Amazon Bedrock's fine-tuning documentation lists exactly one Anthropic model as customizable: Claude 3 Haiku, model ID anthropic.claude-3-haiku-20240307-v1:0:200k, supported in us-west-2 only. No Opus, Sonnet, or Haiku 4.5 appears in that table.

Bedrock's fine-tuning table is not empty. It lists Amazon Nova 2 Lite, Nova Lite, Nova Micro, Nova Pro and Nova Canvas, Titan Image Generator G1 v2, Titan Multimodal Embeddings G1, and several Meta Llama 3.1, 3.2 and 3.3 variants. If the model you intend to ship on is one of those, the branch is genuinely open — with one restriction worth checking before you plan around it. Every row in that table is single-region: the Nova models in us-east-1, the Llama variants in us-west-2, and the two Titan models in either. Fine-tuning on Bedrock pins your custom model to a region, whatever your inference footprint looks like today.

The cost profile everyone prices wrong#

The mistake in the usual comparison is not the arithmetic. It is the billing unit.

Start with the public part. OpenAI lists fine-tuning training at $25.00 per 1M tokens for gpt-4.1 and gpt-4o, $5.00 for gpt-4.1-mini, and $3.00 for gpt-4o-mini (checked August 2026, OpenAI pricing).

Training is the one-time cost. The recurring one is worse.

Base versus fine-tuned inference on OpenAI, per 1M tokens (checked August 2026)

Model	Base input / output	Fine-tuned input / output	Premium
gpt-4o-2024-08-06	$2.50 / $10.00	$3.75 / $15.00	50% on both
gpt-4o-mini-2024-07-18	$0.15 / $0.60	$0.30 / $1.20	Exactly 2x on both

Fine-tuning does not just cost you a training run. It raises your per-token rate for every request the model ever serves. Any payback calculation that ignores that is off by a multiplier, not a rounding error.

Then there is the structural change. AWS states it directly: if you customized a model, you must purchase Provisioned Throughput to use it. Provisioned Throughput is billed hourly, and AWS says billing continues until you delete it — meaning you pay through quiet nights and failed launches alike.

A fine-tuning payback period for Claude cannot be computed from public sources, by anyone. The Amazon Bedrock pricing page lists no Anthropic customization price, no Claude custom model storage price, and no Claude Provisioned Throughput hourly rate. Customization pricing appears for other providers, not for Anthropic.

Every confident TCO table on this topic filled that gap with a guess. If you need real numbers for a budget, ask your account manager and put their answer in the spreadsheet. The guide to estimating LLM API costs covers the token side of the same build.

What fine-tuning needs that retrieval does not#

A pile of 4,000 support documents is a retrieval corpus, not a training set. The gap between those two things is measured in weeks of human labeling.

OpenAI's fine-tuning guide documents methods with genuinely different input requirements. Supervised fine-tuning takes examples of correct responses — human-generated ground truth — and suits classification, specialized translation, format-specific content generation, and correcting instruction-following. Direct preference optimization requires both a correct and an incorrect response for each prompt, and is recommended for summarization and for chat with a specific tone or style. Reinforcement fine-tuning needs an expert grade on each generated response, and targets complex domain reasoning such as medical diagnoses and legal precedent analysis.

Read that DPO requirement again. You need someone qualified to write the good answer and the bad one, for every example, with a defensible view of why one is worse. Retrieval asks only that your documents be findable and correct.

The deciding variable: how often your data changes#

Here is the single question that resolves most of these debates. How often does a fact your users can ask about change?

A document update is a re-index for retrieval and a retraining run for fine-tuning. Those are not comparable operations. Embedding a million tokens with OpenAI's text-embedding-3-small is priced at $0.02; running a million training tokens through gpt-4.1 fine-tuning is priced at $25.00 (both checked August 2026).

That is the same corpus at more than a thousand times the cost, spent on the axis OpenAI's own accuracy guide reserves for consistency of behavior rather than for knowledge. And the money is the small part — the retraining run also needs a new eval pass, a rollout decision, and a rollback plan.

Latency, and the levers that are actually documented#

Retrieval adds a network hop and extra input tokens. That is as far as the structural statement goes: no primary-source measurement of retrieval overhead versus fine-tuned inference is published. Anyone quoting you a millisecond figure should show you the methodology.

The levers with published numbers are pricing levers. Anthropic's pricing page lists prompt caching multipliers against the base input price: 1.25x for a five-minute cache write, 2x for a one-hour cache write, and 0.1x for a cache read. The Batch API applies a 50% discount to both input and output tokens and combines with caching, which is the right answer for anything asynchronous.

Caching has a floor, and it is worth naming. A cache write costs more than sending the same text uncached, so a cache that expires before anything reads it costs you 1.25x the plain rate for no benefit. Anthropic puts the break-even at one read for the five-minute cache and two reads for the one-hour cache. Traffic thinner than that, and caching is a loss.

One caution on cross-model math: Anthropic notes that Claude 4.7 and later use a newer tokenizer producing approximately 30% more tokens for the same text, with the exact increase depending on content and workload shape. Claude Sonnet 4.6 and earlier use the previous tokenizer. Comparing per-token rates across model generations without adjusting for that will mislead you.

Do neither: the case for putting the corpus in the prompt#

Before you stand up a vector store, check the size of the thing you are indexing.

Anthropic's contextual retrieval write-up gives a specific threshold: a knowledge base under about 200,000 tokens, roughly 500 pages, can simply go in the prompt with caching to keep cost sane. Contextual retrieval becomes necessary past that point. Anthropic also notes Claude 4.6 and later include the full 1M-token context window at standard pricing — a 900k-token request bills at the same per-token rate as a 9k-token one.

Past the threshold, fix retrieval before you consider weights. Anthropic published measured results for three techniques: contextual embeddings alone cut the top-20-chunk retrieval failure rate by 35% (5.7% to 3.7%), adding contextual BM25 took it to 49% (2.9%), and adding reranking reached 67% (1.9%). Those are Anthropic's numbers on Anthropic's evaluation — treat the mechanism as general and the percentages as specific to that setup.

The common mistake: fine-tuning to fix a prompting problem#

OpenAI's accuracy guide says prompt engineering is typically the best place to start, and advises squeezing as much accuracy from basic methods as you can before reaching for RAG or fine-tuning. That advice is ignored constantly, usually because fine-tuning feels like the serious engineering answer.

You can tell the difference in one afternoon. Pull twenty real failures and read them by hand. For each one, ask whether the model lacked a fact, or had the fact and used it badly.

Missing facts are a context problem, and no amount of training fixes them. Correct facts rendered in the wrong format, wrong tone, or wrong structure are a behavior problem, and the first three fixes to try are a clearer instruction, a worked example in the prompt, and a constrained output format. The guide to prompts that survive model updates covers how to write that instruction so it does not rot at the next upgrade.

Build the eval set before you pick a technique#

You cannot choose between two techniques without a way to score them, and the eval set is also the cheapest artifact in this whole process. OpenAI makes a useful side observation here: writing the prompt forces you to define what accuracy means for your use case, which is exactly the definition your eval needs.

Retrieval also gives you something a fine-tuned model structurally cannot — a verifiable answer. Anthropic's citations feature returns the exact passages supporting each claim, and all active models support it. Search result content blocks let Claude cite your own content the way it cites web search results, carrying the source and title you supply, with no beta header required and no special prompting needed.

Two honest limitations before you build on that. Anthropic's docs state only text citations are supported and image citations are not yet possible. And citations cannot be combined with structured outputs — enabling both returns a 400 error, a real constraint if your downstream consumer expects strict JSON. For the evaluation harness itself, see the walkthrough on testing AI output quality.

The decision tree#

1. Can you fine-tune the model you intend to ship on?

   Open the provider's own page today, not a summary of it. If you are new to OpenAI, the platform is closed to you. If you are on the Gemini Developer API, no model carries a tuning price. If you are on Claude, there is no first-party option and Bedrock lists only Claude 3 Haiku, us-west-2 only. A "no" here ends the debate in under a minute.

2. Read twenty real failures. Missing knowledge, or wrong behavior?

   Missing knowledge goes in the context window. Wrong behavior goes to the prompt first. Do not proceed on a hunch — the classification of these twenty examples is the whole decision.

3. Is the corpus under about 200,000 tokens?

   Then put all of it in the prompt with caching, per Anthropic's guidance, and skip the retrieval pipeline entirely. Revisit only when the corpus outgrows the threshold.

4. Does any answerable fact change faster than you will retrain?

   If yes, retrieval is the only option that keeps up. Weights are a snapshot; your catalog is not.

5. Do users need to see where an answer came from?

   Retrieval with citations gives you inspectable sources. A fine-tuned model gives you a confident sentence and no provenance.

6. Still failing on behavior after prompting and retrieval?

   Only now does fine-tuning earn a look, and only if you are on a platform where it is available, have labeled pairs in the right shape, and have an eval set that will tell you whether it worked.

Worked example: a support assistant over a weekly-changing catalog#

You are building a support assistant over a product catalog. The full export is roughly 180,000 tokens, and prices and stock change every week. You plan to ship on Claude Sonnet 4.6.

Step one takes thirty seconds. Anthropic sells no first-party fine-tuning, and Bedrock's fine-tunable list does not include Sonnet. The branch is closed and you have saved yourself a two-week evaluation.

Step three resolves the rest. At 180,000 tokens you are under Anthropic's 200,000-token threshold, so the catalog goes in the prompt. Sonnet 4.6 carries the 1M-token context window at standard pricing, so a 180,000-token prefix is not a squeeze. Priced from Anthropic's published per-MTok columns for that model, the arithmetic looks like this.

Cost per request for a 180,000-token catalog on Claude Sonnet 4.6, using Anthropic's published rates (base input $3.00/MTok, checked August 2026). Corpus size is an assumption; the rates are not.

Approach	Rate applied	Cost for 180,000 input tokens
Send the catalog uncached, every request	$3.00 / MTok	$0.540
Five-minute cache write (paid once per cache)	$3.75 / MTok	$0.675
Cache read on every subsequent request	$0.30 / MTok	$0.054
One-hour cache write (paid once per cache)	$6.00 / MTok	$1.080

Anthropic states the five-minute cache pays off after a single read, and the one-hour cache after two. On any real support volume you are paying about a tenth of the uncached rate for context, with output tokens billed separately at the model's standard rate.

The weekly update is a file replacement. You regenerate the prompt from the new export and the next cache write picks it up. No training job, no eval regression suite for a new model artifact, no Provisioned Throughput ticking hourly in us-west-2.

What would fine-tuning have bought here? Not the catalog — this week's prices are proprietary, fast-changing information, which is the context axis by OpenAI's own division of the problem, so a tuned model would still need them in the prompt. You would have paid for training, paid an inference premium, lost citations, and still had a stale catalog. When the corpus outgrows 200,000 tokens, the upgrade path is contextual retrieval with search result blocks, not weights.

Who this is not for#

Teams running open-weight models on their own hardware. Availability is not your constraint and the economics differ entirely. This tree collapses to steps two through five for you.

Anyone without an eval set. Neither technique helps if you cannot measure whether it helped. Build the eval first; it is a day of work and it makes every later decision cheap.

Teams whose source documents are wrong. If your policy pages contradict each other, retrieval will faithfully surface the contradiction and fine-tuning will average it into something worse. Fix the documents.

Existing OpenAI fine-tuning customers. The wind-down changes your roadmap, not your running system — the docs say fine-tuned models stay available for inference until their base models are deprecated. Use the runway to build the retrieval path. The framework for evaluating AI tools before buying applies to platform bets like this one.

What could not be verified#

Google publishes no Vertex AI tuning price on the pages checked, so none is stated here. AWS publishes no Anthropic customization price, no Claude custom model storage price, and no Claude Provisioned Throughput hourly or per-Model-Unit rate — it directs you to your account manager. That makes a Claude fine-tuning payback period uncomputable from public sources, and any article showing you one has invented it.

No primary measurement of retrieval latency overhead could be found, no vendor publishes a minimum training-example count, and OpenAI publishes no date for when it closed the platform to new users. OpenAI's pricing page also carries no headline rate for reinforcement fine-tuning — it states only that tokens used for model grading are billed at that model's per-token rate — so no reinforcement fine-tuning price is quoted here. If a number matters to your budget, get it from the vendor in writing. Before you send a corpus anywhere, run it past the AI tool security checklist.

The verdict#

Default to retrieval, and reach for the context window before you reach for retrieval.

The rule in one line: under about 200,000 tokens, cache the corpus in the prompt; above it, build retrieval and fix retrieval quality first; consider fine-tuning only when the failure is behavior rather than knowledge, the platform still offers it, and you have labeled pairs plus an eval set that can prove it worked.

That sequence is cheaper, faster to ship, and — as of August 2026 — the only one still open to most teams on the biggest APIs. Re-run branch one before you plan a quarter around it. It changed more than anything else in the last twelve months, and it will change again.