This post is for CTOs and founders deciding whether a production AI system should use retrieval-augmented generation, a fine-tuned model, or both. It gives the cost math we run on Aiinfox discovery calls: per-token and infrastructure costs, the eval cycle that dominates the engineering bill, the latency trade-offs, and the four scenarios where fine-tuning earns its price.
The honest answer to "RAG or fine-tune?" almost always starts with "you probably want RAG", and the cases where fine-tuning is right are narrower than the industry conversation suggests. Of the roughly 50 production AI systems we have shipped, around 40 use RAG as the core architecture, around 8 use a fine-tuned model, and a small handful combine both deliberately. Most conventional wisdom on this topic is out of date (it cites 2023 model prices), oversimplified ("fine-tuning is cheaper at scale"), or selling something. The numbers below come from deployments we have shipped or audited.
The pricing landscape in 2026
Foundation-model pricing is materially cheaper per token than it was in 2023, driven by more efficient models, prompt caching, and provider competition. Mid-tier hosted models such as Claude Sonnet sit in the $1-5 per million input tokens range, with output tokens at $3-15. Prompt caching cuts the cached portion of the input cost by up to 90% for repeated system prompts and tool definitions.
Fine-tuning compute has not gotten dramatically cheaper. A LoRA fine-tune of an 8B open-weight model on 5,000 examples runs $100-500 in compute. A full fine-tune of a 70B model runs $5,000-30,000 depending on dataset size and provider. But compute is rarely the binding constraint. The engineering and data costs that surround a fine-tune dominate the total cost of ownership, and those have not gotten cheaper.
Where the money goes in a RAG system
A production RAG system's running cost decomposes into four buckets: LLM tokens, vector store, embedding generation, and observability. For a mid-volume system (100k queries a day, about 3k context tokens per query), the monthly numbers look like this:
- LLM tokens: $3,000-9,000 depending on model choice and prompt-caching efficiency
- Vector store: $200-800 for pgvector on managed Postgres, or $500-2,500 for Pinecone or Qdrant Cloud at this scale
- Embedding generation: $50-200 for incremental embedding of new content (the bulk cost is one-time at corpus ingestion)
- Observability and logging: $200-600 for Langfuse, Datadog, or similar
- Total: roughly $3,500-12,000 a month at this volume
LLM tokens are the dominant cost and the most controllable lever. Top-3 retrieval after re-ranking instead of top-10 unranked, prompt caching for the system prompt and tool definitions, and a right-sized model that clears the eval bar without going to the top tier together typically cut LLM costs by 60-80% versus the naive default. These are the same retrieval disciplines behind our RAG development services.
Where the money goes in a fine-tuning engagement
A fine-tuning engagement decomposes very differently. Compute is the smallest line item; engineering is the largest:
- Data curation: 2-6 weeks of senior engineering and domain-expert review. The bulk of the project cost and the largest source of risk. Typical: $25,000-80,000 in fully loaded hours.
- Eval harness construction: 1-2 weeks of senior engineering. You cannot fine-tune without a measurement instrument. Typical: $10,000-25,000.
- Training pipeline and experiment tracking: 1 week of MLOps for hyperparameter sweeps, checkpointing, and dataset versioning. Typical: $8,000-15,000.
- Compute for training runs: $500-30,000 depending on model size and number of runs. Often the smallest line item.
- Deployment and serving: 1-2 weeks of MLOps for vLLM or TGI, throughput tuning, and integration. Typical: $10,000-25,000.
- Ongoing refresh: every 3-6 months as the domain drifts, a smaller version of all of the above. Typical: $15,000-40,000 a quarter.
Latency: RAG adds milliseconds, fine-tuning subtracts them
RAG introduces retrieval-time latency: typically 20-80ms for hybrid retrieval, 10-30ms for re-ranking, and a little extra prompt processing for the retrieved context. End to end, RAG adds around 50-150ms to first-token latency compared with a non-RAG call. For most use cases this is irrelevant. For voice agents it can matter.
Fine-tuned models, especially smaller ones used as distilled replacements for larger models, can cut latency sharply. A fine-tuned 8B model on vLLM on a single A100 can deliver first-token latency in the 100-200ms range against 300-500ms for a hosted top-tier model. For latency-critical work such as voice agents and sub-second interactive applications, fine-tuning a smaller model on outputs from a larger one is one of the most reliable latency wins.
When to pick RAG: the default case
RAG is the right call when the system needs to ground its outputs in a body of knowledge that changes. "Changes" includes adding new documents, updating existing ones, or scoping to one tenant's data in a multi-tenant deployment. RAG handles all of these naturally; a fine-tuned model handles none of them without retraining.
- Customer support knowledge bases. Knowledge changes weekly.
- Medical inquiry agents grounded in clinical guidelines. Guidelines update, and the system needs to cite them.
- Legal research grounded in case law. Citations need to be verifiable against the source.
- Internal enterprise search and copilots. Per-tenant data, per-permission retrieval.
- Documentation Q&A. The docs are the source of truth and they update.
Our 98.4% citation-accuracy medical-inquiry deployment is RAG. Our legal-research agent is RAG. Our telco support agent handling 110k+ conversations a week is RAG with tool calls. The pattern is consistent: when the question is "what do my documents say", the answer is RAG.
When to pick fine-tuning: the four scenarios that earn it
Fine-tuning is the right call in four scenarios. Outside these, the engagement is almost always better served by RAG plus careful prompting.
Scenario A: Structured output the base model cannot reliably produce
If the system must emit a specific JSON schema, terminology, or format consistently, and prompting gets to 92% but not 99%, fine-tuning closes the gap. Document extractors are the classic case. The base model extracts invoice fields at around 92% accuracy; fine-tuning on 500-2,000 labeled invoices lifts it to 99.5% with reliably shaped JSON.
Scenario B: Cost or latency requires a smaller distilled model
When the top-tier model produces excellent answers at unit economics that do not work, fine-tuning a smaller open-weight model on outputs from the larger model (distillation) can give 90-95% of the quality at roughly a twentieth of the cost and a fifth of the latency. This is the second most common fine-tuning win we ship.
Scenario C: Data residency requires self-hosting with no hosted-API option
For deployments that require zero customer-data egress, common in healthcare, defense, regulated finance, and some EU clients, the system must be self-hosted. Open-weight base models without fine-tuning lag the hosted top tier on specialized tasks, so fine-tuning is often needed to close the quality gap.
Scenario D: Persona, voice, or tone that prompting cannot reliably hold
For consumer-facing applications where the AI's voice is part of the product (adaptive tutors, character agents, brand chatbots), prompting alone drifts across long conversations. Fine-tuning on 1,000-5,000 example dialogues locks in the persona. Our Mockinto deployment, an adaptive AI interviewer that lifted user completion by 47%, uses fine-tuning to hold the interviewer persona across 30-turn conversations where the base model would drift. The full write-up is in our case studies.
Hybrid patterns: when both make sense together
The cases where RAG and fine-tuning combine deliberately are narrow but high-value. The most common hybrid: a fine-tuned model for tone, structure, and refusal behavior, with RAG providing the facts at inference time. The fine-tune handles "how to answer"; retrieval handles "what to answer about".
Concrete example: a healthcare clinical assistant fine-tuned to keep a measured, refusal-prone tone appropriate for clinical context, with RAG supplying the current formulary, guideline, and institutional protocol. The fine-tune gives the system its voice and safety behavior. RAG gives it the facts that change every quarter as the formulary updates. Neither approach alone would deliver the production system.
Cost per incremental accuracy point
The most useful framing for the decision is cost per incremental accuracy point. If your system is at 92% and you need 97% to pass the production bar, the question is which lever gets you there for less.
- Improving retrieval (hybrid retrieval, better chunking, re-ranking): typically 5-15 percentage points of recall lift at 1-2 weeks of engineering. The cheapest lever.
- Tightening the citation requirement and refusal threshold: typically 3-8 points of accuracy lift on grounded responses at about a week of engineering.
- Upgrading from a mid-tier to a top-tier model: typically 1-3 points of lift on a properly engineered RAG system, at materially higher inference cost.
- Fine-tuning on domain data: typically 5-15 points of lift on specific structured-output or persona tasks, at $60,000-180,000 of engagement cost.
The honest order of operations: exhaust the RAG levers first. Retrieval and chunking changes get you most of the way for most use cases. Fine-tuning earns its cost only when the gap is structural (Scenarios A-D above) and the eval set proves the cheaper levers cannot close it. When it does, it runs as a scoped LLM development engagement with the eval harness built first.
The decision tree, condensed
- Answers must be grounded in documents that change: RAG.
- The system needs per-tenant data in a multi-tenant deployment: RAG.
- The system needs a specific structured output the base model cannot reliably produce: fine-tune.
- You need a smaller, cheaper, faster model with quality close to the top tier: distill via fine-tuning.
- The system must be self-hosted with zero data egress: open-weight model plus fine-tune, usually.
- The system needs a consistent persona across long conversations: fine-tune.
- The system needs both grounded facts and a maintained tone: hybrid, fine-tune for tone and RAG for facts.
- None of the above, but the base model is not good enough: improve prompting, retrieval, and tool use first.
Wrapping up
RAG is the default for production AI in 2026 because it handles the most common production constraint, that the knowledge changes, without retraining. Fine-tuning is the right call in four well-defined scenarios: structured output, distillation, self-hosting, and persona consistency. The hybrid pattern is real but narrow. Most engagements are best served by exhausting the RAG levers before considering fine-tuning, because the cost-per-accuracy-point math favors RAG for most use cases.
If you are deciding between RAG and fine-tuning on a specific build and want a 30-minute conversation that runs the decision tree on your actual constraints rather than recites principles, book a discovery call. We will tell you which architecture fits, what the engagement looks like, and deliver a fixed-price scope inside 72 hours.
Frequently asked questions
Is fine-tuning cheaper than RAG at scale?
Usually not. Fine-tuning compute is cheap, but data curation, the eval harness, serving, and the quarterly refresh dominate the cost, and a fine-tuned model still cannot absorb new documents without retraining. RAG's running cost is mostly LLM tokens, which prompt caching and re-ranked top-3 retrieval cut sharply.
When does fine-tuning beat RAG?
In four scenarios: structured output the base model cannot produce reliably, distilling a smaller and faster model, self-hosting with zero data egress, and holding a consistent persona across long conversations.
Can RAG and fine-tuning be combined?
Yes, but the useful cases are narrow. The common pattern is a fine-tuned model for tone, structure, and refusal behavior, with RAG retrieval supplying the facts that change at inference time.