Shipping RAG in Production: What Nobody Tells You

This post is for engineering leads and CTOs who have a RAG prototype that impressed the demo audience and are about to put it in front of real users. It covers the six failure modes we see in almost every production RAG deployment and the fixes, most of which have nothing to do with the LLM.

Most RAG prototypes look great on the demo deck and fall apart in week three of production. The vector store returns the wrong chunks. The model invents a citation that does not match the source document. A user asks something out of scope and the system fabricates an answer instead of refusing. The team patches the prompt, ships the patch, regresses something else. Three months in, nobody trusts the bot.

We have shipped retrieval-augmented generation systems for healthcare, finance, telco, staffing, and EdTech. Across those deployments the failure modes are the same, the fixes are the same, and almost none of them are about the model.

Vector-only retrieval misses keyword-bound queries

Dense embeddings are good at semantic similarity and poor at exact-match retrieval. Your support agent asks about "error code 0x80070005" and the vector store returns documents about general permission errors, because the embedding model never learned that the error code is the signal. The same applies to drug names, SKU codes, legal citation formats, regulation numbers, and product version strings. If your domain has any of those, vector-only retrieval will underperform.

The fix is hybrid retrieval: dense embeddings plus BM25 lexical search plus a re-ranker. In our healthcare medical-inquiry deployment, switching from vector-only to hybrid retrieval lifted top-3 recall from 78% to 96% on the same eval set without changing the LLM.

Chunking matters more than the embedding model

Teams obsess over which embedding model to use and ignore the question that actually moves the needle: how the source documents are broken into chunks. Chunking is where context dies. A naive 512-token sliding window splits a contract clause in half, separates a question from its answer, and orphans a footnote from the table it explains. Retrieval then returns a fragment that means almost nothing on its own.

• Chunk by structural boundary (heading, table, list, section), not arbitrary token count
• Add document-level metadata (title, source, section) to every chunk so retrieval can filter on it
• Use overlapping windows for narrative content and strict-boundary chunks for tabular or reference content
• Re-check chunking first when accuracy regresses; most regressions trace back here before the model

Required citations, or no answer ships

Hallucinations in RAG almost always come from the model answering on its own when retrieval returned nothing relevant. The fix is structural, not a prompt tweak: require a citation on every answer and reject any answer without one. If the model cannot ground its response in a retrieved chunk, the system says "I do not have enough information to answer" and routes to a human or a follow-up question. Users learn to trust this faster than they trust a printed "AI confidence score".

The eval harness is the contract. Write it first

Every team building RAG plans to set up evals later. Later never comes. The team that ships a working RAG system at week six is the team that wrote the golden eval set at week one. The eval set is 200-500 real queries from your real users (or your best simulation of them), each with a correct answer, the source it should cite, and a refusal flag for out-of-scope queries.

Every prompt change, model swap, and chunking tweak runs against that set. The harness reports retrieval recall, citation accuracy, refusal correctness, and end-to-end answer quality, and anything that regresses past threshold is blocked from shipping. This is the single highest-leverage piece of RAG infrastructure and the most commonly skipped.

Cost and latency are decided at retrieval time

The dominant cost in production RAG is usually LLM tokens, which means it is dominated by how many retrieved chunks you put into the prompt. Teams default to top-10 or top-20 retrieval and pay several times the cost of a properly re-ranked top-3 setup that scores higher on the eval set. Smaller context windows are cheaper, faster, and often more accurate because the model has less irrelevant content to weigh.

• Default to top-3 after re-ranking, not top-10 before re-ranking
• Use prompt caching (Anthropic, OpenAI) for the system prompt and tool definitions; the cached portion bills at up to 90% less than base input price
• Cache embeddings server-side and re-embed only on document change, not per query
• Set per-step latency budgets (retrieval, re-rank, LLM) and alert on regressions

Permissions are part of retrieval, not a layer above it

In multi-tenant or permission-aware deployments, which is most enterprise RAG, the temptation is to retrieve from a shared vector store and filter results afterwards based on the calling user's permissions. This is wrong on two levels. The embedding cost is wasted on documents the user will never see, and post-filtering leaks information through retrieval timing and through partial results that show up in logs and analytics.

The right pattern is permission-aware retrieval: the user's permission scope is part of the query, and the vector store filters at the index level before similarity search. In Postgres with pgvector, that is a WHERE clause on a tenant_id or role column. In Qdrant or Weaviate, it is a payload filter combined with the vector search.

Wrapping up

RAG is mostly an engineering-discipline problem, not a model problem. Pick the simplest retrieval architecture that clears your eval bar, require citations, run a refusal layer, and gate every change against your golden set. The teams that get this right ship in 6-8 weeks. The teams that obsess over which LLM provider has the best benchmark numbers ship a brittle demo and rebuild it in month four.

If you have a RAG prototype and want it to survive production, our RAG development services cover hybrid retrieval, citation enforcement, and the eval harness described above, and the same patterns apply to a support-facing AI chatbot. Contact us for a 30-minute review of your current architecture.