There is a familiar arc to most retrieval-augmented generation projects. A team spins up a demo: vector database, embedding model, the company's documentation chunked at 512 tokens, a prompt template, an LLM call at the end. The demo works. Leadership is impressed. Pilot users praise it. Eight weeks after launch, the same team is firefighting daily — wrong answers, missing answers, confidently fabricated answers, latency complaints, and a bill that nobody budgeted for.
The pattern is so common it has become a category. In 2024 RAG moved from research novelty to production reality, with major vendors integrating it into their platforms, but real-world deployment revealed critical gaps including retrieval precision failures, inability to explain answers, and security vulnerabilities where poisoned documents could trigger specific model behaviour. The gap between a polished demo and a system that holds up under real users is not a tuning problem. It is an engineering problem, and it is usually larger than the team expected.
This post is about the failure modes that show up between week two and month two of real use, and the practices that prevent them.
Retrieval is where most failures actually live
The instinct, when a RAG system returns a wrong answer, is to blame the language model. The instinct is usually wrong. Most RAG failures start in retrieval, not generation — when the system retrieves incomplete, irrelevant, or poorly ranked evidence, even a strong LLM will produce weak answers.
A more sobering finding comes from recent academic work auditing the interplay between retriever and generator. Empirical analysis on standard benchmarks revealed that for 47.4% to 66.7% of queries, generators ignored the retriever's top-ranked documents, while 48.1% to 65.9% relied on documents ranked as less relevant. In other words: even when retrieval brings back the right thing, the model frequently uses something else. The two halves of the pipeline disagree about what matters, and nobody is watching.
A practical taxonomy of retrieval-side failure looks like this:
- Retrieval miss. The right chunk was never returned. The model falls back to its parametric memory and invents an answer.
- Context leak. The model pulls in prior knowledge that contradicts the retrieved text and silently overrides the source.
- Generation drift. The retrieved chunk was correct, but the model rephrased it in a way that changed the meaning.
The three modes are described concisely in a recent practitioner guide on evaluation. A retrieval miss means the right chunk was never returned; context leak means the model pulls in prior knowledge that contradicts the retrieved text; generation drift means the retrieved chunk was correct but the model rephrased it in a way that changed the meaning. Each demands a different fix. Conflating them is how teams end up tweaking system prompts for months while the actual problem sits in their vector index.
Chunking is not a setting; it is a design choice
The default in most tutorials is to split documents into fixed 512-token chunks with a small overlap. It is a fine starting point and a terrible final answer.
The problem is that 512 tokens is an arbitrary boundary. It cuts paragraphs in half, separates questions from their answers, and strips out the structural cues — headings, lists, captions — that helped a human reader find their place. Naive RAG is dead — the pattern of chunk documents, embed, cosine similarity, stuff into prompt was always a prototype, not a production system, because chunking destroys context, embedding similarity does not equal relevance, and top-K retrieval is crude.
In 2026 the picture is clearer than it was. Recursive chunking that respects document structure, with metadata enrichment that prepends headings or document titles to each chunk, comfortably outperforms more clever alternatives on most real-world corpora. A well-tuned 512-token recursive splitter with 15% overlap and metadata enrichment outperforms an expensive semantic chunking approach on most real-world document sets, and prepending a document header to each chunk boosted QA accuracy from around 55% to 73% with no changes to retrieval architecture.
The boring lesson: the cheap thing, tuned, beats the clever thing, untuned. Run a sweep of chunk sizes and overlaps against your own queries before reaching for semantic splitters or agentic chunkers. Grab 50 representative questions from your query logs, test four chunk-size and overlap configurations, and you have a data-backed configuration in one afternoon.
Hybrid retrieval matters more than the embedding model
A second instinct, when retrieval is poor, is to upgrade the embedding model. Useful — and not enough. The structural fix is hybrid retrieval: a keyword-based search (BM25) running in parallel with vector search, with the two result sets combined via reciprocal rank fusion or a small reranker on top.
The intuition is simple. Vector search is good at semantic similarity ("how do I cancel my subscription" matches "subscription termination process"). Keyword search is good at exact terms (product names, version numbers, error codes). Each catches what the other misses. Modern production RAG systems run both. Voyage-3-large outperforming OpenAI and Cohere embeddings by 9-20%, semantic chunking improving recall up to 9% over fixed-size approaches — but production challenges shifting from prototypes to scale meant embedding drift, multi-tenancy, and sub-50ms latency requirements drove infrastructure investment.
A reranker is the second cheap upgrade most teams skip. Retrieve more candidates than you need (say, 20), then run a small cross-encoder to score them, then pass the top three or five into the LLM. The reranker has access to the full query–document pair and consistently catches the relevance signal that pure embedding similarity misses.
Embedding drift is real and nobody sets up alerts for it
The vector index is not static, even when the documents in it are. Models change. Your embedding provider releases a new version. Your own documents gain new sections, deprecate old ones, change terminology. Quality degrades quietly. Embedding drift quietly degrades performance over time.
A few of the practices that catch drift before users do:
- Pin embedding model versions explicitly. Do not let a vendor upgrade silently change your retrieval behaviour.
- Re-embed periodically when the source documents change materially.
- Track retrieval quality as a time series. Recall@k against a fixed evaluation set, run weekly, plotted on a dashboard. It helps detect when embeddings or indexes drift.
- Log which chunks were retrieved for every query, with a thumbs-up or thumbs-down feedback signal where the UI allows it.
Evaluation is the difference between shipping and guessing
The single largest gap between demo-quality and production-quality RAG is the absence of an evaluation harness. Without one, every change is a guess, and every regression is discovered by a user.
There are two things to measure, and they need to be measured separately:
- Retrieval quality — did the system return the right context? Standard metrics: recall@k, precision@k, MRR, nDCG.
- Generation faithfulness — did the model use the retrieved context correctly? Standard metric: faithfulness, often scored by an LLM-as-judge against the retrieved passages.
Retrieval precision and contextual relevance curate a high-fidelity knowledge stream, whereas answer relevancy ensures query fidelity and faithfulness grounds outputs in reality. One number cannot tell you what is broken. A system can score perfectly on faithfulness while scoring terribly on recall — accurately reporting what was retrieved, when the wrong things were retrieved. That is exactly the failure mode that produces confident, polite, useless answers.
Concrete thresholds matter too. Enterprises set faithfulness guardrails at 0.7–0.8 minimum, and critical systems in finance or healthcare aim for 0.9+. Pick yours before launch, not after the first incident.
The good news is that you do not need a sophisticated tool to start. A small set of representative queries with known good answers, a script that runs them nightly, and a Slack alert when scores drop below threshold is enough to get the first six months of value. You can add a managed eval platform later, when the volume justifies it.
Latency, cost, and the temptation of bigger context
Larger context windows have made it tempting to skip the hard work of retrieval and just stuff more documents into the prompt. It is a tempting trap.
Bigger context windows have three costs that compound in production: token cost per call, latency per call, and degraded model attention to the middle of the context (the well-documented "lost in the middle" effect). If you push too much text into the LLM, often due to naive "max context" strategies, the model may overlook critical facts or hallucinate — dynamic prioritisation and context filtering with reranking, so only high-value chunks consume tokens, is the production fix.
The same dynamic applies to reranking and multi-step retrieval. Both improve quality. Both add latency and cost. Latency quickly becomes a production bottleneck — adding larger top_k, rerankers, long contexts, and extra retrieval steps may improve recall, but it can also make the system too slow for real users. The production craft is choosing the cheapest pipeline that meets the quality bar, not the most accurate pipeline you can build.
Security: your retriever is now part of the attack surface
The final failure mode is the one most teams discover the hard way. The moment your RAG system retrieves from a corpus that includes any externally-influenced content — uploaded files, scraped web pages, customer-submitted documents — your retriever is a new attack surface.
Security vulnerabilities where poisoned documents could trigger specific model behaviours, including BadRAG and TrojanRAG attacks, are no longer theoretical. An attacker who can get content into your index can plant instructions that the LLM will execute the next time a related query brings that chunk into context.
The mitigations are mundane and effective: treat retrieved content as untrusted input, segment sensitive functions away from the LLM's tool access, apply document-level access controls before retrieval (not after), and run output-side guardrails that catch obvious data-exfiltration patterns. None of this is sexy. All of it is necessary.
What "production-ready" actually looks like
A RAG system that survives real users tends to share a handful of properties:
- Hybrid retrieval (BM25 + vector) with a reranker on top.
- Chunking tuned against the actual document corpus, with structural metadata preserved in each chunk.
- An evaluation harness that runs on every change and on a schedule, splitting retrieval and generation metrics.
- Pinned model versions and pinned embedding model versions, with explicit upgrade procedures.
- Logging of every query, retrieval result, and generation, with user feedback wired in.
- Faithfulness and recall dashboards plotted over time, with alerts when they degrade.
- Document-level access control applied before retrieval.
- Cost and latency budgets enforced at the gateway, not discovered on the bill.
None of this is the part that wins the demo. All of it is the part that lets you keep the customer.
The pattern we see consistently with the teams we work with is that the prototype takes a fortnight and the production-grade version takes a quarter. That ratio is not a failure of engineering — it is the actual shape of the work. Budgeting and staffing accordingly, from the start, is how the demo turns into a shipped feature instead of a graveyard.
References
- DigitalOcean — Why RAG Systems Fail in Production. https://www.digitalocean.com/community/conceptual-articles/why-rag-systems-fail-in-production
- arXiv preprint — RAG-E: Quantifying Retriever-Generator Alignment and Failure Modes (January 2026). https://arxiv.org/abs/2601.21803
- Dev.to / Young Gao — RAG Is Not Dead: Advanced Retrieval Patterns That Actually Work in 2026. https://dev.to/young_gao/rag-is-not-dead-advanced-retrieval-patterns-that-actually-work-in-2026-2gbo
- NStarX — The Next Frontier of RAG: How Enterprise Knowledge Systems Will Evolve (2026-2030). https://nstarxinc.com/blog/the-next-frontier-of-rag-how-enterprise-knowledge-systems-will-evolve-2026-2030/
- Substack / Nandigam Harikrishna — RAG Chunking Strategies & Embeddings Optimization: The 2026 Benchmark Guide. https://nandigamharikrishna.substack.com/p/rag-chunking-strategies-and-embeddings
- Introl — RAG Infrastructure Production Guide. https://introl.com/blog/rag-infrastructure-production-retrieval-augmented-generation-guide
- Dextralabs — Production RAG in 2025: Evaluation Suites, CI/CD Quality Gates & Observability. https://dextralabs.com/blog/production-rag-in-2025-evaluation-cicd-observability/
- Deepchecks — RAG Evaluation Metrics: Answer Relevancy, Faithfulness, and Accuracy. https://deepchecks.com/rag-evaluation-metrics-answer-relevancy-faithfulness-accuracy/
- Scadea — RAG Evaluation Metrics: Detect Hallucinations. https://scadea.com/evaluating-rag-quality-hallucination-detection-and-answer-accuracy-metrics/
- Medium / chenna — The Complete Guide to Evaluating RAG Systems. https://medium.com/@hchenna/the-complete-guide-to-evaluating-rag-systems-d5dd1e5c46e9
Written by JP, Sixees Labs. Last reviewed May 2026.