Seven ways a RAG system fails that a demo never shows

A retrieval demo is a rigged game, and everyone in the room knows it except the person paying. The questions come from the corpus. The corpus was cleaned last week. The person asking already knows which document holds the answer.

Production reverses all three. Here are seven failure modes that survive a good demo, and the check that catches each.

1. The answer straddles two chunks, so neither one ranks

Chunk boundary bleed is the most boring failure in the field and the most common. A fixed 800-token window splits the tolerance value from the part number three paragraphs above it. Each half is individually unremarkable, so neither half ranks, and the system reports that the information is not in the corpus.

Anthropic’s contextual retrieval work quantifies the cost: naive embedding-only retrieval fails 5.7% of queries at top-20 — that is 1 − recall@20 — and prepending 50–100 tokens of model-written document context to each chunk before embedding drops that to 3.7%. The one-time cost is $1.02 per million document tokens. Late chunking — embedding the whole document, then pooling token embeddings into chunks — recovers much of the same benefit at ordinary storage cost, which matters because ColBERT-style late interaction runs roughly 2.46 TB for 100,000 documents at 8k tokens each against about 4.9 GB for plain chunk embeddings.

The check: take 30 real questions, find by hand the passage that answers each, and verify it survives your chunker intact. In the Barnett taxonomy this separates FP1, missing content, from FP3, present but lost during consolidation. They have different fixes.

2. Retrieval is fine. Ranking is not.

These are two numbers and they get reported as one. Recall@20 of 0.94 sounds healthy right up until you notice the correct chunk sits at rank 17 and the prompt is built from the top 5.

Reranking is where the remaining error lives. On the same benchmark, contextual embeddings plus contextual BM25 fail 2.9% of queries; adding a cross-encoder reranker takes it to 1.9% — a 67% reduction against the naive baseline, achieved after retrieval, not during it.

The recurring implementation bug is fusing hybrid results by normalising BM25 scores against cosine similarity. They are not comparable and no scaling makes them so. Reciprocal rank fusion sidesteps it by scoring on position alone, summing 1/(rank_constant + rank) across retrievers; Elasticsearch defaults rank_constant to 60. Keep the lexical leg regardless of how good the embeddings are — part numbers, article codes and headwords need exact matching. On the Ministry of Education and Science dictionary we built at beron.mon.bg, the unit of retrieval is the headword, and no amount of semantic similarity beats matching it exactly.

The check: report recall@k and nDCG@10 separately, and log the rank position of the correct chunk rather than a boolean for whether it appeared at all.

3. Your golden set was written by the person who wrote the chunker

Eighty-five to 95% end-to-end accuracy is a realistic, defensible target on a narrow domain. A reported 99% almost always means the evaluation set was written by whoever wrote the prompt, from the chunks they had just finished tuning.

We have done this. Our first internal eval set was assembled by the engineer who had spent the previous week on the splitter, and it scored high enough that we nearly shipped on it. The questions were phrased in the vocabulary of the chunks. Rewritten by someone who had only read the source PDFs, the same system dropped double digits.

Labels should be binary pass/fail from a named domain expert, not 1–5 Likert scores — a 3 is uninterpretable and unactionable. If an LLM judge does the grading, validate it on true-positive and true-negative rate against those human labels, targeting 0.85+ on each. Raw agreement flatters a judge under class imbalance, and the ceiling is real: MT-Bench put strong judges at roughly 80% agreement with humans, which is what humans reach with each other.

The check: the person who labels the golden set is not the person who built the pipeline. 150–400 cases, and the labelling brief names the expert.

4. The index is a snapshot, and nobody costed the rebuild

Retrieval quality decays with no error anywhere. A superseded SOP is semantically perfect and confidently wrong, so revision and effective-date metadata has to filter it out, not merely rank it lower. Embedding models get deprecated, which forces a full re-embed of the corpus. A full HNSW rebuild past 10M vectors is hours of wall clock, and build speed collapses entirely once the graph exceeds maintenance_work_mem.

The check: measure time from source-document change to searchable as a first-class metric, and put the euro and wall-clock cost of a full reindex into the run-rate before signing anything. A proposal with no line for reindexing and eval maintenance has not been operated.

5. Tenant filters leak in one direction and collapse recall in the other

Filter after retrieval, in the application, and you fetch the global top 50, drop what this user may not see, and hand the model whatever survives — sometimes three chunks, sometimes none. The excluded documents have meanwhile passed through your reranker and your traces, which is a disclosure question as much as a quality one.

Filter inside the index and you hit the other wall: restrictive metadata filters fragment an HNSW graph and recall degrades silently. That is precisely why filterable HNSW and ACORN exist. pgvector’s iterative scan caps at max_scan_tuples = 20,000 by default, so a narrow filter can quietly return fewer than k rows.

The third variant is architectural: a retrieval service holding its own static credentials and querying on behalf of a caller it has not authorised — the confused deputy. MCP forbids token passthrough for this reason. OWASP’s 2025 list carries both halves, LLM02 sensitive information disclosure and LLM08 vector and embedding weaknesses.

The check: eval cases that assert a negative — this identity must not retrieve this document — run on every index, filter or permission change, not just at launch.

6. Two sources disagree and the model picks a tone instead of a side

Give a competent model the 2023 policy and the 2026 policy in the same context and it will not usually refuse. It will produce one fluent paragraph containing both, with both cited, and no reader will notice.

Chroma’s Context Rot study across 18 models found that a single distractor measurably degrades accuracy and four compound it, and that performance falls with input length even on tasks the model handles trivially at short length. So the obvious remedy — pass more context — is not one. Databricks measured generation degrading long before retrieval recall saturates: Claude 3 Sonnet past 16k tokens, GPT-4-0125 past 64k, with DBRX switching to summarising instead of answering 50.4% of the time at 32k and Claude 3 Sonnet’s copyright-refusal rate rising from 3.7% at 16k to 49.5% at 64k.

The check: seed the golden set with known contradictions and require the system to name the conflict rather than resolve it. Then verify citations resolve and that the cited passage actually supports the sentence — broken citations are the most common silent regression in a deployed system.

7. The harness only asks questions somebody already knew the answer to

Every eval set assembled from real usage inherits a survivorship bias: it contains the questions people asked because they expected an answer. The unanswerable question — the policy that does not exist, the year you have no data for — is the one that produces the fabrication that ends the project.

A healthy abstention rate on a real corpus is 5–15%. Under 2% the system is guessing; over 25% retrieval is broken. And measure consistency, not peak: τ-bench found frontier models above 50% single-pass success but under 25% at pass^8 — succeeding on all eight independent runs of the same task. Run each golden case 5–10 times.

The check: ask for the failed traces before you ask for a demo. Error analysis on real failures converges after roughly 30 traces, when no new failure mode appears, and it usually redirects the entire fix.

Where this stops applying

If your corpus is small and stable — a few hundred pages of policy, one language, one permission level — skip retrieval. Long context below the degradation thresholds above is a legitimate architecture and it removes five of these seven failure modes outright. RAG earns its complexity when the corpus is too large, too fresh, or too permissioned to hand over whole.

Notice also what is not on this list. None of these are model problems. Six of the seven are data plumbing and measurement discipline, and the seventh is an argument about who writes the questions. That is the durable part of the system, which is convenient, because the model is the cheapest and most replaceable component you will buy.