RAG in Production: A Failure Taxonomy
A field taxonomy of RAG failure modes as of January 2024: retrieval misses, stale indexes, the lost-in-the-middle effect, and citation drift. We define each mode, cite the measurements behind it, and argue for evaluating retrieval and generation as separate systems — recall@k on one side, faithfulness metrics on the other.
The Demo Works and Production Does Not
Retrieval-augmented generation is almost four years old as a term — Lewis et al. coined it in May 2020 — and roughly one year old as an industry default. In 2023 it became the standard way to ground a language model in private documents. The demos work. Production systems fail in ways the demo never showed, and the failure report is almost always the same word: hallucination.
That word is a diagnosis of the model. In our project work at Blue IT Systems, most RAG failures begin before the model sees a single token: in ingestion, indexing, or retrieval. This article names four failure modes we see repeatedly, cites the measurements behind them, and argues for one discipline: measure retrieval separately from generation.
A Taxonomy of Four Failure Modes
A RAG pipeline has at least four stages: ingestion, indexing, retrieval, and generation. Each stage fails differently, and a wrong answer is a symptom, not a diagnosis. The table assigns each failure mode to the stage that produces it. The assignment matters because the fixes do not transfer across stages.
| Failure mode | Stage | Symptom |
|---|---|---|
| Retrieval miss | Retrieval | The correct passage exists in the corpus but is not in the top-k |
| Stale index | Ingestion and indexing | The answer was correct at index time and is wrong today |
| Lost in the middle | Generation | The correct passage is in the context but is ignored |
| Citation drift | Generation | The cited source does not support the cited statement |
Retrieval Misses Come First
A retrieval miss means the relevant passage is indexed but does not reach the top-k. The generator then answers from parametric memory or refuses. Embedding leaderboard scores do not predict this: BEIR (Thakur et al., 2021) evaluated 10 retrieval systems on 18 datasets and found that dense retrievers trained on MS MARCO often fall behind plain BM25 under domain shift, while BM25 remains a robust zero-shot baseline.
The practical consequence: benchmark your retriever on your corpus, not on its leaderboard. Hybrid retrieval — BM25 plus dense — with a cross-encoder reranker recovers much of the gap. It does not fix chunking that splits an answer across chunk boundaries, and it does not help when the answer is not in the corpus at all.
Stale Indexes Answer Yesterday's Questions
A vector index is a snapshot. Embeddings are computed once; the source documents keep changing. A stale index therefore serves passages that were true at index time — a price list from March, a policy replaced in June. The failure is silent: retrieval metrics stay green because the passage is still relevant to the query. It is merely wrong.
The mitigations are unglamorous: incremental ingestion keyed on source timestamps, freshness budgets per source, and deletion propagation, so that removed documents also leave the index. One trap deserves naming: embedding vectors from different models are not comparable. Changing the embedding model means re-embedding the entire corpus. Plan that cost before it becomes an outage.
Lost in the Middle
Liu et al. (arXiv 2307.03172, July 2023) showed that language models use long contexts unevenly. Across GPT-3.5-Turbo, Claude 1.3, MPT-30B-Instruct, and LongChat-13B, accuracy on multi-document question answering follows a U-shaped curve: highest when the relevant passage is first or last, lowest in the middle. For GPT-3.5-Turbo, performance drops by more than 20 percent; with 20 or 30 documents, middle placement scores below the closed-book baseline of 56.1 percent.
For RAG this inverts an intuition: retrieving more documents can make answers worse. Retrieved-but-ignored is a real failure mode, and it is invisible in retrieval metrics. The lever is ordering — place the strongest passages first or last. A reranker gives you that control; a raw similarity sort does not.
Citation Drift Undermines Trust
Citation drift is the gap between what a system cites and what the citation supports. Liu, Zhang, and Liang (arXiv 2304.09848, April 2023) audited four generative search engines — Bing Chat, NeevaAI, perplexity.ai, and YouChat. On average, only 51.5 percent of generated sentences were fully supported by their citations, and only 74.5 percent of citations supported their sentence. Worse, citation precision correlated negatively with perceived utility (r = −0.96).
The lesson: citations are generated text, not provenance. A footnote does not verify itself. If your product shows sources, verify support explicitly — an entailment check between statement and cited passage — and report citation precision and citation recall as first-class metrics.
Measure Retrieval Separately From Generation
An end-to-end quality score confounds two systems. Split them. Retrieval is measured offline against labeled pairs of question and gold passage: recall@k, MRR, nDCG@10. These metrics are deterministic, cost nothing per run, and belong in CI next to the unit tests. Generation is measured with the retrieved context held fixed: faithfulness, answer relevance, and context relevance, as formalized by RAGAS (Es et al., September 2023) using an LLM as a reference-free judge.
The split also tells you where to invest. Liu et al. found in an open-domain QA case study that reader performance saturates far before retriever recall does — beyond a point, better retrieval buys nothing. Trade-offs in the open: LLM-judged metrics are noisy and cost tokens; retrieval metrics are exact but require labeling work. You need both, and separately.
What We Expect for 2024
As of January 2024, context windows are growing fast: GPT-4 Turbo advertises 128K tokens, Claude 2.1 200K — both announced in November 2023. We do not expect long context to replace retrieval. Cost and latency scale with input tokens, and lost-in-the-middle shows that a longer window is not the same as a used window. Retrieval stays; what changes is how rigorously it must be measured.
Our expectations for 2024: hybrid retrieval with reranking becomes the default rather than an optimization; retrieval evaluation sets become versioned repository artifacts like code; and citation verification moves from UI feature into the pipeline. These are predictions written in January 2024, and some will age badly. One claim will not: a system you can only measure end to end is a system you cannot debug.
Sources
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, May 2020)
- Thakur et al. — BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv, April 2021)
- Liu, Zhang, Liang — Evaluating Verifiability in Generative Search Engines (arXiv, April 2023)
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (arXiv, July 2023)
- Es et al. — Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv, September 2023)