The RAG We Actually Built, Against the One in the Diagrams
Reference diagrams for retrieval pipelines show eight or nine boxes. Ours has four, and two of those exist only because a specific failure forced them. This is the accounting: what each stage cost, what it caught, and what we removed again.
Four Boxes, Not Nine
Our production retrieval pipeline is a chunker, a hybrid index, a cross-encoder reranker and a generation step with a schema on the output. That is the whole thing. Every reference architecture we have been shown in a workshop has at least twice as many stages, and each of those stages is individually defensible.
We did not arrive at four by minimalism. We arrived there by only adding a stage when a measured failure required one, and by removing anything that could not show its contribution on the labelled set. The result is smaller than the diagrams, and the difference is worth explaining rather than apologising for.
The Vocabulary That Helped
Gao and colleagues published a survey that December organising the field into naive, advanced and modular retrieval-augmented generation, with the pre-retrieval, retrieval and post-retrieval stages named and separated. Surveys rarely change what we build, but this one changed how we talk about what we build.
Before it, a conversation about adding query rewriting was a conversation about a specific implementation someone had read about. Afterwards it was a conversation about a named stage with a known purpose, which made it far easier to ask the useful question: what would it catch here, and how would we know.
What Each of Our Stages Earned
Chunking earned its place by being the largest single lever we found. Moving from fixed-size splits to structure-aware ones, respecting headings and table boundaries, changed retrieval quality more than any model swap we tried in the same period.
The hybrid index earned it by covering the failure the vector index alone could not: exact identifiers. Part numbers and contract references are precisely the queries where semantic similarity is useless and keyword matching is exact, and they are also the queries where being wrong is most obvious to a user.
The Stage We Added and Removed
We ran query rewriting for about two months. The idea is sound: expand or rephrase the user question before searching, so that vocabulary mismatches stop costing recall. On our labelled set it produced a small improvement and one new failure mode, where a rewritten query drifted far enough from the original to answer a different question confidently.
The small gain did not survive the new failure being harder to notice. We removed it and put the effort into the chunker instead, which addressed most of the same vocabulary mismatch at the point where the text is indexed rather than at the point where it is queried.
| Stage | Why it is or is not in our pipeline |
|---|---|
| Structure-aware chunking | In. Largest single quality lever we measured |
| Hybrid index | In. Covers exact identifiers that vectors miss |
| Cross-encoder reranking | In. Ordering matters as much as selection |
| Query rewriting | Out. Small gain, one hard-to-notice new failure |
| Multi-hop retrieval | Out. Our questions do not chain that way |
| Knowledge graph layer | Out. Cost never met a failure it would have caught |
The Two We Still Do Not Run
Multi-hop retrieval is absent because our question distribution does not need it. Users ask about one order, one contract, one part. A pipeline that plans a chain of lookups adds latency and failure surface for a case that appears in well under one percent of traffic, and that case is better served by escalating to a person.
A graph layer is absent for a different reason: we have never been able to point at a failure it would have caught. It is a genuinely powerful technique and we expect to use it when a customer corpus has the relationship density to justify it. None of ours has yet, and building it speculatively would be building a component nobody can evaluate.
How We Decide to Add One
A stage goes in when we can name the failure it removes, show that failure in the labelled set, and show the stage removing it without introducing a worse one. That sounds obvious and it is the step most often skipped, because the alternative is adopting a stage on the grounds that a reference architecture contains it.
The cost of skipping the check is not just the stage itself. Each stage is another thing that can drift, another place a regression can hide, and another component whose behaviour someone has to understand at three in the morning when answers go wrong.
What This Costs Us
A small pipeline is not free of downsides. We handle some queries worse than a more elaborate system would, particularly comparative questions spanning several documents, and we know it because those cases fail in our labelled set and we have chosen not to fix them yet.
We also carry the risk of having skipped something that would help. The mitigation is that the labelled set makes adding a stage cheap to evaluate: any of the three we left out can be tried in an afternoon and kept only if it earns the place.
What We Do Not Claim
We do not claim four stages is the right number in general. It is the right number for the corpora and question distributions we run against, and a customer with dense cross-references between documents would likely need more.
We also do not claim the survey taxonomy is prescriptive. It is a map of what people have built, not a checklist to work through, and reading it as the latter is how pipelines end up with stages nobody can justify.