The Question No Single Chunk Can Answer
Ask our system for the torque specification on page two hundred and eleven and it is exact. Ask what the maintenance philosophy of the machine is and it returns three unrelated paragraphs, confidently. The gap between those two failures is not a retrieval quality problem.
Two Failures That Look Alike
A customer support system we run sits on top of technical manuals, some of them several hundred pages. For specific questions it works well: a part number, a tolerance, a step in a procedure. Those answers live in one passage, and finding the right passage is what the pipeline is built to do.
For a second kind of question it failed in a way that looked identical from the outside but has a completely different cause. Asked what a document covers, or how two procedures relate across chapters, it returned plausible passages that individually said nothing wrong and together answered nothing.
Why Better Ranking Cannot Fix It
We spent longer than we should have treating this as a ranking problem. Better embeddings, a stronger reranker, more candidates: each helped the specific questions slightly and the global questions not at all, which in hindsight is exactly what should have happened.
The answer to what is this document about is not written anywhere in the document. It is a property of the whole, and a retriever that returns passages can only ever return pieces. No ordering of pieces produces something that is not in any of them.
What We Built
We added a layer above the chunks. Sections are summarised, the summaries are grouped and summarised again, and those summaries are indexed alongside the original passages. A global question retrieves a summary; a specific one still retrieves the passage. Both are searched, and the reranker sees the mixture.
The structure follows the document rather than a fixed fan-out. Manuals already have chapters and sections, and using that existing hierarchy meant we did not have to invent a grouping and could point at the source of any summary during review.
The Research That Named the Shape
Sarthi and colleagues published RAPTOR that week, describing recursive clustering and summarisation of a corpus into a tree that is retrieved at multiple levels. Reading it clarified something we had been circling: the layers are not an optimisation of retrieval, they are additional content that did not exist before.
The paper builds its tree by clustering rather than by document structure, which is the more general approach and the right one when the corpus has no usable hierarchy. Ours does, so we use it, and we would not have made that choice consciously without a description of the alternative.
| Question type | What answers it |
|---|---|
| A tolerance, a part number, one step | The original passage |
| What this chapter covers | A section summary |
| How two procedures relate | A summary spanning both |
| What changed between revisions | Neither. This needs a diff, not retrieval |
What the Layer Costs
Building it costs model calls proportional to the corpus, which is a one-off expense that is easy to underestimate for a large document set but is genuinely a one-off. The recurring cost is worse and less visible: the summaries are derived content that goes stale when the source changes.
We handle that by treating a summary as a cache keyed on the hash of the text beneath it. When a section changes, its summary and every summary above it are invalidated and rebuilt. Without that, the system develops a specific and nasty failure mode where it answers global questions from a version of the manual that no longer exists.
The Review Step We Insisted On
Summaries are generated text presented to users as if it were sourced. That is a meaningful difference from returning a passage, and we were not comfortable shipping it silently. Every summary carries a link to the range it covers, and the interface shows it as a summary rather than as a quotation.
For the first corpus we also had a domain expert read the top level of the tree before it went live. Eleven of about sixty summaries needed correction, mostly for flattening a conditional into an absolute statement, which is the failure we now specifically look for.
Where We Do Not Build It
Most corpora do not need this. A knowledge base of short articles, a set of contracts queried by clause, a product catalogue: all of these are collections where the useful questions are answered by one record, and a summary layer would be cost and staleness for nothing.
The test we apply is whether users ask questions whose answer spans a document. If they do not, the layer is a solution looking for its problem, and the honest thing is to say so rather than to build the more sophisticated system.
What We Do Not Claim
We do not claim summaries are as trustworthy as passages. They are a lossy derivative, they can flatten nuance, and a user reading one is reading our system rather than the source. That is the trade we accepted for answering a question that was previously unanswerable, and it is why the provenance link is not optional.
We also do not claim our structure-following approach beats clustering. It is easier to explain and to audit on documents that already have a hierarchy, and we have not measured it against the clustering approach on a corpus where both are viable.