The Graph We Finally Built, and What It Cost
We have argued in this blog that a graph layer usually is not worth its cost. Then a corpus arrived where vector retrieval failed on nearly a third of the questions, and the reason was structural rather than fixable by better chunking.
The Position We Held
We have written that a graph layer earns its complexity only when the connection between documents is not already recorded somewhere, and that most corpora we see have identifiers that make a database join the better answer. That has been right for every project until this one.
The corpus is technical specifications for a network of interdependent components, where a single question routinely depends on which version of one component is paired with which revision of another, and those pairings are described in prose rather than held in a field.
Why Chunking Could Not Fix It
The failing questions were not about finding a passage. They were about following a chain: this component requires that interface, which is provided by this other component in revisions after a particular one, which in turn constrains a third.
No chunk contains the chain. Better chunking makes each link easier to find and does nothing about the fact that answering requires three of them together, which is the same argument we have made for years about summary layers, applied to relationships instead of to documents.
The Comparison We Ran
Ahmad and colleagues published a benchmark in 2025 setting vector, graph and hybrid retrieval pipelines against each other on a technical domain, which is the comparison most discussions of this topic assert rather than measure.
We ran our own version on the customer's labelled set. Vector alone answered about seventy percent, graph alone about sixty-four, and hybrid about eighty-eight. The number that mattered was the third one, and the second one is the one that surprised people: the graph on its own was worse than what we already had.
| Question type | What answers it |
|---|---|
| A specification value | Vector. Graph adds nothing |
| A dependency chain | Graph. Vector cannot follow it |
| Which revisions are compatible | Graph, with the version filter |
| Anything in prose only | Vector. The graph never extracted it |
What It Cost to Build
Six weeks, of which four were extraction and validation rather than graph infrastructure. Entities and relations were extracted by a model and then checked: a domain expert reviewed a sample of two hundred relations, of which about one in eight was wrong, mostly by asserting a dependency that the text only implied conditionally.
That error rate is the number nobody puts in a proposal. A graph with one wrong edge in eight answers dependency questions confidently and incorrectly, so the extraction had to be tightened until the sampled error rate fell under three percent, and that tightening is where the four weeks went.
What It Costs to Keep
More than the index. When a specification is revised, the affected subgraph has to be re-extracted and re-validated, and unlike an embedding index that can be rebuilt mechanically, a graph rebuild reintroduces extraction errors that a person previously corrected.
We handle it by keeping the human corrections as a separate layer applied after extraction, so a rebuild does not discard them. That was not in our original design and it is the piece we would tell anyone else to build first.
Why the Answer Was Hybrid
Because the graph is worse at most questions. It answers dependency chains and knows nothing about the prose that was never extracted into a relation, and that prose is where the majority of specification values live.
So retrieval runs both and the reranker sees the union. That is more infrastructure than either alone and it is the only configuration that beat what we started with. Presenting the graph as a replacement would have made the system worse in a way that would have taken months to attribute.
What We Tell Customers Now
The same thing as before, with one addition. Count the questions that need a chain rather than a passage. Below about a fifth, the join or the summary layer is cheaper and we will say so. Above it, and where the links are in prose rather than in fields, a graph is worth measuring.
And build the validation before the graph. The graph is the easy part; the part that decides whether it works is whether someone checked the relations, and whether their corrections survive the next rebuild.
What We Do Not Claim
We do not claim our earlier position was wrong. It was right for the corpora it was about, and this article exists because one corpus was different, not because the general advice changed.
We also do not claim eighty-eight percent is a good result in absolute terms. Twelve percent of questions still fail, most of them where the specification genuinely does not say, and a system that could answer those would be inventing.