We Priced Putting the Whole Corpus in the Window
Every few months someone asks whether growing context windows make our retrieval pipeline unnecessary. It is a fair question with an arithmetic answer, and the arithmetic is specific enough to a corpus that a general opinion about it is worth very little.
The Question in Its Honest Form
Not whether long context works. It does. The question is whether it is the better choice for a given corpus, and that turns on three properties: how large the corpus is, how often it changes, and how many questions per day arrive against it.
Framed that way the answer stops being a matter of taste. We have one customer where we put the entire relevant material in the window on every call, and another where doing so would be indefensible, and the two systems are otherwise built the same way.
The Corpus Where We Do It
A set of internal policies totalling perhaps thirty thousand tokens, updated a few times a year, queried a few dozen times a day. The whole thing fits comfortably, there is no index to maintain, no chunker to tune, and no retrieval failure mode to debug.
That system is simpler than our retrieval pipelines by a wide margin, and simplicity is the real benefit rather than quality. When a policy changes, someone replaces a file. There is no re-indexing, no staleness window and nothing to get out of step.
The Corpus Where We Do Not
Technical documentation of several hundred megabytes, changing weekly, queried a few thousand times a day. Retrieval sends about six passages. The alternative sends orders of magnitude more tokens on every single question, and there is no caching strategy that fixes this when the questions are unrelated to each other.
We costed it properly rather than arguing about it: roughly forty times the per-question cost, for a measured quality difference on our labelled set that was inside the noise of three runs. Paying forty times more for a difference we cannot detect is not a trade-off, it is a mistake with a rationale.
| Corpus property | Points to |
|---|---|
| Fits in the window with room to spare | Put it in the window |
| Changes weekly or faster | Either. Long context avoids re-indexing |
| High query volume | Retrieval. Cost scales with every call |
| Answers need provenance for the user | Retrieval. Passages carry their source |
The Study That Framed the Trade
Li and colleagues published a comparison that July setting retrieval-augmented generation against long-context models on the same tasks, and proposed an approach that routes between them rather than committing to one, on the basis that the cheaper path is sufficient for a large share of queries.
Routing is where we ended up too, from the cost side rather than the quality side. What the study added was the observation that the choice can be made per query rather than per system, which we had not been doing and which is obvious in retrospect.
The Hybrid We Run
For the middle case, a corpus of a few hundred thousand tokens, we retrieve first and fall back to the full document when retrieval returns several passages from the same file with low separation between their scores. That pattern usually means the question is about the document as a whole.
It fires on about one query in twenty. The rest are answered from passages at the normal cost, and the fallback carries the expensive path only where the cheap one has visibly failed rather than on every call.
The Argument That Is Not About Cost
Provenance. A retrieved passage comes with its source, and our interface shows which document and which section an answer rests on. When the entire corpus is in the window, the model can cite, and the citation is generated rather than recorded.
For the policy system that is acceptable because a user can check the whole document in a minute. For the technical documentation it would not be, because the source is the thing an engineer actually wants, and a generated reference that is subtly wrong is worse than no reference.
What Would Change the Answer
Price, mostly. The arithmetic is a ratio between the cost of tokens and the cost of maintaining an index, and both move. We re-run the comparison for a customer when their volume changes materially or when a pricing change makes the ratio look different, which so far has been about once a year.
Caching changes it too, but less than it appears. It helps when many questions share a prefix, and our question distribution is dominated by unrelated one-off queries, which is the case caching handles worst.
What We Do Not Claim
We do not claim retrieval is the better architecture. For a third of the systems we run, putting the material in the window is simpler and we do it. The claim is only that the choice follows from three measurable properties, not from a general position about which approach is more modern.
We also do not claim our forty times figure generalises. It is one corpus at one volume under one price list, and the useful part is the method: measure the ratio for the corpus in front of you before choosing an architecture for it.