The Index Got Expensive Before Anyone Noticed
Nobody plans for the index to become the expensive part. It starts as a rounding error next to model costs, grows with the corpus rather than with usage, and by the time it is visible on an invoice it has been growing for a year.
How It Crept Up
The system started with about two hundred thousand chunks, and the vector index cost so little next to the model calls that nobody put it on a chart. Three years and several document sources later it held roughly four million, and the index was the second largest line in the monthly infrastructure cost.
The growth pattern is the trap. Model cost tracks usage, so it rises when the system is being used and someone is getting value from it. Index cost tracks the corpus, so it rises when documents are added, which happens whether or not anyone queries them.
What We Measured First
How much of the corpus is ever retrieved. We took a year of queries and counted distinct chunks returned in the top results: about eleven percent. Another twenty percent had appeared at least once in a wider candidate set, and the remaining two thirds had never been returned at all.
That number changed the conversation from how do we store four million vectors more cheaply to why are we storing all four million in the same tier. Those are different problems and the second one has cheaper answers.
The Technique and Its Price
Jeong published work in early 2025 on four-bit quantisation of embeddings for retrieval-augmented generation, which is the direct approach: store each vector component at lower precision and accept some loss in the similarity computation.
On our corpus, quantising cut index memory by roughly three quarters. Retrieval quality on the labelled set fell by about one and a half points at the top result and rather less when we looked at whether the correct passage was anywhere in the top ten, which is the number that matters when a reranker runs afterwards.
| Approach | Effect on us |
|---|---|
| Quantise the vectors | Three quarters less memory, small quality cost |
| Two tiers, hot and cold | Large saving, more moving parts |
| Delete what is never retrieved | Tempting and wrong. It is not never, it is not yet |
| Fewer, better chunks | The cheapest fix, and we did it last |
What We Actually Run
Quantised vectors for the whole corpus in the searchable tier, and full-precision vectors kept only for the reranking stage, which sees a few dozen candidates rather than four million. Precision where the decision is fine-grained, compression where the job is to narrow the field.
That arrangement recovered most of the quality loss, because the coarse pass only has to get the right passage into the candidate set, and the expensive comparison happens afterwards on vectors that were never compressed.
The Deletion We Did Not Do
Two thirds of the corpus had never been retrieved and deleting it would have been the largest saving available. We did not, because never retrieved is a statement about the questions asked so far, and the first time someone asks about a document we deleted, the system does not fail visibly. It answers from something else.
This is the same principle as the rest of our work on retrieval: an absent document produces a confident answer from adjacent material, which is worse than an expensive index. We archive rather than delete, and archived material is still findable, just slower.
The Fix We Should Have Tried First
Chunking. When we finally looked, our chunker was producing substantial overlap between adjacent chunks, which is a reasonable default and had been left at a setting chosen years earlier for a different corpus. Reducing the overlap removed nearly a fifth of the vectors with no measurable quality change.
That is the least sophisticated finding in this piece and the one we would repeat first. Before compressing what you store, check whether you are storing the same sentences four times, because a chunker default is exactly the kind of decision that never gets revisited.
What We Watch Now
Index size and cost on the same dashboard as model spend, with a per-thousand-chunks figure so growth is visible before it is expensive. And the share of the corpus retrieved in the last ninety days, which is the signal that a source has been added and is not being used.
That second number found an ingestion job importing a document set nobody had asked for, quietly, for seven months. Turning it off saved more than the quantisation did.
What We Do Not Claim
We do not claim quantisation is free. It costs retrieval quality, we measured how much on our corpus, and a system without a reranker to recover the loss would feel it more than ours did.
We also do not claim our eleven percent retrieval rate is typical. It is one corpus with a long tail of rarely consulted technical documents, and a knowledge base of frequently asked material would look completely different.