Changing the Embedding Model Is a Migration, Not a Setting
The change was one line. The consequence was that half our index answered in the old vector space and half in the new one for most of a working day, and the symptom looked like a quality problem rather than a deployment problem.
One Line, One Bad Afternoon
We pointed the service at a newer embedding model, deployed, and let the re-indexing job run in the background as it had been designed to. Documents were re-embedded in batches, and the search index accepted the new vectors as they arrived alongside the old ones.
Results got strange within the hour. Some queries returned obviously correct passages, others returned material with no visible relationship to the question, and which behaviour you got depended on whether the relevant document had been reached by the batch job yet.
Why the Mixture Is Nonsense
Two embedding models place text in two unrelated coordinate systems. A distance between a vector from one and a vector from the other is a number the database will happily compute and that means nothing at all, which is the worst kind of failure: no error, no warning, just results.
We knew this in the abstract. What we had not thought through is that a gradual migration puts both kinds of vector in the same index at the same time, so the system spends the whole migration comparing quantities that are not comparable.
What We Do Now
The new model builds a second index while the old one keeps serving. Nothing is switched until the new index is complete and has been run against the labelled set, and the switch itself is a single pointer change that is reversible in seconds.
The old index stays for a week. That is not caution about the model, it is caution about everything around it: an index that is complete and correct can still be wrong in its chunking, its metadata or its filters, and having the previous one intact turns a bad discovery into a rollback rather than a rebuild.
| What a re-index costs | Detail |
|---|---|
| Model calls | Every chunk, once, for the whole corpus |
| Storage | Both indexes at once, for the overlap |
| Wall clock | Hours to days, depending on corpus size |
| Risk | A silent quality change if it is not measured |
The Research and What It Changed
Lee and colleagues published Gecko that month, describing a compact text embedding model distilled using an large language model to generate and refine training data, and reporting competitive retrieval quality at smaller embedding dimensions than comparable models.
The dimension point is the one that mattered to us commercially. A smaller vector is cheaper to store and faster to search at the same quality, and at our corpus sizes that is a more tangible benefit than a small gain on a public retrieval benchmark.
How We Decide Whether It Is Worth It
A better benchmark score is not a reason. The question is whether the queries that currently fail on our labelled set would succeed, and that is answerable only by building the second index and running it, which is most of the cost of the migration anyway.
So we sample. Two thousand chunks rather than the whole corpus, embedded with the candidate model into a scratch index, and the labelled queries run against that. It is not conclusive, but it has twice been enough to show that a candidate would not have helped, which saved the full run.
The Part That Is Not the Model
In both migrations we have done, more of the improvement came from things we changed at the same time than from the embedding model itself. Re-indexing forces you to re-chunk, and re-chunking with what you have since learned about the corpus is where the quality actually moved.
That is worth separating rather than celebrating. If the chunker did the work, then the model swap was the occasion and not the cause, and believing otherwise leads to a rule of thumb that says upgrade embeddings for quality when the honest rule is revisit chunking for quality.
What We Watch During the Switch
Three things, all of them cheap. The count of indexed chunks against the count of source chunks, because a migration that silently drops documents is easy to miss. The labelled set, run against the new index before the pointer moves. And the rate of queries returning nothing above the relevance threshold, which is the fastest signal that something is structurally wrong.
That last one caught a filter mismatch on our second migration: metadata field names had changed, the filter matched nothing, and the recall collapse showed up in minutes rather than in a support ticket the following week.
What We Do Not Claim
We do not claim newer embedding models are not worth adopting. They are, and the improvements have been real. Our point is about the shape of the change: it is a data migration with a rollback plan, not a configuration value, and treating it as the latter is what cost us an afternoon.
We also do not claim a two thousand chunk sample is a reliable predictor. It is a cheap negative test. It can show that a candidate is not better; it cannot promise that a candidate is, and we have not tried to make it do more than that.