Same Model Version, Different Answers
On Monday a step that had run at ninety-six percent for months was running at ninety-two. No deployment, no configuration change, no change to the documents, and the same model identifier in every request.
The Monday
The daily evaluation run reported ninety-two percent on a set that had returned ninety-six for four months. Our first hour went into looking for what we had changed, which is the right instinct and produced nothing, because we had not changed anything.
The documents were identical. The prompts were identical. The model identifier in every request was the same pinned string it had been since March. What had changed was on the other side of the network call, and we had no way to observe it directly.
What a Version String Actually Is
A name the provider assigns. It usually corresponds to a set of weights and often corresponds to a serving configuration, and neither correspondence is something we can verify or that most terms of service promise.
Li and colleagues published work in 2025 on evaluating quality of service under token-level behavioural instability, treating the variation of a model's behaviour over time as a service property to be measured rather than as noise to be ignored.
What We Run Daily
The labelled set, every morning, against every model we depend on, with the result on a dashboard and an alert on a drop beyond a threshold. It costs a few euros a day per system and it is the only reason we knew about this within twenty-four hours rather than through a customer.
We also record the full response for twenty fixed inputs, so a change in behaviour can be inspected rather than inferred. That archive is what let us characterise the change as a difference in how optional fields were handled rather than as general degradation.
| What we do | Why |
|---|---|
| Daily labelled run per model | Detects a change within a day |
| Archive responses for fixed inputs | Lets you see what changed, not just that |
| Route through one gateway | One place to switch models under pressure |
| A tested fallback model | So switching is a decision, not a project |
What We Did About It
Adjusted the prompt to make the optional-field behaviour explicit rather than implied, which recovered about three of the four points, and accepted the fourth. The alternative was switching models, which we were ready to do because there was a tested fallback, and which would have cost more than the point was worth.
The important part is that both options existed on Monday. A system whose fallback has never been evaluated does not have a fallback, it has a plan to evaluate one during an incident.
What We Ask Providers Now
Whether a pinned version can change behaviour, what notice we get, and whether previous behaviour remains available. The answers vary and are often unsatisfying, and asking is still worth it, because the answer goes in the customer's system document rather than into an assumption.
Where the answer is that behaviour can change without notice, we say so to the customer and we price the monitoring accordingly. That is a more honest position than implying stability we cannot obtain, and no customer has objected to hearing it.
Why This Argues for Self-Hosting Less Than It Seems
A self-hosted model does not drift, which is a genuine advantage and a smaller one than it looks. The drift we experienced cost us a prompt adjustment and a day; self-hosting would cost us an on-call rotation permanently.
For most customers the daily evaluation is the cheaper answer to this problem, and self-hosting remains a decision about data residency and volume, which is where we have always put it. Drift is a reason to monitor, not a reason to move.
What Customers Should Take From This
That any system built on a hosted model has a dependency that can change without a deployment, and the question to ask a supplier is not whether they pin versions but what they run daily to notice when pinning was not enough.
Our own answer used to be that we pin versions, said with some confidence. It was true and it was not the answer to the question, and being caught out by that is why we now describe the monitoring rather than the pinning.
What We Do Not Claim
We do not claim the provider did anything wrong. Serving infrastructure changes for good reasons, the terms allowed it, and a four-point move on one narrow task is not evidence of a broken commitment.
We also do not claim daily evaluation catches everything. It catches changes large enough to move a set of eighty cases, and a subtle shift affecting one document type could sit below that threshold for a long time.