The Batch Job That Ate the Assistant
At nine in the morning the assistant took nine seconds to answer instead of two. Nothing had been deployed, the model was fine, and the cause was a batch job that had started late and was still running when people arrived.
The Morning
Answers that normally take two seconds took nine. The model was responding normally when tested directly, the index was healthy, and the application logs showed requests queuing rather than failing.
The cause was an ingestion batch that re-embeds newly arrived documents. It normally runs from two until five in the morning. That night a supplier had delivered a large archive, the batch was still running at nine, and it and the assistant share a GPU pool.
Why the Autoscaler Did Not Save Us
It scaled, correctly, against the metric it was given, which was average utilisation over five minutes. Under a batch load that keeps utilisation high and steady, that metric says the cluster is working hard and does not distinguish between work someone is waiting for and work nobody is.
Park published work in early 2026 studying how infrastructure faults lead autoscalers to misallocate resources. Ours had no fault; it had a configuration that assumed one kind of load, which is the more common version of the same outcome and is harder to notice because nothing is broken.
Two Workloads With Opposite Requirements
The assistant needs low latency and has bursty, low-volume demand. The ingestion job needs throughput and does not care about latency at all. Running them on one pool means the pool is tuned for one of them, and it was tuned for neither in particular.
That is the design error, and it was made by us rather than by any component. Sharing capacity between them was a cost decision taken when the ingestion volume was small, and nobody revisited it when the volume grew.
| Workload | What it needs |
|---|---|
| Interactive assistant | Low latency, reserved capacity |
| Nightly ingestion | Throughput, preemptible capacity |
| Evaluation runs | Neither. Runs whenever |
| Shared pool for all three | The configuration that caused this |
What We Changed
Separate pools, with the interactive one small and reserved and the batch one preemptible. The batch job now yields when interactive demand rises, which means a large ingestion takes longer and never again shows up as latency for a person.
We also gave the ingestion job a deadline rather than a schedule. It must finish by seven, and if it will not, it splits and continues the following night, which is a property the customer prefers over an unbounded run that eats the morning.
The Alert We Did Not Have
Time to first token by hour of day, alerting on a shift rather than on a threshold. We had latency monitoring and it was averaged over the day, so a bad hour every few weeks disappeared into a fine daily number.
That alert would have caught this a month earlier, because the overlap had happened twice before at smaller scale and nobody noticed. The incident was the first time the overlap was large enough to be visible to users, not the first time it happened.
Why We Did Not Simply Add Capacity
Because it would have worked and hidden the problem. More capacity moves the threshold at which the two workloads collide, and the collision then happens on a bigger delivery, at a worse moment, with a larger bill in the meantime.
Separation cost about fifteen percent more in reserved capacity and removed the failure mode entirely. That is a trade we can explain, whereas doubling the pool would have been a cost with no accompanying reasoning beyond making a symptom stop.
The General Rule We Took
Do not share capacity between work someone is waiting for and work nobody is. It sounds obvious written down, and it was violated in our own infrastructure for two years by people who would have agreed with it in the abstract.
The reason it survives is that sharing looks efficient on a utilisation graph. High utilisation is a good sign for a batch cluster and a warning for an interactive one, and a single graph covering both tells you neither.
What We Do Not Claim
We do not claim separation is always right. For a small deployment with modest volume, one pool is simpler and the collision may never occur, and we run several like that deliberately.
We also do not claim our alert catches every version of this. It catches a shift in latency by hour, and a degradation spread evenly across the day would pass it unnoticed, which is a limitation we know about and have not closed.