The Cache We Took Out
The cache worked exactly as designed. Similar questions returned instantly, costs fell about twenty percent, and the design review that removed it took fifteen minutes once someone asked what a fast response tells you.
What It Did and What It Leaked
The platform serves several companies from one deployment. A semantic cache sat in front of the generation step: if an incoming question was close enough to one already answered, the stored answer came back without a model call. Costs fell about a fifth and median latency improved sharply.
The problem is not that answers crossed a boundary. They did not; the cache was keyed to include the tenant in the retrieval scope. The problem is timing. A question answered in eighty milliseconds instead of two seconds tells the asker that this question has been asked recently, and on a platform where two of the tenants compete, that is information about a competitor.
Why This Was Easy to Miss
Every review we had done asked whether data could move between tenants. The answer was no, correctly, and we stopped there. The channel is not the content of the response, it is the fact of the response arriving quickly, which no data-flow diagram represents.
That is the general shape of the lesson. Isolation reviews look at what is returned, and a shared resource can disclose through how long something takes, whether it succeeds, or how much it costs, none of which appear in the payload.
The Research That Named It
Song and colleagues published work that September on timing side channels in systems that serve large language models, examining how shared infrastructure such as caching can allow one user to infer things about another user's requests from response time alone.
We had already removed our cache by the time we read it, on the strength of one uncomfortable question in a review. What the paper added was the vocabulary and the confidence that this is a known class rather than a peculiarity of our design, which changed how we now describe it to customers.
| Cache scope | What we run |
|---|---|
| Across tenants, semantic | Never. This is the case we removed |
| Per tenant, semantic | Yes, for high-volume tenants |
| Per tenant, exact repeats | Yes, everywhere |
| Embeddings of our own documents | Yes. No user input involved |
What We Kept
Most of the saving, as it turned out. Caching within a tenant is unobjectionable, because everyone who can observe the timing is already entitled to the content, and for the larger tenants the hit rate inside their own traffic is not far below the shared rate.
We also cache aggressively where no user input is involved at all: embeddings of our own corpus, reranker outputs for fixed passage pairs, and template fragments. That part of the bill never needed a shared cache, and it is where a surprising share of the cost sits.
The Channel That Remains
Per-tenant caching still leaks within a tenant. A user can learn that a colleague recently asked something similar, and in an organisation with an HR or legal function using the same assistant, that is not nothing.
We handle it by scope rather than by removing the cache: assistants serving sensitive functions run with their own cache and their own index, which is more infrastructure and the only answer we found that does not depend on nobody thinking to measure.
What We Now Ask in Design Reviews
Three questions about every shared resource. Can one tenant observe that another used it. Can they observe how much. Can they observe when. If any answer is yes, the resource is either partitioned or it does not carry user-derived data.
That is deliberately blunter than a threat model, and blunt has worked better for us. It catches caches, rate limits shared across tenants, cost dashboards that aggregate at the wrong level, and error messages that distinguish not found from not permitted.
What It Cost to Remove
About six percent on the monthly bill after we rebuilt per-tenant caching, against the twenty percent the shared cache had saved. Median latency rose for the smallest tenants, who benefit least from a cache of their own traffic and had benefited most from everyone else's.
We told those customers why, which was a better conversation than it sounds. A supplier explaining that their system is slightly slower because it stopped exposing them to their competitors is not a difficult message to deliver.
What We Do Not Claim
We do not claim shared caches are always wrong. In a single-tenant deployment, or where all users are inside one organisation with no internal boundaries, the concern does not arise and the saving is real.
We also do not claim we have found every timing channel. We found one because someone asked a good question in a review, and the honest position is that this class is easy to reintroduce whenever a new shared component looks like an obvious optimisation.