It Was Fast and It Felt Slow
Three weeks of work took the median response from four and a half seconds to under three, and the feedback did not change at all. We had optimised the number on our dashboard rather than the one people experience.
The Optimisation That Changed Nothing
We had a latency problem in the sense that users said the assistant was slow. So we did the work: a faster reranker, fewer candidates, a warmed index, parallel retrieval. Median end-to-end response went from about four and a half seconds to under three.
The next round of feedback said the assistant was slow. Not slightly better, not improved: unchanged. That is a disorienting result and it is entirely explicable, because we had never measured the thing users were reacting to.
The Number People Feel
Time to first token. Everything before the first word appears is silence, and silence is where impatience lives. Our pipeline spent nearly all of its time in that window: retrieval, reranking, prompt assembly, and only then a model that streams its answer over a couple of seconds.
So we had cut a second and a half off the streaming portion, which users experience as text arriving, and left the silent portion almost untouched. On the dashboard that is a third faster. In a chat window it is the same wait followed by slightly quicker text.
The Research on That Specific Number
Horton and colleagues published work that October on reducing time to first token specifically, treating the delay before generation begins as its own optimisation target rather than as part of a total.
Seeing it named as a distinct metric was the useful part. We had a general latency budget, and once the budget was split into silence and streaming, it was immediately obvious which half our three weeks had gone into and which half the users were complaining about.
| What we changed | Effect on the silence |
|---|---|
| Faster reranker | Small. It was never the bottleneck |
| Stream retrieval status to the user | Large. Silence became progress |
| Start generation on partial context | Moderate, and risky. We reverted it |
| Cache embeddings of frequent queries | Moderate for repeat questions |
What Actually Worked
Telling the user what is happening. The interface now shows searching the documentation, then found four passages, then the answer streams in. The total time is unchanged and the perceived experience is completely different, because a wait with information is not the same as a wait.
This is not a trick. The messages are real: they reflect the actual stage, and when retrieval takes unusually long, the message stays visible longer, which is honest and also tells the user that something specific is happening rather than that the system is stuck.
The Thing We Tried and Reverted
Starting generation before retrieval finished, on the first few passages, so the first words appear sooner. It shaved close to a second off the silence and it changed answers, because the model committed to a direction before the best passage had arrived.
We caught it on the labelled set rather than in production, which is the one part of that episode we are pleased with. An optimisation that changes what the system says is not a latency change, and it had to be evaluated as a quality change, at which point it failed.
Where Streaming Is the Wrong Answer
Not every response should stream. Where the output is a structured record that a downstream step consumes, streaming buys nothing and costs the ability to validate before anything is shown. Those steps return complete or not at all.
It is also wrong where a partial answer is misleading. An answer that begins the deadline is fourteen days and continues unless the framework contract specifies otherwise reads very differently after two seconds than after four, and a user who acts on the first clause has been misled by our interface rather than by our model.
What We Measure Now
Three numbers per request instead of one: time to first token, total duration, and time spent in each pipeline stage. The first is on the dashboard customers see, because it is the one that corresponds to what their users will tell them.
We also record whether the answer streamed to completion or the user navigated away. Abandonment during the silence is the clearest signal we have that a threshold has been crossed, and it moved when the progress messages went in, which is how we know they were not merely cosmetic.
What We Do Not Claim
We do not claim total latency does not matter. It does, particularly for the steps no human waits for, where it determines throughput and cost. Our point is that the two numbers serve different purposes and we had been using one to reason about the other.
We also do not claim progress messages fix a genuinely slow system. Beyond a few seconds they stop helping, and a system that regularly spends eight seconds searching has an architecture problem that no amount of honest narration will disguise.