The Context Budget Is a Hard Limit, Not a Soft One
Nobody decided to send twenty thousand tokens of context. It accumulated: a few more retrieved passages here, the last ten turns instead of five there, a policy document somebody wanted included. Each change was small, defensible, and shipped on its own.
How It Accumulated
The assistant started with five retrieved passages and the last three turns of conversation. Nine months later it was sending twelve passages, ten turns, a summary of the customer record and a policy excerpt. Every one of those additions had a ticket behind it and a case where it had helped.
Quality on our labelled set had drifted down over the same period in a way nobody could attribute to a specific change, because no specific change had caused it. The regression was in the accumulation, and accumulation is invisible to a review process that looks at one change at a time.
Fitting Is Not the Same as Using
The window was never full. Twenty thousand tokens against a limit far above that felt safe, and that feeling is the trap: the limit tells you what the model will accept, not what it will use well.
We had been treating context capacity as a budget with plenty of headroom, when it behaves more like signal-to-noise. Adding a passage that is merely related rather than needed does not cost nothing; it costs a share of the model's attention, spent on material that cannot help.
What the Measurement Showed
We rebuilt the prompt at four sizes against the same labelled set: three passages, six, nine and twelve, holding everything else fixed. Six was the best on our set. Twelve was measurably worse than six, and the failures at twelve were not omissions but confident answers drawing on a passage that was topically adjacent and factually irrelevant.
That is the part worth emphasising. The extra material was not junk. It was relevant enough to have survived reranking and wrong enough to produce an answer that read well and did not hold up.
The Research That Matched
Levy and colleagues published work that month isolating input length as a variable, holding the reasoning task constant while padding the input, and reported that performance degrades as input grows even when the required information is present and the task is unchanged.
Two things made that useful to us. It separates length from the position effects we already knew about, and it establishes the degradation as a property of length itself rather than of retrieval quality. Our own numbers agreed, and the paper gave us a reason to expect it rather than a coincidence to explain.
| What we cap | How |
|---|---|
| Retrieved passages | Six, chosen by the reranker |
| Conversation history | Four turns, plus a decision store |
| Reference documents | Extract of the relevant clause, not the file |
| Total prompt | Eight thousand tokens, enforced in code |
The Budget as a Rule
The prompt builder now enforces a hard token ceiling and throws if a caller exceeds it. That sounds heavy-handed, and it is deliberately so, because the alternative is what we had: a soft guideline that erodes one justified increment at a time.
The rule that makes it work is that adding something means removing something. When a colleague wants the policy excerpt in the prompt, the question is not whether it helps, it is what comes out to make room. That question is answerable on the labelled set, and it makes the trade explicit instead of letting it happen silently.
Where the Removed Material Went
Most of what we cut did not need to be in the prompt at all. Conversation history became a small typed store of decisions the assistant had made, which is a few hundred tokens rather than several thousand and is more reliable than asking the model to re-read what it said.
The policy document became a lookup. Rather than including the excerpt every time, the assistant calls a function when a policy question arises. That costs a round trip on the small fraction of conversations that need it, instead of costing attention on all of them.
What We Watch Now
Average prompt size is on the same dashboard as latency and cost, and a sustained rise is treated as a defect rather than as growth. It is the only metric we have that catches this class of regression, because none of the quality metrics move enough on any single change to trigger anything.
We also re-run the size sweep after a model change. The optimum is a property of the model as much as of our data, and assuming last quarter's number still holds is how a budget quietly becomes wrong.
What We Do Not Claim
We do not claim six passages or eight thousand tokens are right for anyone else. They are measured against our corpus, our questions and the model we were using at the time, and the only transferable part is the method of measuring rather than the numbers.
We also do not claim long context is not useful. It clearly is, and there are tasks where the whole document genuinely has to be in the window. Our point is narrower: length is a cost that has to be spent deliberately, not a free allowance to be filled.