Lost in the Middle: How Language Models Really Use Long Contexts
A longer context window is not the same as a usable one. In July 2023, Nelson F. Liu and colleagues measured where in the context models actually find information — and the answer changes how you build retrieval.
The Question Nobody Had Asked
In the summer of 2023, vendors are outbidding each other on context length. First 8,000 tokens, then 32,000, then 100,000. The implicit assumption is that whatever fits in the window is something the model can use. The paper "Lost in the Middle: How Language Models Use Long Contexts" tests that assumption directly — and does not find it supported.
The setup is deliberately plain. The authors study two tasks that require identifying relevant information inside the input context: multi-document question answering and key-value retrieval. In both, the location of the correct answer is known. That isolates a variable few had moved systematically before: the position of the relevant passage.
The Finding: a U-Shaped Curve
Move the relevant passage through the context and performance does not hold steady. It is highest when the information sits at the beginning or the end of the input, and degrades significantly when the model has to reach for something in the middle. Plotted against position, the curve looks like a U.
Anyone with a background in cognitive psychology will recognise the shape: primacy and recency, the better recall of the start and end of a list. Whether the underlying cause is the same in language models is not something the paper claims, and neither do we. The measurable effect is the interesting part, not the analogy.
Long-Context Models Are Not Exempt
The single most practical sentence in the paper is a caveat: the degradation in the middle appears even for models built explicitly for long contexts. A bigger window does not solve the problem on its own. It moves the boundary at which the problem shows up.
That is an inconvenient result for a common product decision. "We will just put every document in the prompt" is technically possible with a 100k window. Whether the model still uses the decisive line at position 400 of 800 is a different question — and the paper suggests the answer is no more often than window size would imply.
What It Means for Retrieval
For us the finding has three concrete consequences when building retrieval systems. First, reranking is not a luxury. If position influences the chance of a passage being used, the order in which passages are handed over shapes answer quality — not only which passages are chosen.
Second, less context can be better. Ten precise passages beat fifty mediocre ones when the relevant one lands at position 30. Third, prompt order belongs in evaluation. Measuring only whether the right passage was retrieved measures half the truth.
The Evaluation Protocols
The paper contributes more than a finding: it proposes protocols for evaluating long-context models. The core idea is controlled positional variation — same task, same documents, only the answer's position moves. Whatever performance difference remains is a position effect.
That pattern became the standard test shape. Today's "needle in a haystack" evaluations are a direct descendant. The caveat baked in from the start still holds: a single fact in one place is the easiest version of the task. As soon as several scattered facts must be combined, it gets harder.
What the Paper Does Not Say
| Claim | Supported by the paper? |
|---|---|
| Long contexts are useless | No — what is measured is a position effect, not a total failure |
| This makes RAG unnecessary | No — rather the opposite: selection and ordering matter more |
| The effect disappears in long-context models | No — it explicitly appears there too |
| The cause is settled | No — the paper measures the behaviour, it does not fully explain it |
Why It Landed So Hard in 2023
Timing explains much of the impact. In July 2023 many teams argued that retrieval was a transitional technique that growing context windows would make obsolete. This paper is the first clean measurement against that thesis — not as an opinion, but as a controlled experiment with a reproducible protocol.
That is why it is still cited whenever an architecture decision needs justifying. It does not end the debate, but it shifts the burden of proof: if you want to pour everything into the context, you have to show the model actually uses the middle.
How We Build Since
We have treated position as a quality parameter ever since, not an implementation detail. In practice that means cross-encoder reranking before hand-off, a deliberately small number of passages passed in, and positional variation as part of the test set. The effort is modest. The difference in answer quality is not.