Retrieve Only When It Helps
“Always retrieve” is the default because it is simple, and for a while it was ours. Then we noticed the assistant answering “how do I reset my password” with three loosely related policy excerpts and a hedge.
The Answer That Was Too Long
The complaint was not that answers were wrong. It was that simple questions produced elaborate ones. Asked how to reset a password, the assistant retrieved three policy documents, quoted them, and added a caveat about exceptions that did not apply.
The retrieval was working exactly as built. Every question triggered a lookup, every lookup returned its top passages, and every set of passages ended up in the answer whether or not it added anything.
The Idea We Took From the Research
Asai and colleagues framed the problem clearly: retrieving a fixed number of passages regardless of whether retrieval is needed, or whether the passages are relevant, can reduce a model's versatility and produce unhelpful output. Their answer, Self-RAG, trains a model to decide on demand and to reflect on what it retrieved using special tokens.
The diagnosis matched ours precisely. The remedy did not fit our situation, and the distinction is worth being explicit about.
Why We Kept the Decision Outside the Model
Training the decision into the model means the decision is only as inspectable as the model is. Ours has to be auditable: a regulated customer needs to see why a given answer did or did not consult the document base, and needs that answer to be stable across model versions.
So we put the decision in the pipeline. A small classifier runs before retrieval and outputs one of three routes: answer directly, retrieve then answer, or escalate. It is a component we can test exhaustively, version independently, and explain in a sentence.
| Decision inside the model | Decision in the pipeline |
|---|---|
| Learns subtle cases from data | Only as good as its features |
| Changes when the model changes | Versioned on its own |
| Hard to explain a single case | One classifier output to point at |
| Needs training data and a tuning run | Needs a labelled set and an afternoon |
What Predicts It Well Enough
Two signals carried most of the accuracy. First, whether the question names an entity that exists in the corpus — account numbers, product codes, contract references. If it does, retrieve. Second, whether the question matches a small set of procedural intents that the assistant already knows how to answer, which is the password-reset class.
Everything else defaults to retrieving. We would rather retrieve unnecessarily than miss, because an unnecessary lookup costs latency while a missed one costs a wrong answer.
What Changed in the Output
Short questions started getting short answers, which was the point. The measurable effect was in follow-up rate: fewer clarifying turns, because the answer was no longer padded with material the user had to read past.
Token cost fell as a side effect rather than a goal. The lookups we skipped were the cheap ones, so the saving was modest and not the reason to do it.
What We Are Not Saying
We are not saying learned retrieval decisions are the wrong direction. For open-domain assistants without an audit requirement, letting the model decide is likely better than a classifier, because the judgement is genuinely subtle and a rule set will always be crude.
We are also not claiming three routes are sufficient in general. It is what this assistant needed. A system with several distinct corpora would need the route to name which corpus, and that is a different classifier.