Getting an Assistant to Say It Does Not Know
We measured how often the assistant declined to answer and how often it should have. Two percent against nine. The gap was not a personality problem to be fixed with a firmer instruction, and treating it as one had already cost us three months.
Two Percent Against Nine
The measurement was simple and overdue. We took four hundred logged conversations, had a domain expert mark each answer as supported by the retrieved material or not, and compared that with how often the assistant had declined to answer.
It declined on about two percent. It should have declined on about nine. The remaining seven percent were answers that sounded like every other answer, which is the property that makes this expensive: there is no surface signal separating a supported answer from an invented one.
Why the Prompt Did Not Fix It
Our first three attempts were instructions. Say you do not know when the documents do not contain the answer. Then a stronger version. Then a version with examples. Each moved the abstention rate, and each moved it in both directions at once.
That is the part worth dwelling on. A firmer instruction produced more refusals on questions the system could answer perfectly well, because the model has no reliable internal access to whether its answer is grounded, so it responds to the instruction by becoming generally more hesitant rather than more accurate about when to hesitate.
Two Different Reasons for Not Knowing
Yadkori and colleagues published work that month distinguishing uncertainty that comes from the question genuinely having several valid answers from uncertainty that comes from the model lacking the knowledge, and deriving a measure aimed at detecting the second.
That distinction is what unstuck us, not as a method to implement but as a way of seeing that we had been treating one problem where there were two. The document does not contain this, and the question does not have one answer, need entirely different handling, and we had been asking a single instruction to cover both.
| Reason for not knowing | What we do |
|---|---|
| Nothing retrieved above threshold | Do not call the model. Offer contact |
| Retrieved, but the answer is not in it | Answer with what is there, name the gap |
| Question has several valid answers | Ask which case applies |
| Question is outside the domain | Route to a person immediately |
The Signal That Actually Worked
Not a property of the model. The retrieval score. If nothing comes back above the relevance threshold, we do not call the generation step at all, and the user gets a short message with a contact route instead of a paragraph.
This is deterministic, cheap and explains itself. It also covers a large share of the gap: about four of the seven percentage points were questions where retrieval had returned nothing useful and the model had answered from its general knowledge anyway.
The Harder Remainder
The rest is harder. Retrieval returns something relevant, the answer is genuinely not in it, and the model fills the gap. For this we changed what we ask for rather than how firmly we ask: the answer must quote the passage it relies on, and a post-check verifies that the quoted text appears in the retrieved material.
An answer that cannot produce a supporting quotation is not shown. That converts an unmeasurable property into a string comparison, and it is the single change that moved the number most. It is also imperfect, because a real quotation can be attached to a conclusion that does not follow from it.
What Users Got Instead
Wording matters more here than we expected. An assistant that says it cannot help is worse than useless. Ours names what it did find, says which part of the question it cannot answer from the documents, and offers the contact route in the same message.
We measured the effect on escalations rather than on satisfaction. Contacts went up, which was the point, and the share of contacts that arrived with the relevant document reference already attached went up more, which made them faster to handle at the other end.
What It Cost
Latency for the post-check, which is negligible, and a real reduction in answered questions. About seven percent of conversations that previously ended with an answer now end with a partial answer and a route to a person, and some of those answers had been correct.
That trade is one we would make again in this domain, where a wrong answer about a contractual term is expensive and a redirect is mildly annoying. In a different domain the arithmetic would point the other way, and we would rather state the trade than pretend the change was free.
What We Do Not Claim
We do not claim the quotation check detects hallucination. It detects an unsupported statement of a particular shape. A model can quote accurately and reason wrongly from the quotation, and that failure passes our check unchanged.
We also do not claim nine percent was the true rate. It is one expert's judgement on four hundred conversations, and the boundary between supported and unsupported is not always sharp. It was accurate enough to show the gap was large, which was all the measurement needed to do.