Testing a Conversation, Not a Question
Our labelled set asked a question and checked an answer, which is a fair test of a search engine and a poor test of an assistant. Real conversations have follow-ups, and the follow-up is where ours went wrong.
Where the Failures Were
We reviewed sixty escalated conversations to find out what the assistant had done wrong. In forty-one of them the first answer was fine. The problem appeared at the second or third turn, and in nineteen of those the trigger was a follow-up that referred back to something earlier.
Our evaluation set contained none of this, because every case was a question and an expected answer. We had been measuring the part that worked.
The Two Failures That Need a History
Reference. A user asks about a part, then says and the other one, and the assistant either resolves that correctly or answers about something the user never mentioned. There is no way to test this with a single-question case, because the ambiguity only exists relative to what came before.
Constraint carry-over. A user establishes in turn one that they are working on the older variant, and by turn four the assistant is answering for the current variant because that constraint has aged out of the context. Both failures look like ordinary wrong answers in a log.
The Research on Benchmarking Dialogue
Kranti and colleagues published clem:todd in 2025, a framework for systematically benchmarking task-oriented dialogue systems built on language models, with the conversation rather than the individual response as the unit of evaluation.
The idea we took is that a dialogue test needs a counterpart. Testing a single response only requires an input; testing a conversation requires something that plays the user, and the design of that something is most of the work.
| Single-question case | Conversation case |
|---|---|
| One input, one expected answer | A goal and a sequence of user turns |
| Checks correctness | Checks correctness and carry-over |
| Cheap to write | Needs a simulated or scripted user |
| Misses reference and constraints | Catches both |
How We Build a Conversation Case
A scripted user, not a simulated one, for the gate. Each case is a fixed sequence of user turns written by a domain expert, with expected properties for each assistant turn: the answer must be about the older variant, must name the part the user referred to, must not have invented a constraint.
Scripted means the test is deterministic and the failure is attributable. A simulated user producing different turns each run is useful for exploration and useless as a gate, because a failing run cannot be reproduced.
Where the Simulated User Is Useful
Finding cases we would not have written. We run a model as a user with a goal and a persona against the assistant, collect conversations that end without the goal being met, and a person reads them. The good ones become scripted cases.
That is the correct division: the simulator generates candidates, a human decides what a correct conversation looks like, and the scripted version is what runs in the pipeline. Letting the simulator judge would repeat the mistake we made with generated evaluation data.
What Changed in the System
Two things, both small and both invisible to a single-question test. The assistant now restates the constraint it is working under when a conversation has established one, in a short clause rather than a paragraph. And references to earlier entities are resolved explicitly, with the resolution shown, rather than inferred silently.
Escalations from the multi-turn failure modes fell by roughly two thirds. The single-question accuracy did not move at all, which is exactly what we should have expected and is the clearest evidence that the two tests measure different things.
What It Costs
A conversation case takes a domain expert about three times as long as a question case, and we have forty of them against eighty single-question cases. The suite takes longer to run because each case is several model calls.
We would not replace the single-question set with it. Those cases are cheaper, they cover the retrieval and extraction behaviour thoroughly, and they catch the failures that a conversation test would find only by accident. The two sets test different components.
What We Do Not Claim
We do not claim forty conversation cases is adequate coverage. Conversations branch, ours do not, and a user who takes a path we did not script is untested. It is better than zero, which is where we were, and that is the honest description.
We also do not claim the simulated user finds representative problems. It finds problems that a model playing a user encounters, which is a distribution with its own biases, and the human filtering step exists precisely because we do not trust it to select.