Benchmarks Do Not Survive Contact With a Customer System
We picked a model partly on agent benchmark results. It then failed on step three of a five-step process, in a way no benchmark had asked about. AgentBench explains the gap well — and also why we stopped treating that kind of number as a selection criterion.
The Failure That Started It
The task was a five-step approval process: read a request, look up the account, check an entitlement, write a decision, notify the requester. In isolation the agent did each step. Run end to end, it failed reliably at step three — not by getting the entitlement wrong, but by carrying a stale value from step two into it.
No benchmark we had consulted covered that. They measure whether a model can do a task. Our failure was about what the model carries between tasks.
What AgentBench Actually Measures
Liu et al. built a multi-environment benchmark for LLMs acting as agents, spanning settings from operating systems and databases to knowledge graphs and games. It is a serious piece of work and considerably closer to real usage than single-turn question answering.
It is still a benchmark, which means fixed environments and a comparable score. That is exactly what makes it useful for ranking models against each other, and exactly what makes it silent about a five-step process nobody outside our customer has ever seen.
How We Use It Anyway
We use agent benchmarks as a filter, not as a decision. If a model cannot hold a tool loop together across multiple turns in a public environment, it will not hold ours together either, and we can skip it early. That saves time.
What we do not do is choose between two credible candidates on a benchmark delta. Past a certain point the ranking says more about the benchmark than about how the model will behave inside a process with our data, our tools and our error cases.
What We Test Instead
We build the journey: the actual sequence, with the actual tools, against a counterpart system that behaves like the real one. Then we run it repeatedly and look at where it breaks, not at whether it can.
The interesting output is not a score. It is a list of failure modes with frequencies: stale state carried forward, a tool called with the previous step's argument, a retry that duplicates a side effect. None of those show up as a percentage on a leaderboard.
| What a benchmark answers | What a journey test answers |
|---|---|
| Can the model do this task | Does the process complete on our data |
| How it ranks against others | Which step breaks, and how often |
| Under a fixed environment | Against a counterpart that can say no |
| At one point in time | Every deployment, as a gate |
Why the Simulated Counterpart Matters
The reason we can run this at all is that we build a simulated version of the system the agent talks to. It holds state, it rejects invalid transitions, and it can be driven into the awkward cases on demand — an entitlement that expires mid-process, a duplicate submission, a timeout on the notify step.
Against a real system those cases are either impossible to trigger or expensive to clean up. Against a simulated one they are a test fixture. This is the piece that turns agent testing from a demo into engineering.
What We Are Careful About
A simulation that is too kind is worse than no test, because it produces confidence without coverage. We keep ours honest by replaying recorded interactions from the real system and asserting the simulation would have produced the same rejections.
We also do not pretend the journey test replaces production monitoring. It catches the failure classes we thought of. The ones we did not think of still arrive by the usual route.