Where a Reasoning Model Earns Its Latency
Models that produce an extended chain of reasoning before answering became widely available that January, and the question we were asked was whether we would switch. We ran all nine of our steps against one, and the answer was that two of them should.
What We Tested
Nine steps, each with its own labelled set, each run three times against the existing model and three times against a reasoning model, with prompts adapted only where the output format required it.
The context was that reasoning-focused models with published weights arrived in January 2025, notably the DeepSeek-R1 release, which made this a question customers asked us directly rather than a topic we could defer.
The Two That Improved
The step that plans a sequence of operations across several orders, and the step that reconciles two reports and explains a discrepancy. Both improved by a margin that survived repeats, and both share a property: there is an intermediate decision whose correctness determines the final answer.
That is the pattern, and it is not the same as difficulty. Our hardest step by error rate is document extraction from poor scans, and it did not improve at all, because the difficulty there is perceptual rather than deliberative. More thinking does not make a smudged digit legible.
The One That Got Worse
Classification into eleven categories. Accuracy fell by about two points, and the failures had a consistent shape: the model reasoned itself away from an obvious category by constructing a case for an unusual one.
That behaviour is a reasonable thing for a reasoning process to do and a bad thing for a router to do. We reverted it, and we now describe this to customers as the clearest case where extended reasoning is not an upgrade: a decision that should be immediate does not benefit from deliberation.
| Step type | Effect |
|---|---|
| Multi-step planning | Clear improvement |
| Reconciliation with an explanation | Clear improvement |
| Extraction from documents | No change. The difficulty is perceptual |
| Classification into fixed categories | Worse. Deliberation invents cases |
The Cost Side
Latency roughly tripled on the two steps we switched, and neither is user-facing, which is why the switch was affordable. Had the reconciliation step sat in front of a person waiting for an answer, the quality gain would have been paid for in a wait we could not justify.
Token cost rose more than latency, because the reasoning is generated. We route those two steps to the reasoning model through our gateway, which meant the change was configuration rather than code, and the cost shows up as its own line rather than disappearing into a total.
The Reasoning Is Not an Audit Trail
This is the point customers most often get wrong, and it is easy to see why: the visible chain reads like an explanation of how the answer was produced. Treated that way it invites a conclusion we cannot support, which is that showing it satisfies a requirement to explain a decision.
It is generated text that accompanies an answer. It may describe the process faithfully and we have no way to verify that it does, so we do not show it to end users and we do not put it in records that a customer might rely on. Where a decision must be explainable, the explanation has to come from a deterministic rule, not from a narrative.
What We Do Show
For the reconciliation step, the discrepancy, the two source figures, the rule that flagged them and a short generated summary clearly marked as a summary. That is auditable because the first three items are records rather than text.
The reasoning chain is kept in our own traces for debugging, where it is genuinely useful: reading why the model went wrong on a failing case is faster than inferring it from the output alone. Internal diagnostic, not customer-facing evidence.
How We Advise Now
Test per step, never per system. The nine-step result is the argument: a system-wide switch would have bought two improvements, six unchanged steps at triple the cost, and one regression, which nets out badly and would have been invisible without per-step measurement.
And check where the step sits relative to a waiting human. A quality gain paid for in latency is free when nothing is waiting and expensive when someone is, which is a question about the interface rather than about the model.
What We Do Not Claim
We do not claim reasoning models are narrow. Our nine steps are a particular pipeline of bounded business tasks, and the tasks these models are built for, mathematics and code among them, are not represented in our set at all.
We also do not claim the classification regression generalises. It is one step, one taxonomy, three runs, and the mechanism we describe is our reading of the failures rather than something we established.