Letting a Step Think Longer Instead of Reaching for a Bigger Model
The obvious fix for a step that gets things wrong is a better model. It is also the most expensive fix available, because a model change applies to every call in the system while the failure was concentrated in one step out of nine.
One Step Out of Nine
The pipeline turns a customer request into a sequence of operations against an order system. Eight of its nine steps are dull and reliable. The ninth decides the order of operations when a request touches several orders, and it was getting that wrong often enough to matter.
The reflex proposal was to move the whole pipeline to a stronger model. That would probably have worked, and it would have raised the cost of all nine steps to fix one, which is the kind of trade that is easy to approve and hard to justify afterwards.
What We Did Instead
We let that one step produce three candidate sequences instead of one, ran a deterministic validator over each, and took the first that passed. The validator was already there: it is the same set of rules that would have rejected an invalid sequence downstream.
Failures on the labelled set fell by roughly two thirds. The step became three times more expensive and the pipeline as a whole about three percent more expensive, because that step is a small share of the total work.
Why This Is Not the Voting Pattern
We use majority voting elsewhere, and this is a different thing that is easy to confuse with it. Voting takes several answers and picks the most common, which needs the answers to be comparable and treats agreement as evidence.
Here nothing is compared to anything else. Each candidate is checked against a rule set that knows what a valid sequence is, and the first one that passes wins. There is no agreement involved, and a unanimous set of invalid candidates correctly produces no answer at all.
| Way to spend more | When it fits |
|---|---|
| A stronger model everywhere | Quality is broadly short across steps |
| Several candidates, one verifier | A checkable step fails often |
| Several samples, majority vote | A fixed output set, scattered errors |
| More retrieved context | Rarely. Measure first, it often hurts |
The Research That Named the Choice
Snell and colleagues published work that August on allocating compute at inference time, comparing strategies for spending a fixed additional budget at the point of answering against the alternative of spending it on a larger model, and reporting that the inference-time route can be the more effective one depending on the difficulty of the problem.
The framing we took is that model size and inference compute are two ways of buying the same thing, and that they are substitutable rather than ordered. Before reading it we had treated a bigger model as the upgrade and everything else as a workaround.
The Condition That Makes It Work
A verifier. Our sequence validator is deterministic, complete for the property we care about, and cheap. Generating three candidates is only useful because something other than a model can tell which of them is acceptable.
Where no such check exists, this pattern degrades into generating several answers and choosing by preference, which is the failure we described in an earlier piece on evaluation. The presence of a real verifier is the whole difference, and it is worth building one before reaching for the technique.
What We Watch
The share of requests where all three candidates fail, which is now the interesting signal. It was four percent at first, and when it rises it means the request distribution has moved into territory the step does not handle, which is more actionable than a general quality metric.
Those cases go to a person rather than to a fourth attempt. We tried increasing the candidate count and the return fell away sharply after three, which matches what we would expect: a step that fails three times in a row is usually failing for a structural reason.
Where We Did Upgrade the Model
One place, and it is worth naming so this does not read as an argument against better models. The step that drafts customer-facing text moved to a stronger model, because its output is judged by people and there is no verifier that can decide whether a sentence is good.
That is the honest split. Where a check exists, buy compute at the step. Where quality is a matter of judgement, buy a better model. Most of the arguments we have seen about this treat it as one question when it is two.
What We Do Not Claim
We do not claim three candidates is a general setting. It is where the return flattened on our step, and the number will differ for anyone else. The transferable part is measuring where it flattens rather than picking a number that sounds reasonable.
We also do not claim this substitutes for a better model in general. It substitutes for one where a verifier exists, which is a minority of steps in most systems we have seen, and pretending otherwise would be selling a narrow technique as a broad one.