Asking Twice and Voting: When Repetition Is Worth Paying For
A classification step in one of our pipelines disagreed with itself on about six percent of inputs. We ran it five times and took the majority, and the disagreement went away. We then tried the same trick on a summarisation step and bought nothing at five times the cost.
The Six Percent
The step in question routes an incoming document into one of eleven categories. It ran at what looked like acceptable quality until we started sending the same document through twice as a consistency check, and found it landed on a different category about six percent of the time.
That number was more alarming than a straightforward error rate, because it meant the pipeline could produce two different outcomes for one input depending on nothing at all. A customer seeing that once loses confidence in everything downstream of it.
The Fix That Was Almost Too Simple
We ran the classifier five times and took the most frequent answer. Self-disagreement on repeated runs dropped to well under one percent, and the cases where the five runs split three to two turned out to be genuinely ambiguous documents that a human reviewer also hesitated over.
That second effect was the more valuable one. The vote spread became a usable confidence signal: an even split now routes the document to review rather than guessing, which is something the single confident answer never let us do.
Where It Bought Nothing
Encouraged, we tried the same thing on a step that writes a short summary of each document. Five runs produce five different summaries, all reasonable, none identical, and there is no majority to take. We tried picking the one most similar to the others and could not show that it was better than the first one.
The distinction is not about difficulty. It is that voting needs outputs that can be compared for equality. Eleven categories can be tallied. Free text cannot, and pretending otherwise just adds a similarity metric that nobody can defend.
| Voting helps | Voting does not help |
|---|---|
| Output is one of a fixed set | Output is free text |
| Equality is well defined | Only similarity is available |
| Errors are scattered, not systematic | The model is consistently wrong |
| The spread itself is useful | Cost matters more than the last percent |
The Research and What It Adds
Li and colleagues published a study that month on sampling a model repeatedly and combining the results by majority vote, reporting that performance tends to improve with the number of samples across a range of tasks and that the gain is largest where a task is harder for the model.
What we took from it was permission to treat this as a normal engineering lever rather than a hack. What it does not answer is our second failure: the paper works with tasks that have comparable answers, and the interesting operational question is deciding whether yours does.
The Case That Voting Makes Worse
There is a failure mode that repetition hides rather than fixes. If a model is systematically wrong about something, all five runs are wrong the same way, the vote is unanimous, and the pipeline now reports high confidence in a wrong answer.
We hit this with a date field on one document type, where a particular layout was consistently misread. Before voting it was wrong and uncertain; after voting it was wrong and certain, which is worse. Repetition measures the spread of a model around its own answer, not the distance between that answer and the truth.
How We Decide Now
Two questions. Can the outputs be compared for equality, and are the errors scattered rather than systematic. Both yes, we vote. Either no, we spend the same budget on something else, usually a better schema, a tighter prompt or a check against a system of record.
The cost side is deliberately blunt: five times the calls and five times the latency for that step. We only pay it where a single inconsistent answer is expensive, which in practice is classification and routing, not generation.
What We Run in Production
Three runs rather than five on the classifier. The move from one to three captured most of the benefit, three to five was marginal, and three keeps the latency inside what the pipeline can absorb. Unanimous is accepted, two to one is accepted with a flag, and anything else goes to review.
We log the spread on every run. It has since turned out to be one of the better drift signals we have: when the rate of split votes climbs, something has changed in the incoming documents, and it shows up there before it shows up in quality.
What We Do Not Claim
We do not claim more samples always help. Our gain flattened quickly, and the point at which it does is task-specific rather than something we can predict for a new step without measuring.
We also do not claim vote spread is a calibrated probability. It is a useful ordering, and treating it as an actual confidence value would be over-reading a number that is really about the model agreeing with itself.