The Leaderboard Is Not Your Evaluation
A customer forwarded a ranking and asked why we were not using the model at the top. It was a fair question, we ran the swap on our own set, and the model three places higher was worse for the work we actually do.
The Swap That Went Backwards
The step in question extracts eleven fields from a delivery note and returns them against a strict schema. On a public ranking, the model we were using sat several places below one we had not tried, so we tried it, because refusing to look would have been the wrong answer to the customer.
It produced better prose in every side-by-side we read and failed the schema more often. Optional fields it could not find came back with a plausible value rather than as absent, and that single behaviour was worse for us than every stylistic gain put together.
What a Preference Ranking Measures
Chiang and colleagues described the open platform behind one of the best known rankings that month: users compare two anonymous responses to a prompt of their own choosing and vote for the one they prefer, and the votes are aggregated into a rating.
That design is honest about what it is, and it measures something real that static benchmarks miss. It is also, by construction, an average over the questions people happen to ask, judged on the response they happen to prefer. Neither of those is our field extraction step.
Why the Better Model Lost
Preference favours a response that reads well. A model that fills an uncertain field with a confident value reads better than one that leaves it empty, and a human comparing two answers without the source document in front of them cannot tell which one invented the number.
Our requirement is the reverse. An empty field routes to review and costs a minute; a confidently wrong field enters the customer's system and is discovered weeks later. The ranking was not wrong, it was answering a different question, and we had briefly forgotten to ask ours.
| A public ranking tells you | It cannot tell you |
|---|---|
| General capability, broadly | Behaviour on your schema |
| What people prefer to read | What your downstream system needs |
| Relative standing today | Latency and cost at your volume |
| Which models to shortlist | Which model to ship |
What We Use Rankings For
We do use them, for one thing: deciding what to test. A model that appears near the top of several independent evaluations is worth an afternoon on our own set, and one that appears nowhere probably is not. That is a shortlist, and a shortlist is genuinely useful.
What we stopped doing is treating movement in a ranking as a reason to change anything. A model rising two places is not information about our pipeline, and reacting to it costs a re-evaluation, a deployment and a period where the behaviour everyone had learned is different.
The Eighty Cases
Our decision runs on a labelled set of about eighty cases written by people who know the domain, with expected outcomes checked by hand. It is small enough to be maintained and specific enough that a result on it means something about production.
Every candidate model runs the same set with the same prompts. We look at schema compliance first, then correctness of the extracted values, then the rate of confident invention on fields that are genuinely absent. Prose quality is not scored, because in this step nobody reads the prose.
The Conversation With the Customer
We showed the customer the comparison rather than arguing about the ranking. Fourteen of eighty cases were worse with the higher-ranked model, all of them in the same way, and two of the fourteen were the kind of error that would have reached their ERP unnoticed.
That ended the discussion in about five minutes, and it ended it better than a general argument about benchmarks would have. The number is the point: not a claim that rankings are unreliable, but evidence about this step, this schema and this data.
What This Costs to Maintain
The eighty cases took a domain expert roughly two days to write and check, and they need revisiting when the document types change. That is a real cost and it is the reason most teams do not have such a set, which in turn is why most model decisions are made from rankings.
It pays back the first time a swap is proposed. We have now evaluated eleven candidate models against it, and in three of those eleven the result contradicted what the public standing suggested. Without the set we would have shipped at least one of them.
What We Do Not Claim
We do not claim public rankings are misleading. The one in question is transparent about its method, and the model that lost on our set is genuinely a strong model. It is better at the thing being measured and worse at the thing we need.
We also do not claim our eighty cases are a sufficient evaluation in general. They cover the failures we know about, they are blind to the ones we have not encountered, and they should be read as the minimum bar for a change rather than as proof that a model is good.