Our Test Set Was Too Tidy
We changed the name of one field in a customer schema and a step that had been green for months started failing on a fifth of our cases. Nothing about the task had changed, which is the part that should worry anyone maintaining an evaluation set.
The Change That Should Not Have Mattered
A customer renamed a field in their export from Lieferdatum to Lieferdatum_IST. Our extraction step handles a schema, the schema was updated, and about a fifth of our labelled cases started failing in ways that had nothing to do with that field.
Values that had been extracted correctly for months came back wrong. The task was identical, our prompt was identical, and the only variable was a surface detail that a correct implementation would ignore. Ours was not ignoring it, and neither, it turned out, was our confidence in the step.
What the Set Had Been Measuring
Eighty real cases, drawn from real documents, checked by an expert. All of them collected in the same fortnight, from four customers, in documents produced by two systems. The phrasing, the field ordering and the vocabulary were far more uniform than production traffic.
So the set measured whether the step handled that particular flavour of the task. It had been passing for months, and the passing had been telling us less than we thought, because a step can fit the surface of a set closely enough to pass without handling the task underneath it.
The Research That Made the Point Sharply
Mirzadeh and colleagues published work that October generating variants of established reasoning problems by changing names and numerical values while keeping the underlying problem the same, and reporting that model performance dropped on the variants.
Their setting is mathematical reasoning and ours is field extraction, and the transferable finding is about measurement rather than about models: if performance moves when only the surface moves, then a fixed set with a fixed surface is reporting something narrower than the number suggests.
| What we vary now | Held constant |
|---|---|
| Field names and label wording | The correct extracted values |
| Order of fields on the page | The document meaning |
| Numbers, dates and identifiers | The relationships between them |
| Language of labels, German or English | Everything else |
What We Built
A variant generator. Each labelled case produces several versions with names, numbers and ordering changed by rule, and the expected values transform with them because the transformation is mechanical rather than model-generated.
That last detail is what makes this safe. We wrote elsewhere that we do not generate expected answers, and this does not violate it: the expectation is derived deterministically from a human-checked original, so no model is deciding what correct means.
What It Found Immediately
Three real defects on the first run. The step was matching a label by position when the label was absent, treating a date as European format based on a heading rather than on the value, and reading the last of two totals rather than the one it was asked for.
All three had been present the entire time and all three were invisible because our eighty cases never presented the conditions. The tidiness of the set was not a small limitation, it was the reason we believed a step worked.
How We Report It Now
Two numbers instead of one. Accuracy on the original cases, which is what we promise customers, and accuracy across variants, which is what tells us whether the step is robust. When the second falls well below the first, the step is fitted to a surface.
The gap is the useful signal. It sat around eighteen points when we started and is now under five, and watching it is how we notice a step drifting towards our phrasing rather than towards the task.
The Cost and the Trap
Cheap to run and easy to misuse. The generator produces a lot of cases and a large green number is reassuring in a way that is not always earned, so we cap what it can claim: a variant failure is a defect to investigate, a variant pass is not evidence of anything on its own.
The original human-checked cases remain the gate. Variants sit alongside them as a robustness measure, which is the same division we use for generated inputs everywhere else in our evaluation work.
What We Do Not Claim
We do not claim variants cover the real distribution. They vary what we thought to vary, and the next surprise will almost certainly be a dimension we did not think to change, which is a limitation no amount of generation removes.
We also do not claim a small gap means a step is robust. It means the step is robust to the transformations we apply, which is a narrower statement, and stating it narrowly is the only way the number stays worth having.