Showing More Is Not the Same as Helping
We believe in showing our work, and this blog is full of it: the source passage, the SQL, the row count, the retrieval score. Then we measured what happens when a reviewer sees all of it at once, and the result argued against us.
What We Had Built
The review screen for extracted records showed, for each field, the extracted value, the passage it came from, a retrieval score, whether the field had been marked as absent, and a short explanation of how the value was determined. Every one of those was added because someone asked for it.
Reviewers using it corrected fewer errors than reviewers using the previous, plainer screen. Not dramatically, but consistently, and the difference held when we swapped which group used which.
The Explanation We Did Not Want
Margondai and Mouloua published work in January 2026 developing a theoretical account of when AI explanations improve human decisions and when they impair them, framed around cognitive load and the depletion of the reviewer's own judgement.
The uncomfortable part for us is that our design assumed transparency is monotonic: more visible reasoning, better decisions. What we had built was a screen where finding the one field that was wrong required reading five annotations for the eleven fields that were right.
What We Removed
The retrieval score, which no reviewer could interpret and which we had added because it was easy. The per-field explanation, which was accurate and long. And the confidence indication, which turned out to be the most harmful of the three.
Confidence was harmful in a specific way. A field marked high confidence was checked less carefully, and the errors that survive review are by definition the ones a model was confident about, so the indicator was steering attention away from exactly the fields that needed it.
| Element | What we do now |
|---|---|
| Extracted value and field name | Always visible |
| Source passage | One click away, not on screen |
| Confidence indication | Removed. Steered attention wrongly |
| Cross-field check result | Always visible. The one signal that helps |
What We Kept and Made Louder
The reconciliation check. A record where quantities and totals do not agree is marked at the top of the screen, and that single indicator does what the confidence score was supposed to do, without the failure mode, because it is computed from the data rather than expressed by the model.
Review accuracy on the simplified screen with a prominent reconciliation flag was better than either previous version. The reviewers were never the problem; the screen was asking them to do a filtering job that a rule could do.
The Principle That Needed a Limit
We still believe a system should be able to show its work. What we got wrong is that being able to show is not the same as showing by default, and a system that displays everything it could justify is optimising for its own defensibility rather than for the person using it.
The rule we use now is that the default screen shows what a decision needs, and everything else is one click away and always available. Nothing was removed from the system; it was removed from the first thing a person sees.
Where More Detail Does Help
Disputes. When a controller challenges a figure or a customer challenges an answer, the full provenance is exactly what settles it, and that is a different task performed by a different person with time to read.
That is why we did not simply delete the annotations. The information has a genuine user; it is just not the reviewer working through a queue at four in the afternoon, and designing one screen for both readers served neither.
How We Test This Now
With reviewers and a stopwatch, on real records with known errors, comparing screens. It is a small study each time, eight to twelve people, and it has now twice reversed a change that everyone in the room thought was obviously an improvement.
That is the argument for doing it at all. Interface changes to a review screen look like taste, they are measurable, and the measurement disagrees with the room often enough to be worth the afternoon.
What We Do Not Claim
We do not claim transparency is harmful. It is what makes a disputed answer resolvable and an audit possible, and a system that cannot show its work is one we would not ship. The claim is narrower: what to show by default is a design decision that has to be measured, not derived from a principle.
We also do not claim our result generalises to other interfaces. It is one review screen, one document type and a small number of reviewers, and the useful part is the method rather than which elements we removed.