Making Invalid Output Impossible Instead of Retrying Until It Is Valid
A two percent parse failure rate with a retry looks like a solved problem on a dashboard. It is a solved problem for the pipeline and an unsolved one for the record, because a retry that succeeds tells you nothing about why the first attempt did not.
The Failure We Had Made Invisible
The extraction step returns a JSON object against a schema. About two percent of responses did not parse: a trailing comma, an unquoted key, a fragment of prose before the object. The pipeline retried once and almost always succeeded, so the metric that reached anyone was the post-retry rate, which was excellent.
That arrangement is comfortable and it discards information. A model producing malformed output on a particular document is telling you something about that document, and we had built a mechanism that silently asked it to try again until it stopped telling us.
Constraining Instead of Correcting
The alternative is to make the malformed output unreachable. Generation is restricted so that only tokens permitted by the schema at that point can be produced, which means the result parses by construction and no retry is needed for structural reasons.
Dong and colleagues published XGrammar that winter, an engine for structured generation aimed at making this kind of constraint efficient enough to be used routinely rather than as a special case. Efficiency is the whole question here: constrained generation has been possible for a long time and expensive enough to avoid.
What It Revealed
Once structural failures could not occur, we compared the documents that had previously failed to parse against the rest. The extraction error rate on that group, after retry, had been roughly five times the rate on documents that parsed first time.
The malformed output had been a signal. The documents producing it were the unusual ones: a scanned page with a second table, a supplier using a different date format, a note in the margin. The retry had been converting a loud failure into a quiet wrong answer.
| Approach | What you get |
|---|---|
| Retry until it parses | Valid records, and a lost signal |
| Constrain generation to the grammar | Valid by construction, signal preserved elsewhere |
| Constrain and log confidence per field | Both, at the cost of a wider schema |
| Free text and parse leniently | Neither. We have stopped doing this |
What We Kept From the Old Behaviour
The signal, deliberately reconstructed. The schema now includes a per-field marker for not present in the document, and the step is instructed to use it rather than to guess. Records where several fields come back marked are routed to review, which is the population the parse failures used to identify.
That is the general lesson we took. When you remove a failure mode, check what it had been telling you, because a metric that improves because a symptom was suppressed is worse than the symptom.
Where Constraining Makes Things Worse
Where the schema is wrong. A constrained model cannot tell you that your enumeration is missing a value; it will pick the nearest permitted one, confidently, and the resulting record is valid and wrong. We hit this with a document type that had a status our enumeration did not contain.
So we include an other value with a free-text companion field in every enumeration, and we watch how often it is used. A rise in that rate is how we learn that a schema has fallen behind the documents, which is a thing schemas do quietly.
The Cost of Constraint
Some latency, and a real constraint on where it can be used. It requires a serving stack that supports it, which rules it out for some hosted providers, and it applies to the model call rather than to the pipeline, so a step that composes several calls needs the constraint at each one.
For the steps that produce text for people we do not constrain at all. A grammar over prose is either so loose that it guarantees nothing or so tight that the writing suffers, and the validation for those steps belongs with the person reading them.
What Changed in the Numbers
Parse failures went to zero, which was the least interesting result. The useful one was that flagged records rose to about three percent, slightly above the old parse failure rate, and the error rate within that flagged group is high enough that reviewing it is clearly worth the time.
Overall extraction accuracy improved by roughly a point and a half, entirely from the previously hidden group. Nothing about the model changed. We had simply stopped throwing away the cases it was struggling with.
What We Do Not Claim
We do not claim constrained generation improves correctness. It guarantees shape. The accuracy improvement in our numbers came from routing difficult documents to review, not from the model extracting better, and conflating those two would be the easiest mistake to make with this technique.
We also do not claim retries are always wrong. For a transient network error a retry is exactly right. Our objection is to retrying a semantic failure, where the second attempt is not a second chance at the same conditions but a fresh sample from a model that already found this input hard.