When the Page Is a Picture, Text Extraction Loses the Table
The scan was clean, the recognition was accurate, and the output was wrong. Reading a table left to right across a page with three columns produces a sequence of correct words in an order that means something entirely different.
Correct Characters, Wrong Data
A delivery note came through with a three-column table: article, quantity, unit. The scan was clean and the character recognition made no mistakes we could find. The extracted quantities were nonetheless attached to the wrong articles, consistently, on every document of that layout.
The cause is that text extraction produces a stream and a table is a two-dimensional arrangement. Flattening one into the other discards exactly the information that made the table a table, and a model reading the flattened stream has no way to recover it.
Why We Did Not Notice Sooner
Every individual value was plausible. Quantities were quantities, article numbers were article numbers, and a validation rule checking types and ranges passed all of them. The error was in the association between fields, which is precisely what field-level validation cannot see.
This is worth stating plainly because it generalises. Validation that checks each field in isolation will pass a record where every value is real and the combination is invented, and that is one of the more expensive failure modes in document processing.
Sending the Image Instead
For that document type we stopped converting to text and started sending the page image to a model that accepts images, asking for the same schema. The columns are visible in the image, the association is preserved, and the same layout that broke the text pipeline is unremarkable for the visual one.
That is not a free upgrade. Image input costs more per document, is slower, and the failure modes are different rather than absent: a model reading a page can still misread a handwritten annotation or a stamp overlapping a figure, and it does so without the layout excuse.
The Research Behind the Shift
McKinzie and colleagues published MM1 that spring, a study of what actually matters when pre-training multimodal models, reporting on the effects of image resolution, the composition of the training mixture and the design of the connection between the visual encoder and the language model.
The finding we carried into practice was about resolution. If the visual detail available to the model constrains what it can read, then our scanning settings are part of the system rather than a preprocessing detail, and we now treat resolution as a parameter to test instead of a default to inherit.
| Document trait | What we use |
|---|---|
| Plain prose, single column | Text extraction. Cheaper and sufficient |
| Tables, multi-column layout | Page image to a multimodal model |
| Forms with checkboxes or stamps | Page image, plus a review threshold |
| Digital documents with real structure | Neither. Parse the structure directly |
The Row That Is Not Worth Losing
The last row of that table matters more than the others. A large share of what customers call scanned documents are digitally generated files that were never printed, and those carry their table structure inside the file.
Extracting it properly is unglamorous work with no model in it at all, and it is more accurate than either alternative. We check for this first on every new document type, and it has removed the problem entirely for about a third of them.
The Check That Makes Either Path Safe
Field validation was not enough, so we added a cross-field check specific to the document. On a delivery note, quantities times unit prices must reconcile against the stated total, and a record where every field is individually valid but the arithmetic does not close goes to review.
That check would have caught the original failure on the first document rather than the fortieth. It is also the piece we now insist on before any extraction runs unattended, regardless of whether the input is text or an image, because both paths can produce a well-formed record that is wrong.
What It Cost to Change
Roughly four times the per-document cost for the affected layouts, and a slower step in a pipeline where nothing was waiting on it, so the latency was irrelevant. Against that, the corrections had been costing a person about two hours a week, and the errors that reached the customer cost considerably more than that.
We did not convert everything. Two thirds of the document types still go through text extraction because they are prose or simple key-value layouts where it is accurate and much cheaper. Choosing per document type rather than globally is the whole saving.
What We Do Not Claim
We do not claim visual models make text extraction obsolete. For most of our documents it remains the right tool, and replacing a cheap accurate step with an expensive one because it handles a case you do not have is not an upgrade.
We also do not claim that sending an image removes the need for verification. It changes which errors occur, not whether they occur, and the reconciliation check stayed in place for exactly that reason.