13 Jul 2024 AI Engineering 5 min read

Structured Document Extraction with Vision LLMs

How invoices and contracts become schema-validated JSON with vision LLMs in 2024: what GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro deliver on documents, which accuracy ranges are realistic in production, why the validator is written before the prompt, and where human review remains structurally necessary.

Invoices arrive as pixels

An invoice is not text. It is a layout: a supplier block in the top left, an amount table in the middle, payment terms in a footnote. Plain PDF text extraction discards this geometry. Scanned pages and smartphone photos contain no text layer at all. Contracts add a second problem: the clause that matters sits somewhere in thirty pages of boilerplate.

The classical answer is OCR plus templates — per-supplier rules that map page coordinates to fields. Templates work until a supplier changes its layout, and maintaining hundreds of them is a permanent operational cost. The task worth solving is different: arbitrary document in, schema-validated JSON out, including layouts the system has never seen.

We define the task precisely: given a document image and a target schema, produce a JSON object that conforms to the schema, contains only values evidenced by the document, and marks everything else as null. Every design decision in this article follows from that definition.

A document arrives — scan, mail or upload. 1/4

Vision models read the page directly

In 2024 this became practical. GPT-4o (13 May 2024), the updated Gemini 1.5 Pro (generally available since May 2024) and Claude 3.5 Sonnet (20 June 2024) all accept page images directly. The separate OCR step disappears. The model sees what a human clerk sees: tables, stamps, checkboxes, handwritten margins.

DocVQA (Mathew et al., 2021) is the standard benchmark for document understanding: 50,000 questions over 12,767 document images, scored by ANLS. The figures below are those reported in Anthropic's model card addendum of June 2024. One caveat: DocVQA asks single questions. Extracting a complete schema with dozens of interdependent fields is a harder task, and no benchmark number transfers to it one to one.

Model	Release	DocVQA (test, ANLS)
Claude 3.5 Sonnet	Jun 2024	95.2%
Gemini 1.5 Pro	May 2024 (GA)	93.1%
GPT-4o	May 2024	92.8%
GPT-4 Turbo	Apr 2024	87.2%

Schema first not prompt first

We recommend a fixed order: schema first, prompt second. The target schema is JSON Schema — types, enums, format constraints, required fields. A field the document does not contain must be null, never a guess; that rule goes into the schema description and into the prompt. The schema is also the contract the downstream system tests against, which makes it the natural place for domain knowledge such as allowed currency codes.

The 2024 APIs help but do not close the loop. OpenAI's JSON mode (November 2023) guarantees syntactically valid JSON — not conformance to your schema. Function calling (June 2023) and Anthropic's tool use (generally available since the end of May 2024) accept a schema and bias the output strongly toward it. None of them guarantees conformance. Every response must pass a validator; that is not paranoia but the design.

Accuracy in benchmarks and in production

Benchmark and production are different distributions. Production documents are skewed scans, faxes and photographed paper; production schemas hold 20 to 50 fields per document instead of one question. Our observations across projects, stated deliberately as ranges: header fields of digitally created invoices — supplier, date, gross amount — land above 95 percent; line items, handwriting and stamps land noticeably lower; degraded scans lower still.

The dominant error class is long identifiers: IBANs, invoice numbers, order references. A single transposed character is invisible in fluent output. Vision LLMs also fail differently from OCR. They do not produce garbled characters — they produce plausible wrong values, and occasionally a value for a field the document never mentions. A plausible error is worse than an obvious one, because it reads as correct.

Validation before trust

We build these pipelines validation-first: the validator exists before the first prompt is written. Three layers. Syntactic — the output parses and conforms to the JSON Schema. Semantic — IBAN check digits per ISO 13616, VAT ID patterns, parseable dates, known currency codes. Arithmetic — net plus tax equals gross, line items sum to the invoice total, contract dates are ordered.

A failed validation triggers exactly one retry with the validator's error message in context; a second failure routes the document to a human. State the limit clearly: validation detects inconsistent values, not wrong ones. A misread delivery date passes every rule. Validation shrinks the undetected error space. It does not close it.

Human review where confidence is low

Vision LLMs return no calibrated per-field confidence. OpenAI's GPT-4 technical report (March 2023) showed calibration degrading after RLHF, and token log probabilities are at best a weak proxy at field level. Specialised services — Amazon Textract, Azure AI Document Intelligence — do return per-field confidence scores. That remains a real argument for hybrid pipelines.

A workable substitute is agreement. Extract every document twice — two runs or two different models — and compare per field. Agreement plus passed validation: auto-accept. Disagreement, a validation failure or a business-critical field such as the payee IBAN: a review queue, where a person sees document and extracted value side by side. The goal is that reviewers handle flagged fields, not every document. What fraction gets flagged depends on document quality — honest projects measure it instead of promising it.

Outlook from July 2024

Three expectations from where we stand in July 2024. First, schema conformance moves server-side. Constrained decoding is proven in open source — llama.cpp grammars, the Outlines library — and we expect hard schema guarantees from the large API providers, replacing retry loops on the syntactic layer. The semantic and arithmetic layers remain our job.

Second, price per page keeps falling. GPT-4o already costs half of GPT-4 Turbo in the API, and Gemini 1.5 Flash and Claude 3 Haiku mark the small-model trajectory. That makes two-model agreement checks a default rather than a luxury. Third, human review does not disappear. It moves up the value chain — from typing values off paper to arbitrating flagged disagreements. We design systems for that role, not for its absence.