The Document That Gave Orders
It was not an attack. A supplier had pasted a note meant for a colleague into the footer of a delivery note, and our extraction step read it as an instruction and followed it. Nothing malicious had to happen for us to learn that we had no boundary at all.
What the Footer Said
A delivery note arrived with a sentence in its footer along the lines of ignore the quantities in the table, the corrected figures follow by email. It had been written by a person for a person, pasted into the wrong template, and it was perfectly reasonable prose.
Our extraction step returned the eleven fields with the quantities left empty and a note in the free-text field repeating the sentence. It had done exactly what the document told it to, and every individual component had behaved as designed.
Why This Is Not a Prompt Problem
Our first instinct was to add an instruction telling the model that document content is data and must never be treated as a command. We did add it. It helps a little and it is not a boundary, because both our instruction and the document text arrive at the model as the same kind of thing.
Anything that depends on the model reliably distinguishing our text from the document text is a preference, not a control. That is the sentence we now open with when a customer asks how we protect against this, because it sets up every other measure correctly.
The Research That Made It Concrete
Clusmann and colleagues published a study that month demonstrating prompt injection against vision-capable models in an oncology setting, where instructions hidden in the content given to the model changed its output in a domain where a wrong answer carries real clinical weight.
Two things in that were useful to us. It is a demonstration on realistic material rather than on a toy prompt, and it lands in exactly the place our own pipeline had just moved to: we had recently started sending page images to a model, and an image can carry text that no text-based filter ever sees.
| Measure | What it is worth |
|---|---|
| Instruction telling the model content is data | Helps a little. Not a boundary |
| Filtering for injection patterns | Catches the obvious. Trivially evaded |
| Schema-only output, no free text | Strong. Removes most of the channel |
| No tools on the step that reads content | The actual boundary |
The Change That Actually Held
The extraction step has no tools, no network access and no ability to do anything except return a value for each field in a schema. Whatever a document says to it, the most it can do is produce wrong field values, which our cross-field reconciliation check is there to catch.
That is the whole design principle and it is not new: the component that reads untrusted input is not the component that can act. Our mistake was not having thought about it in the first place, because a step that only reads documents did not feel like an attack surface.
What We Removed
The free-text field. It existed so that an operator could see anything unusual on the document, which is a reasonable feature, and it was also the channel through which the footer text reached a human as if our system had said it.
It now returns a flag rather than text: unusual content present, with a link to the page image. A person looks at the source rather than at a repetition of the source, which turns out to be both safer and more useful, since the layout is often the point.
Where the Risk Really Sits
Not in extraction. Extraction is bounded by its schema and produces data that other checks can verify. The risk sits in any step where a model reads content and then calls a tool, because that is where a sentence in a document becomes an action in a system.
We have one such step, and it is where our approval rules live. Anything with an external effect names the action and its arguments and waits for a person. Slow, deliberately, and the slowness is concentrated on the small share of operations that can actually cause harm.
What We Tell Customers to Ask
One question, and it works without any knowledge of models. Which component reads untrusted content, and what can that component do? If the answer is that it can do things, the design is wrong regardless of how good the prompt is.
The follow-up is what happens when it is wrong rather than when it is attacked. A pipeline built so that a mistaken extraction is caught downstream is also a pipeline where an injected instruction has nothing to reach, and the ordinary-failure case is much easier to reason about than the adversarial one.
What We Do Not Claim
We do not claim we are safe from injection. We claim the step that reads documents cannot act, which bounds the damage rather than preventing the influence. A document can still produce wrong values, and the defence against that is verification, not filtering.
We also do not claim pattern filtering is worthless. We run it, it catches clumsy attempts, and it is a reasonable outer layer. It is simply the weakest thing in the list, and presenting it as the answer would be misleading about where the security actually comes from.