Why We Do Not Automate by Clicking Through Screens
A customer showed us a recording of an agent operating their ERP through the screen, filling fields and clicking buttons, and asked how quickly we could put it into production. The honest answer involved asking what was wrong with the API.
What the Demonstration Showed
It genuinely worked. The agent read the screen, found the order search, typed an order number, waited for the results and clicked into the right row. Watching it is more convincing than any diagram of an integration, and that is precisely the problem with it as a basis for a decision.
The demonstration proves that the task is possible. It says nothing about the failure rate over ten thousand runs, what happens when a dialog appears that was not in the recording, or who is responsible when a click lands on the wrong row.
A Screen Is Not the System
The user interface is a rendering of the system for a person. It changes when the vendor decides a button belongs elsewhere, it varies with screen size and locale, and nothing in it is a contract. An API, however badly documented, at least implies a promise about names and shapes.
That distinction is the whole argument. Automating against a screen means our system depends on a surface that nobody has agreed to keep stable, and the dependency is invisible until a routine update moves a field and a nightly job silently starts doing something else.
The Research and What It Establishes
Wu and colleagues published OS-ATLAS that October, an action model for agents operating graphical interfaces, with attention to grounding, meaning the mapping from a described element to the place on screen where it actually is.
The reason grounding is the hard part is the reason we are cautious. Interface automation depends on locating the right element reliably, and the research treating that as a substantial open problem is a better guide to production risk than a successful recording.
| Question | If the answer is unhelpful |
|---|---|
| Is there an API or a database? | Use it. This is most cases |
| Can the vendor commit to the layout? | Screens will move. Plan for breakage |
| Is the action reversible? | No unattended runs |
| Is the volume worth the fragility? | Below a few hundred a day, use a person |
The Project Where We Said Yes
A supplier portal with no API, no export, and a legal requirement to retrieve documents that only exist there. Twice a day, roughly forty documents, and the alternative was a person doing it by hand for an hour each time.
We built it, and the acceptance criteria were unusual: it must fail loudly rather than continue, every run produces a screenshot log, and a human checks the count against an expected range. It has broken twice in eighteen months, both times after a portal redesign, and both times it stopped rather than guessed.
What We Insisted On
Read-only, wherever possible. Downloading a document is recoverable; submitting a form is not, and the agent that submits is a different proposition from the agent that fetches. Our portal agent reads and downloads and does nothing else.
Where a write is unavoidable, it produces a draft that a person confirms. That halves the value proposition and it is the only arrangement we have been willing to run, because a misplaced click in a system of record is not a failure anyone can see in a log until it matters.
The Cost Nobody Budgets
Maintenance. Interface automation does not degrade gracefully; it works until it does not, and the repair is not a code fix but a re-derivation of how the screen is now laid out. We price it at a day a quarter per portal and have been roughly right.
There is also the test environment problem. Testing a screen agent means having a test instance of somebody else's portal, which frequently does not exist, so verification happens against production with all the care that implies.
What We Recommend Instead
In order: use the API. If there is none, ask for a scheduled export, which vendors often provide and customers often do not know about. If that fails, ask the vendor directly, because a customer requesting an integration is a stronger signal to them than a workaround they never hear about.
Screen automation is fourth, and it is a reasonable fourth. Our objection is not to the technique, it is to reaching for it before the first three have been tried, which happens because the demonstration is more compelling than a conversation about export formats.
What We Do Not Claim
We do not claim interface agents do not work. Ours works, it saves an hour a day, and the technology has improved considerably. The claim is about ordering: it is the option you take when the others are unavailable, not the one you take because it demonstrates well.
We also do not claim our thresholds are principled. A few hundred operations a day is where fragility stopped being worth it for us, and someone with a more stable portal or a higher tolerance for breakage would reasonably draw the line elsewhere.