We Do Not Ship Systems That Change Themselves
A customer asked for an assistant that learns from corrections and adjusts its own instructions. It is an appealing idea, we have built a version of it, and we do not ship it, for a reason that has nothing to do with whether it works.
The Request and the Prototype
The idea is straightforward. When a reviewer corrects an extraction, feed the correction back so the instruction improves, and over months the system fits the customer's documents better without anyone maintaining a prompt.
We built it. It worked, in the sense that accuracy on recent documents rose over six weeks. We ran it in a development environment, nowhere else, and the reason we stopped is not in the accuracy numbers.
The Question That Ends It
What version is running? On any other system the answer is a commit hash. On a self-modifying one the answer is a description of a history, and a description of a history is not something a customer can put in a change record, an auditor can inspect, or an engineer can roll back to.
Everything downstream assumes a version exists. Our system documents describe behaviour, our evaluations certify a state, our incident process reverts to a known good. A system that rewrites its own instructions invalidates all three quietly, without anyone deciding to give them up.
The Research on Where It Drifts
Han and colleagues published work in late 2025 on what they call an alignment tipping process, examining how self-evolution can push agents away from their intended behaviour over the course of operation.
That is a stronger objection than ours and it points the same way. Our concern is administrative and immediate: nobody can say what the system does today. Theirs is that the direction of the drift may not be the one anyone intended, which is the same problem with a longer time constant.
| Adaptive mechanism | Our position |
|---|---|
| Prompt rewrites itself from corrections | No. No nameable version |
| Corrections collected for a human to apply | Yes. This is the version we ship |
| Retrieval index updated continuously | Yes. Data, not behaviour |
| Per-user preferences stored | Yes, with limits, written about separately |
What We Ship Instead
The collection half without the application half. Corrections are gathered, grouped, and presented monthly as a proposed prompt change with the cases that motivated it. A person reads it, accepts or rejects, and the change goes through the normal deployment path with an evaluation run.
That captures most of the value. The learning is real, it just lands as a reviewed change rather than as a silent one, and the review takes about an hour a month. Two of the last twelve proposals were rejected, both because the correction pattern reflected one reviewer's preference rather than a document property.
The Distinction That Matters
Data changing is fine. Our indexes update continuously, documents are added and superseded daily, and nobody considers that a version problem, because the behaviour is fixed and the inputs vary.
Behaviour changing is the line. When the thing that decides how to interpret an input changes on its own, the system has no stable description, and every artefact that describes it becomes a historical document. That is the distinction we use in design reviews and it has been easy to apply.
What We Say to Customers
That we can give them the improvement without the unpredictability, that the cost is an hour a month of someone's attention, and that this is not us being conservative for its own sake: the same customer who wants a self-improving assistant is usually the one whose auditor will ask what it did in March.
That framing has persuaded every customer who has asked. Nobody actually wants a system whose behaviour last quarter cannot be reconstructed; they want the improvement, and the improvement is available with a review step attached.
Where We Would Reconsider
If the adaptation were confined to something with a version. A learned routing table that is versioned, evaluated and deployable like any other artefact is not self-modification in the sense that worries us, even though the values in it were derived automatically.
The property we need is that a state can be named, stored, evaluated and restored. Any adaptive mechanism with those four properties is one we would build. The ones we decline are the ones where the state is only ever the current one.
What We Do Not Claim
We do not claim self-improving systems cannot work. Ours worked in development and the accuracy trend was real, and there are settings, particularly ones with no audit requirement and a fast feedback loop, where the trade would be sensible.
We also do not claim our monthly review captures everything a continuous loop would. It is slower and it applies a human filter that will reject some genuine improvements, which is a cost we accept in exchange for a system that can be described.