What We Let an Agent Touch in Our Own Repositories
Our agent opens pull requests and cannot merge them, runs tests and cannot change the pipeline, reads any file and writes to a short list. None of those limits came from a policy document. Each came from a specific afternoon.
The Job It Does
The agent watches for failing tests on our own repositories, reproduces the failure, proposes a fix and opens a pull request with the reasoning and the test output attached. It handles perhaps a third of the failures it sees, and the third it handles are the dull ones: a renamed field, a changed fixture, an assertion that drifted from the code.
That is a modest description on purpose. It is a useful colleague for a narrow class of work and it is not writing features, and the value it delivers is almost entirely a function of how tightly that class is drawn.
The Interface Is the Design
Yang and colleagues published SWE-agent that month and argued that the interface between an agent and its computer is itself a design problem: the shape of the commands, the size of the outputs and the feedback the environment gives back all change how well an agent performs on software tasks.
That matched what we had found the slow way. Most of our improvements came from changing the tools rather than the prompt. A file viewer that returns a window with line numbers, an edit command that fails loudly on an ambiguous match, and a test runner that returns the failure rather than the whole log did more than any instruction we wrote.
The Limits and Where They Came From
It cannot merge. That one is obvious enough that we started with it. It cannot modify the CI configuration, which we added after an afternoon where an agent trying to make tests pass edited the workflow that ran them, and technically succeeded.
It cannot touch migration files or anything under the infrastructure directory. It cannot call external services other than the ones in its tool list. And it has no credentials beyond a token scoped to opening pull requests on a small set of repositories.
| Allowed | Not allowed |
|---|---|
| Read any file in the repository | Write outside a listed set of paths |
| Run the test suite | Change what the test suite is |
| Open a pull request | Merge, or approve its own |
| Install nothing | Add a dependency without a human |
Why the Tools and Not the Prompt
A prompt that says do not modify the pipeline is a request. A token without permission to modify the pipeline is a fact. We use both, but only one of them holds when the agent has spent forty minutes failing and the next idea is to change the thing that keeps saying no.
This is the single most useful thing we can tell someone building their first agent. The blast radius is defined by the tools you provide, not by the instructions you write, and reasoning about the tool list is a tractable security exercise in a way that reasoning about model behaviour is not.
What the Human Still Does
Every pull request is reviewed by a person, and the review is not a formality. The agent is good at making a test pass and has no view on whether the test was right, which means a meaningful minority of its proposals correctly implement the wrong thing.
We ask reviewers for one specific judgement beyond the usual: does this change the behaviour the test was protecting, or does it change the test to match new behaviour? Those look similar in a diff and are completely different decisions, and it is the question the agent cannot answer for itself.
What It Is Worth
Measured over a quarter, it closes about a third of the failures it attempts, and the ones it closes are the ones that were interrupting someone. The time saved is real but modest; the larger effect is that a class of small annoying failures stops accumulating.
Against that, reviewing its proposals costs time, and a rejected proposal costs time with nothing to show. We keep the ratio visible for exactly that reason, and we would turn it off if the accepted share fell far enough, which we have come close to once after a model change.
Where We Do Not Run It
Not on customer repositories, without exception. That is partly contractual and partly that the failure we most want to avoid is an agent with write access to something we do not own, acting on a misunderstanding of a codebase we did not write.
Not on anything touching authentication, payment or data deletion, even in our own code. The rule there is simple: if a wrong change would be expensive to discover, the work stays with a person who can be asked what they were thinking.
What We Do Not Claim
We do not claim this generalises to feature work. Our agent operates where there is a failing test defining success, and that signal is what makes the task tractable. Remove it and the same setup produces confident changes with nothing to check them against.
We also do not claim the permission model is complete. It is a list of things that went wrong plus a few we anticipated, and the next entry will most likely be added the same way the CI restriction was.