Alignment Is Not a Security Boundary
A customer wanted the model's refusal to be the safeguard: the agent could call any internal API, and the system prompt would keep it in line. We said no. Zou et al. published the reason a few days later.
The Request We Turned Down
The design on the table was simple and, on the surface, elegant. One agent, access to every internal endpoint, and a system prompt describing which operations were allowed for which role. No separate authorisation layer, because the model would decline anything outside its brief.
We refused to build it. Not because we had a specific exploit in hand, but because the design put a probabilistic component where a deterministic one belongs. Refusal is a behaviour. An access rule is a guarantee. They are not interchangeable.
Why the Argument Got Easier to Make
Shortly after, Zou and colleagues published a method that finds adversarial suffixes automatically, using greedy and gradient-based search rather than human ingenuity. Attach the suffix to a request and an aligned model becomes markedly more likely to comply instead of refusing.
The part that mattered for our argument is transferability. The suffixes were trained against open models and still induced the behaviour through the public interfaces of several commercial systems. That turns jailbreaking from a craft into a procedure, and procedures scale.
What We Build Instead
Every consequential action an agent can take runs through an authorisation check the model cannot influence. The agent proposes; a separate component decides. It receives the caller's identity, the operation and the target, and answers yes or no without ever seeing the prompt.
That component is boring on purpose. It is the same check the application would need if a human were driving it through a form. The agent does not get a wider permission set because it is an agent.
Where the Model Still Helps
None of this makes alignment worthless. A model that declines obviously harmful requests removes a great deal of noise, and good refusal behaviour makes a product pleasant rather than hostile to use.
The distinction we hold is between a filter and a boundary. A filter reduces how often something bad reaches the next stage. A boundary determines what is possible at all. Alignment is a good filter. It is not a boundary, and designing as though it were is where the risk enters.
How This Shows Up in a Design Review
We ask one question of every agent design: if the model were fully cooperative with an attacker, what could it reach? If the answer includes anything irreversible, the design is not finished. Deleting records, moving money, sending external mail and changing permissions all sit behind a deterministic rule or a human approval.
This is also why we keep toolboxes small. A tool the agent does not have is the cheapest control available, and it cannot be talked around.
What We Do Not Claim
We do not claim this makes a system safe. Prompt injection through retrieved content remains unsolved, and an authorisation layer only limits blast radius; it does not stop an agent being manipulated into a permitted but unwanted action.
We also do not treat the specific attack as the point. Particular suffixes get patched. What does not get patched is the underlying property: model behaviour is steerable by input, and inputs are not always ours.
| Design assumption | How we treat it |
|---|---|
| The model will refuse misuse | A filter, not a control. Never load-bearing |
| The system prompt is confidential | Assume it is readable |
| Only our text reaches the model | Assume retrieved content is attacker-controlled |
| Alignment improves every release | Likely, but not something to design around |
What It Costs
An external authorisation layer is a few hundred lines and one call per tool invocation. Against the effort of the agent itself, that is negligible. The expensive part is organisational: someone has to write down which role may do what, work that many systems have quietly avoided for years.
In our experience that write-up is worth doing on its own merits. The agent only makes the omission visible.