We Put an Assistant on Our Own Incidents
At three in the morning the hard part of an incident is not the fix, it is orientation: what changed, what is affected, what has happened before. That is a retrieval problem, and it is where we let an assistant help.
What Is Actually Hard at Three in the Morning
Not the repair. By the time someone knows which component is failing and why, the fix is usually a documented procedure. The expensive part is the twenty minutes before that, spent reading dashboards, correlating a deployment time against an error spike, and remembering whether this has happened before.
All of that is retrieval over things we already store: change records, alerts, logs, previous incident write-ups. It is the kind of work an assistant is genuinely good at, and it is not the kind of work anyone had thought to automate because it feels like thinking rather than looking.
What It Does
When an alert fires, it assembles a briefing: what deployed in the last six hours, which alerts are firing together, which services depend on the affected one, and any previous incident whose description is similar. Each item carries a link to its source.
It takes about twenty seconds and it is on the screen before a person has finished logging in. That timing is most of its value: it is not smarter than the engineer, it is earlier.
The Research on the Same Shape
Xiang and colleagues published work in 2025 on root cause analysis in Kubernetes combining a state graph of the cluster with a language model, which is the same division of labour we settled on: something deterministic that knows the topology, and a model that reads and summarises across it.
The part we took most directly is that the structure comes from the system rather than from the model. Our dependency information is read from configuration, not inferred, so an assertion about which service depends on which is a fact rather than a guess.
| Assistant does | Assistant does not |
|---|---|
| Assemble what changed and what is affected | Say what caused it |
| Surface similar past incidents | Say this is the same incident |
| Propose hypotheses with evidence | Recommend an action |
| Read logs, metrics, change records | Touch anything. Read-only, always |
The Rule That Keeps It Honest
It may not say what to do. It proposes hypotheses, each with the evidence that supports it, and stops. That is a deliberate limit and it survived an argument, because a system that says restart the service is right often enough to be trusted and wrong at exactly the moments when trusting it is expensive.
The hypotheses are also ranked by evidence rather than by confidence. A hypothesis supported by a deployment at the right time and a matching error signature is presented above one supported by textual similarity to an old incident, and the ordering is computed rather than expressed by the model.
Where It Is Useless
The last ten minutes. Once the cause is known, the work is a judgement about a fix under time pressure with knowledge of what else is running and what the customer can tolerate, and none of that is in any system it can read.
It is also useless on genuinely novel failures, which is a limitation worth stating because those are the incidents that matter most. Its similar-incident retrieval is best exactly when the problem has happened before, which is when a person would probably have remembered anyway.
What We Measured
Time from alert to a correct hypothesis being written in the incident channel, before and after, across about forty incidents. The median fell from roughly eleven minutes to four. Time to resolution fell by less, which is the honest shape of the result: it accelerates the beginning, not the end.
We also counted how often its top hypothesis was the eventual cause: a bit under half. That number is low enough that presenting it as an answer would be wrong and high enough that reading it first is worth twenty seconds.
Why It Is Read-Only
Because an incident is the worst possible moment for an automated action, and because we would not have got it past our own review otherwise. The credentials it holds can read metrics, logs and configuration, and can do nothing else.
That constraint also made it easy to deploy. A read-only tool with no write path is a component our own security review approved in an afternoon, and the same tool with the ability to restart a pod would have been a month of discussion for a benefit nobody could quantify.
What We Do Not Claim
We do not claim it diagnoses anything. It retrieves and arranges, and the diagnosis is done by a person reading what it retrieved. The distinction sounds pedantic and it is exactly what stops the tool from being trusted more than it deserves.
We also do not claim the seven-minute median saving generalises. It is our infrastructure, our alerting and our incident history, and a team with better dashboards would find less to gain because their orientation problem is already smaller.