Technical analyses
Since 2023, we have analyzed what works reliably in AI systems — using numbers, sources, and clearly stated technical limits.
Sixty tool calls produce a history longer than any window, so something has to go. Summarising the oldest part discards the decisions and the failed attempts, which is exactly what stops an agent repeating itself. Kang et al. optimise what to compress for long-horizon agents. Our rule: observations compress, decisions and errors stay verbatim, and the reason is measurable.
A customer asked why our estimate did not reflect a published productivity gain. The study measured new code with no existing constraints; our work is a fifteen-year-old system with undocumented conventions and downstream dependents. Sharma names brownfield as its own category. What actually accelerates in that setting, what does not, and the number we quote instead.
We added sources, confidence, the query and the reasoning to a review screen, and review accuracy went down. Margondai and Mouloua develop a theory of when explanations impair rather than improve decisions through cognitive load. What we removed, the one thing worth showing by default, and why our own principle needed a limit rather than an exception.
Procurement documents ask for it by name with a target percentage attached. Sinha derives limits on what embedding similarity between an answer and its sources can certify, which matches what we measured: a similarity check catches the copy that drifted and passes the conclusion that does not follow. What we build instead, and the three checks that each catch a different, nameable failure.
An overnight ingestion run overlapped with the morning and the assistant took nine seconds instead of two. Park studies how infrastructure faults make autoscalers misallocate resources. Ours was not faulty, it was configured for average load by a team that did not know two workloads shared a pool. What we separated, what we capped, and the alert that would have caught it a month earlier.
We were asked why one assistant had degraded and found eleven across the organisation, with a named owner for two. Acharya proposes a maturity model for managing agent sprawl. The four questions we now ask about every deployed system, the handover document we insist on, and why an assistant nobody owns is worse than no assistant.
One corpus, three departments, and filtering by permission after retrieval. A colleague showed in a morning that the filter could be inferred around: what comes back when a restricted document is the best match tells you something even when the document does not. Tang et al. work on privacy guarantees for retrieval. What we changed, and why per-permission indexes cost more and are the only version we would defend.
We have called chunking the biggest lever in a dozen posts and never written about it. What we actually do: structure first, one idea per chunk, headings carried into the text, tables never split. Koutsiaris segments by predicted information need, which is the direction we arrived at by accident. Two things we got wrong, and the setting nobody should copy from anyone.
The assistant resolved forty percent of requests and misrouted six, and support throughput did not improve. Dada et al. model the gatekeeper who either resolves or refers. A misroute costs a full handling plus the original, so the break-even is not where intuition puts it. What we changed in routing rather than in the model, and the metric that finally made it visible.
A user realised ninety seconds in that the filter was wrong and said so. The agent finished the original job correctly and reported success, because nothing in the design allowed a running task to be interrupted. Zou et al. evaluate interruptible agents on long-horizon tasks. Checkpoints between steps, what an interruption costs, and why undo is a different problem we did not solve.
Told it was wrong, our assistant apologised and produced a different answer. We checked fifty such exchanges: the first answer had been correct in about four out of five. Zhang et al. study whether agents repair when challenged or merely reply. What we changed so a challenge triggers a re-check against the source rather than a new answer, and why the polite version was the dangerous one.
The number in the answer matched the document, and the document had been wrong since 2019. Every check we built asks whether the answer matches the source; none asks whether the source is right. Patil works on detecting manipulated numerical claims in retrieval systems. What we can check across documents, why numbers deserve different treatment from prose, and the part that stays a data quality problem.