Technical analyses
Since 2023, we have analyzed what works reliably in AI systems — using numbers, sources, and clearly stated technical limits.
Ask in German, get a number from the ERP: the demonstration is irresistible and the production version looks nothing like it. We expose curated views rather than tables, because a correct query over the wrong table is the failure that has no error message. What we forbid outright, why every answer shows the query, and the number that was right and still wrong.
A customer wanted an assistant to modernise a fifteen-year-old codebase, and we spent the first six weeks writing tests. Translating code is the part models are good at; knowing whether the translation preserved behaviour is the part that decides whether the project ships. What we test first, why we migrate leaves before trunks, and the change we refused to let an assistant make.
Models are better at knowing what they do not know when the subject is widely documented. Ni et al. connect that self-assessment to how popular the knowledge is, which has an uncomfortable consequence for specialists: the assistant is least able to flag its own uncertainty precisely where a customer's expertise lives. What follows for architecture, and the demonstration trap it explains.
A planner, a researcher, a writer, a critic and a supervisor produced plausible plans and wrong results that nobody could attribute to a component. Cemri et al. build a taxonomy of why multi-agent systems fail, and most of the categories are coordination rather than capability. What we replaced with ordinary code, the two places a second agent still earns its place, and why the diagram is so persuasive.
A technician asked what happens if two cleaning agents are mixed and the safety filter refused, which is the one question the safety data sheet exists to answer. Kumar et al. show guardrails trade false positives against false negatives with no free option. What we measure on both sides, why a refusal is a defect with a ticket, and where we accept over-blocking deliberately.
After eighteen months of running our adversarial prompt set in English, we translated it. Roughly twice as many got through, on a system where most users write German. Kumar et al. build multilingual moderation across seventeen languages precisely because coverage is uneven. What we changed, why translation is not enough to build the test set, and the general lesson about which language a safety property was measured in.
Extraction accuracy fell four points over a weekend with no deployment and the same model identifier in every request. A version string is a name the provider controls, not a guarantee about behaviour. What we run daily to catch this, the contractual question we now ask before signing, and what we do when the answer is unsatisfying, which it usually is.
Our evaluation set asked one question and checked one answer, and most real failures happened on the third turn. Kranti et al. benchmark task-oriented dialogue systems as whole conversations. What a multi-turn case has to specify, why a simulated user is a test fixture rather than a judge, and the two failure modes that only appear once a conversation has a history.
A self-improving assistant that adjusts its own instructions from corrections is the most requested feature we decline. Han et al. describe how self-evolution can push agents away from their intended behaviour over time. Our objection is narrower and harder to argue with: a system that rewrites itself has no version anyone can name, and every process around it assumes one exists.
We spent two years telling customers a knowledge graph was overkill, and then a corpus arrived where vector retrieval failed on nearly a third of questions for structural reasons. Ahmad et al. benchmark vector, graph and hybrid pipelines side by side. What made this corpus different, what the graph cost to build and keep current, and why the answer was hybrid rather than a replacement.
At three in the morning the hard part is orientation, not the fix: what changed, what is affected, what happened before. Xiang et al. combine cluster state with a model for root cause analysis. Ours is read-only, proposes hypotheses with the evidence attached and never a conclusion, and the rule that keeps it honest is that it may not say what to do.
The technicians who most need the documentation work where there is no signal. What fits on a handheld device is a small model and a subset of the corpus, which forced a decision nobody wanted to make: which questions this thing is allowed to be asked. The scoping that made it useful, and why the honest version says out loud what it cannot see.