The Answer Was Correct and Nobody Understood It
The answers were accurate, sourced and largely unread. Escalations from one user group stayed flat while accuracy improved for a year, which is the pattern you get when the failure is not in the content.
Two Populations, One Voice
The assistant serves technicians who maintain the equipment and office staff who handle orders and complaints about it. Both ask about the same documents, and both received answers written in the register of those documents, which is technical.
For the technicians that is correct and welcome. For the office staff it produced answers that were accurate and not actionable: a sentence containing three part designations and a conditional clause is a good answer to a question the reader did not have the vocabulary to ask precisely.
Why the Metrics Missed It
Our accuracy measures whether the answer is supported by the source. It says nothing about whether the person reading it could act on it, and those are different properties that happen to be measured by the same word in most project conversations.
The signal was in the escalation rate by user group, which we had not been segmenting. Once we did, technicians escalated on about four percent of conversations and office staff on nineteen, against the same corpus and the same accuracy.
The Research on Whether Simplification Helps
Guidroz and colleagues published work in May 2025 measuring the effect of language-model-based text simplification on user comprehension and cognitive load, which is the question that matters and is not the same as whether the simplified text scores as simpler.
That distinction is why we cite it. Readability scores are easy to move and easy to move without helping anyone, and a study that measures comprehension rather than a readability index is the kind of evidence that justifies changing a production system.
| We simplify | We do not simplify |
|---|---|
| Sentence structure and connectives | Part numbers and designations |
| Ordering, with the action first | Numeric values and units |
| Vocabulary where a common word exists | Conditions and qualifiers |
| Explaining what a term means | The quoted source passage |
What Went Wrong First
Our first attempt was an instruction to answer in plain language. It worked, in the sense that the text got simpler, and it lost conditions. The sentence unless the machine is fitted with the older controller lost its unless clause often enough that we found it within a week, and the resulting answers were wrong rather than simple.
That is the specific danger with this task. Simplification removes subordinate structure, and the subordinate structure is where the conditions live, so careless simplification systematically deletes the part that makes an answer true in the reader's particular case.
What We Do Now
Two answers from one retrieval. The technical answer is generated first, then a second step rewrites it under an explicit rule set: keep every condition, keep every number and designation exactly, shorten sentences, put the action first, and add a one-line explanation for any term that survived because it could not be replaced.
The quoted source passage is never rewritten. It stays as it is, below the answer, in the words of the manual, so a reader who wants the original has it and a rewritten sentence is never mistaken for a citation.
How We Checked It Worked
Not with a readability score. We took forty answers, in both versions, and asked eight people from the office group what they would do next. Correct action rose from about half to about four fifths, and the cases still failing were mostly questions where the correct action was to ask someone.
Escalations from that group fell from nineteen percent to eleven over the following quarter. That is the number we report, and it is worth more than any property of the text itself.
Who Chooses the Version
The interface, from the user's role, with a control to switch. That was contested internally: the alternative was to let the model judge from the question, which is elegant and gets it wrong for the technician who asks a simple question and the office user who quotes a part number from an email.
Role plus an override has been unremarkable in operation, which is the outcome we wanted. About one user in fourteen switches, and most of them switch once and stay.
What We Do Not Claim
We do not claim simplified answers are safe by default. Ours are safe because the rewriting step is constrained to preserve conditions and values and is checked against the same labelled set as the technical answer, and an unconstrained rewriting step is a way of producing confident, readable, wrong text.
We also do not claim our two-population split generalises. Some deployments have one audience and need one voice, and producing two answers where one would do is a cost with no return.