Forty Thousand Tickets, Two Thousand Worth Using
The archive was the asset, or so everyone assumed. Forty thousand resolved tickets going back six years, and an obvious plan to put all of it to work. The version built on two thousand of them was better, and explaining why took longer than building it.
The Asset Everyone Assumed
Six years of resolved support tickets, each with a customer question and an agent answer. It is the kind of archive that makes a project look easy: the questions users ask and the answers that worked, already paired, already in the domain language.
The first version indexed all of it. It answered fluently, it answered confidently, and a review by two experienced agents found that roughly one answer in six was wrong in a way that would have caused a callback.
What Was Actually in the Archive
Advice that had been correct in 2019 and was now wrong. Answers referring to a product configuration that no longer exists. Agents guessing, politely and plausibly, in the way people do when a customer is waiting. Tickets closed with a note that the customer had called instead.
None of this is a criticism of the support team. It is what a support archive is: a record of conversations, not a maintained body of knowledge. We had treated a log as a knowledge base, which is the same category error as treating a chat transcript as a record of facts.
What We Filtered On
Four rules, all deliberately blunt. The ticket must be closed as resolved rather than as no reply. The answer must be at least two sentences, which removes the ones that say as discussed by phone. The product or component referenced must still exist. And the answer must not have been superseded by a later documented change.
That last rule did most of the work and needed the customer's change history to evaluate. It also cannot be done by recency alone: a four-year-old answer about a stable component is fine, and a six-month-old answer about a component that changed last month is not.
| Kept | Discarded |
|---|---|
| Resolved, with a substantive written answer | Closed with no reply or by phone |
| References a component that still exists | About a discontinued configuration |
| Not contradicted by a later change | Superseded advice, any age |
| Reviewed by an agent for the top clusters | Everything else, listed and inspectable |
The Research That Framed It
Li and colleagues published DataComp-LM that month, a benchmark where the model training procedure is held fixed and the dataset is the variable, so that filtering and curation strategies can be compared directly against each other.
The idea of holding the method constant and varying the data is what we borrowed. We had been comparing retrieval settings against a fixed corpus, when the corpus was the largest lever available, and simply inverting which side of that we treated as fixed changed what we spent the next fortnight on.
What Changed in the Numbers
Wrong answers on the review set fell from about one in six to about one in twenty-five. Coverage fell too: the filtered system had nothing to say on a set of questions the full archive would have answered, and about half of those had been answered correctly before.
We accepted that trade because the failure modes are not symmetrical. A question routed to a person costs a few minutes. A confident answer citing a policy that changed two years ago costs a callback, a correction and some trust.
Making the Discard Inspectable
The customer was, reasonably, uneasy about discarding ninety-five percent of their archive. So the filter is not a black box: every excluded ticket carries the rule that excluded it, and the counts per rule are on a page they can look at.
That turned an argument into a review. They looked at the largest exclusion group, disagreed with one rule, and we changed it, which added about four hundred tickets back. A filter someone can inspect and contest is a different object from a filter that just produces a smaller number.
What We Do With the Rest
It is not deleted, and this is worth being precise about. The full archive stays searchable for agents, who can read a superseded answer and understand the context around it. What changed is that it does not feed the automated assistant.
That distinction resolved most of the discomfort. Nobody was proposing to throw away six years of work; the proposal was that a log of past conversations is the wrong input to a system that speaks to customers as if it were authoritative.
What We Do Not Claim
We do not claim five percent is a useful ratio. It is what four rules produced on one archive, and a well-maintained knowledge base would keep almost all of itself. The number says something about that archive, not about support archives.
We also do not claim our filter is correct. It encodes a view of what makes an answer usable, that view was contested once and changed, and it will be wrong again in ways we will discover from the questions it fails to answer.