Synthetic Data: Where We Use It and Where We Will Not
We once generated four hundred test cases in an afternoon and felt productive. They agreed with the system on nearly everything, which we mistook for quality until a real regression walked straight through them.
The Afternoon That Felt Productive
We needed a larger evaluation set and had a model available, so we generated four hundred question and answer pairs from the corpus. The coverage looked excellent, the distribution looked plausible, and the suite went green almost everywhere.
Some weeks later a retrieval change introduced a genuine regression, and the generated set did not notice. The reason is uncomfortable in hindsight: the cases had been produced by a model reading the same documents with the same assumptions, so they encoded the system's understanding rather than the users' expectations.
Why the Agreement Was the Warning
A test set that agrees with the system almost everywhere is either measuring a very good system or measuring the wrong thing. The prior should be the second, and we had treated a high pass rate as reassurance rather than as a signal to look harder.
Human-written cases are noisier and disagree more, and that disagreement is where their value sits. A domain expert asks a question in the way a customer would ask it, including the ambiguity and the missing context that a generated question tidies away.
The Line We Draw
We generate inputs. We do not generate expected answers. A model is useful for producing many plausible phrasings of a question, for varying tone and length, and for surfacing formulations nobody on the team would have written. That widens coverage without deciding what correct means.
The expected answer stays human, or it stays absent and the case is checked by a person. The moment the same family of model both produces the answer and judges it, the test measures internal consistency and stops measuring correctness, which is the failure we had shipped.
| We generate | We do not generate |
|---|---|
| Question phrasings and paraphrases | The expected answer |
| Adversarial and edge-case inputs | The pass or fail judgement |
| Volume for load and latency tests | Ground truth for correctness |
| Data for a schema, with fixed fields | Anything that becomes a gate |
Where It Genuinely Earns Its Place
Adversarial inputs are the clearest case. Asking a model to produce a hundred ways of phrasing a request that should be refused finds formulations we would not have thought of, and the expected outcome is fixed in advance, so the model is not being asked to decide anything.
Load and latency testing is the other. There the content barely matters, and generated volume is simply cheaper than collecting it. Neither of these has the property that made our evaluation set useless, because in both the correct outcome is known before the data exists.
What the Research Suggests Is Missing
Singh and colleagues examined self-training on model-generated data and made the dependency explicit: what carries the method is a reliable signal for whether an output is actually correct, rather than the generation itself.
That reframes the question usefully. Self-training works where correctness is checkable, as it is when code compiles and passes tests or a computation can be verified. Our evaluation set had no such check, which is precisely why generating more of it produced confidence rather than information.
What We Do Now
The labelled set stayed human and small, around eighty cases, and it is the gate. Generated inputs sit alongside it as a wider sweep that can flag something worth looking at but never blocks a release on its own.
We also record where every case came from. A case with a generated input and a human expectation is a different kind of evidence from one where both are human, and mixing them without labelling makes it impossible to reason about what a green suite means.
The Cost of Being Strict
The strictness is expensive. A human-written case with a checked expectation takes a domain expert perhaps ten minutes, which is why our labelled set is eighty cases rather than eight hundred. We accept a smaller set that means something over a larger one that does not.
The compensation is that a small set gets maintained. Nobody revisits four hundred generated cases when the domain shifts; eighty are reviewed in an afternoon, and that review is where most of their value accumulates over time.
What We Do Not Claim
We do not claim synthetic data is a poor tool. Where a verifier exists it is clearly powerful, and the research direction is real. Our rule is about the absence of a verifier, not about the technique.
We also do not claim human-written cases are correct by virtue of being human. They carry their own bias, particularly towards the cases the writer already has in mind. They are simply biased differently from the system under test, and that independence is the whole point.