Golden Test Sets: Ground Truth for LLM Products
How to build and maintain golden test sets for LLM products in 2024: sampling real production queries, labeling with domain experts despite criteria drift, covering known failure modes, and calibrating automated judges against expert labels. And why dataset maintenance — not tooling — is the real evaluation bottleneck. With concrete numbers and dated sources.
Why Vibe Checks Stop Working
Most teams shipping LLM features in 2024 evaluate them the same way: an engineer changes a prompt, reads five outputs, and declares the change an improvement. This works for a demo. It fails as soon as the product covers more than one task, because a fix in one place silently breaks another. Hamel Husain's widely read essay from 29 March 2024 names the pattern: unsuccessful LLM products almost always share one root cause — the lack of a robust evaluation system.
Public benchmarks do not close the gap. MMLU measures accuracy on 57 academic tasks; it says nothing about whether your assistant extracts the right invoice fields. What a product needs is a golden test set: ground truth for its own traffic.
What a Golden Test Set Is
A golden test set is a fixed, versioned collection of real inputs paired with expert-verified reference outputs or explicit grading criteria. It is product-specific ground truth. Every change — prompt, retrieval logic, model version — is scored against the same cases, so a score delta is attributable to the change and nothing else.
The definition includes what it is not. A golden set is not a benchmark for comparing foundation models, not a load test, and not a substitute for A/B tests with real users. It answers exactly one question: did this change make the system better or worse on the cases we have declared to matter.
Sampling Real Queries
Golden cases must come from production traffic, not from what engineers imagine users will type. Real queries are shorter, more ambiguous, and worse-spelled than invented ones. Sample from logs, stratify by feature, intent, language, and length, and deduplicate near-duplicates via embedding similarity — ten phrasings of the same request must not occupy ten slots.
Before launch there are no logs; synthetic cases derived from the specification fill the gap and are replaced as real traffic arrives. Start small. In our projects, 50 to 200 labeled cases per feature reliably catch regressions. A set the team cannot fully re-read in one day is too large to start with.
Labeling with Domain Experts
Labels define the ground truth, so who labels matters more than how many labels exist. Whether an answer in a tax, medical, or logistics product is correct can only be decided by domain experts. Even experts disagree: in the MT-Bench study (June 2023), human experts agreed with each other on only 81% of non-tie judgments. Measure inter-annotator agreement before trusting labels; a written labeling guideline and an adjudication step for conflicts are mandatory.
Expect the criteria to move. Shankar et al. (April 2024) call this criteria drift: graders need criteria to judge outputs, but judging outputs changes the criteria. Treat the labeling guideline as a living document with a changelog, and re-adjudicate old labels whenever the guideline changes.
Covering Failure Modes
A golden set made of happy paths certifies nothing. Build an explicit failure taxonomy from observed incidents: retrieval misses, hallucinated fields, format violations, unjustified refusals, out-of-scope requests, prompt-injection attempts. Reserve a fixed share of the set — we use roughly a third — for these categories, and add every confirmed production failure as a labeled case, exactly like regression tests in classical software.
Scope honestly: the set covers known failure modes. Unknown ones are absent by definition, and Anthropic's write-up from October 2023 shows how hard robust evaluation remains even with dedicated teams. A green golden set lowers risk; it does not prove safety.
Calibrating Automated Judges
Expert time is the scarcest resource, so automated grading is unavoidable. Programmatic assertions — regex, schema checks, exact match — cover deterministic criteria. For open-ended answers, LLM-as-a-judge works, under conditions. Zheng et al. measured 85% agreement between GPT-4 and human experts, slightly above the 81% human-human rate, but also documented position, verbosity, and self-enhancement biases.
The golden set is the calibration instrument: a judge is validated against expert labels, its agreement rate is measured and reported, and only then does it grade unlabeled traffic. A judge without a measured agreement rate is just another unevaluated model in the pipeline.
Maintenance Is the Real Bottleneck
Building the first set takes weeks. Keeping it truthful takes years — and in our experience this is where evaluation efforts stall, not at judge quality or tooling. Products change, models get swapped, traffic drifts, and test cases leak into prompts or fine-tuning data, silently inflating scores. Version the dataset like code: reviewed changes, a changelog, immutable releases.
Budget for this explicitly. A golden set without an owner degrades into a benchmark of last year's product.
| Trigger | Maintenance action |
|---|---|
| Confirmed production failure | Add as a labeled case within the sprint |
| Guideline change after adjudication | Re-label affected cases and bump the version |
| Model or prompt-architecture swap | Re-run judge calibration against expert labels |
| Quarterly | Resample traffic and retire stale cases |
| Fine-tuning on internal data | Audit training data for golden-case leakage |
Outlook from May 2024
We expect three developments. First, eval maintenance becomes a product category of its own; assistants in the style of EvalGen, which co-evolve criteria with the graders, point the way. Second, synthetic data generation will scale case creation but will not remove the expert from labeling — someone still has to decide what correct means. Third, regulation forces the issue: the EU AI Act, given final approval by the Council on 21 May 2024, requires documented testing for high-risk systems. Golden test sets are the artifact that requirement will ask for.
Our bet: within two years, the question in reviews will no longer be which model you use, but: show me your golden set and its changelog. Teams that start sampling and labeling now will own that conversation.
Sources
- Hamel Husain — Your AI Product Needs Evals (29 March 2024)
- Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv, 9 June 2023)
- Ganguli et al. / Anthropic — Challenges in Evaluating AI Systems (4 October 2023)
- Shankar et al. — Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences (arXiv, 18 April 2024)
- Council of the EU — Artificial Intelligence Act: Council Gives Final Green Light (21 May 2024)