The Metric That Agreed With Us Too Often
For six weeks our faithfulness score climbed and our support tickets climbed with it. The metric was not broken. It was answering a narrower question than we thought we had asked.
Two Lines Going the Wrong Way Together
We had automated evaluation on a retrieval assistant and were tuning against it weekly. Faithfulness rose, answer relevance rose, and the team was pleased. In the same period, tickets asking a variant of “this is not what I asked” also rose.
Both measurements were correct. They were about different things, and we had let one stand in for the other because it was the one that produced a number every night.
What the Metric Actually Asks
Es and colleagues set out a reference-free framework for evaluating retrieval-augmented pipelines: judging the generated answer against the retrieved context without needing a human-written ground truth for every case. That is a real advance, because ground truth is the expensive part and its absence is why most teams evaluate nothing.
Faithfulness asks whether the answer is supported by the retrieved passages. It is a question about internal consistency. It cannot ask whether those passages were the right ones to retrieve, because the passages are its yardstick.
The Failure It Cannot See
Our regression was retrieval drift. An index change had quietly started favouring a newer document set, and answers were being grounded, faithfully and relevantly, in the wrong records. Every automated signal said the system was improving, because every answer matched its context.
This is not a flaw in the method. A reference-free metric is by construction blind to the case where the context is wrong, and the papers proposing these metrics do not claim otherwise. The mistake was ours: we treated a consistency check as a correctness check.
| What the automated score answers | What it cannot answer |
|---|---|
| Is the answer grounded in the context | Was this the right context |
| Does the answer address the question | Is the underlying record current |
| Every night, on every case | Whether the corpus changed underneath |
| Without human labelling | What the user actually wanted |
What We Put Underneath It
A small labelled set, maintained by hand: around eighty real questions with the record that should answer them, chosen to cover the awkward parts of the domain rather than the average case. It is checked against every release.
Eighty is small enough that a domain expert can revisit it in an afternoon and large enough that a retrieval regression shows up as several failures rather than one. The automated metrics still run nightly on everything. The labelled set decides whether we ship.
How We Read the Two Together
The automated score is a trend line and a smoke alarm: a sudden drop means something broke, and that is worth waking up for. It is not evidence that a change was an improvement.
The labelled set is the gate. When the two disagree — scores up, labelled set down — we trust the labelled set and go looking for what moved in retrieval. That has happened three times, and each time the labelled set was right.
What We Do Not Conclude
We do not conclude that automated RAG evaluation is not worth running. It is cheap, it covers cases a labelled set never will, and it caught two genuine prompt regressions for us before any human noticed.
We also do not claim eighty labelled cases are enough to be confident. They are enough to stop us shipping the obvious mistakes, which is a lower bar and an honest one.