Training the Retriever Instead of the Writer
When a customer asks about fine-tuning, they mean the model that writes the answer. On the two projects where training something on customer data actually paid, it was the retriever, and the difference in cost between those two options is large.
Which Half Is Failing
Before considering training anything, we split the failures. Take the cases the system got wrong and ask, for each, whether the correct passage was in the retrieved set. If it was, the generation step failed. If it was not, retrieval failed.
On the project that prompted this, seventy-one percent of failures were retrieval. The model wrote perfectly reasonable answers from material that did not contain the answer, which is a retrieval problem wearing a generation problem's clothes and had been discussed for weeks as though it were the latter.
Why the Retriever Is the Cheaper Thing to Train
It is a much smaller model, the training data is pairs of question and correct passage rather than question and ideal answer, and the correct passage is something a domain expert can mark in seconds while an ideal answer takes minutes to write and is contestable.
Béchard and colleagues published work in early 2025 on fine-tuning a retriever for domain-specific use with multiple tasks in one model, which matches the shape of the problem we had: several kinds of query against one corpus, where a general-purpose embedding model treats domain terms as ordinary words.
What Our Training Data Looked Like
About twelve hundred pairs, collected from real queries rather than invented. Where the system had failed, an expert marked which passage should have been returned; where it succeeded, the returned passage was confirmed. Two afternoons a week for a month, done alongside normal support work.
The collection method matters more than the count. Pairs drawn from real failures teach the retriever the vocabulary gap that is actually costing money, and a set assembled by writing plausible questions would have taught it the vocabulary we already handle.
| Option | Cost and effect |
|---|---|
| Fine-tune the answering model | Expensive, and our failures were elsewhere |
| Fine-tune the retriever | Cheap, and aimed at seventy-one percent |
| Better chunking | Free, and we did it first |
| Multilingual embeddings | Configuration, and it helped separately |
The Order We Do Things In
Chunking first, because it is free and it was the largest lever on every project we have measured. Then a general multilingual model if the queries and documents differ in language. Then, and only where the failure split justifies it, training the retriever.
Training came last on both projects that ended up doing it, and on four other projects the earlier steps closed the gap and training was never needed. That ordering is the practical content of this piece more than the training itself.
What It Gained
Retrieval failures fell by a bit over half on the labelled set, and end-to-end accuracy rose by nine points, which is a larger improvement than any model change we have made on that system. Latency was unchanged, because the trained model is the same size as the one it replaced.
It also generalised worse than the general model on queries unlike anything in the training set, which is expected and worth stating. A retriever trained on one customer's vocabulary is a component that belongs to that customer's deployment, not a general asset.
The Maintenance Nobody Mentions
A trained retriever is a second artefact that can drift. When the corpus changes substantially, the training set describes a corpus that no longer exists, and the retriever quietly gets worse in a way that looks like ordinary quality decay.
We re-check it quarterly against the labelled set and retrain about once a year, and we said so in the proposal. A customer who accepts a nine-point gain should know it comes with an artefact that needs feeding, because the alternative is discovering that obligation eighteen months later.
When We Still Say No
Below a few thousand documents, or where the retrieval failure share is under about a third. In both cases the effort goes further elsewhere, and we have talked two customers out of retriever training on those grounds after doing the failure split for them.
We also say no when nobody on the customer side can spend the afternoons marking passages. The data cannot be outsourced to us, because the whole value is that a domain expert knows which passage is the right one, and that is precisely the knowledge we do not have.
What We Do Not Claim
We do not claim fine-tuning the answering model is never right. Where the failure is style, format or a house vocabulary that no prompt reliably enforces, it is the correct tool, and one project of ours does exactly that.
We also do not claim our nine points transfer. It is one corpus with a heavy domain vocabulary, which is the best case for this technique, and a corpus of ordinary prose would show considerably less.