When We Fine-Tune, and When We Refuse To
Fine-tuning became affordable enough that customers started asking for it by name. That made the conversation harder, not easier: the question is no longer what it costs but what it is for.
The Request
The customer wanted their product catalogue inside the model. Prices, availability, compatibility rules — trained in, so the assistant would simply know them without a lookup. The reasoning was that retrieval added latency and a moving part, and tuning had become cheap.
The catalogue changed weekly. That single fact decided the design, and it is the fact the request had not accounted for.
Why Cheap Tuning Changed the Question
Dettmers and colleagues had shown that fine-tuning a large model could be done on far more modest hardware than before, by tuning adapters over a quantised base. QLoRA is a genuine engineering result and it moved fine-tuning from a capital decision to an ordinary one.
The consequence was not that everyone should tune. It was that cost stopped being the filter. When something is expensive, the expense does the deciding for you. When it is cheap, you have to decide on merit — and many teams had never had to articulate what tuning was actually for.
The Rule We Use
We separate knowledge from behaviour. Facts that change — prices, stock, policies, people — belong in retrieval, because a training run is a snapshot and a snapshot of a moving target is wrong by the time it ships. Behaviour that is stable — tone, output format, domain vocabulary, a house style of reasoning — is what tuning is good at.
Stated as a question: if this changes next month, does the model have to be retrained? If yes, it is knowledge and it does not belong in weights.
| Belongs in retrieval | Belongs in tuning |
|---|---|
| Prices, stock, availability | Output format and structure |
| Customer and contract data | Tone and register |
| Policies that get revised | Domain vocabulary and abbreviations |
| Anything with an audit trail | A consistent way of reasoning through a task |
The Two Cases Where We Do Tune
The first is format adherence at scale. When every output must match a strict structure and prompt-based enforcement keeps costing tokens and still drifting, tuning fixes the shape reliably and cheaply per call. That is a behaviour, and it does not go stale.
The second is domain language. Where a field has vocabulary the base model reads wrong — abbreviations that collide with common words, terms that mean something specific in that industry — tuning teaches the reading. Retrieval cannot, because the problem is comprehension rather than lookup.
What We Built Instead
For the catalogue, retrieval with a strict schema on the extracted fields, and a nightly index rebuild. The assistant does not know the price; it looks it up and cites the record it used. When a price changes, nothing is retrained and nothing is stale.
The customer's original objection was latency. Measured, the lookup cost less than the round of clarification that a confidently wrong price produced.
What We Do Not Claim
We do not claim tuning and retrieval are alternatives. Most of the systems we run use both, for different jobs, and the interesting engineering is the boundary rather than the choice.
We also do not claim our rule is complete. There is a genuine middle ground — knowledge that is stable over years, like a regulation that rarely changes — where either approach works and the decision comes down to operational preference rather than principle.