The Small Model That Stayed in Production
The plan was to swap it out once we had budget. Two years later it is still there, doing the same job, and the reason is not that it got better — it is that we finally scoped the job properly.
Why It Went In
The step was routing: decide which of six categories an incoming request belongs to, so the right downstream handler picks it up. We were prototyping and did not want to spend on a frontier model for something that felt like a classifier.
The intention was explicitly temporary. It is still running because every time we revisited the decision, the honest answer was that the step did not need more.
What Changed in the Field
Late 2023 is when small open models stopped being a compromise you had to defend. Jiang and colleagues released Mistral 7B into that moment, and its significance for us was less the specific model than the message: this size class had become genuinely usable for bounded work.
That reframed the question from “how small can we get away with” to “how well have we scoped the step”, which turns out to be the more productive version.
The Scoping That Made It Work
A routing step fails in two ways: the category set is ambiguous, or the input carries information the classifier cannot see. We fixed both before touching the model. Two categories that overlapped got merged. One that depended on account history got a lookup instead of a guess.
After that the step was genuinely easy, and an easy step does not need a large model. Most of the accuracy we gained came from the category redesign rather than from anything model-shaped.
| Where a small model holds up | Where we take a large one |
|---|---|
| Bounded classification with clear categories | Open-ended reasoning over a long chain |
| Extraction against a fixed schema | Judgement calls with unclear criteria |
| High volume, latency sensitive | Low volume, high stakes |
| Failure is cheap and visible | Failure is expensive or silent |
How We Decide Now
We do not choose a model and then design around it. We scope the step, build the labelled set, and then run the cheapest model that clears the bar. If nothing small clears it, that is information: usually the step is doing two jobs and should be split.
The measurement that settles arguments is accuracy on the labelled set at production latency. Discussion about which model is better in general does not survive contact with a number on our own data.
What It Costs to Keep
Running a small model has its own overhead: it is another thing to host, patch and monitor, and it does not benefit from a provider's improvements arriving for free. For a step this size that trade is fine. It would not be for six of them.
We also keep the swap path open. The step sits behind an interface, and moving it to a hosted model is a configuration change rather than a rewrite. We have not needed it.
What We Do Not Claim
We do not claim small models generally substitute for large ones. Where the work is open-ended, they do not, and pretending otherwise produces systems that fail quietly on the cases that matter most.
We also do not present this as a cost story. The saving is real but modest at our volume. The reason the small model stayed is that the step was right-sized, and that would have been worth doing regardless of which model ran it.