Host It or Call It: The Question Behind the Model Choice
Every few months a customer asks whether they should run the model themselves. The question is almost never about capability, and answering it as though it were is how organisations end up operating infrastructure they did not want.
The Question as It Arrives
It usually arrives phrased as a technical comparison: is the open model good enough. By January 2024 that framing had largely stopped being interesting, because for the bounded work most business systems do, several open models were plainly good enough and the gap that remained was not where these customers were operating.
What the question is actually about is control, cost at volume, and where data goes. Those are three different conversations with three different answers, and merging them into a capability question produces a decision nobody can defend six months later.
What Changed in the Field
Jiang and colleagues published Mixtral of Experts that January, a sparse mixture-of-experts model whose released weights put strong capability in reach of anyone willing to host it. The architectural interest is real, but the effect on our conversations was commercial rather than technical.
Once a credible open model exists, self-hosting stops being a compromise you argue for and becomes an option you have to evaluate honestly. That is a better position to be in, and it is also more work, because the honest evaluation is not about benchmark scores.
The Four Things We Weigh
Data residency comes first, because it can end the discussion. If a customer cannot send the data outside their own infrastructure for regulatory reasons, hosting is not a preference, and the remaining questions are about how rather than whether.
Then volume, latency and operational capacity. Volume decides whether per-call pricing or fixed infrastructure is cheaper. Latency decides whether a network hop is acceptable. Operational capacity decides whether anyone will actually be available when a GPU node fails at the wrong hour, which is the question customers most often answer optimistically.
| Points toward hosting it yourself | Points toward calling a service |
|---|---|
| Data cannot leave the environment | No residency constraint |
| High steady volume | Spiky or low volume |
| Existing platform team on call | No one to page at night |
| A fixed model version is required | Provider improvements are welcome |
What Self-Hosting Actually Costs
The compute is the part everyone estimates and it is rarely the part that hurts. What hurts is that a hosted model becomes a component with an on-call rotation: version upgrades, capacity planning, a rollback path, and someone who understands why throughput collapsed after a driver update.
We ask one question that tends to settle it. If this node fails at two in the morning on a Sunday, who is woken up and what do they do? Where there is no good answer, we advise against, regardless of how attractive the per-token arithmetic looks on a slide.
Where We Recommend It Without Hesitating
The first case is regulatory: data that genuinely cannot leave, where the alternative is not building the system at all. Then the operational burden is simply the price of the capability, and the conversation moves to making that burden as small as possible.
The second is high steady volume on a narrow task. A classification step running millions of times a month against a stable prompt is exactly where fixed infrastructure beats per-call pricing, and exactly where a small model on modest hardware does the job, which keeps the operational surface small.
The Hybrid We Build Most Often
In practice most systems we deliver do both. The high-volume bounded steps run on something small and self-hosted, and the open-ended reasoning calls out to a hosted frontier model. The split follows the work rather than a policy.
That arrangement needs one piece of discipline to stay manageable: every model call goes through one interface with a named capability rather than a model identifier. Which model serves which capability is configuration, so moving a step between hosted and self-hosted is a deployment change and not a rewrite.
What We Got Wrong Early
We used to treat this as a cost optimisation and led with the arithmetic. That was a mistake, because per-token comparisons ignore the staffing cost and therefore always favour hosting it yourself, which is how you talk a customer into a platform team they did not budget for.
We now lead with data residency and operational capacity, and only reach for cost once those two have narrowed the field. The conversations are shorter and the decisions have held up better.
What We Do Not Claim
We do not claim open and hosted models are interchangeable. For the hardest reasoning work the frontier models remain ahead, and a system whose quality depends on that margin should not be moved for operational convenience.
We also do not claim our four criteria are exhaustive. Licensing terms matter for some customers, and so does the political question of provider dependency, which is a legitimate concern we are not well placed to weigh on their behalf.