One Door for Every Model Call
A customer asked what their share of our model spend was. We had one invoice from one provider covering everything, no way to attribute it, and a week of work to fix a question that should have been a database query.
The Question We Could Not Answer
It was a fair question and a routine one: what does our system cost you to run each month? We could produce a total across all customers and we could not split it, because eleven places in our code called a model provider directly with the same key.
Reconstructing it from provider logs was possible and unpleasant, and the estimate we eventually gave carried a caveat about its accuracy. That is a bad position for a supplier who bills for the service and asks the customer to trust the numbers.
What We Built
One internal service that every model call goes through. It takes a named capability rather than a model identifier, a tenant, and a request. It returns the response and records the tenant, the capability, the model actually used, the token counts and the latency.
That is all it does. It is not a framework and it deliberately has no opinion about prompts, retries or orchestration, because a gateway that starts making decisions becomes a component every team has to understand rather than one they can ignore.
The Four Things It Paid For
Attribution first, which is what we built it for. Cost per tenant, per capability and per request is now a query, and the customer conversation that started this takes a minute rather than a day.
Then substitution: because callers ask for a capability, changing which model serves it is configuration. Then a single place to enforce timeouts and rate limits per tenant. And finally an audit trail, which turned out to be what our security reviews had wanted all along.
| Recorded per call | Why it matters |
|---|---|
| Tenant and capability | Attribution without guessing |
| Model and version actually used | So a silent provider change is visible |
| Token counts in and out | Cost, and prompt size drift |
| Latency, and whether it was cached | Performance, and cache hit rates |
The Research on the Same Idea
Martin and colleagues published work that October describing a brokering layer between applications and model providers, aimed at reducing the cost of access by routing requests and handling the mechanics centrally rather than in each application.
We had built ours for accounting rather than for routing, and reading it prompted us to add the routing we had been avoiding. The gateway now sends the bounded high-volume capabilities to a small model and the open-ended ones to a frontier model, which is a change we could make because there was one place to make it.
What It Refuses to Do
It does not retry. Retry policy depends on what the caller is doing, and a gateway that retries silently turns one visible failure into three invisible ones plus a cost the caller did not authorise.
It does not cache across tenants, for reasons we wrote about separately. And it does not modify prompts, which sounds obvious and is the feature most often requested, usually as a well-meant idea to inject a standard instruction everywhere.
The Number That Surprised Us
Once we could see spend by capability, one step accounted for thirty-eight percent of the bill. It was a summarisation step running on every document at ingestion, which nobody had costed because it runs in the background and nobody waits for it.
We moved it to a smaller model and the bill fell by about a quarter with no measurable quality change on the labelled set. That single finding covered the cost of building the gateway several times over, and we would never have found it from an invoice.
What We Would Tell Someone Starting
Build it before you need it, and keep it stupid. Ours took a week at eleven call sites; at forty it would have taken a month and would probably have been postponed again in favour of whatever was more urgent.
The discipline that makes it work is that callers name capabilities. If callers name models, the gateway is only an accounting layer, and every model decision remains scattered across the code exactly as it was before.
What We Do Not Claim
We do not claim a gateway improves quality. It improved our visibility and our ability to change things, and the quality improvements that followed came from decisions the visibility made possible rather than from the component itself.
We also do not claim ours is a product. It is a few hundred lines specific to how we work, and the reason we did not adopt something larger is that most of what larger frameworks provide is exactly the opinionated behaviour we wanted the gateway not to have.