Where Our Models Come From, and What We Check
An inventory is a boring exercise that occasionally produces an uncomfortable number. Ours was eleven: eleven model and embedding artefacts pulled from public registries into systems we run, of which we could name the provenance of four.
The Assumption We Had Made
We deliver systems around models rather than training them. From that it followed, we thought, that questions about training data belonged to the organisations doing the training. It is a reasonable division of labour and it quietly skips a step.
The step is that we still pull artefacts. Embedding models, rerankers, a small classifier, tokenisers: each arrives from a registry, gets loaded into a process we operate, and processes customer data. That is a supply chain whether or not we call it one.
What the Inventory Found
Eleven artefacts across the systems we run. Four had a pinned version and a recorded source. Three were pinned to a moving tag, which means the thing we tested and the thing that runs are related only by convention. Four were pulled at container build time with no pin at all.
Nothing had gone wrong. That is worth saying plainly, because the finding is not an incident, it is the absence of a control, and those are much easier to leave in place because nothing draws attention to them.
Why the Research Made Us Look
Bowen and colleagues published work that August on how data poisoning scales in language models, examining the relationship between the amount of poisoned data and the effect on the resulting model, and finding that larger models were not inherently more resistant in the settings they measured.
We are not in a position to detect any of that in a downloaded artefact, and that is precisely why it changed our behaviour. If we cannot inspect what we load, the only remaining controls are about where it came from and what it is allowed to touch.
| Control | Where we apply it |
|---|---|
| Pin to a content hash | Every model artefact, no exceptions |
| Mirror into our own registry | Anything in a production path |
| Record the source and licence | All eleven, in one file |
| No network from the inference process | Everything we self-host |
What We Changed
Every artefact is pinned to a digest rather than a tag, and mirrored into a registry we control. That is not sophisticated and it removes the failure where the thing that runs in March is not the thing that was tested in January.
The inference processes lost outbound network access. A self-hosted model does not need to reach the internet to answer a question, and removing that possibility means an artefact behaving badly has considerably less to work with than one that can open a connection.
The Licence Half of It
The same inventory served a second purpose that we had also been putting off. Four of the eleven had licence terms nobody on the project could state, and one had terms that restricted a use case a customer had asked about.
That is a commercial risk rather than a security one, and it is much more likely to bite. We now record the licence next to the digest, which costs nothing at pull time and answers a question that otherwise arrives during a contract negotiation.
What We Cannot Do
We cannot inspect weights for anything meaningful. There is no practical check we can run on a downloaded model that would tell us whether it behaves badly on some rare input, and pretending otherwise would be the worst outcome of writing this piece.
So our position is about consequence rather than detection. A model runs with no network, no credentials and no tools, its output goes through the same validation as any other untrusted input, and what it can affect is bounded by design rather than by trust.
What We Ask Providers
Three questions, and we ask them of hosted providers too. Which version am I calling and can I pin it. What is your notice period for retiring it. And what is your policy on changing behaviour within a version.
The third question is the one that gets the least satisfying answers, and it is the one that matters most operationally: a system tuned against a model that quietly changes underneath it is a system whose evaluation set is measuring history.
What We Do Not Claim
We do not claim pinning and mirroring make an artefact trustworthy. They make it consistent, which is a different property, and a consistently compromised artefact is exactly as compromised as an inconsistent one.
We also do not claim eleven is a lot or a little. It is what we had, we had not counted before, and the value of the exercise was mostly in discovering that seven of them were not pinned rather than in the total.