A Model Judges Itself Worst Exactly Where Your Business Is
A model asked about a widely documented standard will often tell you when it is unsure. Asked about a component made by one supplier for one industry, it answers with the same confidence and is wrong more often, and that asymmetry is not a detail.
The Pattern in Our Own Numbers
We keep a small set of questions we ask every candidate model without any retrieval, purely to see what it knows unaided. Questions about widely documented standards produce either a correct answer or a clear hedge. Questions about a particular supplier's component produce a confident answer that is wrong about a third of the time.
Both sets are answered in the same tone. There is no linguistic signal separating the reliable answers from the unreliable ones, and a user cannot tell which kind of question they have asked, because the distinction is about the training data rather than about the subject.
The Research That Names the Mechanism
Ni and colleagues published work in 2025 examining how the popularity of knowledge influences a model's perception of its own knowledge boundary, that is, how well it can judge whether it knows something.
The consequence for our customers is uncomfortable and worth saying plainly. Their competitive advantage is knowledge that is not widely documented, by definition, and that is precisely the region where a model's self-assessment is least reliable. The assistant is most confidently wrong exactly where their business lives.
| Subject | What the model does |
|---|---|
| A widely documented standard | Often correct, hedges when unsure |
| A common product from a large vendor | Usually correct, occasional confident error |
| One supplier's component | Confident, wrong about a third of the time |
| The customer's own process | Confident, wrong nearly always |
Why This Settles an Architecture Argument
It is the reason we ground everything in documents rather than relying on what the model knows, even for subjects where its unaided answers look good. The unaided answers look good on the popular half of the question distribution, and that half is not where the value is.
It also explains a pattern in sales conversations. A demonstration built on general questions performs impressively and predicts nothing about performance on the customer's own material, which is why our first meeting now uses their questions and their documents even when that makes the demonstration worse.
What We Do About It
Retrieve, always, for anything domain-specific, and refuse to answer when retrieval returns nothing rather than falling back on the model's knowledge. That fallback is tempting because it produces an answer, and it produces the least reliable class of answer available.
We also separate the two kinds of question in the interface where a system serves both. General questions about a standard can be answered without retrieval and are marked as such; anything about the customer's equipment or process is grounded or declined.
The Uncomfortable Corollary for Evaluation
A model comparison on general benchmarks tells you about the popular region. We have written before about why leaderboards did not predict our results, and this is the mechanism underneath that observation rather than a separate point.
It also means our own labelled sets are more important than they would otherwise be, because they are the only measurement located in the region we actually operate in, and there is no public substitute for them by construction.
What It Means for Scoping
The more specialised a customer's domain, the more of the project is corpus work rather than model work. That is the opposite of what most people expect: a highly technical niche sounds like it needs a more capable model, and it needs better documents and a stricter refusal path.
We say this in first meetings now, and it changes what customers expect to spend money on. Two projects have started with three months of documentation work before any assistant existed, on our recommendation, and both went better than the ones that did not.
Where the Effect Works in Our Favour
General knowledge that surrounds the specialist knowledge. Units, materials, regulatory vocabulary, common procedures: the model handles those well and hedges appropriately, which is why an assistant can explain what a term means while refusing to say what value it takes for a particular machine.
That split is a useful shape for an interface. Explain freely, assert only from documents, and make the boundary visible rather than blending both into one confident paragraph.
What We Do Not Claim
We do not claim our third is a measured rate in any rigorous sense. It comes from a small internal question set, it is not a benchmark, and its only role is to make an argument we then act on conservatively.
We also do not claim models cannot improve here. Self-assessment is an active research area and the situation may look different in a year, and our architecture does not depend on the effect persisting: grounding is the right design even for a model that judges itself perfectly.