What We Do When a Paper Goes Viral
A new neural network architecture reached the general technology press, and within days we were asked three times whether it changed our recommendations. It did not, and being able to say why in two sentences turned out to be more valuable than having an opinion about the architecture.
Three Calls in One Week
Liu and colleagues published a paper on Kolmogorov-Arnold Networks at the end of that April, proposing learnable activation functions on the edges of a network in place of fixed activations on the nodes, and reporting favourable accuracy and interpretability on the small scientific problems they study.
It spread far beyond its field within days. We were asked about it three times in a week, and the question each time was some version of whether our customers should wait, reconsider, or budget for something different. That question deserves a real answer rather than a shrug.
The Layer Question
Our first question about any result is which layer it belongs to. There is the architecture of the network, the training of a specific model, the system built around a model, and the data and workflow the system serves. Most work that reaches a general audience concerns the first layer, and almost nothing we build touches it.
That is not dismissal. Architecture research is where the field's long-term progress comes from. It is a statement about the distance between that layer and a delivery date: results at the bottom reach a customer through several intermediate steps, each of which takes time and may not happen at all.
The Four Questions
We ask what would have to be true for this to reach production, whether anything in our current pipeline is limited by what the result improves, what it would cost to find out, and who else has reproduced it. Four questions, twenty minutes, and usually a clear answer.
For that particular architecture the answers were: it would need to train at scale, be implemented in the frameworks we use, and produce a model someone hosts. Nothing in our pipelines is limited by activation function design. Finding out would cost weeks. And at that point it was days old.
| Where a result sits | How fast it can reach us |
|---|---|
| Network architecture | Years, through several intermediate steps |
| Training method | Months, once someone ships a model |
| System technique, for example retrieval | Weeks. We can test it directly |
| Prompting or workflow | Days. Often an afternoon |
What We Actually Told Customers
That it is interesting work, that it is at the architecture layer, that nothing in their system is currently limited by that layer, and that we would notice if it produced a model worth using because we watch what gets hosted rather than what gets published.
One of the three pushed back, reasonably: how do we know we are not missing the next big thing? The honest answer is that we might be, and that a consultancy which reacts to every paper would be worse at delivering than one that is occasionally late to a genuine shift.
Where We Move Fast Instead
The third and fourth rows of the table are where our attention goes. A retrieval technique or a prompting pattern can be tested against our labelled set in an afternoon, and the cost of being wrong is that afternoon. We try considerably more of those than we adopt.
That is where most of our actual improvements have come from. None of them made the general press, and several of them were unglamorous: better chunking, a tighter schema, removing a stage. The visible research and the useful changes have been largely disjoint sets.
What Would Change Our Mind
A trained model at a useful scale that we can call, an independent reproduction of the headline claim, and a benchmark that resembles work our customers do. Any one of those alone is not enough; the combination is, and it is roughly what we look for before spending time.
We keep a short list of results we would like to be wrong about and revisit it about twice a year. Two items have moved off it into production use since we started keeping it, which is a low hit rate and exactly the rate we would expect.
Why This Is a Service and Not a Reflex
Customers are subjected to a great deal of noise, and much of it is designed to create urgency. Being able to say clearly that a widely discussed result does not affect their roadmap, and to explain the reasoning in a way they can check, is a substantial part of what they are paying for.
The alternative is a supplier who agrees that everything is important, which produces a budget shaped by press coverage. We would rather be the ones who say that this month nothing changed, when nothing changed.
What We Do Not Claim
We do not claim the architecture in question is unimportant. The paper is careful, its authors are explicit about the settings they tested, and whether the approach scales is an open question we are not qualified to settle.
We also do not claim our filter is safe. It is biased against exactly the kind of change that eventually matters most, and the mitigation is the revisit list rather than any confidence that we would catch a shift early.