iiinova — where your digital self keeps evolving

Russell J. Funk @russellfunk.bsky.social · 18/09/2026

For anyone following our 2023 disruption paper: at our request, Nature has released the peer review file for Holst et al.’s Comment and our Reply. Reviews of the Comment were not shared with our team during review. Both sets are now public. www.nature.com/articles/s41...

nature.com

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

000

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

I think there is a less bleak interpretation—if the constraint is how much of the literature actually gets used, that’s more fixable than the claim that the ideas have run out.

200

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

That’s great that you were teaching this and discussing it with students—that’s the side bibliometric indicators can’t see at all. We measure what gets cited, never what got read, or what someone was taught to look for.

010

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

Thanks for engaging; there’s a lot of room for future work here.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

One explanation we suggested is there is more to build on, but a narrower slice of it is actually getting used.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

The puzzle was, knowledge growth should be self-reinforcing—the more you have, the more you can build. Yet across many aggregate measures, from research productivity to disruption, there seems to be a conversion problem.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

That’s close to why we looked at this. In our 2023 paper we found greater citation concentration and aging reference lists, despite enormous growth in publishing.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

If new work is cited briefly then forgotten, the citations that persist concentrate on an entrenched canon—which pulls mean reference age up. Faster decay of the new is part of what makes reference lists older on average. Chu & Evans (2021) show this in large fields.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

Great, thought provoking question. I don’t think they necessarily conflict. Half-life measures how fast attention to a given paper fades. Citation lag measures the age mix of reference lists. Those can move oppositely if the distribution splits.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

And the model they did propose—which in their own supplement still shows a large, significant decline—they distanced themselves from two days after publication, writing that the Matters Arising "was not intended to... defend any particular regression analysis." Our Reply: doi.org/10.1038/s415...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

Their own Comment argues the problem is specific to zero-reference works; the change in their model is "not matched by the inclusion of a dummy variable for any other number of references." So nothing in their own paper justifies cutting at ten.

100

Russell J. Funk @russellfunk.bsky.social · 18/08/2026

Some context that figure is missing, which the FAQ doesn't make clear—it isn't in their peer-reviewed Comment. It appears only on that page, after publication, and the authors say it isn't their conclusion. It cuts the sample to papers with ≥10 references and ≥10 citations—about 18% of works.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

That correspondence was confidential and mediated by Nature, and we intend to respect that, so we have reached out to Nature's editorial team on how to proceed before saying more here. The Reply: doi.org/10.1038/s415... Thread: bsky.app/profile/russ...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Their FAQ also gives an account of the 32-month editorial process—one that inverts the record. The delays it attributes to our team, and the transparency concerns it raises, run in the other direction, as the correspondence documents in detail.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

The critics of our 2023 paper have posted an unreviewed FAQ on our exchange. The reviewed record is the two papers Nature published together as a pair—rerun with their own dataset, metric, exclusions, and regression model, the decline persists, and their own regression shows it.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

For readers following this—Nature published our peer-reviewed Reply alongside the Comment, as a pair. Using their own dataset, metric, exclusions, and regression model, a large decline remains. Reply: doi.org/10.1038/s415... Thread: bsky.app/profile/russ...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Until then, the Reply is here: doi.org/10.1038/s415... the Comment's own dataset, metric, exclusions, and regression model, a large and substantively meaningful decline remains. Summary with figures: bsky.app/profile/russ...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

The practical effect is several journalists have written to me for comment unaware that a Reply exists at all, and some coverage has presented only one half of the exchange. Nature's editors have been responsive and are working on a fix.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

A note for anyone following our 2023 Nature paper on declining disruptiveness. Nature published a Comment by Holst et al. alongside our peer-reviewed Reply. But the Comment's page gives no sign a Reply exists, and the shared PDF opens on the Comment, with the Reply at the end.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Worth adding—PLF 2023 claimed a large, robust, persistent decline across data, measures, specifications, not a point estimate. Raw index values carry little meaning alone, which is why we benchmarked on documented shifts in science. The exclusion moves a number within that family. The scale holds.

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Why does their raw number move so much? Their adjustment mostly measures the effect of proper subsetting. Their sample keeps millions of docs that by definition cite nothing—editorials, obituaries—plus fields w/ metadata issues. We dropped these at the start; removing them at the end looks dramatic.

110

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Thanks for engaging. Here is the comparison on a common scale, which is what the attenuation question needs. Include 0-ref works and the decline rivals or exceeds the growth of team size, arguably the largest shift in science we can measure in metadata. Exclude them and it is second only to that.

110

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

The other half, published alongside it: our peer-reviewed Reply. Using the Comment’s own dataset, metric, exclusion criteria, and regression model, a large and substantively meaningful decline remains—and their own regression model shows the same. Thread: bsky.app/profile/russ...

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

We included them on the basis of the broader literature on works without references. Reasonable people can disagree on that choice and the Reply discusses it. But the finding doesn’t rest on it…drop every 0-ref work, using the Comment’s own criteria and their own model, and a large decline remains.

110

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Nature published our peer-reviewed Reply alongside this, as a pair. Using the Comment’s own dataset, metric, exclusion criteria, and regression model, we find a large and substantively meaningful decline remains—and their own regression model shows the same. doi.org/10.1038/s415...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

010

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Appreciate you saying so. The whole exchange is admittedly complex with a lot of moving parts and can be easy to misread—thanks for taking a look.

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

And on the “180 degrees”—Holst et al. do not report an increase in disruption anywhere in their commentary.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

On the measure: PLF 2023 and the Reply both test alternatives that weight citation types differently, plus different windows and binary versions. On WoS with zero-reference works excluded, four independently developed measures decline 7.5–36 percentile points (ED Fig 1, all P<0.001).

000

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

Both were done. The inclusion criterion at issue is doc type: standard practice drops editorials, obituaries, books and reviews, which by nature cite nothing. Their sample keeps them—one in five items—hence 32% 0-ref works vs our 12%. Citing norms also vary by field, so we analyze fields separately.

100

Russell J. Funk @russellfunk.bsky.social · 14/08/2026

This has the direction backwards. Inclusion criteria are exactly what our Reply evaluates. Their sample contains 3x as many 0-ref works as ours (32% vs 12%), bc one in five items in it is not research at all. Drop every 0-ref work—theirs, ours, either dataset—a large decline remains. No reversal.

000

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The size of the shift tracks how much such content a sample contains—larger in theirs, smaller in ours. Either way, a large and meaningful decline remains: on our data, and on theirs with every zero-reference work removed. Their adjustment is a fact about their dataset, not about science.

000

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

Their design treats 0-reference works as the artefact, in a dataset where such works are nearly 3x more common than in ours (32% vs 12%), without separating true metadata errors from genuine non-research—editorials, books—that legitimately cites nothing. Removing all of it moves the estimate a lot.

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

To be clear about where we agree: if the concern is that metadata noise can affect bibliometric estimates, we are fully on board—PLF 2023 ran 20+ analyses addressing exactly that concern, and the Reply adds more. The question is what their specific design can show.

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The most interpretable anchor—under the regression proposed in the MA, run on WoS, the decline equals the gap between an average paper and a paper by a Nobel Prize winner (our Fig 1a). Fig 1d (attached) plots the decline against the benchmark shifts on a common scale; exact values are in the Reply.

000

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

Yes, thanks! Same rank of magnitude, though the two papers report it somewhat differently. What the 0-reference debate comes down to is whether the decline is at or above the largest widely recognized transformation of modern science—the growth of team size—or second only to it. Either way, large.

110

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

Their Fig 1c drops ALL top-scoring works, beyond the 0-reference works their critique concerns. A cut that removes more than the works at issue cannot attribute the change to those works, so it cannot adjudicate the claim. The designs that isolate the concern show a substantively meaningful decline.

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

Our Reply also applies the exclusion proposed in the Comment, dropping zero-reference works entirely. On WoS, four independent disruption measures decline by 7.5–36 percentile points (ED Fig 1, all P<0.001). In SciSciNet, the decline is 15.5 points, second only to the growth of team size (Fig 1d).

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

On WoS, we estimated the regression model proposed in the MA, which adds a control for zero-reference works. The decline is large and significant—equal to the gap between an average paper and a paper by a Nobel Prize winner (our Fig 1a). The same estimate appears in their supp. tab 1 (on SciSciNet).

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

To clarify, by "their sample" I meant SciSciNet. But the question is the same on any dataset: is the decline we documented an artifact of zero-reference works? Our Reply addresses that concern with their model and their exclusion, on both WoS and SciSciNet. A substantial decline remains.

100

Reposted by Russell J. Funk

Stephen Curry @scurry.bsky.social · 13/08/2026

In the interests of fuller discussion of innovation/disruption…

001

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The books are illustration, not the argument. The artefact question is whether 0-ref works are metadata errors. In their sample, 2.8M editorials, obituaries, comments—20% of the total—genuinely have no references. That non-research content fell from 40% to 8%, producing the trend they call error.

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

Yes. The Comment and our Reply went through Nature's Matters Arising peer review together, as a pair, over the same multi-year process, and were published together.

020

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The model also controls for changing citation, authorship, and publication practices, following our 2023 approach. So the side-by-side is apples to oranges—a raw, over-exclusion next to a model-adjusted trend. Model to model, their own specification shows the decline (P<0.01).

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The curves answer different questions. Red (Holst) removes ALL CD=1 work—beyond the scope of their critique—deleting disruptive papers with >0 references and inflating the apparent effect of their adjustment. Blue is us (Park) running their own regression on WoS, with their zero-reference control.

100

Russell J. Funk @russellfunk.bsky.social · 13/08/2026

The peer-reviewed Reply, published alongside: we reran their analysis with their own dataset, metric, exclusions, and regression model—the decline persists, and their own regression shows it (P<0.01). One in five works in their sample is not research. doi.org/10.1038/s415...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

101

Russell J. Funk @russellfunk.bsky.social · 12/08/2026

18/ Evidence of declining disruptiveness now spans nearly 100 studies—across databases, metrics, and measures that use no citation data at all. Every number in this thread is in today's Reply and its supplement. doi.org/10.1038/s415...

doi.org

Reply to: Dataset artefacts can partially drive the measured decline in disruption - Nature

Nature - Reply to: Dataset artefacts can partially drive the measured decline in disruption

000

Russell J. Funk @russellfunk.bsky.social · 12/08/2026

17/ As our Reply records, 'Nature, through which all inquiries were mediated, declined to pursue further clarification.' We cannot assess the analysis. Neither can anyone else. It was published today anyway.

100