iiinova — where your digital self keeps evolving

vincentholst.bsky.social @vincentholst.bsky.social · 17/08/2026

Hi Eric! Yes, I presented this at STI 2024 in Berlin, the slides from back then are also available online: data.research.vub.be/sites/defaul...

data.research.vub.be

010

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

You can find our comments to the most questionable arguments on this website. We might update this more in the future. The Curious Case of the Declining Disruption’s Disappearance vincentholst.github.io

vincentholst.github.io

010

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

In the spirit of full transparency, you can find our comments to the most questionable arguments on this website. We might update this more in the future. The Curious Case of the Declining Disruption’s Disappearance vincentholst.github.io

vincentholst.github.io

110

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

You can find our comments to the most questionable arguments on this website. We might update this more in the future. But Brandolini's law also takes its toll on us.

vincentholst.github.io

The Curious Case of the Declining Disruption’s Disappearance

110

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

You can find our comments to the most questionable points on this website vincentholst.github.io/the_curious_...

vincentholst.github.io

The Curious Case of the Declining Disruption’s Disappearance

000

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

So how can the process be improved? Some ideas to initiate a discussion: substantiated critiques should be handled faster than original research, not twice as slowly, and articles under substantiated critique should carry a visible flag while the exchange is under review. 7/7

1165

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

We know that it is difficult for outsiders to grasp the full details of all analyses, but we believe everyone can see what they are doing here. As much as we want to focus on the science, we cannot let these these tactics go uncommented. At some point, Brandolini’s law takes its toll on us. 6/7

2133

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

Additionally, they use questionable language. Every children's book they name is classified as a ‘Book’ in SciSciNet. We ran multiple robustness checks, including one that removes all books, all of which can be found in our supplementary information. Our results remain unchanged. 5/7

182

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

As another example, Park et al. stated the raw rewired curves to be available upon request in their original peer review file, never supplied them despite repeated requests, and now question the reconstruction this made necessary. They have still not shared their figure. 4/7

193

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

For example, the reply of Park et al. omits every analysis that could directly test the artefact explanation for Web of Science, such as a random source-document audit of papers with zero recorded references. 3/7

172

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

To explain this a bit more, a core tension in this exchange was that only the critiqued authors had access to metadata for Web of Science, such as the number of cited references. Every fast step in this story ran through open code and data. And every slow step ran through closed ones. 2/7

1114

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

There are multiple reasons for the delay. A Matters Arising is peer reviewed. Park et al. kept changing their defence, and every iteration took many months. The original authors had no incentive to reply swiftly and even outright refused to share data ‘available upon request’. 1/7

2146

vincentholst.bsky.social @vincentholst.bsky.social · 14/08/2026

As promised, we now share some details why it took 32 months to get this Matters Arising published, and, since many asked, share some comments on the reply to our critique. All details can be found in this small FAQ vincentholst.github.io/the_curious_.... Below is a summary 🧵

vincentholst.github.io

The Curious Case of the Declining Disruption’s Disappearance

48348

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

Shoutout to my co-authors @andresalgaba.bsky.social, FlorianoTori, @sylviawenmackers.be, and @vincentginis.bsky.social. This entire saga required incredible teamwork. Full paper, code, and data below Paper: rdcu.be/fzwzq Code: github.com/VincentHolst... Data: zenodo.org/records/1512... 8/8

rdcu.be

Dataset artefacts can partially drive the measured decline in disruption

Nature - Dataset artefacts can partially drive the measured decline in disruption

54210

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

Instead of a decline in the disruptiveness of scientific and technological knowledge, the data mainly show an increasing metadata quality over time: relatively fewer papers with zero references due to database errors. 7/8

The decline in disruption (including zero reference papers, left panel) mirrors the relative decrease of zero reference papers which were wrongly counted as being maximally disruptive (right panel).

14013

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

The Monte Carlo simulations randomly rewired citations while preserving each paper's number of references. Zero-reference papers are thus preserved, and the rewired networks inherit the peak at CD=+1. Their decline mirrors the observed decline in disruption. 6/8

Randomly re-arranging the citations between papers preserves the number of references made (top left). This means the zero reference papers are mapped one-to-one in the observed and randomly rewired network, as visible by the peak at one in both distributions (bottom left). The randomly rewired CD index over time almost perfectly mirrors the observed trend, and the gap is closing (right).

1112

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

Why didn’t the robustness checks catch this? The linear regression did not control for the discontinuity of the CD index at zero references. Simply adding a dummy for zero references improves the adjusted R^2 from 0.15 to 0.95, and the adjusted CD index becomes essentially flat. 5/8

The Park et al. regression adjustment did not control for the disconinuity of the CD index appearing at zero references. Adding a dummy to control for it substantially boosts the adjusted R^2 and the regression adjusted curves become nearly flat.

1131

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

Why so many papers with a CD index of exactly 1? In the open-source SciSciNet database, 97% of them make zero references. Of 100 randomly sampled source documents, 93 do contain references. They are dataset artefacts. 4/8

The CD index has a discontinuity at zero references, as every paper gets automatically assigned a maximum value of +1. This correspondence is almost one-to-one: the majority of papers with a CD index of exactly equal to +1 have zero references, an observation that is stable over time.

1155

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

When removing the hidden outliers with CD = +1, the reported decline in disruption largely disappears across all databases used in the Nature study. For Web of Science, it reduces by 93 %. 3/8

Removing the hidden outliers negates the decline in disruption across all databases used in the original study. Compared the the original scale (on the left), all curves depicting scientific papers become flat (on the right).

1185

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

Due to a (then unreported) bug in the seaborn plotting software, the published histograms silently dropped the largest data points. A peak of papers with the maximum disruption score of CD = +1 was hidden from view, while these papers were kept in the analysis. 2/8

Due to a bug in the seaborn plotting software, the largest data points (+1 in this case) were omitted in the CD index distributions shown in the original study (on the left). Correct plot settings (using bins instead of binwidth) reveal a huge peak at papers with a CD index of exactly equal to one (on the right).

23111

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

The study is based on the CD index, a citation-based measure of how much the successors of a work still rely on its predecessors, running from -1 (consolidating) to +1 (disruptive). Key detail: a paper with zero references automatically gets CD = +1 as soon as it is cited once.

The CD index aims to rely the successors of a given work to its predecessors by counting how many citing papers still refer to the references of a given work. In case zero references and at least one citation, the measure defaults to +1.

1284

vincentholst.bsky.social @vincentholst.bsky.social · 13/08/2026

In 2023, @nature.com published 'Papers and patents are becoming less disruptive over time', receiving world-wide media attention. Our Matters Arising, published after a 32 month delay (more on that soon), shows that the reported decline can largely be attributed to dataset artefacts. 🧵

The average CD_5 index per year for Web of Science. The original Park et al. decline (top curve) becomes essentially flat (bottom curve) when removing papers with CD_5=1. Those papers largely correspond to dataset artefacts.

9352173