iiinova — where your digital self keeps evolving

Institute for Replication @i4replication.bsky.social · 29/09/2026

We had an amazing time at the University of Münster. Our replicators covered papers across psychology, public health and political science! Huge thanks to @bschlipphak.bsky.social, Katrin Schmietendorf, @aufdroeseler.bsky.social and Oliver Kamps for organizing.

0102

Institute for Replication @i4replication.bsky.social · 25/09/2026

See our disclosure statements on pages 8-9

120

Institute for Replication @i4replication.bsky.social · 25/09/2026

Preregistrations were generally strong on the basics (hypotheses, outcomes, models, and sample size) but less detailed on control variables, data cleaning, and robustness checks. Multiple-hypothesis corrections appeared in fewer than 10%.

130

Institute for Replication @i4replication.bsky.social · 25/09/2026

Preregistration was common: 74.6% of articles included at least one preregistered study. Of those, 42 (84%) had at least one deviation from the preregistered plan. And 17 of those 42 disclosed none of their deviations.

151

Institute for Replication @i4replication.bsky.social · 25/09/2026

One of the clearest patterns concerned documentation. Share of packages fully reproducible from final analysis data: 📘: Clear README: 87.5% 📙: Unclear README: 48.3% 📕: No README: 50.0% An unclear README appeared to offer little advantage over having none.

270

Institute for Replication @i4replication.bsky.social · 25/09/2026

Coding and statistical reporting issues were also common. About 63% of reproduced articles had at least one coding error or reporting inconsistency. Most flagged cases were minor (59.5%), but 40.5% had an issue that could affect estimates, standard errors, significance, or interpretation.

151

Institute for Replication @i4replication.bsky.social · 25/09/2026

Reproducibility depends heavily on where you start. From authors' final analysis data: ✅: 58.2% fully reproducible 🟡: 32.8% partially reproducible ❌: 9.0% not reproducible From raw data: ✅: 28.4% fully reproducible 🟡: 34.3% partially reproducible ❌: 37.3% not reproducible

196

Institute for Replication @i4replication.bsky.social · 25/09/2026

Over the past two years, I4R partnered with Psychological Science (@psychscience.bsky.social) on a large reproducibility project. 67 independent teams reproduced 64 articles published in 2024–2025, about 44% of eligible papers. Paper: www.econstor.eu/bitstream/10... Here's what we learned 🧵

19059

Institute for Replication @i4replication.bsky.social · 22/09/2026

And here is something particularly unusual. Two of the original authors — Abu Siddique and Tabassum Rahman — formally requested that their names be removed from the paper. REStat has now published those requests as a separate item. direct.mit.edu/rest/article...

110

Institute for Replication @i4replication.bsky.social · 22/09/2026

Our comment is here: direct.mit.edu/rest/article... The response from the three remaining authors is here: direct.mit.edu/rest/article...

130

Institute for Replication @i4replication.bsky.social · 19/09/2026

Some of us are having fun today in Germany at the Freiburg Replication Games! Meanwhile, others are in Montreal for a workshop on AI and research integrity that we’re co-organizing. A busy and entertaining weekend for I4R!

051

Institute for Replication @i4replication.bsky.social · 10/09/2026

Podcast: open.spotify.com/episode/3qAa... www.youtube.com/watch?v=xP6U... Blog: www.i4replication.org/blog/not-wro... Paper: arxiv.org/abs/2609.09190

000

Institute for Replication @i4replication.bsky.social · 09/09/2026

The PII rates are also much higher in the JDE than other journals. If we focus on direct PII, 33.3% for JDE vs 6.9% for EE. Most authors of JDE articles do not share any data and do not use author-collected data, so these rates are for articles that fit our inclusion criteria

100

Institute for Replication @i4replication.bsky.social · 09/09/2026

We ran a couple of regressions. Dependent variable is a dummy variable for whether the package includes any PII (col 1-4), direct PII (5) or indirect/IP (col 6). The difference remains significant at the 1% level.

100

Institute for Replication @i4replication.bsky.social · 09/09/2026

Among packages containing data only from participants in low- or lower-middle-income countries, 35.0% contained PII. For upper-middle- and high-income countries, the rate was 16.5%. Figure shows split by type of experiments.

100

Institute for Replication @i4replication.bsky.social · 09/09/2026

We show coefficients+SE for the control variables here.

100

Institute for Replication @i4replication.bsky.social · 09/09/2026

We ran a couple of regressions. Dependent variable is a dummy variable for whether the package includes any PII (col 1-2), direct PII (3-4) or indirect/IP (col 5-6). Almost all differences across journals are gone.

100

Institute for Replication @i4replication.bsky.social · 09/09/2026

PII rates do vary a lot across journals, no doubt. If we focus on direct PII, 33.3% for JDE vs 6.9% for EE. But this could be explained by data types (lab vs field vs online experiments), rep package characteristics, etc.

100

Institute for Replication @i4replication.bsky.social · 08/09/2026

28/ We have no funding to run our PII tool or scale up this project. We wrote a blog post with approximate numbers on how much it would cost to check for PII for the universe of social science packages on Dataverse: www.i4replication.org/blog/what-wo...

130

Institute for Replication @i4replication.bsky.social · 08/09/2026

21/ We asked our data editors friends about their current PII practices. It varies a lot. Have a look!

120

Institute for Replication @i4replication.bsky.social · 08/09/2026

16/ What about data editors? Overall PII prevalence was almost identical: 21.2% of packages handled by a data editor contained PII vs 21.1% of packages without data editor. Existing comp reproducibility checks therefore do not appear to systematically prevent PII disclosure.

110

Institute for Replication @i4replication.bsky.social · 08/09/2026

13 We also find a striking difference by study setting. Among packages containing data only from participants in low- or lower-middle-income countries, 35.0% contained PII. For upper-middle- and high-income countries, the rate was 16.5%.

121

Institute for Replication @i4replication.bsky.social · 08/09/2026

12/ Type of data collection matters a lot. PII appeared in: 29.2% of field-experiment packages 26.6% of online-experiment/survey packages 8.3% of laboratory-experiment packages The risk appears closely related to how data are collected and how complex the resulting packages are.

110

Institute for Replication @i4replication.bsky.social · 08/09/2026

11/ Differences across journals are mostly explained by data type. See the next two posts. Also, most articles at some journals (eg JDE) do not share any data and do not use author-collected data, so these rates are for articles that fit our inclusion criteria.

101

Institute for Replication @i4replication.bsky.social · 08/09/2026

10/ PII is present across all three disciplines we studied: Economics: 25.9% Political science: 17.8% Psychology: 16.7% Across individual journals, prevalence ranged from 6.9% to 46.7%.

111

Institute for Replication @i4replication.bsky.social · 08/09/2026

8/ Out of 327 packages, we found: • Names in 24 packages • IP addresses in 23 • Prolific/MTurk IDs in 14 • Email addresses in 12 • Phone numbers in 7 • Precise GPS locations or addresses in 6

112

Institute for Replication @i4replication.bsky.social · 08/09/2026

5/ Sample: 327 randomly selected public replication packages from 11 leading journals. ONLY AUTHOR-collected data = field/lab/online experiments and surveys that were published in 2020-2024. The packages were drawn from repositories (e.g. Dataverse, OSF) and journal websites

101

Institute for Replication @i4replication.bsky.social · 08/09/2026

2/ We find that 21.1% of replication packages with author-collected data contain PII • 14.4% with direct identifiers • 7.0% with IP addresses • 7.0% with indirect identifiers These estimates are likely lower bounds, especially for indirect PII Paper: www.econstor.eu/bitstream/10...

171

Institute for Replication @i4replication.bsky.social · 08/09/2026

1/ What is the prevalence of personally identifiable information (PII) in author-collected data publicly shared in the social sciences? We audited 327 replication packages from 11 leading journals in economics, political science, and psychology Answer: surprisingly high 🧵

25537

Institute for Replication @i4replication.bsky.social · 04/09/2026

11/ The broader point: sampling variation in the data itself is a quantitatively important contributor to the fragility of empirical results.

100

Institute for Replication @i4replication.bsky.social · 04/09/2026

9/ Noisier inputs, weaker results. Papers using daily series have relative t-values 57% lower, and the average cross-resample correlation strongly predicts robustness. The least robust papers used the noisiest sampling distributions.

110

Institute for Replication @i4replication.bsky.social · 04/09/2026

8/ Fragility is predictable. Each additional "exercised" researcher degree of freedom — multiple queries, transformations, subsampling, dropping zeroes — predicts ~14% smaller t-stats and a 12pp lower chance of significance. Multiple queries matters most.

120

Institute for Replication @i4replication.bsky.social · 04/09/2026

6/ Both margins move: point estimates fall ~12%, standard errors rise ~40%.

130

Institute for Replication @i4replication.bsky.social · 04/09/2026

5/ 70% of resamples produce a smaller t-stat than the original — not the 50% you'd expect if published results were representative draws. 9% flip sign. Over half of those flips are themselves significant at the 5% level.

130

Institute for Replication @i4replication.bsky.social · 04/09/2026

4/ Headline: t-statistics shrink by 31% on average. 55% of resamples are significant at 5%, versus 85% of the original published estimates.

131

Institute for Replication @i4replication.bsky.social · 04/09/2026

2/ The setup: Google Trends indices aren't built from all searches. Google draws a random sample. Run the identical query on a different day and you get a different realization of the same data-generating process. So the data can be resampled. DP: ideas.repec.org/p/zbw/i4rdps...

140

Institute for Replication @i4replication.bsky.social · 04/09/2026

🧵 New DP from I4R co-director Lester Lusher, with Sonia Ale, Md Shafiqui Islam, Huda Osman & Jacob Stenstrom (Pitt): "When Samples Shape Significance." What happens when you re-estimate published results using fresh draws of the same data? Some evidence from Google Trends!

12914

Institute for Replication @i4replication.bsky.social · 24/08/2026

The Institute for Replication and University of Freiburg are jointly organizing the Replication Games on Saturday, September 19th. The Games will focus on replicating papers on the causes of political violence.

143

Institute for Replication @i4replication.bsky.social · 14/07/2026

The Institute for Replication (I4R) and UCD School of Economics are jointly organizing the Replication Games (RGs) alongside the EEA-ESEM Congress on 𝐒𝐮𝐧𝐝𝐚𝐲, 𝐀𝐮𝐠𝐮𝐬𝐭 16.

162

Institute for Replication @i4replication.bsky.social · 12/06/2026

📢 Call for Papers! We are partnering with the Journal of Economic Surveys on a special issue: Reproducibility and Meta-Science in Economics. Submission deadline: 1 March 2027. Details in thread 🧵

1811

Institute for Replication @i4replication.bsky.social · 08/06/2026

We're having an amazing time at @mcgilluniversity.bsky.social. Our CAnD3 replication games today are focused on health! Huge thanks to Arianne Rodriguez-Saltron and Amélie Quesnel Vallée for organizing.

010

Institute for Replication @i4replication.bsky.social · 04/06/2026

11/ The stars help us navigate. Good scientists change their minds. May the Force be with all Jedi replicators ⭐ And all our love to Melody Gardot.

010

Institute for Replication @i4replication.bsky.social · 04/06/2026

10/ And closes with a quote attributed to Leonardo da Vinci "He who is fixed to a star does not change his mind." For 13 years, we thought that was a fitting ending. Now, we are even more convinced that science works best when we are willing to do the opposite.

110

Institute for Replication @i4replication.bsky.social · 04/06/2026

3/ We worked together with Jedi Roodman to write the corrigendum, but he had the last word on whether he thought our results and claim were robust and valid. The corrigendum is now officially posted by the AEA: www.aeaweb.org/content/file...

120

Institute for Replication @i4replication.bsky.social · 04/06/2026

🚨 CORRIGENDUM ALERT 🚨 A decade after Star Wars: The Empirics Strike Back was published in AEJ: Applied, we've just published an official corrigendum with the AEA. This is a love story. 🧵

1217

Institute for Replication @i4replication.bsky.social · 04/06/2026

We're having fun replicating papers today. Thanks to everyone participating to the Utrecht Replication Games!

2100

Institute for Replication @i4replication.bsky.social · 31/05/2026

So we are actually quite happy with the retraction outcome at PLOS One :) We simply want to highlight how mission impossible it is to publish comments. P.S. See also this blog post on the American Economic Review: www.i4replication.org/blog/the-van....

0102

Institute for Replication @i4replication.bsky.social · 31/05/2026

Incentives to reproduce and replicate articles are so bad. Even when your comment leads to the retraction of a PLOS One article, the reward is a (10 days late) email with a thank you at the bottom. Retraction notice: journals.plos.org/plosone/arti.... But we are not complaining about PLOS One 🧵

12412

Institute for Replication @i4replication.bsky.social · 29/05/2026

Ongoing replication games at CERDI, Clermont-Ferrand and games yesterday at the Canadian Economic Association in Vancouver! Thanks to the local organizers (Simone Bertoli and Kevin Schnepel) and everyone one who participates! 2 more games within the next 10 days :)

020