iiinova — where your digital self keeps evolving

Reposted by @renebekkers.bsky.social

Ian Hussey @ianhussey.mmmdata.io · 09/10/2026

Our reflections on the recent "AI and Research Integrity" meeting in Montreal: We are somewhat worried that many AI trustworthiness assessment tools are not trustworthy. @lhdjung.bsky.social @jamiecummins.bsky.social trustworthy.scientific.claims/posts/trustw...

trustworthy.scientific.claims

Trustworthiness assessment tools must be trustworthy

03517

renebekkers.bsky.social @renebekkers.bsky.social · 10/10/2026

How complete and accurate are AI tools to find research papers? Thousands of scientists around the globe are trying out new tools every day. But it’s difficult to know how complete and accurate they are. In the spirit of open science, here's a rare case where we do know. And the news is not so good.

101

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 25/09/2026

Reproducibility depends heavily on where you start. From authors' final analysis data: ✅: 58.2% fully reproducible 🟡: 32.8% partially reproducible ❌: 9.0% not reproducible From raw data: ✅: 28.4% fully reproducible 🟡: 34.3% partially reproducible ❌: 37.3% not reproducible

196

Reposted by @renebekkers.bsky.social

NBER @nber.org · 28/09/2026

An LLM workflow that reproduces, improves, and extends economics research has been applied to over 4,000 published replication packages, from Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro www.nber.org/papers/w35782

0197

Reposted by @renebekkers.bsky.social

Dr Vanessa @hpsvanessa.bsky.social · 25/09/2026

Cannot tell you how much I hate this & how ashamed & angry I am that @ukri.org have taken this step. We KNOW this sort of gAI is socially biased, we KNOW it favours slopAI output over human-written. Our research landscape is falling apart due to underfunding, & they do this? it's *appalling*

718690

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 25/09/2026

Over the past two years, I4R partnered with Psychological Science (@psychscience.bsky.social) on a large reproducibility project. 67 independent teams reproduced 64 articles published in 2024–2025, about 44% of eligible papers. Paper: www.econstor.eu/bitstream/10... Here's what we learned 🧵

19260

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 24/09/2026

Episode 2 from our 1-day I4R workshop is out. @janmarcus.de, @josefinaw.bsky.social & René Bekkers ask: How can we link publications to replications, so anyone reading a paper can instantly see whether its findings have been retested? 🧵

234

Reposted by @renebekkers.bsky.social

Jamie Cummins @jamiecummins.bsky.social · 24/09/2026

I sincerely hope this paper will be seen as a turning point in how the medical sciences think and approach the issue of trustworthiness assessment (and I hope the social sciences follow suit!). Congrats to @jdwilko.bsky.social and colleagues (incl. @ianhussey.mmmdata.io and @lhdjung.bsky.social)!

12210

Reposted by @renebekkers.bsky.social

Dan Quintana @dsquintana.bsky.social · 21/09/2026

Is it worth putting in all the effort needed to make your materials open if there’s a chance that no one will download them? I still think so, for two reasons:

12512

Reposted by @renebekkers.bsky.social

Bart Penders @penders.bsky.social · 21/09/2026

Back of the envelope calculations: 3600 years of research time went into the collective application process. The research will fund 3200 years of research. We are 400 research years poorer as a result of this. 1/2

316486

Reposted by @renebekkers.bsky.social

Dan Quintana @dsquintana.bsky.social · 19/09/2026

New preprint! 🎉 I analysed 1660 papers from 4 psychology journals and found materials sharing went from 9% of papers in 2015 to 82% in 2025, and these materials *do* get downloaded — a median of 135 times each. BUT shared code is often hard to run. doi.org/10.31234/osf... Let's walk through it 🧵

Two-panel figure. Panel a is a flow diagram tracking 1,611 empirical psychology articles from publication year (545 in 2015, 552 in 2020, 514 in 2025) to repository-link type: 785 link an OSF project, 85 link another platform, and 741 link no repository. Of those with an OSF link, download counts were retrieved for 670 and not retrieved for 115. Panel b is a line chart of the share of empirical papers linking OSF across 2015, 2020 and 2025. The overall rate, shown as a dashed black line, rises from 9% to 57% to 82%. All four journals rise steeply and end close together: Psychological Science 92%, JESP 89%, JML 83%, Cognition 76%, with Psychological Science highest throughout.

Three-panel figure. Panel a: ridgeline plot of downloads per file by material type on a log scale, with the percentage never downloaded labelled for each — archive 16% of 545 files, documents 20% of 2,970, code 12% of 3,471, other 17% of 1,646, data 21% of 10,051, media 35% of 2,120, images 35% of 4,102. Most files cluster between 1 and 10 downloads, with long right tails past 100. Panel b: ridgeline plot of downloads per paper by journal, log scale, with dashed median lines; Psychological Science is highest, then JESP, JML and Cognition. Panel c: stacked bars showing, for documents, data and code separately, the share of papers by download band (0, 1–10, 11–100, more than 100) in 2015, 2020 and 2025. The share exceeding 100 downloads falls sharply over time in all three types, from roughly two-thirds in 2015 to a quarter or less in 2025, as the 1–10 band grows.

Four-panel figure. Panel a: statistical languages detected among 333 papers with retrievable code — R 88%, SPSS 12%, Stata 4%, SAS 1%. Panel b: code red flags among those 333 papers — 34% hard-code an absolute path, 40% reference a missing file — above documentation among 672 OSF-linked papers — 21% have a README, 30% are documented by README, description or wiki. Panel c: among 562 Elsevier papers with no repository link, 44% (245) host at least one journal supplementary file but only 14% (77) host data, code or an archive. Panel d: composition of those 448 hosted files — documents 52%, data 21%, media 9%, other 7%, archive 6%, code 3%, images 1%. The code panels are green, the journal-supplement panels blue.

Coefficient plot (download predictors)

Dot-and-whisker plot of three standardised predictors of OSF download volume, each with a 95% confidence interval. Repository size (number of files) has the largest effect at about 0.76, citations about 0.38, and altmetric attention about 0.12. All three intervals sit entirely above zero, so each predicts more downloads, with repository size roughly twice the effect of citations and six times that of attention. X-axis: standardised effect, −0.2 to 1.0.

4267113

Reposted by @renebekkers.bsky.social

John Holbein @johnholbein1.bsky.social · 14/09/2026

Generative AI makes replication research even more important. Academia should take this opportunity to boost the incentives for replication. It’d be a huge missed opportunity if we don't.

142

Reposted by @renebekkers.bsky.social

Framework for Open and Reproducible Research Training @forrt.bsky.social · 14/09/2026

Peer review is a gift researchers give to the field, worth >$1B globally in 2020. Imagine if more of that generosity flowed straight back into the scholar-led journals that make research free to read and free to publish. Help build that future, become a diamond reviewer forrt.org/diamond-reviewer

A hexagonal badge with a diamond and the words "diamond reviewer" below it.

22824

Reposted by @renebekkers.bsky.social

Alex Holcombe @alexh.bsky.social · 12/09/2026

It's Peer Review Week, and I have pledged to review at least one article per year for a diamond open access journal; one that charges nothing to publish and nothing to read, and is run by and for the scholarly community, using the new Diamond Reviewer Pledge (forrt.org/diamond-revi...).

forrt.org

Diamond Reviewer Pledge

Signatories of the Diamond Reviewer Pledge commit to reviewing at least one article in the next 12 months for a diamond open access journal.

23110

Reposted by @renebekkers.bsky.social

Per Engzell @pengzell.bsky.social · 08/09/2026

Causal inference is the art of knowing exactly why your estimate isn't causal

611318

Reposted by @renebekkers.bsky.social

Johannes Breuer @johannesbreuer.com · 05/09/2026

Social psychology in the lab: Telling undergrad psych students to imagine being well-off and take some candy. Social psychology in the press: Rich people steal kids' candy!

082

Reposted by @renebekkers.bsky.social

Jörg Ankel-Peters @jrgptrs.bsky.social · 04/09/2026

We have an excellent line-up for our Leibniz Open Science Day on Nov 17 in Berlin - including an exciting keynote by Felix Holzmeister. Come join us and catch up on the latest in meta-scientific research! www.zbw.eu/de/ueber-uns...

065

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 04/09/2026

4/ Headline: t-statistics shrink by 31% on average. 55% of resamples are significant at 5%, versus 85% of the original published estimates.

131

Reposted by @renebekkers.bsky.social

ZBW – Leibniz Informationszentrum Wirtschaft @zbw-leibniz.bsky.social · 03/09/2026

Leibniz Open Science Day 2026: Scientific Rigor and Collaborative Research in the Age of AI Berlin, 17 November Keynote speaker: Felix Holzmeister @flxhlzmstr.bsky.social Further information and registration www.zbw.eu/de/ueber-uns...

022

Reposted by @renebekkers.bsky.social

Lisa DeBruine @debruine.bsky.social · 01/09/2026

We’re so close to a solution! Most unis have an institutional repository. If outputs were always published on the lead author’s repo, learned societies reviewed and curated them, their overlay journals just pointed at them, and everything had a DOI, what would we need commercial publishers for?

4245

Reposted by @renebekkers.bsky.social

MetaArXivBot @metaarxivbot.bsky.social · 28/08/2026

How Much Replication Is Enough? Balancing Novelty and Replication in a Truth-Driven Evolutionary Simulation: osf.io/x7n9h

073

Reposted by @renebekkers.bsky.social

Lukas Röseler @aufdroeseler.bsky.social · 24/08/2026

Wherever you go to store data instead of #OSFprojects, please choose a non-commercial provider. Commercial interests are not aligned with scientific interests: They aim at turning your data into money by selling it, training AI models with it and locking us into an ecosystem: doi.org/10.1098/rsos...

1168

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 15/08/2026

New Blog post: Which Data Repository Should you Use? In light of OSF closing down, I compare Zenodo, Dataverse, ResearchBox, PsychArchive, and local repositories on six important dimensions. If you want to know which to pick: It depends! daniellakens.blogspot.com/2026/08/whic...

daniellakens.blogspot.com

Which Data Repository Should You Use?

The Center for Open Science has announced that from November 16, 2026, no new projects can be created on the Open Science Framework. After F...

4198112

Reposted by @renebekkers.bsky.social

Lukas Röseler @aufdroeseler.bsky.social · 13/08/2026

New CC0-licensed badges for numerical reproducibility and an openly accessible peer review report available at doi.org/10.5281/zeno...

Image for four hexagonal badges with speaking bubles and gear symbols.

Meaning of the badges

Open Review Report: A peer review report is openly accessible.
Reproducibility Certificate (Human): A human-led execution of the code yielded identical results.
Reproducibility Certificate (AI): An AI-led execution of the code yielded results that are identical to the reported numbers.
Reproducibility Certificate (Human & AI): A human-led and an AI-led execution of the code yielded results that are identical to the reported numbers.

012

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 14/08/2026

The Metacheck Shiny app to download all private OSF Repositories is done. The manual is updated, and everything is in the dev branch on GitHub. As a bonus, we explain how you can run repo_check, data_check, code_check, and codebook_check on your repository with 1 line of code. Launch next week.

14019

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 12/08/2026

The Metacheck team is busy building an improved version of the osf_download function (www.scienceverse.org/metacheck_bo... ). You will soon be able to use metacheck to download all files, wikis, and meta-data. You will also be able to upload everything to Zenodo.

scienceverse.org

36 Checking Local Files with repo_check and code_check – The Metacheck Manual

A guide to Metacheck, an R package providing a module based tool for checking scientific manuscripts and data repositories for best practices using text search, R code, and optional LLM queries.

03217

Reposted by @renebekkers.bsky.social

Crystal Lewis @cghlewis.bsky.social · 27/07/2026

Always share a data dictionary along with your data so that people aren't guessing what a variable represents. There are many options for creating one (see thread for examples). Choose which one works best for you and share it alongside your data files. :)

512926

Reposted by @renebekkers.bsky.social

Jamie Cummins @jamiecummins.bsky.social · 14/07/2026

Today we launch the first stable release of RegCheck: v1.0.0. RegCheck makes it easier and quicker to compare study registrations to published papers for consistency - something we know is important in principle, but rarely done in practice. A 🧵 on what's new: regcheck.app

regcheck.app

RegCheck

RegCheck is an AI tool to compare preregistrations with papers instantly.

512883

renebekkers.bsky.social @renebekkers.bsky.social · 11/07/2026

In a new preprint with Max Littel, Stephanie Koolen-Maas, Hoda Fahimpour, Arthur Gautier and Wahideh Achbari, we revisit research on charitable giving. While replications and retractions are rare, we show how science can be self-correcting through replications. #metascience osf.io/preprints/me...

osf.io

OSF

000

renebekkers.bsky.social @renebekkers.bsky.social · 08/07/2026

Progress! The metacheck team led by @lakens.bsky.social and @debruine.bsky.social has worked hard to develop software that runs automated checks of research outputs for best practices. Improve your preprint - fix references, links, p-values, and much more. renebekkers.wordpress.com/2026/07/08/r...

renebekkers.wordpress.com

Research Transparency Check Update

Here’s an update on Research Transparency Check: a a software tool that provides automatic assessments of the transparency of data and methods in research reports before they are published an…

000

Reposted by @renebekkers.bsky.social

Jörg Ankel-Peters @jrgptrs.bsky.social · 07/07/2026

Our next Leibniz Open Science Day is coming up on Nov 17 in Berlin! Submit your paper if you do research on science. We welcome submissions across all areas of meta-science. Keynote by @flxhlzmstr.bsky.social! www.zbw.eu/en/about-us/...

043

Reposted by @renebekkers.bsky.social

Framework for Open and Reproducible Research Training @forrt.bsky.social · 14/06/2026

How can AI & LLMs improve how science works? FORRT is delighted to host the AI in Metascience online conference. Please join us! 🗓️ 28–30 Sep 2026 · online · free Talks, workshops & hackathons — all career stages welcome. 👉 Submit by 31 Jul: forrt.org/ai-metascience #Metascience #OpenScience

forrt.org

FORRT AI in Metascience Online Conference | FORRT - Framework for Open and Reproducible Research Training

Call for submissions — FORRT AI in Metascience Online Conference, 28–30 September 2026, online. Deadline 31 July 2026.

02215

Reposted by @renebekkers.bsky.social

Brian Nosek @briannosek.bsky.social · 30/06/2026

"Federal research funding has produced exceptional returns precisely because decisions about what to fund have rested on merit and independent expert judgment, rather than political alignment." @webmz.bsky.social comments on OMB's proposal rule for @cos.io www.cos.io/blog/what-om...

cos.io

What OMB’s Proposed Grantmaking Revisions Could Mean for Research Integrity

The Center for Open Science (COS) submitted a public comment opposing several proposed revisions in the White House Office of Management and Budget’s proposed Regulation for Federal Financial Assistan...

02010

Reposted by @renebekkers.bsky.social

Ingo Rohlfing @ingorohlfing.bsky.social · 01/07/2026

The emerging disenchantment with retrospective meta-analysis www.nature.com/articles/s41... Retrospective meta analysis lacks prior unobservability of data, hence all problems known from empirical research apply. Multilab preregistered studies are seen as best alternative.

1217

Reposted by @renebekkers.bsky.social

Tobias Dienlin @tobiasdienlin.com · 29/06/2026

New publication by Jasmijn Krujit, @bnbakker.bsky.social et al. in @cogitatiomac.bsky.social: Only a minority of studies in Communication transparently share research materials or data. Way to go ... #openscience www.cogitatiopress.com/mediaandcomm...

153

Reposted by @renebekkers.bsky.social

RigorLabs @rigorlabs.bsky.social · 29/06/2026

AUTOMATING COMPUTATIONAL REPRODUCIBILITY My colleague @philipjakobbln.bsky.social and I are currently engaged in a research project where we reproduce scientific results en masse. To that end, we built rigor.me, a platform for automatically reproducing papers using agents 🧵

1177

renebekkers.bsky.social @renebekkers.bsky.social · 28/06/2026

New findings from Giving in the Netherlands: in 2024, the Netherlands donated € 5.49 billion. That's $ 6.38 billion or £ 4.77 billion. An increase from 2022, but in terms of GDP a decline from 0.58% to 0.54%. 70% of households give, and 45% of the population volunteers. osf.io/8fvbc/files/...

010

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 26/06/2026

Our metacheck package now has an online manual! In 29 short chapters we explain how to set up metacheck (running all modules 100% locally if you want), and explain the 18 available modules. www.scienceverse.org/metacheck_bo... We are still in beta, and would love to get feedback!

scienceverse.org

The Metacheck Manual

A guide to Metacheck, an R package providing a module based tool for checking scientific manuscripts and data repositories for best practices using text search, R code, and optional LLM queries.

04316

Reposted by @renebekkers.bsky.social

The Ill @the-ill.bsky.social · 17/06/2026

Anakin&Padme 4 panel meme, with Anakin announcing “We’re using AI instead of biased humans”. Palme asks “What did you train the AI on?”. Anakin looks on blankly. Palme repeats, “What did you train the AI on?”

81225270

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 12/06/2026

📢 Call for Papers! We are partnering with the Journal of Economic Surveys on a special issue: Reproducibility and Meta-Science in Economics. Submission deadline: 1 March 2027. Details in thread 🧵

1811

Reposted by @renebekkers.bsky.social

Pieter Hermans @pjeh.eurosky.social · 10/06/2026

What happens when researchers stop competing for funding? Twenty Dutch metascientists replaced competition with collaboration for NWO grant applications, proving collective approaches can outperform individual ones. @sajedehra.bsky.social @lakens.bsky.social www.tue.nl/en/news-and-...

tue.nl

What happens when researchers stop competing for funding? - tue.nl/stories | funding | open science | Daniel Lakens | Sajedeh Rasti

Twenty Dutch metascientists explore whether collaboration can outperform competition.

0197

Reposted by @renebekkers.bsky.social

Christopher Barrie @cbarrie.bsky.social · 10/06/2026

Very pleased to have been on the leadership team for this paper! LLMs are already being used all over behavioural science. But it is often pretty hard to work out exactly what has been done, and therefore how much confidence to place in the results. www.nature.com/articles/s41...

nature.com

A reporting checklist for large language models in behavioural science - Nature Human Behaviour

Large language models offer new opportunities for behavioural science, but their rapid evolution poses challenges for research rigour. We introduce a consensus-based reporting checklist to improve tra...

16127

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 10/06/2026

My PhD Sajedeh Rasti has published a World View piece in Nature today: "Don’t compete, collaborate: why collective funding applications are the future" www.nature.com/articles/d41... She discusses how she successfully brought together a team of meta-scientists to collaboratively apply for funding.

nature.com

Don’t compete, collaborate: why collective funding applications are the future

Scientists with disparate expertise writing grants together can identify knowledge gaps and drive progress — but systems must change to incentivize them.

16324

Reposted by @renebekkers.bsky.social

John Holbein @johnholbein1.bsky.social · 05/06/2026

Only 9.9% of top Sociology articles from 2019-2024 provided replication packages. And ≈ half of those with replication packages could not be verified due to missing or incomplete materials. www.econstor.eu/bitstream/10...

13310

Reposted by @renebekkers.bsky.social

Institute for Replication @i4replication.bsky.social · 29/05/2026

1/ Can AI help researchers check whether published social science results actually reproduce? In our new PNAS paper, we tested this directly in the AI Replication Games: 288 researchers, 103 teams, and real replication packages from quantitative social science.

36337

Reposted by @renebekkers.bsky.social

Michèle Nuijten @michelenuijten.bsky.social · 27/05/2026

Should automated tools for peer review come with a "nutritional label" for validity?🥪 And if so, what should be on it? Join my workshop on June 5th @ the META/e Conference on Transparency, Technology and AI in Peer Review in Eindhoven! Results will inform our FAIR validation framework. Details >>

A table with the same layout as a nutritional label, but instead filled out with Validation Facts. Categories include tool information, ground truth, performance, and limitations & generalizability.

1176

Reposted by @renebekkers.bsky.social

Daniel Lakens @lakens.bsky.social · 27/05/2026

Excellent news! Nature is expanding Registered Reports to all the fields in which they publish! A great result by all those proposing this strong solution to publication bias and selective reporting (and getting expert feedback before data collection)! www.nature.com/articles/d41...

114273