Reposted by Jamie CumminsLisa DeBruine @debruine.bsky.social · 13/10/2026This post does a great job of articulating why the development and release of metacheck has been pretty slow compared to other tools. One of our core values is validation, which is one of the things that has not been made a lot faster by LLMs. www.scienceverse.org/metacheck/#o... 2376
Reposted by Jamie CumminsNeuroskeptic @neuroskeptic.bsky.social · 12/10/2026This is an astonishing @pubpeer.com post: pubpeer.com/publications... The author compares data points from a figure against random numbers produced with a certain seed, finding r > 0.999999996! 🫠 Could there be any possible innocent explanation? The paper: www.nature.com/articles/s41...nature.comTowards a general-purpose foundation model for functional MRI analysis - Nature Biomedical EngineeringThe Neuroimaging Foundation Model with Spatial–Temporal Optimized and Representation Modelling (NeuroSTORM) learns generalizable representations from 4D functional MRI volumes, enabling accurate perfo... 46219
Reposted by Jamie CumminsLukas Jung @lhdjung.bsky.social · 12/10/2026Many new forensic meta-science tools aim to detect trustworthiness problems in published research at scale. Yet how trustworthy are these tools themselves? We – @ianhussey.mmmdata.io, @jamiecummins.bsky.social, and I – have some concerns. 071
Reposted by Jamie CumminsDaniel Lakens @lakens.bsky.social · 07/10/20269 days left to submit an abstract to the 9th edition of the Perspectives on Scientific Error conference that will take place in Copenhagen, Denmark, 27–29 January 2027, errorsin.science. It's my favorite conference combining philosophy of science, metascience, and statistics.errorsin.sciencePerspectives on Scientific Error 2027 · PSE9Johanna Einsiedler, University of Copenhagen & University of Basel, johanna.einsiedler@sodas.ku.dk 0139
Reposted by Jamie CumminsIan Hussey @ianhussey.mmmdata.io · 09/10/2026Our reflections on the recent "AI and Research Integrity" meeting in Montreal: We are somewhat worried that many AI trustworthiness assessment tools are not trustworthy. @lhdjung.bsky.social @jamiecummins.bsky.social trustworthy.scientific.claims/posts/trustw...trustworthy.scientific.claimsTrustworthiness assessment tools must be trustworthy 17834
Jamie Cummins @jamiecummins.bsky.social · 09/10/2026If (forensic) metascientists want to hold others to certain standards of rigor based on the tools they build, then we must also hold ourselves to these same standards, as we (@ianhussey.mmmdata.io, @lhdjung.bsky.social, and I) argue in this blog post. 081
Reposted by Jamie Cumminssimine vazire @simine.com · 07/10/2026Another special issue at Psych Science! We invite “submissions that advance the field through critical appraisal of articles previously published by Psychological Science” Please share! www.psychologicalscience.org/publications...psychologicalscience.orgSpecial Issue Call for Submissions: “Appraising the Foundations of Psychological Science”Editors: Tom Beckers and Heather UrryOne way a journal can earn its reputation for quality is to take active steps to vet the articles that appear within its pages after publication. To that end, Psyc... 610361
Reposted by Jamie CumminsMartin Plöderl @ploederl.bsky.social · 01/10/2026🚨 Paper Alert 🚨 Our new study on the impact of a single small zombie trial on fluoxetine's overall efficacy estimation is out. With great colleagues @floriannaudet.bsky.social @richlyus.bsky.social and Gert van Valkenhoef 🧟♂️🧟♀️🧟 A longer 🧵 (1/n)onlinelibrary.wiley.comA Re‐Appraisal of Three Network Meta‐Analyses to Explain the Discrepancy in Findings for the Efficacy of Fluoxetine for the Treatment of Depression in Children and AdolescentsObjective To explain discrepant findings for fluoxetine's efficacy in three influential network meta-analyses (NMAs) of treatments for pediatric depression, which led to conflicting clinical recomme..... 52112
Reposted by Jamie CumminsRichard Van Noorden @richvn.bsky.social · 02/10/2026NEW: researchers say one 'zombie' study in Iran, 20 years ago, has skewed the world's medical evidence base for using Prozac to treat depression in children. They stop short of saying don't use Prozac this way, but want clinical guidelines revised. @nature.com www.nature.com/articles/d41...nature.comProzac use for childhood depression skewed by single flawed medical trialOne small study in Iran swayed meta-analyses in favour of antidepressant use in children, researchers say. 12510
Reposted by Jamie CumminsBennett Institute for Applied Data Science @bennettoxford.bsky.social · 02/10/2026📣Cory Doctorow will deliver the closing keynote at the 2026 Bennett Oxford Symposium! Open to the public. Already attending? No need to register again. 🔗 www.eventbrite.co.uk/e/cory-doctoro… 📸 Anna Olthoff 052
Reposted by Jamie CumminsHenry Carnell @hencarnell.bsky.social · 01/10/2026New research: “AI-generated survey results consistently differ from human surveys“pewresearch.orgHow well synthetic samples replicate public opinionSynthetic survey respondents tend to exaggerate differences between demographic subgroups and portray some responses as nearly universal. 12612
Jamie Cummins @jamiecummins.bsky.social · 01/10/2026Looking sharp (the building too!). Congrats Dirk! 030
Reposted by Jamie CumminsDirk Wulff @dirkwulff.bsky.social · 01/10/2026📣 Delighted to share that I have joined WU (Vienna University of Economics and Business) as Professor of Artificial Intelligence and Behavioral Science. Announcements for at least one postdoc-level position will follow soon. Please reach out if you are interested. 68016
Reposted by Jamie CumminsIan Hussey @ianhussey.mmmdata.io · 29/09/2026Hayes has been banging this drum about idionomic analyses for 6+ years, with folk like @jamiecummins.bsky.social pointing out that its reinventing-the-wheel-but-worse for just as long. My experience is that the ACT community is insufficiently interested and aware of existing methods+stats work. 12312
Jamie Cummins @jamiecummins.bsky.social · 28/09/2026My lukewarm take is that when people disagree about whether LLMs can do X (be it reasoning, creativity, etc), the substantive disagreement is really about what X means, rather than about LLMs themselves. Aka Wittgenstein was right 1184
Reposted by Jamie CumminsMagnus Johansson @pgmj.bsky.social · 28/09/2026Our commentary "A Revolution in Search of an Estimand: On the Idionomic Proposal of Hayes et al. (2026)" is now available as a preprint: osf.io/preprints/ps... With @rpsychologist.com, @matti.vuorre.com, and Nils Hentati Isacsson.osf.ioOSF 03218
Reposted by Jamie CumminsInstitute for Replication @i4replication.bsky.social · 25/09/2026Over the past two years, I4R partnered with Psychological Science (@psychscience.bsky.social) on a large reproducibility project. 67 independent teams reproduced 64 articles published in 2024–2025, about 44% of eligible papers. Paper: www.econstor.eu/bitstream/10... Here's what we learned 🧵 19260
Jamie Cummins @jamiecummins.bsky.social · 25/09/2026Right side of the photo - the middle finger has clear morphing on it. Definitely AI-generated/manipulated. 110
Reposted by Jamie CumminsSaloni @scientificdiscovery.dev · 25/09/2026Happy Friday! Iʼve created a game for you to play! Will AI cure all diseases in 10 years? Inspired by bottlenecks discourse, I created a game to help build your intuitions. Play it here: game.clinicaltrialsabundance.blog Naturally, it’s bottle-themed with bubbles and there are leaderboards too. 33810
Jamie Cummins @jamiecummins.bsky.social · 24/09/2026I sincerely hope this paper will be seen as a turning point in how the medical sciences think and approach the issue of trustworthiness assessment (and I hope the social sciences follow suit!). Congrats to @jdwilko.bsky.social and colleagues (incl. @ianhussey.mmmdata.io and @lhdjung.bsky.social)! 12210
Reposted by Jamie CumminsThe Cochrane Collaboration @cochrane.org · 24/09/2026INSPECT-SR – a tool for assessing the trustworthiness of randomized controlled trials – has been published today in @bmj.com, giving researchers a structured way to identify problematic studies before they're included in systematic reviews. 🔗 www.cochrane.org/about-us/new... #ResearchIntegrity 03224
Reposted by Jamie CumminsRandy Ellis @randalljellis.bsky.social · 23/09/2026Here’s my conversation with Chris Chambers on Metascience Matters. Chris was the first journal editor to offer registered reports, starting in 2013 at Cortex. youtu.be/PgZjgZlxsYw?...youtu.beRegistered reports | Chris ChambersYouTube video by Metascience Matters 11912
Reposted by Jamie CumminsAlexander Wuttke @kunkakom.bsky.social · 23/09/2026We are hiring at LMU Munich: two PhD positions and one postdoc on democracy, political psychology and computational social science. Start: January 2027 or as soon as possible. We review applications on a rolling basis until October 20. Details below. Reposts very welcome! 1/7 26082
Jamie Cummins @jamiecummins.bsky.social · 23/09/2026We just published a new paper (led by Love von Schreeb) demonstrating how large language models can be used to link published results to trial registrations with very high accuracy! Now out in Journal of Clinical Epidemiology - check it out: www.jclinepi.com/article/S089...jclinepi.comTrialScout links published results to trial registrations using a large language modelMultiple stakeholders need to locate results of registered clinical trials but frequently struggle to find them. Summary results of clinical trials are often not published in trial registries, and pub... 1187
Reposted by Jamie CumminsEileen Clancy @eileenclancy.bsky.social · 16/09/2026RegCheck leverages LLMs and AI to automatically compare preregistration plans with scientific publications. This lets researchers effortlessly identify if and how executed studies deviated from the initial plan regcheck.appregcheck.appRegCheckRegCheck is an AI tool to compare preregistrations with papers instantly. 021
Jamie Cummins @jamiecummins.bsky.social · 15/09/2026I mean, I would never miss such an opportunity ;) 110
Reposted by Jamie CumminsRafael M Batista @rafmbatista.bsky.social · 13/09/2026I’ve got a @semble.so Collection of #AI tools I’ve come across specifically for academic research. Most of these Ive tried and found them user-friendly-ish. If you use something not on the list, please share it :) semble.so/profile/rafm...semble.soAI Tools for Academic Research (by Rafael M Batista) — SembleView Rafael M Batista's collection on Semble 195
Jamie Cummins @jamiecummins.bsky.social · 13/09/2026I actually did know that but just didn’t want to seem like I was bragging 🙇 120
Jamie Cummins @jamiecummins.bsky.social · 13/09/2026Makes sense, I similarly list “TED speaker” on my CV because @scientificdiscovery.dev once used RegCheck 150
Reposted by Jamie CumminsSandra Grinschgl, (she/her) @grinschglsandra.bsky.social · 13/09/2026Happy Sunday morning! This article was finally retracted: psycnet.apa.org/fulltext/202... Unfortunately the author continues as is - no accountability. Curious if we will see more sloppy work from her published in Psych journals soon-ish 🧐psycnet.apa.orgAPA PsycNet 094
Reposted by Jamie CumminsLuna Fazio @lunafazio.bsky.social · 09/09/2026Had a Zoom meeting earlier today where a lot of great exchanges were happening in the chat, so we wanted to save the log for later. Zoom exports a plain text file, so I made a tool to convert that into something that's more convenient to interact with. Sharing in case anyone else finds it useful! 153
Reposted by Jamie CumminsRuben C. Arslan @ruben.the100.ci · 11/09/2026An interesting thing is happening to me at the moment. I have a standing bug bounty policy for my first authored scientific work. That means, I've committed to pay people who report bugs in my code, up to 750€ for severe errors. Since GPT-6 Astra came out, I'm now getting several bug reports daily. 39334
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026Machine learning and generative LLMs are subject to many of the same drawbacks and issues as general empirical research in the typical publishing system. In this piece now published at NHB we argue that, like for empirical research, ML and LLMs can benefit from the use of Registered Reports. 02210
Reposted by Jamie CumminsNature Human Behaviour @nathumbehav.nature.com · 10/09/2026How Registered Reports can improve trust in machine learning researchdlvr.itHow Registered Reports can improve trust in machine learning researchNature Human Behaviour, Published online: 10 September 2026; doi:10.1038/s41562-026-02586-2How Registered Reports can improve trust in machine learning research 084
Reposted by Jamie CumminsKristin Jankowsky @krijanko.bsky.social · 10/09/2026Out now, with @jamiecummins.bsky.social & @uschroeders.bsky.social: why ML research benefits from Registered Reports, and where the format reaches its limits. Table 1 goes through 7 scenarios, from confirmatory prediction to climbing a benchmark leaderboard. Free view-only link: rdcu.be/MCamygrKHKxardcu.be 042
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026Or is your point indeed that the variation in LLM outputs will largely be a function of the changing context retrieved through web search (in which case, perhaps a control arm without web search would serve to strengthen this point)? Anyway, super interesting work! 100
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026Very cool! Do you also run each configuration multiple times in a given day to handle/model same-day stochasticity? And would you have thoughts about potentially constraining web search, given that the context retrieved from this search during CoT will necessarily vary? 100
Jamie Cummins @jamiecummins.bsky.social · 09/09/2026Thanks! I was happy to be a small part of this work, definitely have an eye on some integrations with RegCheck going forward too 020
Reposted by Jamie CumminsJournal of Clinical Epidemiology @jclinepi.bsky.social · 07/09/2026TrialScout links published results to trial registrations using a large language model - Journal of Clinical Epidemiology www.jclinepi.com/article/S089...jclinepi.comTrialScout links published results to trial registrations using a large language modelMultiple stakeholders need to locate results of registered clinical trials but frequently struggle to find them. Summary results of clinical trials are often not published in trial registries, and pub... 153
Reposted by Jamie CumminsInstitute for Replication @i4replication.bsky.social · 08/09/20261/ What is the prevalence of personally identifiable information (PII) in author-collected data publicly shared in the social sciences? We audited 327 replication packages from 11 leading journals in economics, political science, and psychology Answer: surprisingly high 🧵 25537
Reposted by Jamie CumminsJoachim Baumann @joachimbaumann.bsky.social · 04/09/2026After Science News, our recent ICML paper also got featured in @science.org! Article: www.science.org/content/arti... Paper: arxiv.org/pdf/2605.03202 @dirkhovy.bsky.social @sanmikoyejo.bsky.social #science #PeerReview #research 1104
Reposted by Jamie CumminsSabrina Norwood @sabrinanorwood.bsky.social · 03/09/2026Here’s a video showing how it looks for the user! With example survey questions at open and close 1144
Reposted by Jamie CumminsSabrina Norwood @sabrinanorwood.bsky.social · 03/09/2026Early in this study I knew I’d want to run many more observational and intervention studies using this method. And I wanted to make it easier for myself and other researchers to do it So @ianhussey.mmmdata.io & I spent time developing an iOS app that can collect these data and deliver interventions 1223
Reposted by Jamie CumminsSabrina Norwood @sabrinanorwood.bsky.social · 02/09/2026🚨 New preprint: In this paper we make the case for measuring social media use at the level of naturally occurring sessions (i.e., beginning when the app/site is opened and ending when it is closed). Across 7 days and 161 participants this gave us nearly 4,000 paired sessions of use 🧵 25916
Reposted by Jamie Cumminssimine vazire @simine.com · 02/09/2026Psych Science has retracted Ariely & Wertenbroch (2002). The second author requested the retraction. Both authors agreed to it. journals.sagepub.com/doi/full/10.... So grateful to @datacolada.bsky.social for these blog posts and the work behind them: datacolada.org/138 and datacolada.org/139 1/2journals.sagepub.comSage Journals: Discover world-class researchSubscription and open access journals from Sage, the world's leading independent academic publisher. 426694
Reposted by Jamie CumminsJsonGeller @jgeller1phd.bsky.social · 02/09/2026Are preregs just a way for people to virtual signal now? Comparing pregregs to published papers is very sobering. Also shout out to @debruine.bsky.social and @lakens.bsky.social for metacheck and regcheck @ianhussey.mmmdata.io for making this easy! 5175