Sign in

Jamie Cummins

@jamiecummins.bsky.social
3.1K followers 819 following 1.1K posts

Advanced postdoc at University of Bern. Also sometimes @bennettoxford.bsky.social. Metascience stuff. Creator and developer of RegCheck (regcheck.app). STAR Editor at Psych Science. @error.reviews 🇮🇪

PostsRepliesMedia
Reposted by Jamie Cummins
Lisa DeBruine @debruine.bsky.social · 13/10/2026
This post does a great job of articulating why the development and release of metacheck has been pretty slow compared to other tools. One of our core values is validation, which is one of the things that has not been made a lot faster by LLMs. www.scienceverse.org/metacheck/#o...
2376
Reposted by Jamie Cummins
Neuroskeptic @neuroskeptic.bsky.social · 12/10/2026
This is an astonishing @pubpeer.com post: pubpeer.com/publications... The author compares data points from a figure against random numbers produced with a certain seed, finding r > 0.999999996! 🫠 Could there be any possible innocent explanation? The paper: www.nature.com/articles/s41...
nature.com
Towards a general-purpose foundation model for functional MRI analysis - Nature Biomedical Engineering
The Neuroimaging Foundation Model with Spatial–Temporal Optimized and Representation Modelling (NeuroSTORM) learns generalizable representations from 4D functional MRI volumes, enabling accurate perfo...
46219
Reposted by Jamie Cummins
Lukas Jung @lhdjung.bsky.social · 12/10/2026
Many new forensic meta-science tools aim to detect trustworthiness problems in published research at scale. Yet how trustworthy are these tools themselves? We – @ianhussey.mmmdata.io, @jamiecummins.bsky.social, and I – have some concerns.
071
Reposted by Jamie Cummins
Daniel Lakens @lakens.bsky.social · 07/10/2026
9 days left to submit an abstract to the 9th edition of the Perspectives on Scientific Error conference that will take place in Copenhagen, Denmark, 27–29 January 2027, errorsin.science. It's my favorite conference combining philosophy of science, metascience, and statistics.
errorsin.science
Perspectives on Scientific Error 2027 · PSE9
Johanna Einsiedler, University of Copenhagen & University of Basel, johanna.einsiedler@sodas.ku.dk
0139
Reposted by Jamie Cummins
Ian Hussey @ianhussey.mmmdata.io · 09/10/2026
Our reflections on the recent "AI and Research Integrity" meeting in Montreal: We are somewhat worried that many AI trustworthiness assessment tools are not trustworthy. @lhdjung.bsky.social @jamiecummins.bsky.social trustworthy.scientific.claims/posts/trustw...
trustworthy.scientific.claims
Trustworthiness assessment tools must be trustworthy
17834
Jamie Cummins @jamiecummins.bsky.social · 09/10/2026
If (forensic) metascientists want to hold others to certain standards of rigor based on the tools they build, then we must also hold ourselves to these same standards, as we (@ianhussey.mmmdata.io, @lhdjung.bsky.social, and I) argue in this blog post.
081
Reposted by Jamie Cummins
simine vazire @simine.com · 07/10/2026
Another special issue at Psych Science! We invite “submissions that advance the field through critical appraisal of articles previously published by Psychological Science” Please share! www.psychologicalscience.org/publications...
psychologicalscience.org
Special Issue Call for Submissions: “Appraising the Foundations of Psychological Science”
Editors: Tom Beckers and Heather UrryOne way a journal can earn its reputation for quality is to take active steps to vet the articles that appear within its pages after publication. To that end, Psyc...
610361
Reposted by Jamie Cummins
Martin Plöderl @ploederl.bsky.social · 01/10/2026
🚨 Paper Alert 🚨 Our new study on the impact of a single small zombie trial on fluoxetine's overall efficacy estimation is out. With great colleagues @floriannaudet.bsky.social @richlyus.bsky.social and Gert van Valkenhoef 🧟‍♂️🧟‍♀️🧟 A longer 🧵 (1/n)
onlinelibrary.wiley.com
A Re‐Appraisal of Three Network Meta‐Analyses to Explain the Discrepancy in Findings for the Efficacy of Fluoxetine for the Treatment of Depression in Children and Adolescents
Objective To explain discrepant findings for fluoxetine's efficacy in three influential network meta-analyses (NMAs) of treatments for pediatric depression, which led to conflicting clinical recomme.....
52112
Reposted by Jamie Cummins
Richard Van Noorden @richvn.bsky.social · 02/10/2026
NEW: researchers say one 'zombie' study in Iran, 20 years ago, has skewed the world's medical evidence base for using Prozac to treat depression in children. They stop short of saying don't use Prozac this way, but want clinical guidelines revised. @nature.com www.nature.com/articles/d41...
nature.com
Prozac use for childhood depression skewed by single flawed medical trial
One small study in Iran swayed meta-analyses in favour of antidepressant use in children, researchers say.
12510
Reposted by Jamie Cummins
Bennett Institute for Applied Data Science @bennettoxford.bsky.social · 02/10/2026
📣Cory Doctorow will deliver the closing keynote at the 2026 Bennett Oxford Symposium! Open to the public. Already attending? No need to register again. 🔗 www.eventbrite.co.uk/e/cory-doctoro… 📸 Anna Olthoff
052
Reposted by Jamie Cummins
Henry Carnell @hencarnell.bsky.social · 01/10/2026
New research: “AI-generated survey results consistently differ from human surveys“
pewresearch.org
How well synthetic samples replicate public opinion
Synthetic survey respondents tend to exaggerate differences between demographic subgroups and portray some responses as nearly universal.
12612
Jamie Cummins @jamiecummins.bsky.social · 01/10/2026
Looking sharp (the building too!). Congrats Dirk!
030
Reposted by Jamie Cummins
Dirk Wulff @dirkwulff.bsky.social · 01/10/2026
📣 Delighted to share that I have joined WU (Vienna University of Economics and Business) as Professor of Artificial Intelligence and Behavioral Science. Announcements for at least one postdoc-level position will follow soon. Please reach out if you are interested.
68016
Reposted by Jamie Cummins
Ian Hussey @ianhussey.mmmdata.io · 29/09/2026
Hayes has been banging this drum about idionomic analyses for 6+ years, with folk like @jamiecummins.bsky.social pointing out that its reinventing-the-wheel-but-worse for just as long. My experience is that the ACT community is insufficiently interested and aware of existing methods+stats work.
12312
Jamie Cummins @jamiecummins.bsky.social · 28/09/2026
My lukewarm take is that when people disagree about whether LLMs can do X (be it reasoning, creativity, etc), the substantive disagreement is really about what X means, rather than about LLMs themselves. Aka Wittgenstein was right
1184
Reposted by Jamie Cummins
Magnus Johansson @pgmj.bsky.social · 28/09/2026
Our commentary "A Revolution in Search of an Estimand: On the Idionomic Proposal of Hayes et al. (2026)" is now available as a preprint: osf.io/preprints/ps... With @rpsychologist.com, @matti.vuorre.com, and Nils Hentati Isacsson.
osf.io
OSF
03218
Reposted by Jamie Cummins
Institute for Replication @i4replication.bsky.social · 25/09/2026
Over the past two years, I4R partnered with Psychological Science (@psychscience.bsky.social) on a large reproducibility project. 67 independent teams reproduced 64 articles published in 2024–2025, about 44% of eligible papers. Paper: www.econstor.eu/bitstream/10... Here's what we learned 🧵
19260
Jamie Cummins @jamiecummins.bsky.social · 25/09/2026
Right side of the photo - the middle finger has clear morphing on it. Definitely AI-generated/manipulated.
110
Reposted by Jamie Cummins
Saloni @scientificdiscovery.dev · 25/09/2026
Happy Friday! Iʼve created a game for you to play! Will AI cure all diseases in 10 years? Inspired by bottlenecks discourse, I created a game to help build your intuitions. Play it here: game.clinicaltrialsabundance.blog Naturally, it’s bottle-themed with bubbles and there are leaderboards too.
33810
Jamie Cummins @jamiecummins.bsky.social · 24/09/2026
I sincerely hope this paper will be seen as a turning point in how the medical sciences think and approach the issue of trustworthiness assessment (and I hope the social sciences follow suit!). Congrats to @jdwilko.bsky.social and colleagues (incl. @ianhussey.mmmdata.io and @lhdjung.bsky.social)!
12210
Reposted by Jamie Cummins
The Cochrane Collaboration @cochrane.org · 24/09/2026
INSPECT-SR – a tool for assessing the trustworthiness of randomized controlled trials – has been published today in @bmj.com, giving researchers a structured way to identify problematic studies before they're included in systematic reviews. 🔗 www.cochrane.org/about-us/new... #ResearchIntegrity
“Systematic reviews are only as credible as the studies they include.” — phrased as a Cochrane senior research integrity editor; a magnifying glass rests on stacked papers in the background to emphasize careful scrutiny of research.
03224
Reposted by Jamie Cummins
Randy Ellis @randalljellis.bsky.social · 23/09/2026
Here’s my conversation with Chris Chambers on Metascience Matters. Chris was the first journal editor to offer registered reports, starting in 2013 at Cortex. youtu.be/PgZjgZlxsYw?...
youtu.be
Registered reports | Chris Chambers
YouTube video by Metascience Matters
11912
Reposted by Jamie Cummins
Alexander Wuttke @kunkakom.bsky.social · 23/09/2026
We are hiring at LMU Munich: two PhD positions and one postdoc on democracy, political psychology and computational social science. Start: January 2027 or as soon as possible. We review applications on a rolling basis until October 20. Details below. Reposts very welcome! 1/7
26082
Jamie Cummins @jamiecummins.bsky.social · 23/09/2026
We just published a new paper (led by Love von Schreeb) demonstrating how large language models can be used to link published results to trial registrations with very high accuracy! Now out in Journal of Clinical Epidemiology - check it out: www.jclinepi.com/article/S089...
jclinepi.com
TrialScout links published results to trial registrations using a large language model
Multiple stakeholders need to locate results of registered clinical trials but frequently struggle to find them. Summary results of clinical trials are often not published in trial registries, and pub...
1187
Reposted by Jamie Cummins
Eileen Clancy @eileenclancy.bsky.social · 16/09/2026
RegCheck leverages LLMs and AI to automatically compare preregistration plans with scientific publications. This lets researchers effortlessly identify if and how executed studies deviated from the initial plan regcheck.app
regcheck.app
RegCheck
RegCheck is an AI tool to compare preregistrations with papers instantly.
021
Jamie Cummins @jamiecummins.bsky.social · 15/09/2026
I mean, I would never miss such an opportunity ;)
110
Jamie Cummins @jamiecummins.bsky.social · 14/09/2026
Thanks for the inclusion!
010
Reposted by Jamie Cummins
Rafael M Batista @rafmbatista.bsky.social · 13/09/2026
I’ve got a @semble.so Collection of #AI tools I’ve come across specifically for academic research. Most of these Ive tried and found them user-friendly-ish. If you use something not on the list, please share it :) semble.so/profile/rafm...
semble.so
AI Tools for Academic Research (by Rafael M Batista) — Semble
View Rafael M Batista's collection on Semble
195
Jamie Cummins @jamiecummins.bsky.social · 14/09/2026
JCBS
120
Jamie Cummins @jamiecummins.bsky.social · 13/09/2026
I actually did know that but just didn’t want to seem like I was bragging 🙇
120
Jamie Cummins @jamiecummins.bsky.social · 13/09/2026
Makes sense, I similarly list “TED speaker” on my CV because @scientificdiscovery.dev once used RegCheck
150
Reposted by Jamie Cummins
Sandra Grinschgl, (she/her) @grinschglsandra.bsky.social · 13/09/2026
Happy Sunday morning! This article was finally retracted: psycnet.apa.org/fulltext/202... Unfortunately the author continues as is - no accountability. Curious if we will see more sloppy work from her published in Psych journals soon-ish 🧐
psycnet.apa.org
APA PsycNet
094
Reposted by Jamie Cummins
Luna Fazio @lunafazio.bsky.social · 09/09/2026
Had a Zoom meeting earlier today where a lot of great exchanges were happening in the chat, so we wanted to save the log for later. Zoom exports a plain text file, so I made a tool to convert that into something that's more convenient to interact with. Sharing in case anyone else finds it useful!
Example of what a zoom chat log looks like (fictional conversation)The chat log after being converted into an interactable html
153
Reposted by Jamie Cummins
Ruben C. Arslan @ruben.the100.ci · 11/09/2026
An interesting thing is happening to me at the moment. I have a standing bug bounty policy for my first authored scientific work. That means, I've committed to pay people who report bugs in my code, up to 750€ for severe errors. Since GPT-6 Astra came out, I'm now getting several bug reports daily.
39334
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026
Machine learning and generative LLMs are subject to many of the same drawbacks and issues as general empirical research in the typical publishing system. In this piece now published at NHB we argue that, like for empirical research, ML and LLMs can benefit from the use of Registered Reports.
02210
Reposted by Jamie Cummins
Nature Human Behaviour @nathumbehav.nature.com · 10/09/2026
How Registered Reports can improve trust in machine learning research
dlvr.it
How Registered Reports can improve trust in machine learning research
Nature Human Behaviour, Published online: 10 September 2026; doi:10.1038/s41562-026-02586-2How Registered Reports can improve trust in machine learning research
084
Reposted by Jamie Cummins
Kristin Jankowsky @krijanko.bsky.social · 10/09/2026
Out now, with @jamiecummins.bsky.social & @uschroeders.bsky.social: why ML research benefits from Registered Reports, and where the format reaches its limits. Table 1 goes through 7 scenarios, from confirmatory prediction to climbing a benchmark leaderboard. Free view-only link: rdcu.be/MCamygrKHKxa
rdcu.be
042
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026
Or is your point indeed that the variation in LLM outputs will largely be a function of the changing context retrieved through web search (in which case, perhaps a control arm without web search would serve to strengthen this point)? Anyway, super interesting work!
100
Jamie Cummins @jamiecummins.bsky.social · 10/09/2026
Very cool! Do you also run each configuration multiple times in a given day to handle/model same-day stochasticity? And would you have thoughts about potentially constraining web search, given that the context retrieved from this search during CoT will necessarily vary?
100
Jamie Cummins @jamiecummins.bsky.social · 09/09/2026
Thanks! I was happy to be a small part of this work, definitely have an eye on some integrations with RegCheck going forward too
020
Reposted by Jamie Cummins
Journal of Clinical Epidemiology @jclinepi.bsky.social · 07/09/2026
TrialScout links published results to trial registrations using a large language model - Journal of Clinical Epidemiology www.jclinepi.com/article/S089...
jclinepi.com
TrialScout links published results to trial registrations using a large language model
Multiple stakeholders need to locate results of registered clinical trials but frequently struggle to find them. Summary results of clinical trials are often not published in trial registries, and pub...
153
Reposted by Jamie Cummins
Institute for Replication @i4replication.bsky.social · 08/09/2026
1/ What is the prevalence of personally identifiable information (PII) in author-collected data publicly shared in the social sciences? We audited 327 replication packages from 11 leading journals in economics, political science, and psychology Answer: surprisingly high 🧵
25537
Jamie Cummins @jamiecummins.bsky.social · 04/09/2026
Nice!
010
Reposted by Jamie Cummins
Joachim Baumann @joachimbaumann.bsky.social · 04/09/2026
After Science News, our recent ICML paper also got featured in @science.org! Article: www.science.org/content/arti... Paper: arxiv.org/pdf/2605.03202 @dirkhovy.bsky.social @sanmikoyejo.bsky.social #science #PeerReview #research
Screenshot of an article in Science Magazine on the opportunities and risks of AI used for peer review featuring an ICML conference paper by Joachim Baumann et al., 2026.
1104
Reposted by Jamie Cummins
Sabrina Norwood @sabrinanorwood.bsky.social · 03/09/2026
Here’s a video showing how it looks for the user! With example survey questions at open and close
1144
Reposted by Jamie Cummins
Sabrina Norwood @sabrinanorwood.bsky.social · 03/09/2026
Early in this study I knew I’d want to run many more observational and intervention studies using this method. And I wanted to make it easier for myself and other researchers to do it So @ianhussey.mmmdata.io & I spent time developing an iOS app that can collect these data and deliver interventions
1223
Reposted by Jamie Cummins
Sabrina Norwood @sabrinanorwood.bsky.social · 02/09/2026
🚨 New preprint: In this paper we make the case for measuring social media use at the level of naturally occurring sessions (i.e., beginning when the app/site is opened and ending when it is closed). Across 7 days and 161 participants this gave us nearly 4,000 paired sessions of use 🧵
Depicts the data collection process. 1) participant opens TikTok, 2) open survey is prompted, 3) participant uses TikTok as normal, 4) participant closes TikTok, 5) close survey is prompted
25916
Reposted by Jamie Cummins
simine vazire @simine.com · 02/09/2026
Psych Science has retracted Ariely & Wertenbroch (2002). The second author requested the retraction. Both authors agreed to it. journals.sagepub.com/doi/full/10.... So grateful to @datacolada.bsky.social for these blog posts and the work behind them: datacolada.org/138 and datacolada.org/139 1/2
journals.sagepub.com
Sage Journals: Discover world-class research
Subscription and open access journals from Sage, the world's leading independent academic publisher.
426694
Jamie Cummins @jamiecummins.bsky.social · 02/09/2026
Thanks for the shout out :)
000
Reposted by Jamie Cummins
JsonGeller @jgeller1phd.bsky.social · 02/09/2026
Are preregs just a way for people to virtual signal now? Comparing pregregs to published papers is very sobering. Also shout out to @debruine.bsky.social and @lakens.bsky.social for metacheck and regcheck @ianhussey.mmmdata.io for making this easy!
5175