Matthew B. Jané @matthewbjane.bsky.social · 20/09/2026I will make a separate post summarizing the findings for this report. 020
Matthew B. Jané @matthewbjane.bsky.social · 20/09/2026Some additional things: Version control through Zenodo is fully set up. Each report has a change log and a DOI for each version. New cover pages and some new aesthetic changes to the PDF versions of the audit reports. Thanks @gidmk.bsky.social for posting the report on pubpeer! 130
Matthew B. Jané @matthewbjane.bsky.social · 20/09/2026The Audit Clinical Evidence (ACE) Initiative posted an update to the first audit report (wrt Abdelbasset et al 2019). @gidmk.bsky.social found some additional errors in the baseline tables that have now been included in the report. auditclinicalevidence.org/reports/ACE2...auditclinicalevidence.orgAudit of Abdelbasset et al. (2019) – ACEThree 2019 RCTs on exercise for depression in heart failure: overlapping trial arms, duplicated PHQ-9 data, impossible statistics and baseline p-values. 25 integrity violations; retraction recommended... 1134
Matthew B. Jané @matthewbjane.bsky.social · 19/09/2026Yes I will be including that in the next update today! It already had that functionality set up but zenodo was sporadically down. 111
Matthew B. Jané @matthewbjane.bsky.social · 15/09/2026Oh yes these papers were identified along the way when looking at the field, but the MEP report had a different focus (particularly on one meta-analysis) 010
Reposted by Matthew B. JanéHealth Nerd @gidmk.bsky.social · 15/09/2026New piece on a great project from @matthewbjane.bsky.social looking at problematic research. Also, extremely worrisome issues in studies of exercise for depression. gidmk.substack.com/p/how-do-we-...gidmk.substack.comHow Do We Deal With Seriously Problematic Research?A new solution to a very serious issue 12910
Matthew B. Jané @matthewbjane.bsky.social · 15/09/2026So one of the other articles has a different person listed as the editor that’s not in the author list so I don’t think so 100
Matthew B. Jané @matthewbjane.bsky.social · 15/09/2026No connection. I no longer work at the medical evidence project. Just finally finishing my PhD now! 110
Matthew B. Jané @matthewbjane.bsky.social · 15/09/2026Believe it or not it’s explicitly stated in the article page for the journal 110
Matthew B. Jané @matthewbjane.bsky.social · 15/09/2026I’ve always disagreed with his premise. This concept of “methodological noise” is ill-defined. He used it once in the context of meta-analysis which I found particularly odd because that is explicitly is being modeled within the heterogeneity component. He suggested it was some unobserved component 120
Reposted by Matthew B. JanéMatthew B. Jané @matthewbjane.bsky.social · 13/09/2026I’ve been quiet on social media lately, but today I’m launching Audit Clinical Evidence (ACE), an initiative I’ve been developing over the past year. Take a look: AuditClinicalEvidence.orgauditclinicalevidence.orgACEIndependent audits of published clinical research for statistical errors, data inconsistencies, and research integrity violations. 89232
Matthew B. Jané @matthewbjane.bsky.social · 14/09/2026Yes I have read that. I actually should have mentioned that in the report to add some circumstantial evidence for the publication mill section. I will update that once I have the version control with zenodo set up. 010
Matthew B. Jané @matthewbjane.bsky.social · 13/09/2026Hahaha yes this and my dissertation had me taking a bit of a break from social media. I probably won’t be doing much blogging and instead focusing on this venture. 221
Matthew B. Jané @matthewbjane.bsky.social · 13/09/2026There is an RSS feed, but also I’ll post each report with its corresponding ID. We may make an ACE account on here as well, but that’s TBD 030
Matthew B. Jané @matthewbjane.bsky.social · 13/09/2026We use the ACE Research Integrity Violations (ACE-RIV) taxonomy to classify and document these issues systematically and consistently. Many more posts to come! 190
Matthew B. Jané @matthewbjane.bsky.social · 13/09/2026ACE documents errors, inconsistencies, and potential research misconduct, including falsification, in clinical and medical research. Our audit reports are evidence-based and openly available, with supporting analyses and materials shared for transparency and independent scrutiny. 1110
Matthew B. Jané @matthewbjane.bsky.social · 13/09/2026I’ve been quiet on social media lately, but today I’m launching Audit Clinical Evidence (ACE), an initiative I’ve been developing over the past year. Take a look: AuditClinicalEvidence.orgauditclinicalevidence.orgACEIndependent audits of published clinical research for statistical errors, data inconsistencies, and research integrity violations. 89232
Matthew B. Jané @matthewbjane.bsky.social · 21/07/2026Agreed. People seem to not like the fact that the meta-analytic estimand is the *distribution* of effects. The mean effect is often meaningless and therefore prediction intervals should be seen as the primary statistical result of a meta-analysis. 0182
Matthew B. Jané @matthewbjane.bsky.social · 14/06/2026Yeah. The meta-analysis was put on the backburner for the time being. But this is a good summary. Also, like a lot of the RCTs are just not very good. They almost all are “high risk of bias” 020
Matthew B. Jané @matthewbjane.bsky.social · 15/05/2026This should be mandatory reading for every graduate student. 0191
Matthew B. Jané @matthewbjane.bsky.social · 28/04/2026Thank you! Any ideas of what my next blog post should be? 240
Matthew B. Jané @matthewbjane.bsky.social · 15/04/2026In response to the OP: I think this would be very context dependent so it’s kind of hard to give an answer 140
Matthew B. Jané @matthewbjane.bsky.social · 15/04/2026Just redownloaded this app. I’m just now getting back to being active on my socials 150
Reposted by Matthew B. JanéThe Unjournal (Unjournal.org) @unjournal.bsky.social · 10/02/2026As specified and posted here: info.unjournal.org/evaluator-pr... I am making the draw for these now using the closing value of S&P 500 today to 2 decimal places, 6,941.81. --> Winners: @matthewbjane.bsky.social and Anon evaluator (of Wellbeing paper). Will make the payments soon. 021
Matthew B. Jané @matthewbjane.bsky.social · 11/02/2026Thank you so much @unjournal.bsky.social !! 040
Reposted by Matthew B. JanéThe Unjournal (Unjournal.org) @unjournal.bsky.social · 23/12/2025Congrats on being nominated Edoardo Masset, Hugh Sharma-Waddington, @davidreiley.bsky.social , @cannoncloud.bsky.social , @matthewbjane.bsky.social, & anon. hero. And thanks for the strong work! 031
Reposted by Matthew B. JanéThe Unjournal (Unjournal.org) @unjournal.bsky.social · 23/12/2025We want your feedback – which evaluations merit the top award, and why? Total prize pool: $6600 www.unjournal.org/news/unjourn... Anonymous: coda.io/form/The-Unj...unjournal.orgUnjournal 2024-25 Evaluator Prize Nominations; inviting feedback — The UnjournalThe Unjournal commissions expert public evaluation of research with potential for global impact. We recognize and reward the most useful, insightful, and credible evaluation work, including through ou... 123
Matthew B. Jané @matthewbjane.bsky.social · 08/11/2025I think what ultimately will matter in this context is that it needs to be consistent for each study. I see the argument with either the resid SD or the raw within group SD, but I would worry about a meta-analyst mixing and matching them in a meta-analysis 010
Matthew B. Jané @matthewbjane.bsky.social · 28/06/2025Wanna write a meta-analysis error report? matthewbjane.quarto.pub/meta-analysi...matthewbjane.quarto.pubMeta-Analysis Error Reports 071
Matthew B. Jané @matthewbjane.bsky.social · 19/06/2025Update on Error Report #2: The impact of acoustic stimulation during sleep on memory and sleep architecture: A meta-analysis The article has been retracted by agreement with the journal editor and authors #metascience 071
Reposted by Matthew B. JanéNick Holmes @thehandlab.bsky.social · 26/05/2025"Meta-meta-analyses trade precision & rigor for expediency. If your aim is rigor, there’s no substitute for doing the hard work." Preach! I once read a #meta-analysis that mistook SE for SD. Since then I am suspicious of all such studies. Do the hard work! matthewbjane.github.io/blog-posts/b...matthewbjane.github.ioWhen Evidence Synthesis Obscures: A Critique of Meta-Meta-Analyses – Matthew B. JanéMeta-analyses are often seen as the gold standard of evidence synthesis–systematic, quantitative, and rigorous (at least when done well). So it’s no surprise that some researchers have taken things a ... 162
Reposted by Matthew B. JanéPaul Whaley @dangerwhale.bsky.social · 25/05/2025This is very nice from @matthewbjane.bsky.social as to why umbrella reviews are basically useless (sorry!), essentially because they compound up the issues with the meta analyses they include. I have the same concerns but articulate them less eloquently. matthewbjane.github.io/blog-posts/b...matthewbjane.github.ioWhen Evidence Synthesis Obscures: A Critique of Meta-Meta-Analyses – Matthew B. JanéMeta-analyses are often seen as the gold standard of evidence synthesis–systematic, quantitative, and rigorous (at least when done well). So it’s no surprise that some researchers have taken things a ... 0183
Matthew B. Jané @matthewbjane.bsky.social · 24/05/2025I think my reaction to the first time I saw it was something along the lines of “wait you can just do that?” 010
Matthew B. Jané @matthewbjane.bsky.social · 24/05/2025Yeah I might do that. Maybe I’ll make it into a pre-print for now 180
Reposted by Matthew B. JanéMatthew B. Jané @matthewbjane.bsky.social · 23/05/2025Happy Friday everyone! I just posted what I think is an important blog post on my website. It is a critique of meta-meta-analyses: meta-analyses of meta-analyses. Link: matthewbjane.github.io/blog-posts/b... #stats #metascience 68727
Matthew B. Jané @matthewbjane.bsky.social · 23/05/2025Happy Friday everyone! I just posted what I think is an important blog post on my website. It is a critique of meta-meta-analyses: meta-analyses of meta-analyses. Link: matthewbjane.github.io/blog-posts/b... #stats #metascience 68727
Matthew B. Jané @matthewbjane.bsky.social · 23/05/2025I’ve been hearing it’s good. No spoilers please, still gotta watch it 010
Matthew B. Jané @matthewbjane.bsky.social · 29/04/2025Most meta-analyses are bad, some are useful 0152
Reposted by Matthew B. JanéBrendan Nyhan @brendannyhan.bsky.social · 29/04/2025I'm calling for a total and complete shutdown of JAMA Pediatrics until we can figure out what is going on goodscience.substack.com/p/the-resear... I mean, come ON 1017445
Matthew B. Jané @matthewbjane.bsky.social · 28/04/2025(#metascience) A good article by @stuartbuck.bsky.social on the Fluoride and IQ meta-analysis: goodscience.substack.com/p/the-resear.... Summarizes some of the points in our forensic review of it (see the forensic review as a pre-print here: osf.io/preprints/os...). Very nice excerpt below: 174
Matthew B. Jané @matthewbjane.bsky.social · 25/04/2025We tried to. Best we can do for now is submit to meta-psychology and post a pub-peer comment 010
Reposted by Matthew B. JanéMatthew B. Jané @matthewbjane.bsky.social · 23/04/2025Error Report #4: matthewbjane.quarto.pub/meta-analysi... The meta-analysis on Fluoride and IQ by Taylor et al (2025) is methodologically flawed and has data integrity issues. The full report by me, @jamesheathers.bsky.social, and @drg.bsky.social is on OSF: osf.io/preprints/os... #metascience 46821
Matthew B. Jané @matthewbjane.bsky.social · 23/04/2025Error Report #4: matthewbjane.quarto.pub/meta-analysi... The meta-analysis on Fluoride and IQ by Taylor et al (2025) is methodologically flawed and has data integrity issues. The full report by me, @jamesheathers.bsky.social, and @drg.bsky.social is on OSF: osf.io/preprints/os... #metascience 46821
