iiinova — where your digital self keeps evolving

František Bartoš @fbartos.bsky.social · 04/09/2026

And another one: > As Fig. 2 indicates, the funnel plot had a symmetrical distribution, indicating no publication bias in this meta-analysis. Besides, the results of the classic fail-safe N ... reveals that 6619 missing studies would be needed to nullify the effect size ...

110

František Bartoš @fbartos.bsky.social · 04/09/2026

Another meta-analyses on AI in education. > Fig. 2 provides a visual representation of the distribution of effect sizes, indicating that the effect sizes are nearly symmetrically distributed... This distribution suggests that there is no evidence of publication bias present...

121

František Bartoš @fbartos.bsky.social · 11/07/2026

@jaspstats.bsky.social released a new of the Meta-Analysis module -- if you have the latest version of JASP you should see a small blue download icon next to the Meta-Analysis module on the ribbon. You can update it via the Module Store (see screenshot below).

1100

František Bartoš @fbartos.bsky.social · 02/07/2026

I was presenting a poster at ISBA overviewing the RoBMA R package for Bayesian meta-analysis. Check it below to see a brief summary of the latest changes with the new 4.0 version and the future directions!

170

František Bartoš @fbartos.bsky.social · 18/06/2026

Out of curiosity I checked a the first author's google scholar. He seemed to have co-authored additional three meta-analyses last year. Two of those show quite unbelievable effect sizes again. (I guess the d = 2.5 is fine when the rest contains d around 4)

110

František Bartoš @fbartos.bsky.social · 18/06/2026

Currenly revising our critique of meta-analyses of AI/LLM on learning. Some of the original meta-analyses are beyond ridicolous. This one has 87 citations since published in 2025. (doi.org/10.1177/0266...)

150

František Bartoš @fbartos.bsky.social · 13/05/2026

Interestingly, we find that in a presence of a correlation between cluster size and effect size (as we observed in our second example), most existing methods become severely biased; multilevel RoBMA was the only method to consistently recover the true effect across all simulated conditions.

120

František Bartoš @fbartos.bsky.social · 13/05/2026

We illustrate the method on two re-analyses (see vignettes: cran.r-project.org/web/packages... and cran.r-project.org/web/packages...) and two simulation studies.

100

František Bartoš @fbartos.bsky.social · 13/05/2026

The multilevel extension to RoBMA was just published in BRM (link.springer.com/article/10.3...). Together with @maxmaier.bsky.social and EJ Wagenmakers, we developed a 3-level version of RoBMA which allows analysts to adjust for publication bias in meta-analyses with estimates nested within studies.

3163

František Bartoš @fbartos.bsky.social · 07/05/2026

What makes it even more efficient is combining this trick with inverse weighted marginal density estimator (IWMDE). In out settings IWMDE becomes trivial to implement and improves the density estimation tremendously so only a couple thousands posterior samples are needed.

110

František Bartoš @fbartos.bsky.social · 07/05/2026

In our new preprint we show how to efficiently compute Bayes factor sensitivity analyses for any model comparison. The computation trick is simple -- it rellies on extend the model by a hyper-prior on the parameter(s) for which sensitivity is desired and using transitivity of Bayes factors.

140

František Bartoš @fbartos.bsky.social · 28/01/2026

Meta-analysis level re-analysis then further highlights the issue of publication bias. Extremely overstated evidence (left) and mean effect size estimates (middle) due to a large degree of publication bias (right).

110

František Bartoš @fbartos.bsky.social · 28/01/2026

We explored several moderators and compared results of studies published before and after 2023 (to assess older AI systems and modern LLMs) but we did not find any meaningful difference.

100

František Bartoš @fbartos.bsky.social · 28/01/2026

We managed to collect 1,840 effect size estimates from 67 meta-analyses. The distribution of study-level effect sizes shows both a notable skew (funnel plot on the left) and clear selection for positive effects (z-curve plots on the right).

100

František Bartoš @fbartos.bsky.social · 28/01/2026

We just posted a preprint with a comprehensive meta-meta-analysis of the effects of AI/LLMs on learning. TLDR: - 1,840 effect sizes - extreme between-study heterogeneity - extreme publication bias - small average effects (three times lower than usually reported) (osf.io/preprints/ps...)

181

František Bartoš @fbartos.bsky.social · 23/12/2025

"we did not find any evidence for publication bias (p=0.077)"

130

František Bartoš @fbartos.bsky.social · 01/12/2025

We also re-analyzed all of the original meta-analyses individually. Many of them are consistent with publication bias: the evidence for and the degree of the pooled effects decrease once publication bias is adjusted for.

120

František Bartoš @fbartos.bsky.social · 01/12/2025

We run subgroup analyses for each outcome/population/intervention. We found that most results are too heterogeneous to tell (see wide prediction intervals), but some interventions seem to be promising and some have substantive evidence against them. See figures for each outcome.

110

František Bartoš @fbartos.bsky.social · 01/12/2025

First, we found notable publication bias, especially in studies on general cognition and executive function. Importantly, there was extreme between-study heterogeneity (tau ~ 0.3-0.6!). This means that the results were consistent with both large benefit but also large harm.

130

František Bartoš @fbartos.bsky.social · 23/10/2025

We developed PublicationBiasBenchmark R package (github.com/FBartos/Publ...) that can be easily extended with new methods and measures. It also automatically generates a webpage with summary reports (fbartos.github.io/PublicationB...). All the raw data, results, and measures are available on OSF.

020

František Bartoš @fbartos.bsky.social · 23/10/2025

Our proposal addresses other issues of current simulation studies (incomparability, irreproducibility...). We demonstrate the living synthetic benchmark methodology on the publication bias adjustment literature. See how previous simulations use different methods and measures.

120

František Bartoš @fbartos.bsky.social · 23/10/2025

We want to separate those two steps. New simulations should be published without new methods. Instead, they should evaluate all existing methods. New methods should be published without new simulations. Instead, they should be assessed on all existing simulations.

120

František Bartoš @fbartos.bsky.social · 23/10/2025

Simulation studies have a conflict of interest problem. The same team: - develops a new method - designs a simulation study to evaluate it However, the new method has to show good performance to get published. We propose living synthetic benchmarks to address the issue (doi.org/10.48550/arX...).

21910

František Bartoš @fbartos.bsky.social · 24/09/2025

Can anyone point me to the simulation studies showing that p-curve performs well under realistic conditions? And any done by someone else than pcurve authors? As far as I know, p-curve fails horrendously as long as any heterogeneity is involved... doi.org/10.1177/1745... doi.org/10.1002/jrsm...

191

František Bartoš @fbartos.bsky.social · 10/09/2025

We derive posterior predictive distributions for many meta-analytic models. Importantly, meta-analytic models that ignore these discontinuities misfit the data and should not be used for inference; models that respect them provide a better basis for inference. (see a couple of examples attached)

110

František Bartoš @fbartos.bsky.social · 10/09/2025

Publication bias is usually indicated by sharp discontinuities—typically at the significance threshold (selection for significance) or at zero (selection for positive results). Similar plots are often used in metaresearch, we bring them to meta-analysis!

101

František Bartoš @fbartos.bsky.social · 10/09/2025

Z-curve plot is a new visual model fit diagnostic for #metaanalysis with an emphasis on #publicationbias. In contrast to funnel plots, z-curve plots - visualize the distribution of z-statistics (where bias usually occurs) - compare the fit of multiple models simultaneously

2187

František Bartoš @fbartos.bsky.social · 10/09/2025

011

František Bartoš @fbartos.bsky.social · 11/08/2025

Importantly, there are considerable differences in the degree of the same-side bias between our coauthor-participants. As the previous figure suggests, people can differ by 1-2%. Further analyses showed that the bias seems to decrease over time, possibly due to practice effects.

100

František Bartoš @fbartos.bsky.social · 11/08/2025

We found that (as the title suggests) coins flipped by humans tend to land on the same side they started. The probability of the same side (50.8%) almost exactly matches the theoretical prediction of Diaconis, Holmes, and Montgomery from 2007.

100

František Bartoš @fbartos.bsky.social · 11/08/2025

Fair coins tend to land on the same side they started: evidence from 350,757 flips. That's the title of our paper summarizing ~650 hours of coin-tossing experimentation just published in the Journal of the American Statistical Association. doi.org/10.1080/0162...

1199

František Bartoš @fbartos.bsky.social · 09/08/2025

Tbh, third party simulations were showing issues with p-curve for quite some time-this shouldn't have come as a surprise doi.org/10.1002/jrsm...

171

František Bartoš @fbartos.bsky.social · 28/07/2025

The new version of JASP (0.95) containing another significant update to the Meta-Analysis module is out. You can perform state-of-the-art Bayesian publication bias-adjusted meta-regression in only a few clicks. A couple of additional clicks get you publication-ready figures!

081

František Bartoš @fbartos.bsky.social · 08/07/2025

It performs well in a wide range of simulation conditions and protects against reaching over-confident conclusions in the presence of model uncertainty and nested effect sizes (i.e., estimates within studies).

111

František Bartoš @fbartos.bsky.social · 08/07/2025

We finally extended robust Bayesian meta-analysis to multilevel settings. Now, you can fit 3-level publication bias-adjusted model-averaged meta-regression models in R (and in about a week in @jaspstats.bsky.social too!)

2203

František Bartoš @fbartos.bsky.social · 21/05/2025

In summary, we found that publication bias exaggerated the evidence in favor of SC-BCT to such a degree that once this bias is properly adjusted for, the effect disappears entirely. In fact, the data show moderate evidence against the presence of an effect.

151

František Bartoš @fbartos.bsky.social · 21/05/2025

A recently published meta-analysis in Nature Human Behaviour "found evidence supporting the efficacy of social comparison as a behaviour change technique in shaping behaviour in the desired direction". I was curious, so I re-analyzed the manuscript, but the funnel plots below say it all.

1228

František Bartoš @fbartos.bsky.social · 14/05/2025

We are running the summer JASP workshops again this year. We offer both on-site/online participation, see more details at: jasp-stats.org/workshops/

132

František Bartoš @fbartos.bsky.social · 12/05/2025

Does ChatGPT help with students' learning performance, learning perception, and higher-order thinking? We re-analyzed a recently published meta-analysis and found that the original conclusion is almost entirely driven by publication bias.

11915

František Bartoš @fbartos.bsky.social · 01/05/2025

Looking for recommendations to finish my #statistics #mugs collection!

280

František Bartoš @fbartos.bsky.social · 17/04/2025

The upcoming version of @jaspstats.bsky.social is gonna feature an even better version of the meta-analysis module. For example, the forest plot will allow aggregating effect size estimate information with just a couple click!

1187

František Bartoš @fbartos.bsky.social · 08/04/2025

Pretty cool! Already got a feature request to add this in JASP :) In the meantime, you can also use Bubble Plots with categorical variables

140

František Bartoš @fbartos.bsky.social · 16/11/2024

We are in the final phases of preparing a new update of JASP (@jaspstats.bsky.social). I'm creating examples for the completely revamped Meta-Analysis module - check out how stunning forest and funnel plots can be created with only a couple of mouse clicks! #meta-analysis

1304