iiinova — where your digital self keeps evolving

36/ Instead, we believe that the corrected results of the NMAs need to be published, ideally in the journals where the NMAs appeared. This way guideline committees, clinicians, and patients have access to more reliable estimations of efficacy.

050

35/ Second, even if the zombie trial is retracted/corrected, how will the impact of this study be known if the evidence syntheses in which it is included are unchanged?

34/ First, we believe it *does* make a difference if efficacy melted down into the range of clinically unimportance, especially if it was considered as a first-line drug!

130

33/ Furthermore, we were recommended to contact the journal where the zombie trial was published. We regret that Lancet decided this way, because we disagree with both arguments.

32/ But this ignores that the numerical difference is well within the range of clinical equivalence. We think clinicians and families should be able to decide for themselves using the most accurate estimates of efficacy

31/ Reviewers argued that corrected results don’t change the clinical conclusions because fluoxetine’s efficacy was still statistically significant

30/ We then contacted Lancet about an appropriate format to submit our analysis and were invited to submit it as a full paper and not just as a commentary, as we had intended. Our study was rejected after peer review.

29/ This shows that even a small zombie trial can severely “infect” the overall results of a NMA. Furthermore, this shows that unreliable/zombie trials can be undetected with classic risk of bias analysis.

131

28/ In summary, excluding the zombie trial lead to an efficacy estimation of fluoxetine which aligns across the NMAs and which are largely in the range of clinical equivalence.

27/ Nonetheless, we were able to closely reproduce the findings of the three NMAs. !!! Main finding: excluding the zombie trial resolved inconsistencies between the Lancet and Cochrane NMAs and also inconsistencies between direct and indirect evidence in the Lancet NMAs

121

26/ Hetrick’s group replied that the data and code were not accessible anymore. For data, Hetricks’s group provided some pre-processed data, as the final data-set was not accessible anymore. Cipriani’s data can be found online, and Zhou’s data was made public in another publication.

25/ We then tried to re-analyze all three NMAs to see if excluding the zombie trial could resolve the inconsistencies. The codes in Cipriani/Zhou’s appendix were not sufficient for reproduction and we did not get the full code upon request.

"Zombie trials" refer to untrustworthy clinical trials. The RestoRes initiative is committed to accurately describing these trials, thoroughly analyzing their design and results to highlight their cha...

24/ 🧟 Zombie trials are “randomised controlled trials that appear to be false and those where the data lack credibility so blatantly that they can be called ‘zombies’”. The term does not imply anything about the authors' intent. It focuses on the quality of the papers”

restores.univ-rennes.fr

Zombie trials | RestoRes

INSPECT-SR: a tool for assessing trustworthiness of randomised controlled trials

23/ We used the new INSPECT-SR checklist, leading to a “serious concerns” judgement regarding trustworthiness. Such trials may be labelled as “Zombie” trials. We contacted the authors but got no response. It is not possible to post on PubPeer as the study is not listed in PubMed and has no DOI 🧟

medrxiv.org

Precis The integrity of evidence synthesis is threatened by problematic randomised controlled trials (RCTs). These are RCTs where there are serious concerns about the trustworthiness of the data or fi...

22/ There were also other numerical discrepancies. The small trial also replaced drop-outs with new cases, without noting if this was pre-specified and if these patients were randomized, too. These and other problematic features call into question the trustworthiness of the data in Attari et al.

21/ Calculating the SMD from the endpoint scores or from baseline-endpoint score differences lead to much smaller effects, at least for the usual range of correlations between pre-post assessments. An SMD of 4.2 can only be achieved for implausibly large correlations > 0.97.

20/ We inspected this trial and found several problematic features. We noted unusually small SDs for the change scores and larger SDs (in the usual range) based on pre- and post-scores.

Sage Journals: Discover world-class research

19/ for context effect sizes exceeding one are hardly found for psychiatric treatments. Where can you find such enormous effects? For example, if you ask people if they prefer shit over chocolate, SMD 4.5 💩🍫 journals.sagepub.com/doi/10.1177/...

journals.sagepub.com

Subscription and open access journals from Sage, the world's leading independent academic publisher.

18/ We then found that this large indirect effect may have been the result of a single small trial (n=40) comparing fluoxetine with nortriptyline, with an extremely large effect of SMD 4.2. This is obviously an outlier (also confirmed with formal analysis methods).

17/ Therefore, we dug deeper into the two Lancet NMAs and found that there was a large difference between direct evidence (SMD -0.26 in Cipriani) and indirect evidence (SMD -1.40 in Cipriani) for fluoxetine. This was acknowledged but remained unexplained. Same in Zhou’s NMA

16/ However, we only later found out that this cannot explain all the discrepancies between the two Lancet NMAs and the Cochrane NMA, because the later Lancet NMA by Zhou already included most of the recent trials, and adding trials since then did not alter the overall effect

15/ We also included some new trials appearing since the Cochrane meta-analysis, but this didn’t change the overall results. Paper here: linkinghub.elsevier.com/retrieve/pii...

linkinghub.elsevier.com

Redirecting

14/ Additionally, efficacy was also larger in two-arm trials (fluoxetine vs. placebo), compared to multiarm trials where the chance to receive placebo was < 50% (eg duloxetine 30mg - duloxetine 60mg - fluoxetine - placebo), which can lead to different expectation effects and better blinding

13/ We found evidence for two mechanisms: first, efficacy was larger in trials sponsored by the producer of fluoxetine (or with associated COIs with the sponsor), as compared to trials where fluoxetine was the comparator drug, investigated in addition to a competing drug.

12/ So we aimed at explaining the discrepancies. In a first attempt we did our own pairwise meta-analysis and found that efficacy estimates for fluoxetine reduced over time. This is likely related to the “novelty bias” where several mechanisms can be at work to explain a decline of efficacy.

11/ Therefore, resolving the discrepancy and/or updating the guidelines has some urgency as kids may be exposed to a drug which lacks clinical meaningful efficacy but adverse effects (sexual dysfunctions, sleep problems, suicidality). Now already 5 years have passed…

130

10/ It also has not been acknowledged in guidelines that, with the reduced efficacy estimates for fluoxetine reported in the Cochrane NMA, the evidence does not meet established thresholds of clinical relevance.

9/ However, the discrepant findings for fluoxetine between the two Lancet NMAs and the Cochrane NMA remained unexplained in the Cochrane NMA and unacknowledged in clinical guidelines which appeared since then (or were updated afterwards).

7/ For background, they used the Childhood Depression Rating Scale CDRS-R (range 17 to 113) as outcome, and the area of equivalence was +/- 5 points. This can be transformed to a SMD with their suggested standard devia notion. Here the results:

6/ In 2021, a Cochrane network analysis (Hetrick et al., 2021) reported a substantially smaller effect for fluoxetine: SMD -0.20 (-0.28 to -0.11). The confidence rating was “moderate”. They defined the area of equivalence with placebo as +/- SMD 0.35. www.cochranelibrary.com/cdsr/doi/10....

cochranelibrary.com