Retraction: Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis

Bias propagation in network meta-analysis models

Network meta-analysis combines direct and indirect evidence to compare multiple treatments. As direct evidence for one treatment contrast may be indirect evidence for other treatment contrasts, biases in the direct evidence for one treatment contrast may affect not only the estimate for this particular treatment contrast but also estimates of other treatment contrasts. Because network structure determines how direct and indirect evidence are combined and weighted, the impact of biased evidence will be determined by the network geometry. Thus, this study's aim was to investigate how the impact of biased evidence spreads across the whole network and how the propagation of bias is influenced by the network structure. In addition to the popular Lu & Ades model, we also investigate bias propagation in the baseline model and arm-based model to compare the effects of bias in the different models. We undertook extensive simulations under different scenarios to explore how the impact of bias may be affected by the location of the bias, network geometry and the statistical model. Our results showed that the structure of a network has an important impact on how the bias spreads across the network, and this is especially true for the Lu & Ades model. The impact of bias is more likely to be diluted by other unbiased evidence in a well-connected network. We also used a real network meta-analysis to demonstrate how to use the new knowledge about bias propagation to explain questionable results from the original analysis.

A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects

OBJECTIVE

To explore the impact of methodological choices on the results of meta-analyses (MAs), with acupuncture for smoking cessation as a case study.

STUDY DESIGN AND SETTING

After performing an umbrella review (using MEDLINE, the COCHRANE Library, the Wan Fang database, and the Chinese Journal Full-text Database/March 2018) of MAs exploring the use of acupuncture for smoking cessation, we extracted all randomized controlled trials. Numerous MAs were performed as per every possible combination of various methodological choices (e.g., characteristics of the intervention and control procedures, outcome, publication status, language) to assess their vibration of effects or more precisely the existence of a Janus effect, that is, whether the 10th and 90th percentiles in the distribution of effect sizes were in opposite directions.

RESULTS

After including 7 MAs and 39 randomized controlled trials, we performed 496,528 MAs. The effect size was negative at the 10th percentile (-0.1, favoring controls) and positive at the 90th percentile (1.17, favoring acupuncture). In all, 104,491 MAs showed a statistically significant difference in favor of acupuncture, whereas 392,037 failed to demonstrate the efficacy of acupuncture (including 96 that showed a statistically significant difference in favor of the control).

CONCLUSION

The methodological choices made in performing pairwise MAs can result in substantial vibration of effects, occasionally leading to opposite results.

Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis

BACKGROUND

Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We explored the range of such differences, using several methodological choices for indirect comparison meta-analyses to compare nalmefene and naltrexone in the reduction of alcohol consumption as a case study.

METHODS

All double-blind randomized controlled trials (RCTs) comparing nalmefene to naltrexone or one of these compounds to a placebo in the treatment of alcohol dependence or alcohol use disorders were considered. Two reviewers searched for published and unpublished studies in MEDLINE (August 2017), the Cochrane Library, Embase, and ClinicalTrials.gov and contacted pharmaceutical companies, the European Medicines Agency, and the Food and Drug Administration. The indirect comparison meta-analyses were performed according to different inclusion/exclusion criteria (based on medical condition, abstinence of patients before inclusion, gender, somatic and psychiatric comorbidity, psychological support, treatment administered and dose, treatment duration, outcome reported, publication status, and risk of bias) and different analytical models (fixed and random effects). The primary outcome was the vibration of effects (VoE), i.e. the range of different results of the indirect comparison between nalmefene and naltrexone. The presence of a "Janus effect" was investigated, i.e. whether the 1st and 99th percentiles in the distribution of effect sizes were in opposite directions.

RESULTS

Nine nalmefene and 51 naltrexone RCTs were included. No study provided a direct comparison between the drugs. We performed 9216 meta-analyses for the indirect comparison with a median of 16 RCTs (interquartile range = 12-21) included in each meta-analysis. The standardized effect size was negative at the 1st percentile (- 0.29, favouring nalmefene) and positive at the 99th percentile (0.29, favouring naltrexone). A total of 7.1% (425/5961) of the meta-analyses with a negative effect size and 18.9% (616/3255) of those with a positive effect size were statistically significant (p < 0.05).

CONCLUSIONS

The choice of inclusion/exclusion criteria and analytical models for meta-analysis can result in entirely opposite results. VoE evaluations could be performed when overlapping meta-analyses on the same topic yield contradictory result.

TRIAL REGISTRATION

This study was registered on October 19, 2016, in the Open Science Framework (OSF, protocol available at https://osf.io/7bq4y/ ).