Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis

Tipping point analysis for the between-arm correlation in an arm-based evidence synthesis

Systematic reviews and meta-analyses are essential tools in contemporary evidence-based medicine, synthesizing evidence from various sources to better inform clinical decision-making. However, the conclusions from different meta-analyses on the same topic can be discrepant, which has raised concerns about their reliability. One reason is that the result of a meta-analysis is sensitive to factors such as study inclusion/exclusion criteria and model assumptions. The arm-based meta-analysis model is growing in importance due to its advantage of including single-arm studies and historical controls with estimation efficiency and its flexibility in drawing conclusions with both marginal and conditional effect measures. Despite its benefits, the inference may heavily depend on the heterogeneity parameters that reflect design and model assumptions. This article aims to evaluate the robustness of meta-analyses using the arm-based model within a Bayesian framework. Specifically, we develop a tipping point analysis of the between-arm correlation parameter to assess the robustness of meta-analysis results. Additionally, we introduce some visualization tools to intuitively display its impact on meta-analysis results. We demonstrate the application of these tools in three real-world meta-analyses, one of which includes single-arm studies.

Synthesis of sexual selection: a systematic map of meta-analyses with bibliometric analysis

Sexual selection has been a popular subject within evolutionary biology because of its central role in explaining odd and counterintuitive traits observed in nature. Consequently, the literature associated with this field of study became vast. Meta-analytical studies attempting to draw inferences from this literature have now accumulated, varying in scope and quality, thus calling for a synthesis of these syntheses. We conducted a systematic literature search to create a systematic map with a report appraisal of meta-analyses on topics associated with sexual selection, aiming to identify the conceptual and methodological gaps in this secondary literature. We also conducted bibliometric analyses to explore whether these gaps are associated with the gender and origin of the authors of these meta-analyses. We included 152 meta-analytical studies in our systematic map. We found that most meta-analyses focused on males and on certain animal groups (e.g. birds), indicating severe sex and taxonomic biases. The topics in these studies varied greatly, from proximate (e.g. relationship of ornaments with other traits) to ultimate questions (e.g. formal estimates of sexual selection strength), although the former were more common. We also observed several common methodological issues in these studies, such as lack of detailed information regarding searches, screening, and analyses, which ultimately impairs the reliability of many of these meta-analyses. In addition, most of the meta-analyses' authors were men affiliated to institutions from developed countries, pointing to both gender and geographical authorship biases. Most importantly, we found that certain authorship aspects were associated with conceptual and methodological issues in meta-analytical studies. Many of our findings might simply reflect patterns in the current state of the primary literature and academia, suggesting that our study can serve as an indicator of issues within the field of sexual selection at large. Based on our findings, we provide both conceptual and analytical recommendations to improve future studies in the field of sexual selection.

Subjective evidence evaluation survey for many-analysts studies

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Vibration of effects resulting from treatment selection in mixed-treatment comparisons: a multiverse analysis on network meta-analyses of antidepressants in major depressive disorder

OBJECTIVE

It is frequent to find overlapping network meta-analyses (NMAs) on the same topic with differences in terms of both treatments included and effect estimates. We aimed to evaluate the impact on effect estimates of selecting different treatment combinations (ie, network geometries) for inclusion in NMAs.

DESIGN

Multiverse analysis, covering all possible NMAs on different combinations of treatments.

SETTING

Data from a previously published NMA exploring the comparative effectiveness of 22 treatments (21 antidepressants and a placebo) for the treatment of acute major depressive disorder.

PARTICIPANTS

Cipriani et al explored a dataset of 116 477 patients included in 522 randomised controlled trials.

MAIN OUTCOME MEASURES

For each possible treatment selection, we performed an NMA to estimate comparative effectiveness on treatment response and treatment discontinuation for the treatments included (231 between-treatment comparisons). The distribution of effect estimates of between-treatment comparisons across NMAs was computed, and the direction, magnitude and statistical significance of the 1st and 99th percentiles were compared.

RESULTS

4 116 254 different NMAs concerned treatment response. Among possible network geometries, 172/231 (74%) pairwise comparisons exhibited opposite effects between the 1st and 99th percentiles, 57/231 (25%) comparisons exhibited statistically significant results in opposite directions, 118 of 231 (51%) comparisons derived results that were both significant and non-significant at 5% risk and 56/231 (24%) treatment pairs obtained consistent results with only significant differences (or only non-significant differences) at 5% risk. Comparisons based on indirect evidence only were associated with greater variability in effect estimates. Comparisons with small absolute values observed in the complete NMA more frequently obtained statistically significant results in opposite directions. Similar results were observed for treatment discontinuation.

CONCLUSION

In this multiverse analysis, we observed that the selection of treatments to be included in an NMA could have considerable consequences on treatment effect estimations.

TRIAL REGISTRATION

https://osf.io/mb5dy.

Quality of systematic reviews with meta-analyses of resveratrol: A methodological systematic review

Recently, several meta-analyses (MAs) have focused on the health effects of resveratrol. However, the methodological and reporting quality of these MAs has not yet been fully evaluated so far. Therefore, the present study evaluated the quality of these MAs through a methodological systematic review. Systematic searches were conducted in PubMed, Embase, Web of Science, and Cochrane Library from inception until May 20, 2022, and PubMed was used to update the search until September 6, 2023. The methodological and reporting quality of the selected MAs was evaluated using AMSTAR-2 and PRISMA 2009. Fifty-one MAs published during 2013-2023 were included. In each review, the number of primary studies ranged from 3 to 37, and the number of participants ranged from 50 to 2114. Among the first-listed primary outcomes, only 23 (45.10%) were "positive." As for the methodological quality, most MAs (44, 86.27%) on resveratrol were rated critically low. Inadequate reporting of the included MAs mainly involved items 2 ("Structured summary"), 5 ("Protocol and registration"), 8 ("Search"), 9 ("Study selection"), 10 ("Data collection process"), 12 ("Risk of bias in individual studies"), and 24 ("Summary of evidence") based on the PRISMA 2009. Additionally, journal's impact factor, number of authors, and funding support were positively associated with the overall methodological quality but were not statistically significant (p > 0.05). Future MAs on resveratrol require better design, implementation, and reporting by following the Cochrane Handbook, AMSTAR-2, and PRISMA.

Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals

BACKGROUND

Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important.

METHODS

We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent.

RESULTS

A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%-9% for code sharing, 5%-25% for data sharing, 1%-31% for registration, 7%-100% for conflicts of interest, and 65%-100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%).

CONCLUSIONS

Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

BACKGROUND

Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses.

METHODS

In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality.

RESULTS

Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate.

CONCLUSIONS

The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.

A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results

OBJECTIVES

To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures.

STUDY DESIGN AND SETTING

A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates.

RESULTS

We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%-99%; quality of life 58%-76%; adverse events 87%-91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%-78%, quality of life; 10%-24%; adverse events 10%-13%).

CONCLUSION

Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects

OBJECTIVE

To explore the impact of methodological choices on the results of meta-analyses (MAs), with acupuncture for smoking cessation as a case study.

STUDY DESIGN AND SETTING

After performing an umbrella review (using MEDLINE, the COCHRANE Library, the Wan Fang database, and the Chinese Journal Full-text Database/March 2018) of MAs exploring the use of acupuncture for smoking cessation, we extracted all randomized controlled trials. Numerous MAs were performed as per every possible combination of various methodological choices (e.g., characteristics of the intervention and control procedures, outcome, publication status, language) to assess their vibration of effects or more precisely the existence of a Janus effect, that is, whether the 10th and 90th percentiles in the distribution of effect sizes were in opposite directions.

RESULTS

After including 7 MAs and 39 randomized controlled trials, we performed 496,528 MAs. The effect size was negative at the 10th percentile (-0.1, favoring controls) and positive at the 90th percentile (1.17, favoring acupuncture). In all, 104,491 MAs showed a statistically significant difference in favor of acupuncture, whereas 392,037 failed to demonstrate the efficacy of acupuncture (including 96 that showed a statistically significant difference in favor of the control).

CONCLUSION

The methodological choices made in performing pairwise MAs can result in substantial vibration of effects, occasionally leading to opposite results.

A meta-epidemiological assessment of transparency indicators of infectious disease models

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Further validation was performed by manual assessment of 10% of the articles identified by text mining as fulfilling transparency indicators and of 10% of the articles lacking them. Correcting estimates for validation performance, 26.0% of papers shared code and 41.1% shared data. On manual assessment, 5/6 articles identified as registered had indeed been registered. Of articles containing COI and funding statements, 95.8% disclosed no conflict and 11.7% reported no funding. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

Vibration of effect in more than 16 000 pooled analyses of individual participant data from 12 randomised controlled trials comparing canagliflozin and placebo for type 2 diabetes mellitus: multiverse analysis

Objective

To evaluate the impact of conducting all possible pooled analyses across different combinations of randomised controlled trials and endpoints.

Design

Multiverse analysis, consisting of numerous pooled analyses of individual participant data.

Setting

Individual patient data from 12 randomised controlled trials comparing canagliflozin treatment with placebo, shared on the Yale University Open Data Access project (https://yoda.yale.edu/) platform, up to 16 April 2021.

Participants

15 094 people with type 2 diabetes mellitus.

Main outcome measures

Pooled analyses estimated changes in serum glycated haemoglobin (HbA1c), major adverse cardiovascular events, and serious adverse events at weeks 12, 18, 26, and 52. The distribution of effect estimates was calculated for all possible combinations, and the direction and magnitude of the first and 99th centiles of effect estimates were compared.

Results

Across 16 332 distinct pooled analyses comparing canagliflozin with placebo for changes in HbA1c, standardised effect estimates were in favour of canagliflozin treatment at both the first centile (-0.75%) and 99th centile (-0.48%); 15 994 (97.93%) analyses showed significant results (P<0.05) in favour of canagliflozin. For major adverse cardiovascular events, estimated hazard ratios were 0.20 at the first centile and 0.90 at the 99th centile; 2705 of 8144 analyses (33.21%) were significant, all of which were in favour of canagliflozin treatment. For serious adverse events, estimated hazard ratios were 0.59 at the first centile and 1.14 at the 99th centile; 5793 of 16 332 (35.47%) analyses were significant, with 5754 in favour of canagliflozin and 39 in favour of placebo.

Conclusion

Results from pooled analyses can be subject to vibration of effects and should be critically appraised, especially regarding the risk for selection and availability bias in individual participant data retrieved.

Systematically assessing microbiome–disease associations identifies drivers of inconsistency in metagenomic research

Evaluating the relationship between the human gut microbiome and disease requires computing reliable statistical associations. Here, using millions of different association modeling strategies, we evaluated the consistency—or robustness—of microbiome-based disease indicators for 6 prevalent and well-studied phenotypes (across 15 public cohorts and 2,343 individuals). We were able to discriminate between analytically robust versus nonrobust results. In many cases, different models yielded contradictory associations for the same taxon–disease pairing, some showing positive correlations and others negative. When querying a subset of 581 microbe–disease associations that have been previously reported in the literature, 1 out of 3 taxa demonstrated substantial inconsistency in association sign. Notably, >90% of published findings for type 1 diabetes (T1D) and type 2 diabetes (T2D) were particularly nonrobust in this regard. We additionally quantified how potential confounders—sequencing depth, glucose levels, cholesterol, and body mass index, for example—influenced associations, analyzing how these variables affect the ostensible correlation between Faecalibacterium prausnitzii abundance and a healthy gut. Overall, we propose our approach as a method to maximize confidence when prioritizing findings that emerge from microbiome association studies.

The human microbiome has been associated with many aspects of our health, but how many of these associations are truly reproducible? This study attempts to address this question by systematically testing the robustness of 581 microbial features that have been reported as being disease-associated.

rTMS for depression: The difficult transition from research to clinical practice

The publication of the 2020 Royal Australian and New Zealand College of Psychiatrists clinical practice guidelines for the management of mood disorders was followed by a series of correspondences discussing the place of repetitive transcranial magnetic stimulation (rTMS) in the sequence care for the routine treatment of depression. The controversy has led to an intense debate about the positioning of this neuromodulation technique in the clinical armamentarium for depression in the Australian and New Zealand Journal of Psychiatry. The many arguments presented by the authors to defend their respective points of view can be summarised in two main key issues: (i) the interpretation of the level of evidence of rTMS for depression and (ii) the characterisation of patients who could benefit from this treatment. In this viewpoint, we discuss the difficult transition from research to clinical practice regarding the use of rTMS to treat depression.

Region-of-interest analysis approaches in neuroimaging studies of body ownership: An activation likelihood estimation meta-analysis

How do we feel that we own our body? By manipulating the integration of multisensory signals and creating the illusory experience of owning external body parts and entire bodies, researchers have investigated the neurofunctional correlates of body ownership. Recent attempts to synthesize the neuroimaging literature of body ownership through meta-analysis have shown partly inconsistent results. A large proportion of functional magnetic resonance imaging (fMRI) findings on body ownership include analyses based on regions of interest (ROIs). This approach can produce inflated findings when results are synthesized in meta-analyses. We conducted a systematic search of the fMRI literature of ownership of body parts and entire bodies. Three activation likelihood estimation (ALE) meta-analyses were conducted, testing the impact of including ROI-based findings. When both whole-brain and ROI-based results were included, frontal and posterior parietal multisensory areas were associated with body ownership. When only ROI-based results were included, larger areas of the frontal and posterior parietal cortices and the middle occipital gyrus were associated with body ownership. A whole-brain meta-analysis, excluding ROI-based results, found no significant convergence of activation across the brain. These findings highlight the difficulty of quantitatively synthesizing a neuroimaging field where a large part of the literature is based on findings from ROI-based analyses. We discuss these findings in the light of current practices within this field of research and highlight current problems of meta-analytic approaches of body ownership. We recommend the sharing of unthresholded data as a means to facilitate future meta-analyses of the neuroimaging literature of body ownership.

Variation in preoperative stress testing by patient, physician and surgical type: a cohort study

OBJECTIVES

To describe variation in and drivers of contemporary preoperative cardiac stress testing.

SETTING

A dedicated preoperative risk assessment and optimisation clinic at a large integrated medical centre from 2008 through 2018.

PARTICIPANTS

A cohort of 118 552 adult patients seen by 104 physicians across 159 795 visits to a preoperative risk assessment and optimisation clinic.

MAIN OUTCOME

Referral for stress testing before major surgery, including nuclear, echocardiographic or electrocardiographic-only stress testing, within 30 days after a clinic visit.

RESULTS

A total of 8303 visits (5.2%) resulted in referral for preoperative stress testing. Key patient factors associated with preoperative stress testing included predicted surgical risk, patient functional status, a previous diagnosis of ischaemic heart disease, tobacco use and body mass index. Patients living in either the most-deprived or least-deprived census block groups were more likely to be tested. Patients were tested more frequently before aortic, peripheral vascular or urologic interventions than before other surgical subcategories. Even after fully adjusting for patient and surgical factors, provider effects remained important: marginal testing rates differed by a factor-of-three in relative terms and around 2.5% in absolute terms between the 5th and 95th percentile physicians. Stress testing frequency decreased over the time period; controlling for patient and physician predictors, a visit in 2008 would have resulted in stress testing approximately 3.5% of the time, while a visit in 2018 would have resulted in stress testing approximately 1.3% of the time.

CONCLUSIONS

In this large cohort of patients seen for preoperative risk assessment at a single health system, decisions to refer patients for preoperative stress testing are influenced by various factors other than estimated perioperative risk and functional status, the key considerations in current guidelines. The frequency of preoperative stress testing has decreased over time, but remains highly provider-dependent.

Efficacy and safety of clozapine in psychotic disorders-a systematic quantitative meta-review

A recent increase in the literature regarding the evidence base for clozapine has made it increasingly difficult for clinicians to judge "best evidence" for clozapine use. As such, we aimed at elucidating the state-of-the-art for clozapine with regard to efficacy, effectiveness, tolerability, and management of clozapine and clozapine-related adverse events in neuropsychiatric disorders. We conducted a systematic PRISMA-conforming quantitative meta-review of available meta-analytic evidence regarding clozapine use. Primary outcome effect sizes were extracted and transformed into relative risk ratios (RR) and standardized mean differences (SMD). The methodological quality of meta-analyses was assessed using the AMSTAR-2 checklist. Of the 112 meta-analyses included in our review, 61 (54.5%) had an overall high methodological quality according to AMSTAR-2. Clozapine appears to have superior effects on positive, negative, and overall symptoms and relapse rates in schizophrenia (treatment-resistant and non-treatment-resistant subpopulations) compared to first-generation antipsychotics (FGAs) and to pooled FGAs/second-generation antipsychotics (SGAs) in treatment-resistant schizophrenia (TRS). Despite an unfavorable metabolic and hematological adverse-event profile compared to other antipsychotics, hospitalization, mortality and all-cause discontinuation (ACD) rates of clozapine surprisingly show a pattern of superiority. Our meta-review outlines the superior overall efficacy of clozapine compared to FGAs and most other SGAs in schizophrenia and suggests beneficial efficacy outcomes in bipolar disorder and Parkinson's disease psychosis (PDP). More clinical studies and subsequent meta-analyses are needed beyond the application of clozapine in schizophrenia-spectrum disorders and future studies should be directed into multidimensional clozapine side-effect management to foster evidence and to inform future guidelines.

Evaluation of a Meta-Analysis of Ambient Air Quality as a Risk Factor for Asthma Exacerbation

Background: An irreproducibility crisis currently afflicts a wide range of scientific disciplines, including public health and biomedical science. A study was undertaken to assess the reliability of a meta-analysis examining whether air quality components (carbon monoxide, particulate matter 10 µm and 2.5 µm (PM10 and PM2.5), sulfur dioxide, nitrogen dioxide and ozone) are risk factors for asthma exacerbation. Methods: The number of statistical tests and models were counted in 17 randomly selected base papers from 87 used in the meta-analysis. Confidence intervals from all 87 base papers were converted to p-values. p-value plots for each air component were constructed to evaluate the effect heterogeneity of the p-values. Results: The number of statistical tests possible in the 17 selected base papers was large, median = 15,360 (interquartile range = 1536–40,960), in comparison to results presented. Each p-value plot showed a two-component mixture with small p-values < 0.001 while other p-values appeared random (p-values > 0.05). Given potentially large numbers of statistical tests conducted in the 17 selected base papers, p-hacking cannot be ruled out as explanations for small p-values. Conclusions: Our interpretation of the meta-analysis is that random p-values indicating null associations are more plausible and the meta-analysis is unlikely to replicate in the absence of bias.

Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension

Since the early 1990s, ecologists and evolutionary biologists have aggregated primary research using meta-analytic methods to understand ecological and evolutionary phenomena. Meta-analyses can resolve long-standing disputes, dispel spurious claims, and generate new research questions. At their worst, however, meta-analysis publications are wolves in sheep's clothing: subjective with biased conclusions, hidden under coats of objective authority. Conclusions can be rendered unreliable by inappropriate statistical methods, problems with the methods used to select primary research, or problems within the primary research itself. Because of these risks, meta-analyses are increasingly conducted as part of systematic reviews, which use structured, transparent, and reproducible methods to collate and summarise evidence. For readers to determine whether the conclusions from a systematic review or meta-analysis should be trusted - and to be able to build upon the review - authors need to report what they did, why they did it, and what they found. Complete, transparent, and reproducible reporting is measured by 'reporting quality'. To assess perceptions and standards of reporting quality of systematic reviews and meta-analyses published in ecology and evolutionary biology, we surveyed 208 researchers with relevant experience (as authors, reviewers, or editors), and conducted detailed evaluations of 102 systematic review and meta-analysis papers published between 2010 and 2019. Reporting quality was far below optimal and approximately normally distributed. Measured reporting quality was lower than what the community perceived, particularly for the systematic review methods required to measure trustworthiness. The minority of assessed papers that referenced a guideline (~16%) showed substantially higher reporting quality than average, and surveyed researchers showed interest in using a reporting guideline to improve reporting quality. The leading guideline for improving reporting quality of systematic reviews is the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement. Here we unveil an extension of PRISMA to serve the meta-analysis community in ecology and evolutionary biology: PRISMA-EcoEvo (version 1.0). PRISMA-EcoEvo is a checklist of 27 main items that, when applicable, should be reported in systematic review and meta-analysis publications summarising primary research in ecology and evolutionary biology. In this explanation and elaboration document, we provide guidance for authors, reviewers, and editors, with explanations for each item on the checklist, including supplementary examples from published papers. Authors can consult this PRISMA-EcoEvo guideline both in the planning and writing stages of a systematic review and meta-analysis, to increase reporting quality of submitted manuscripts. Reviewers and editors can use the checklist to assess reporting quality in the manuscripts they review. Overall, PRISMA-EcoEvo is a resource for the ecology and evolutionary biology community to facilitate transparent and comprehensively reported systematic reviews and meta-analyses.

The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

BACKGROUND

Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision-making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions.

METHODS

The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by 1 team each and 2 reviews will be replicated by 15 teams (Study 4).

DISCUSSION

The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.

Comparative efficacy of placebos in short-term antidepressant trials for major depression: a secondary meta-analysis of placebo-controlled trials

BACKGROUND

The issue of unblinded outcome-assessors and patients has repeatedly been stressed as a flaw in allegedly double-blind antidepressant trials. Unblinding bias can for example result from a drug's marked side effects. If such unblinding bias is present for a given drug, then it might be expected that the placebos of that drug are rated significantly less effective than that of other antidepressants.

METHODS

To test this hypothesis, the present exploratory analysis conducted a Bayesian network meta-analysis (NMA) comparing the efficacy of 19 different placebos in placebo-controlled trials provided in the dataset by Cipriani et al. (Lancet 2018; 391: 1357-66). Primary outcome was efficacy (continuous) estimated on the standardized mean difference (SMD) scale and defined as the pre-post change on the Hamilton Depression scale (HAMD-17), on which information was available in N = 258 trials.

RESULTS

Comparative placebo ranking suggested mirtazapine-placebo (SMD -2.0 [- 5.0-1.0 95% CrI]) to be the most, and amitriptyline- (SMD 1.2 [- 1.6-3.9 95% CrI]) and trazodone- (SMD 2.1 [- 0.9-5.2 95% CrI]) placebos to be the least effective placebos. Other placebos suggested to be more effective than amitriptyline- and trazodone-placebos (based on 95% CrIs excluding zero) were citalopram, desvenlafaxine, duloxetine, escitalopram, fluoxetine, sertraline, and venlafaxine placebos. These NMA results were corroborated by the observation that the relative efficacy between drug and placebo was considerably larger for amitriptyline and trazodone than for instance mirtazapine, duloxetine, and venlafaxine, supported by a small and insignificant correlation between drug-efficacy and placebo-efficacy (r = - 0.202, p = 0.408).

DISCUSSION

The present exploratory NMA indicates that distinguishable side effects of older drugs may unblind outcome-assessors thus resulting in overestimation of the average drug-placebo difference and underrating bias in placebo-arms, particularly for the older antidepressant drugs amitriptyline and trazodone. If confirmed in prospective studies, these findings suggest that efficacy rankings for antidepressants are susceptible to bias and should be considered unreliable or misleading. The analysis is limited by the focus on the single-comparison placebos (76%, i.e., placebos assessed in two-arm trials), since double-comparison placebos (25%, i.e., placebos assessed in three-arm trials) are hard to interpret and therefore not included in the present interpretation. Another limitation is the problem of multiplicity, which was only approximately accounted for in the Bayesian NMA by modelling treatment effects as exchangeable.

"One more time": why replicating some syntheses of evidence relevant to COVID-19 makes sense

•Given the urgent need for credible answers to high-priority questions about the health and social impacts of COVID-19, many systematic reviewers seek to contribute their skills and expertise. •Rather than embarking on unnecessary, duplicate reviews, we encourage the evidence synthesis community to prioritise purposeful replication of systematic reviews of evidence relevant to COVID-19. •We explain why replication of systematic reviews is important, how to carry out a replication, and when to consider replication of reviews.

Fragility index of network meta-analysis with application to smoking cessation data

BACKGROUND AND OBJECTIVES

The network meta-analysis (NMA) is frequently used to synthesize evidence for multiple treatment comparisons, but its complexity may affect the robustness (or fragility) of the results. The fragility index (FI) is recently proposed to assess the fragility of the results from clinical studies and from pairwise meta-analyses. We extend the FI to NMAs with binary outcomes.

METHODS

We define the FI for each treatment comparison in NMAs. It quantifies the minimal number of events necessary to be modified for altering the comparison's statistical significance. We introduce an algorithm to derive the FI and visualizations of the process. A worked example of smoking cessation data is used to illustrate the proposed methods.

RESULTS

Some treatment comparisons had small FIs; their significance (or nonsignificance) could be altered by modifying a few events' status. They were related to various factors, such as P-values, event counts, and sample sizes, in the original NMA. After modifying event status, treatment ranking measures were also changed to different extents.

CONCLUSION

Many NMAs include insufficiently compared treatments, small event counts, or small sample sizes; their results are potentially fragile. The FI offers a useful tool to evaluate treatment comparisons' robustness and reliability.