Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

Transcatheter versus surgical aortic valve replacement for severe aortic valve stenosis: Meta-analysis with trial sequential analysis

OBJECTIVES

Randomized controlled trials comparing transcatheter aortic valve implantation with surgical aortic valve replacement demonstrate conflicting evidence, particularly in low-risk patients. We aim to reevaluate the evidence using trial sequential analysis, balancing type I and II errors, and compare with conventional meta-analysis.

METHODS

Databases were searched for randomized controlled trials, which were divided into higher-risk and lower-risk randomized controlled trials according to a pragmatic risk classification. Primary outcomes were death and a composite end point of death or disabling stroke assessed at 1 year and maximum follow-up. Conventional meta-analysis and trial sequential analysis were performed, and the required information size was calculated considering a type I error of 5% and a power of 90%.

RESULTS

Eight randomized controlled trials (n = 5274 higher-risk and 3661 lower-risk patients) were included. Higher-risk trials showed no significant reduction in death at 1 year with transcatheter aortic valve implantation (relative risk, 0.93, 95% CI, 0.81-1.08, P = .345). Lower-risk trials suggested lower death risk on conventional meta-analysis (relative risk, 0.67, 95% CI, 0.47-0.96, P = .031), but trial sequential analysis indicated potential spurious evidence (P = .116), necessitating more data for conclusive benefit (required information size = 5944 [59.8%]). For death or disabling stroke at 1 year, higher-risk trials lacked evidence (relative risk, 0.90, 95% CI, 0.79-1.02, P = .108). In lower-risk trials, transcatheter aortic valve implantation indicated lower risk in conventional meta-analysis (relative risk, 0.68, 95% CI, 0.50-0.93, P = .014), but trial sequential analysis suggested potential spurious evidence (P = .053), necessitating more data for conclusive benefit (required information size = 5122 [69.4%]). Follow-up results provided inconclusive evidence for both primary outcomes across risk categories.

CONCLUSIONS

Conventional meta-analysis methods may have prematurely declared an early reduction of negative outcomes after transcatheter aortic valve implantation when compared with surgical aortic valve replacement.

Trial sequential analysis: A simple guide for judging the conclusiveness of the effect

Meta-analysis is a statistical method used in systematic reviews to provide a quantitative estimate of the effect. However, including very few studies and participants may increase the risk of spurious findings. Trial sequential analysis (TSA) has been introduced to enhance the robustness of meta-analysis. TSA is a cumulative meta-analysis method that weighs type I and II errors while estimating the effect. The application of TSA can lead to a more accurate estimation of the clinical effectiveness of the intervention. The aim of the present paper was to introduce the TSA to orthodontic clinicians and researchers using continuous data from an orthodontic systematic review.

An Updated Trial Sequential Meta-analysis of Vitamin D Receptor Gene Polymorphism (Fok1, Bsm1, Taq1 and Apa1) and Risk to Tuberculosis

Vitamin D receptor (VDR) is one of the most widely studied genes for the Tuberculosis (TB) susceptibility. Several studies have been conducted to establish some association between them but most of the time they are contradictory and underpowered. So, a trial sequential meta-analysis between VDR gene polymorphisms and TB susceptibility can provide a better understanding of the relationship. A meta-analysis was carried out using a total of 17 case-control studies which includes Fok1 (14 Studies), Bsm1 (8 Studies), Apa1 (8 Studies) and Taq1 (12 Studies) polymorphisms in the VDR gene searched from Pubmed and Google Scholar. Pooled Odds Ratio (OR) and 95% Confidence Interval (CI) were calculated using StatsDirect Version 3, using random effects model. Trial sequential analysis (TSA) was also performed to assess if the statistical significance of the meta-analysis was within monitoring boundaries. It was found that the individuals with BB genotype of Bsm1 polymorphism with OR = 0.713 (95%CI = 0.521, 0.974; p value < 0.05) and FF genotype of Fok1 polymorphism with pooled OR = 0.716 (95%CI = 0.523, 0.979; p value < 0.05) had decreased incidence of TB. Also, the aa genotype of Apa1 gene polymorphism increases susceptibility to TB with pooled OR = 1.997 (95%CI = 1.121, 3.558; p value < 0.05). All these analyses reached the required information size through TSA analysis. No statistically significant result was found for Taq1 polymorphisms and TB susceptibility. VDR polymorphisms in Fok1 and Bsm1 played protective roles against development of TB infection, while Apa1 appeared to have a significant association to TB susceptibility.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12291-022-01091-3.

Environmental Risk Factors, Protective Factors, and Biomarkers for Allergic Rhinitis: A Systematic Umbrella Review of the Evidence

Many potential environmental risk factors, protective factors, and biomarkers of AR have been published, but so far, the strength and consistency of their evidence are unclear. We conducted a comprehensive review of environmental risk, protective factors, and biomarkers for AR to establish the evidence hierarchy. We systematically searched Embase, PubMed, Cochrane Library, and Web of Science electronic database from inception to December 31, 2022. We calculated summary effect estimate (odds ratio (OR), relative risk (RR), hazard ratio (HR), and standardized mean difference (SMD)), 95% confidence interval, random effects p value, I2 statistic, 95% prediction interval, small study effects, and excess significance biases, and stratification of the level of evidence. Methodological quality was assessed by AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews 2). We retrieved 4478 articles, of which 43 met the inclusion criteria. The 43 eligible articles identified 31 potential environmental risk factors (10,806,206 total population, two study not reported), 11 potential environmental protective factors (823,883 total population), and 34 potential biomarkers (158,716 total population) for meta-analyses. The credibility of evidence was convincing (class I) for tic disorders (OR = 2.89, 95% CI 2.11-3.95); and highly suggestive (class II) for early-life antibiotic use (OR = 3.73, 95% CI 3.06-4.55), exposure to indoor dampness (OR = 1.49, 95% CI 1.27-1.75), acetaminophen exposure (OR = 1.54, 95% CI 1.41-1.69), childhood acid suppressant use (OR = 1.40, 95% CI 1.23-1.59), exposure to indoor mold (OR = 1.66, 95% CI 1.26-2.18), coronavirus disease 2019 (OR = 0.11, 95% CI 0.06-0.22), and prolonged breastfeeding (OR = 0.72, 95% CI 0.65-0.79). This study is registered in PROSPERO (CRD42022384320).

Importance of sequential methods in meta-analysis: implications for postoperative mortality, delirium, and stroke management

Trial sequential analysis is an adaptation of frequentist sequential methods that can be used to improve inferences from meta-analysis. Trial sequential analysis can help preserve type I and type II error rates at desired levels for analyses conducted before the required information size. Through three case studies recently published in the British Journal of Anaesthesia, we show how trial sequential analysis can inform the interpretation of meta-analyses. Limitations of trial sequential analysis, which also include those of the meta-analysis to which it is applied, must be carefully considered alongside its benefits.

Trial Sequential Analysis and Updated Meta-Analysis of Fluvoxamine on Clinical Deterioration in Adult Patients with Symptomatic COVID-19 Infection

Preliminary meta-analyses suggested that fluvoxamine was effective in treating COVID-19 infection. However, the reliability of this evidence has not yet been examined. MEDLINE, CENTRAL, EMBASE, PsycINFO, and ClinicalTrials.gov were searched to identify any randomized controlled trials (RCTs) from the inception of the databases to 5 February 2023. We used trial sequential analysis (TSA) to examine the reliability of the current existing evidence on the benefits of fluvoxamine on COVID-19 infection. The primary outcome was clinical deterioration, as defined in the original study (reported as odds ratio (OR), with 95% confidence intervals), and the secondary outcome was hospitalization. In the TSA, we used the relative risk reduction thresholds of 10, 20, and 30%. The updated meta-analysis of the five RCTs showed that fluvoxamine was not associated with lower odds of clinical deterioration when compared with a placebo (OR: 0.81; 0.59-1.11). The effect of fluvoxamine lay within the futility boundary (i.e., lack of effect) when using a 30% relative risk reduction threshold. The effect estimates lay between the superiority and futility boundary using the 10% and 20% threshold, and the required size of information was not reached for these two thresholds. The effect of fluvoxamine on the odds of hospitalization was not statistically significant (0.76; 0.56-1.03). In conclusion, there is no reliable evidence that fluvoxamine, when compared to a placebo, reduces the relative risk of clinical deterioration among adult patients with COVID-19 infection by 30%, and a relative risk reduction of 20% or 10% is still uncertain. The role of fluvoxamine as a COVID-19 treatment cannot be justified.

Subcutaneous fast-acting insulin analogues, alone or in combination with long-acting insulin, versus intravenous regular insulin infusion in patients with diabetic ketoacidosis: protocol for an updated systematic review and meta-analysis of randomised trials

INTRODUCTION

Diabetic ketoacidosis (DKA) is traditionally managed using intravenous regular insulin infusion (RII) in intensive care unit (ICU)/high dependency unit (HDU). Subcutaneous fast-acting insulin analogues (FAIAs) may help to manage DKA outside ICU/HDU. Furthermore, combining subcutaneous long-acting insulin (LAI) with subcutaneous FAIAs may accelerate ketoacidosis resolution. The latest (2016) Cochrane review was inconclusive regarding subcutaneous FAIAs versus intravenous RII in DKA. It was limited by small sample sizes, unclear risk of bias (RoB) in primary trials and did not examine subcutaneous FAIAs with subcutaneous LAI versus intravenous RII in DKA. We report the protocol for an updated meta-analysis on the safety and benefits of subcutaneous FAIAs with/without subcutaneous LAI versus intravenous RII in DKA.

METHODS AND ANALYSIS

We will search Medline, Embase, CINAHL and Cochrane Library, from inception until December 2022, without language restrictions, for randomised trials on subcutaneous FAIAs with/without subcutaneous LAI versus intravenous RII in DKA. We also search ClinicalTrials.gov, ClinicalTrialsRegister.eu and reference lists of included trials. Primary outcomes include all-cause in-hospital mortality, time to DKA resolution, in-hospital DKA recurrence and hospital readmission for DKA post-discharge. Secondary outcomes include resource utilisation and patient satisfaction. Safety outcomes include important complications of DKA and insulin. Reviewers will extract data, assess overall RoB and quality of evidence using Grading of Recommendations, Assessment, Development and Evaluation. We will assess statistical heterogeneity by visually inspecting forest plots and the I² statistic. We will synthesise data using the random-effects model. Predefined subgroup analyses are: mild versus moderate versus severe DKA; age <20 vs ≥20 years; pregnant versus non-pregnant; infective versus non-infective DKA precipitating cause; subcutaneous FAIAs alone versus subcutaneous FAIAs and subcutaneous LAI; and high versus low overall RoB. We will also perform trial sequential analysis for primary outcomes.

ETHICS AND DISSEMINATION

Ethics board approval is not required. Results will be disseminated through publication in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42022369518.

Shared and distinct neural activity during anticipation and outcome of win and loss: A meta-analysis of the monetary incentive delay task

Reward and punishment motivate decision making and behavioral changes. Numerous studies have examined regional activities during anticipation and outcome of win and loss in the monetary incentive delay task (MIDT). However, the great majority of studies reported findings of anticipation or outcome and of win or loss alone. It remains unclear how the neural correlates share and differentiate amongst these processes. We conducted an Activation Likelihood Estimation meta-analysis of 81 studies of the MIDT (5,864 subjects), including 24 published since the most recent meta-analysis, to identify and, with conjunction and subtraction, contrast regional responses to win anticipation, loss anticipation, win outcome, and loss outcome. Win and loss anticipation engaged a shared network of bilateral anterior insula (AI), striatum, thalamus, supplementary motor area (SMA), and precentral gyrus. Win and loss outcomes did not share regional activities. Win and loss outcome each engaged higher activity in medial orbitofrontal cortex (mOFC) and dorsal anterior cingulate cortex. Bilateral striatum and right occipital cortex responded to both anticipation and outcome of win, and right AI to both phases of loss. Win anticipation vs. outcome engaged higher activity in bilateral AI, striatum, SMA and precentral gyrus and right thalamus, and lower activity in bilateral mOFC and posterior cingulate cortex as well as right inferior frontal and angular gyri. Loss anticipation relative to outcome involved higher activity in bilateral striatum and left AI. These findings collectively suggest shared and distinct regional responses during monetary wins and losses. Delineating the neural correlates of these component processes may facilitate empirical research of motivated behaviors and dysfunctional approach and avoidance in psychopathology.

Decisive gene strategy on osteoporosis: a comprehensive whole-literature-based approach for conclusive candidate gene targets

Purpose: Previous meta-analyses only examined the association between single gene polymorphisms and osteoporosis; there is no compilation of all gene loci that correlate with osteoporosis in the literature. In this study, we develop a new literature-based approach, a decisive gene strategy (DGS), to examine the sufficiency of the cumulative sample size for each gene locus and to assess whether a definite conclusion of the association between the gene locus and osteoporosis can be drawn.

Methods: The DGS was used to search PubMed, Embase, and Cochrane databases for all meta-analyses that correlated gene polymorphisms with osteoporosis. Trial sequential analysis was employed to examine the sufficiency of the cumulative sample size. Finally, we assessed the importance of gene loci in osteoporosis based on whether there were enough sample sizes and the heterogeneity of the literature with the I² value.

Results: After excluding 169 irrelevant publications, 39 meta-analysis papers were obtained. Among Caucasians, in 17 gene loci, there were eight gene loci (e.g., vitamin D Receptor ApaI rs7975232) with sufficient cumulative sample size to confirm that they were unrelated to the disease. Among Asians, in 15 gene loci, four gene loci that had sufficient sample sizes were risk factors: VDR FokI rs2228570 (odds ratio (OR) = 1.44, 95% confidence interval (CI) = 1.22–1.70), TGF β1 rs1800470 (OR = 1.35, 95% CI = 1.10–1.65), IGF1 rs2288377 (OR = 1.44, 95% CI = 1.28–1.62), and IGF1 rs35767 (OR = 1.20, 95% CI = 1.06–1.36), respectively, whereas one gene locus, ESR2 RsaI rs1256049 (OR = 0.69, 95% CI = 0.59–0.81), was a protective factor.

Conclusions: The DGS successfully identified five gene loci in osteoporosis that will apply to other diseases to find causal genes, which may contribute to further genetic therapy.

An umbrella review of effect size, bias, and power across meta-analyses in emergency medicine

OBJECTIVES

The objective of this study was to conduct an umbrella review of therapeutic studies relevant to emergency medicine, analyzing patterns in effect size, power, and signals of potential bias across an entire field of clinical research.

METHODS

We combined topic- and journal-driven searches of PubMed and Google Scholar for published articles of systematic reviews and meta-analyses (SRMA) relevant to emergency medicine (last search in November 2020). Data were screened and extracted by six investigators. Redundant meta-analyses were removed. Whenever possible for each comparison we extracted one meta-analysis on mortality with the most events and one meta-analysis on a nonmortality outcome with the most studies. From each meta-analysis we extracted all individual study effects; outcomes were converted to odds ratios (ORs) and placed on a common scale where an OR < 1.0 represents a reduction in a harmful outcome with an experimental treatment versus control. Outcomes were analyzed at the level of individual studies and at the level of summary effects across meta-analyses.

RESULTS

A total of 332 articles contained 431 eligible meta-analyses with a total of 3,129 individual study outcomes; of these, 2,593 (83%) were from randomized controlled trials. The median OR across all studies was 0.70. Within each meta-analysis, the earliest study effect on average demonstrated larger benefit compared to the overall summary effect. Only 57 of 431 meta-analyses (13%) both favored the experimental intervention and did not show any signal of small study effects or excess significance, and of those only 12 had at least one study with 80% or higher power to detect an OR of 0.70. Of these, no interventions significantly decreased mortality in well-powered trials. Although the power of studies increased somewhat over time, the majority of studies were underpowered.

CONCLUSIONS

Few interventions studied within SRMAs relevant to emergency medicine seem to have strong and unbiased evidence for improving outcomes. The field would benefit from more optimally powered trials.

The Unknown Prevalence of Postrandomization Bias in 15 Physical Therapy Journals: A Methods Review

OBJECTIVES

To determine the prevalence of prospective clinical trial registration and postrandomization bias in published musculoskeletal physical therapy randomized clinical trials (RCTs).

DESIGN

A methods review.

LITERATURE SEARCH

Articles indexed in MEDLINE and published between January 2016 and July 2020 were included.

STUDY SELECTION CRITERIA

Two independent blinded reviewers identified the RCTs using Covidence. We included RCTs related to musculoskeletal interventions that were published in International Society of Physiotherapy Journal Editors member journals.

DATA SYNTHESIS

Data were extracted independently for the variables of interest from the identified RCTs by 2 blinded reviewers. The data were presented descriptively or in frequency tables.

RESULTS

One hundred thirty-eight RCTs were identified. One third of RCTs were consistent with their prospectively registered intent (49/138); consistency with prospectively registered intent could not be determined for two thirds (89/138) of the RCTs. Four RCTs (8%)reported inconsistent results with the primary aims and 7 (14%) with the outcomes from the prospective clinical trial registry, despite high methodological quality (Physiotherapy Evidence Database [PEDro] scale score). Differences between prospectively registered and non-prospectively registered RCTs for PEDro scale scores had a medium effect size (r = 0.30). Two of 15 journals followed their clinical trial registration policy 100% of the time; in 1 journal, the published RCTs were consistent with the clinical trial registration.

CONCLUSION

Postrandomization bias in musculoskeletal physical therapy RCTs could not be ruled out, due to the lack of prospective clinical trial registration and detailed data analysis plans. J Orthop Sports Phys Ther 2021;51(11):542-550. Epub 21 Sep 2021. doi:10.2519/jospt.2021.10491.

Association between long non-coding RNA (lncRNA) GAS5 polymorphism rs145204276 and cancer risk

OBJECTIVE

The long non-coding RNA (lncRNA) growth arrest‑specific transcript 5 (GAS5) plays an important role in various tumors, and an increasing number of studies have explored the association of the GAS5 rs145204276 polymorphism with cancer risk with inconclusive results.

METHODS

PubMed, Medline, EMBASE, Cochrane databases, and Web of Science were searched, and nine studies involving 6107 cases and 7909 controls were deemed eligible. Odds ratios (ORs) and corresponding 95% confidence intervals (CIs) were calculated to evaluate the relationship between rs145204276 and cancer risk in six genetic models.

RESULTS

The pooled results suggest that the variant allele del was not associated with overall cancer risk. However, the subgroup analysis showed that allele del was significantly associated with a 22% decreased risk of gastrointestinal cancer (OR = 0.78, 95% CI: 0.72-0.85). Both sensitivity analyses and trial sequential analyses (TSA) demonstrated that the subgroup results were reliable and robust. Moreover, False-Positive Report Probability (FPRP) analysis indicated that the results had true significant correlations.

CONCLUSION

These findings provide evidence that the GAS5 rs145204276 polymorphism is associated with the susceptibility to gastrointestinal cancer. Further studies with different ethnicities and larger sample sizes are warranted to confirm these results.

Applications of simple and accessible methods for meta-analysis involving rare events: A simulation study

Meta-analysis of clinical trials targeting rare events face particular challenges when the data lack adequate number of events and are susceptible to high levels of heterogeneity. The standard meta-analysis methods (DerSimonian Laird (DL) and Mantel–Haenszel (MH)) often lead to serious distortions because of such data sparsity. Applications of the methods suited to specific incidence and heterogeneity characteristics are lacking, thus we compared nine available methods in a simulation study. We generated 360 meta-analysis scenarios where each considered different incidences, sample sizes, between-study variance (heterogeneity) and treatment allocation. We include globally recommended methods such as inverse-variance fixed/random-effect (IV-FE/RE), classical-MH, MH-FE, MH-DL, Peto, Peto-DL and the two extensions for MH bootstrapped-DL (bDL) and Peto-bDL. Performance was assessed on mean bias, mean error, coverage and power. In the absence of heterogeneity, the coverage and power when combined revealed small differences in meta-analysis involving rare and very rare events. The Peto-bDL method performed best, but only in smaller sample sizes involving rare events. For medium-to-larger sample sizes, MH-bDL was preferred. For meta-analysis involving very rare events, Peto-bDL was the best performing method which was sustained across all sample sizes. However, in meta-analysis with 20% or more heterogeneity, the coverage and power were insufficient. Performance based on mean bias and mean error was almost identical across methods. To conclude, in meta-analysis of rare binary outcomes, our results suggest that Peto-bDL is better in both rare and very rare event settings in meta-analysis with limited sample sizes. However, when heterogeneity is large, the coverage and power to detect rare events are insufficient. Whilst this study shows that some of the less studied methods appear to have good properties under sparse data scenarios, further work is needed to assess them against the more complex distributional-based methods to understand their overall performances.

Chinese Herbal Medicine for the Treatment of Children and Adolescents With Refractory Mycoplasma Pneumoniae Pneumonia: A Systematic Review and a Meta-Analysis

Objectives: Chinese herb medicine (CHM) is one of the most popular complementary and alternative therapies, which has been widely used to treat Refractory Mycoplasma Pneumoniae Pneumonia (RMPP). However, the effect and safety of CHM remain controversial. Hence, we conducted this meta-analysis to evaluate whether CHM combination therapy could bring benefits to children and adolescents with RMPP. Methods: Seven databases were used for data searching through November 11, 2020 following the PRISMA checklist generally. Review Manager 5.3, Trial sequential analysis 0.9.5.10 Beta software and Stata16.0 were applied to perform data analyses. Mean difference or risk ratio was adopted to express the results, where a 95% confidence interval (CI) was applied. Results: In general, this research enrolled 17 trials with 1,451 participants. The overall pooled results indicated that CHM was beneficial for children and adolescents with RMPP by improving the clinical efficacy rate [RR = 1.20, 95% CI (1.15, 1.25), p < 0.00001], shortening antipyretic time [MD = -2.60, 95% CI (-3.06, -2.13), p < 0.00001], cough disappearance time [MD = -2.77, 95% CI (-3.12, -2.42), p < 0.00001], lung rale disappearance time [MD = -2.65, 95% CI (-3.15, -2.15), p < 0.00001], lung X-ray infiltrates disappearance time [MD = -2.75, 95% CI (-3.33, -2.17), p < 0.00001], reducing TNF-α level [MD = -5.49, 95% CI (-7.21, -3.77), p < 0.00001]. Moreover, subgroup results suggested that removing heat-phlegm and toxicity therapy had more advantages in shortening antipyretic time, cough disappearance time, lung X-ray infiltrates disappearance time and reducing TNF-α level. Meanwhile promoting blood circulation therapy seemed to be better at relieving lung rale. However, regarding adverse events, the two groups displayed no statistical difference [RR = 0.97, 95% CI (0.60, 1.57), p = 0.91]. Conclusion: Despite of the apparently positive results in relieving clinical symptoms, physical signs and reducing inflammation, it is premature to confirm the efficacy of CHM in treating RMPP because of the limitation of quality and the number of the included studies. More large-scale, double-blind, well-designed, randomized controlled trials are needed in future research.

The Rollercoaster of Paclitaxel in the Lower Limbs and Skeletons in the Closet: An Opinion Review

There have been concerns about the long-term risk of all-cause death with the use of paclitaxel-coated devices in the lower limbs. Results from a 2018 meta-analysis were corroborated by the US Food and Drug Administration and later confirmed by an individual patient data meta-analysis. However, population-based observational studies have produced contradictory results and often claimed a survival benefit with the use of paclitaxel. The recently published Swedish drug-elution trial in peripheral arterial disease did not confirm a significant mortality risk. In this review, the authors discuss the key elements of the identified mortality signal and stress important facts and figures that remain underrecognized and elusive. They also highlight the important types of epidemiological bias that pertain to the ongoing debate on paclitaxel.

Trial sequential analysis: novel approach for meta-analysis

Systematic reviews and meta-analyses rank the highest in the evidence hierarchy. However, they still have the risk of spurious results because they include too few studies and participants. The use of trial sequential analysis (TSA) has increased recently, providing more information on the precision and uncertainty of meta-analysis results. This makes it a powerful tool for clinicians to assess the conclusiveness of meta-analysis. TSA provides monitoring boundaries or futility boundaries, helping clinicians prevent unnecessary trials. The use and interpretation of TSA should be based on an understanding of the principles and assumptions behind TSA, which may provide more accurate, precise, and unbiased information to clinicians, patients, and policymakers. In this article, the history, background, principles, and assumptions behind TSA are described, which would lead to its better understanding, implementation, and interpretation.

Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in PLoS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in PLoS One, 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.

The Empirical Status of Mindfulness-Based Interventions: A Systematic Review of 44 Meta-Analyses of Randomized Controlled Trials

In response to questions regarding the scientific basis for mindfulness-based interventions (MBIs), we evaluated their empirical status by systematically reviewing meta-analyses of randomized controlled trials (RCTs). We searched six databases for effect sizes based on four or more trials that did not combine passive and active controls. Heterogeneity, moderators, tests of publication bias, risk of bias, and adverse effects were also extracted. Representative effect sizes based on the largest number of studies were identified across a wide range of populations, problems, interventions, comparisons, and outcomes (PICOS). A total of 160 effect sizes were reported in 44 meta-analyses (k = 336 RCTs, N = 30,483 participants). MBIs showed superiority to passive controls across most PICOS (ds = 0.10-0.89). Effects were typically smaller and less often statistically significant compared with active controls. MBIs were similar or superior to specific active controls and evidence-based treatments. Heterogeneity was typically moderate. Few consistent moderators were found. Results were generally robust to publication bias, although other important sources of bias were identified. Reporting of adverse effects was inconsistent. Statistical power may be lacking in meta-analyses, particularly for comparisons with active controls. Because MBIs show promise across some PICOS, future RCTs and meta-analyses should build on identified strengths and limitations of this literature.

Pooling resources to enhance rigour in psychophysiological research: Insights from open science approaches to meta-analysis

Recent years have witnessed calls for increased rigour and credibility in the cognitive and behavioural sciences, including psychophysiology. Many procedures exist to increase rigour, and among the most important is the need to increase statistical power. Achieving sufficient statistical power, however, is a considerable challenge for resource intensive methodologies, particularly for between-subjects designs. Meta-analysis is one potential solution; yet, the validity of such quantitative review is limited by potential bias in both the primary literature and in meta-analysis itself. Here, we provide a non-technical overview and evaluation of open science methods that could be adopted to increase the transparency of novel meta-analyses. We also contrast post hoc statistical procedures that can be used to correct for publication bias in the primary literature. We suggest that traditional meta-analyses, as applied in ERP research, are exploratory in nature, providing a range of plausible effect sizes without necessarily having the ability to confirm (or disconfirm) existing hypotheses. To complement traditional approaches, we detail how prospective meta-analyses, combined with multisite collaboration, could be used to conduct statistically powerful, confirmatory ERP research.

How Can Intranasal Oxytocin Research Be Trusted? A Systematic Review of the Interactive Effects of Intranasal Oxytocin on Psychosocial Outcomes

Over the past two decades, research about the role of oxytocin (OT) in human behavior has grown exponentially. However, a unified theory of OT effects has yet to be developed. Relatedly, growing concerns about the robustness of conclusions drawn in the field have been raised. The current article contributes to this debate by reporting on and discussing key conclusions from a systematic review of published studies addressing the interactive effects of intranasal OT (IN-OT) administration on psychosocial outcomes in a healthy population. The review indicates that (a) tested interactive IN-OT effects were highly heterogeneous; (b) for most published interactions, no replication was attempted; (c) when attempted, replications were largely unsuccessful; (d) significance was unrelated to sample size; (e) statistical power was critically low and unrelated to the rate of significant results; and (f) research practices were characteristic of an exploratory approach. This concerning state of affairs makes it virtually impossible to tease apart true from false interactive IN-OT effects. We provide constructive directions on the basis of this observation and positive predictive value simulations for future research that should help extract true effects from noise and move the IN-OT field forward.

Individual Patient Data Pooled Analysis of Randomized Trials of Bivalirudin versus Heparin in Acute Myocardial Infarction: Rationale and Methodology

BACKGROUND

 Individual randomized controlled trials (RCTs) of periprocedural anticoagulation with bivalirudin versus heparin during percutaneous coronary intervention (PCI) have reported conflicting results. Study-level meta-analyses lack granularity to adjust for confounders, explore heterogeneity, or identify subgroups that may particularly benefit or be harmed.

OBJECTIVE

 To overcome these limitations, we sought to develop an individual patient-data pooled database of RCTs comparing bivalirudin versus heparin.

METHODS

 We conducted a systematic review to identify RCTs in which ≥1,000 patients with acute myocardial infarction (AMI) undergoing PCI were randomized to bivalirudin versus heparin.

RESULTS

 From 738 identified studies, 8 RCTs met the prespecified criteria. The principal investigators of each study agreed to provide patient-level data. The data were pooled and checked for accuracy against trial publications, with discrepancies addressed by consulting with the trialists. Consensus-based definitions were created to resolve differing antithrombotic, procedural, and outcome definitions. The project required 3.5 years to complete, and the final database includes 27,409 patients (13,346 randomized to bivalirudin and 14,063 randomized to heparin).

CONCLUSION

 We have created a large individual patient database of bivalirudin versus heparin RCTs in patients with AMI undergoing PCI. This endeavor may help identify the optimal periprocedural anticoagulation regimen for patient groups with different relative risks of adverse ischemic versus bleeding events, including those with ST-segment and non-ST-segment elevation MI, radial versus femoral access, use of a prolonged bivalirudin infusion or glycoprotein inhibitors, and others. Adherence to standardized techniques and rigorous validation processes should increase confidence in the accuracy and robustness of the results.

European Medicines Agency's Priority Medicines Scheme at 2 Years: An Evaluation of Clinical Studies Supporting Eligible Drugs

The Priority Medicines (PRIME) scheme was launched by the European Medicines Agency (EMA) in 2016 to expedite the development and approval of promising products targeting conditions with high unmet medical need. Manufacturers of PRIME drugs receive extensive regulatory advice on their trial designs. Until June 2018, the EMA granted PRIME status to 39 agents, evaluated in 138 studies (102 initiated before and 36 after PRIME eligibility). A third of the studies forming the basis of PRIME designation were randomized controlled trials, and a quarter of the studies were blinded. There was no statistically significant difference between trials initiated before and after PRIME designation in terms of randomized design and use of blinding. However, significantly more efficacy studies included a clinical end point after PRIME designation than before, and significantly fewer included surrogate measures alone. There were no statistically significant differences between the trial designs of PRIME and non-PRIME-designated products.

Accumulation Bias in meta-analysis: the need to consider time in error control

Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias —Accumulation Bias— and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds — each with their own timing — or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advice against future studies.

Systematic reviews with meta-analysis of clinical trials: Is there enough evidence?

Systematic reviews with meta-analysis are essential tools to reliably summarize the effectiveness of health interventions. A meta-analysis based exclusively on a small number of trials will often not be conclusive or may yield false positive results. The estimation of the optimal information size (OIS) can reduce the risk of obtaining false positive results and determine if additional clinical trials are required. The estimation of the OIS is very similar to that used to estimate the sample size of a clinical trial. The required number of participants for the meta-analysis should be at least as large as a single trial with adequate power. The OIS concept provides a statistical framework in which the accumulated information is convincing to indicate whether more clinical trials are needed to address the effects of the intervention.

The fragility of statistically significant results from clinical nutrition randomized controlled trials

BACKGROUND & AIMS

Recently, a parameter called "Fragility index" (FI) has been proposed, which measures how many events the statistical significance relies on. The lower the FI the more "fragile" the results, and thus more care should be taken when interpreting the results. Our aim in this study was to check FI of nutritional trials.

METHODS

We conducted a systematic review of human clinical nutrition RCTs that report statistically significant dichotomous primary outcomes. We searched the EMBASE, MEDLINE, and Scopus databases. The FI of primary outcomes using the Fisher exact test was calculated and checked the correlations of FI with the number of randomised trials, the p-value of primary outcomes, the publication date, the journal impact factor and the number of patients lost to follow-up.

RESULTS

The initial database search revealed 5790 articles, 37 of which were included in qualitative synthesis. The median (IQR) FI for all studies was 1 (1-3). 28 studies (75.7%) had an FI lower or equal to 2, and in 12 (32.43%) articles, the FI was lower than the number of patients lost to follow-up. No correlations were found between FI and the study characteristics (number of randomized patients, p value of primary outcome, event ratio in experimental group, event ratio in control group, publication date, journal impact factor, lost to follow-up).

CONCLUSION

The results of RCTs in nutritional research often rely on a small number of events or patients. The number of patients lost to follow-up is frequently higher than the FI calculation. Formulating recommendations based on RCTs should be done with caution and FI may be used as auxiliary parameter when assessing the robustness of their findings.

Testosterone, myocardial function, and mortality

The cardiovascular system is particularly sensitive to androgens, but some controversies exist regarding the effect of testosterone on the heart. While among anabolic abusers, cases of sudden cardiac death have been described, recently it was reported that low serum level of testosterone was correlated with increased risk of cardiovascular diseases (CVD) and mortality rate. This review aims to evaluate the effect of testosterone on myocardial tissue function, coronary artery disease (CAD), and death. Low testosterone level is associated with increased incidence of CAD and mortality. Testosterone administration in hypogonadal elderly men and women has a positive effect on cardiovascular function and improved clinical outcomes and survival time. Although at supraphysiologic doses, androgen may have a toxic effect, and at physiological levels, testosterone is safe and exerts a beneficial effect on myocardial function including mechanisms at cellular and mitochondrial level. The interaction with free testosterone and estradiol should be considered. Further studies are necessary to better understand the interaction mechanisms for an optimal androgen therapy in CVD.

Evidence for the effectiveness of chlorhexidine bathing and health care-associated infections among adult intensive care patients: a trial sequential meta-analysis

Background

Health care associated infections (HAI) among adults admitted to the intensive care unit (ICU) have been shown to increase length of stay, the cost of care, and in some cases increased the risk of hospital death (Kaye et al., J Am Geriatr Soc 62:306-11, 2014; Roberts et al., Med Care 48:1026-35, 2010; Warren et al., Crit Care Med 34:2084-9, 2006; Zimlichman et al., JAMA Intern Med 173:2039-46, 2013). Daily bathing with chlorhexidine gluconate (CHG) has been shown to decrease the risk of infection in the ICU (Loveday et al., J Hosp Infect 86:S1-S70, 2014). However, due to varying quality of published studies, and varying estimates of effectiveness, CHG bathing is not universally practiced. As a result, current opinion of the merit of CHG bathing to reduce hospital acquired infections in the ICU, is divergent, suggesting a state of ‘clinical equipoise’.

This trial sequential meta-analysis aims to explore the current status of evidence for the effectiveness of chlorhexidine (CHG) bathing, in adult intensive care patients, to reduce hospital acquired infections, and address the question: do we need more trials?

Methods

A systematic literature search was undertaken to identify trials assessing the effectiveness of chlorhexidine bathing to reduce risk of infection, among adult intensive care patients. With particular focus on: (1) Blood stream infections (BSI); (2) Central Line Associated Blood Stream Infections (CLABSI); (3) Multi-Resistant Drug Organism (MRDO); (4) Ventilator Associated Pneumonia; and, Catheter Associated Urinary Tract Infections (CAUTI). Only randomised-control or cluster randomised cross-over trials, were include in our analysis. A Trial Sequential Analysis (TSA) was used to describe the current status of evidence for the effectiveness of chlorhexidine (CHG) bathing, in adult intensive care patients, to reduce hospital acquired infections.

Results

Five trials were included in our final analysis - two trials were individual patient randomised-controlled, and the remaining cluster-randomised-crossover trials. Daily bathing with CHG was estimated to reduce BSI in the ICU by approximately 29% (Der-Simonian and Laird, Random-Effects.

(DL-RE) Incidence Rate Ratio (IRR) = 0.71, 95% confidence interval (CI) 0.51, 0.98); reduce CLABSI in the ICU by approximately 40% (DL-RE IRR = 0.60, 95% CI 0.34, 1.04); reduce MDRO in the ICU by approximately 18% (DL-RE IRR = 0.82, 95% CI 0.69, 0.98); no effect in reducing VAP in the ICU (DL-RE IRR = 1.33, 95% CI 0.81, 2.18); and, no effect in reducing CAUTI in the ICU (DL-RE IRR = 0.77, 95% CI 0.52, 1.15). Upper (superiority) monitoring boundaries from TSA were not crossed for all five specific infections in the ICU.

Conclusion

Routine bathing with CHG does not occur in the ICU setting, and TSA suggests that more trials are needed to address the current state of ‘clinical equipoise’. Ideally these studies would be conducted among a diverse group of ICU patients, and to the highest standard to ensure generalisability of results.

Expanding our perspectives on research in musculoskeletal science and practice

INTRODUCTION

The quantity and quality of scientific research within physiotherapy has unquestionably grown and matured over the last few decades, especially since the "formal" onset of evidence-based physiotherapy in the 1990s. The urgent need to evaluate our practice for effectiveness and efficiency has been responded to with thought and respect to both scientific integrity and shop-floor clinical needs. However, after thirty years or more of a professionally-governed and strategic approach to research activity, it is now timely to reflect, review, and consider the next chapter in the relationship between scientific research and clinical practice.

PURPOSE

This masterclass aims to develop a research vision for the future of physiotherapy. It is proposed that a crisis is evident within evidence-based physiotherapy, particular so given the assumed complexity and context-sensitivity of our clinical practice. This crisis period has highlighted fundamental limitations within the way research and practice are currently related. These limitations are presented and framed within the problematisation of empirical and philosophical concerns. As research becomes increasingly aligned to traditional scientific principles, examination of the real world context in which its outcomes are intended expose critical challenges for both research and clinical practice.

IMPLICATIONS

A reconceptualisation of fundamental elements of scientific research may allow more meaningful relationships between research and clinical practice. A proposed research vision encourages scientific activity to embrace real-world complexity in a way that it is presently unable to. An enhanced person-centered, scientifically-informed world of effective musculoskeletal practice is envisaged.

Interventions for treating patients with chikungunya virus infection-related rheumatic and musculoskeletal disorders: A systematic review

Background

Chikungunya virus infection (CHIKV) is caused by a mosquito-borne alphavirus. CHIKV causes high fever and painful rheumatic disorders that may persist for years. Because little is known about interventions for treating CHIKV-related illness, we conducted a systematic review.

Methods

We used Cochrane methods. We searched PubMed, EMBASE, Cochrane Library, LILACS and other sources from the earliest records to March 2016. We had no language restrictions. We included randomized controlled trials assessing any intervention for treating acute or chronic CHIKV-related illness. Our primary outcomes were pain relief, global health status (GHS) or health related quality of life (HRQL), and serious adverse events (SAEs). We assessed bias risk with the Cochrane tool and used GRADE to assess evidence quality.

Results

We screened 2,229 records and found five small trials with a total of 402 participants. Patients receiving chloroquine (CHQ) had better chronic pain relief than those receiving placebo (relative risk [RR] 2.67, 95% confidence interval [CI] 1.23 to 5.77, N = 54), but acute pain relief was marginally not different between groups (mean difference [MD] 1.46, 95% CI 0.00 to 2.92, N = 54). SAEs were similar (RR = 15.00, 95% CI 0.90 to 250.24, N = 54). Comparing CHQ with paracetamol (PCM), CHQ patients had better pain relief (RR = 1.52, 95% CI 1.20 to 1.93, N = 86). Compared with hydroxychloroquine (HCHQ), disease-modifying anti-rheumatic drugs (DMARDs) reduced pain (MD = -14.80, 95% CI -19.12 to -10.48, N = 72). DMARDs patients had less disability (MD = -0.74, 95% CI -0.92 to -0.56, N = 72) and less disease activity (MD = -1.35; 95% CI -1.70 to -1.00; N = 72). SAEs were similar between DMARDs and HCHQ groups (RR = 2.84, 95% CI 0.12 to 67.53, N = 72). Comparing meloxicam (MXM) with CHQ, there was no difference in pain relief (MD = 0.24, 95% CI = -0.81 to 1.29; p = 0.65, N = 70), GHS or HRQL (MD = -0.31, 95% CI -2.06 to 1.44, N = 70) or SAEs (RR = 0.85, 95% CI 0.30 to 2.42, N = 70). Finally, a four-arm trial (N = 120) compared aceclofenac (ACF) monotherapy to ACF+HCHQ, ACF+ prednisolone (PRD), or ACF+HCHQ+PRD. Investigators found reduced pain (p<0.001) and better HRQL (p<0.001) in the two patient groups receiving PRD, compared to those receiving ACF monotherapy or ACF+HCHQ. Trials were at high risk of bias. GRADE evidence quality for all outcomes was very low.

Conclusion

Results from these small trials provide insufficient evidence to draw conclusions about the efficacy or safety of CHIKV interventions. Physicians should be cautious in prescribing and policy-makers should be cautious in recommending any intervention reviewed here. Rigorous trials with sufficient statistical power are urgently needed, with results stratified by disease stage and symptomology.

Trial Sequential Analysis in systematic reviews with meta-analysis

BACKGROUND

Most meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors).

METHODS

We developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached.

RESULTS

The Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D²) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis.

CONCLUSIONS

Trial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.

Low statistical power in biomedical science: a review of three human research domains

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0-10% or 11-20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review

OBJECTIVE

Many published meta-analyses are underpowered. We explored the role of trial sequential analysis (TSA) in assessing the reliability of conclusions in underpowered meta-analyses.

METHODS

We screened The Cochrane Database of Systematic Reviews and selected 100 meta-analyses with a binary outcome, a negative result and sufficient power. We defined a negative result as one where the 95% CI for the effect included 1.00, a positive result as one where the 95% CI did not include 1.00, and sufficient power as the required information size for 80% power, 5% type 1 error, relative risk reduction of 10% or number needed to treat of 100, and control event proportion and heterogeneity taken from the included studies. We re-conducted the meta-analyses, using conventional cumulative techniques, to measure how many false positives would have occurred if these meta-analyses had been updated after each new trial. For each false positive, we performed TSA, using three different approaches.

RESULTS

We screened 4736 systematic reviews to find 100 meta-analyses that fulfilled our inclusion criteria. Using conventional cumulative meta-analysis, false positives were present in seven of the meta-analyses (7%, 95% CI 3% to 14%), occurring more than once in three. The total number of false positives was 14 and TSA prevented 13 of these (93%, 95% CI 68% to 98%). In a post hoc analysis, we found that Cochrane meta-analyses that are negative are 1.67 times more likely to be updated (95% CI 0.92 to 2.68) than those that are positive.

CONCLUSIONS

We found false positives in 7% (95% CI 3% to 14%) of the included meta-analyses. Owing to limitations of external validity and to the decreased likelihood of updating positive meta-analyses, the true proportion of false positives in meta-analysis is probably higher. TSA prevented 93% of the false positives (95% CI 68% to 98%).

Comparative effectiveness of exercise and drug interventions on mortality outcomes: metaepidemiological study

Objective

To determine the comparative effectiveness of exercise versus drug interventions on mortality outcomes.

Design

Metaepidemiological study.

Eligibility criteria

Meta-analyses of randomised controlled trials with mortality outcomes comparing the effectiveness of exercise and drug interventions with each other or with control (placebo or usual care).

Data sources

Medline and Cochrane Database of Systematic Reviews, May 2013.

Main outcome measure

Mortality.

Data synthesis

We combined study level death outcomes from exercise and drug trials using random effects network meta-analysis.

Results

We included 16 (four exercise and 12 drug) meta-analyses. Incorporating an additional three recent exercise trials, our review collectively included 305 randomised controlled trials with 339 274 participants. Across all four conditions with evidence on the effectiveness of exercise on mortality outcomes (secondary prevention of coronary heart disease, rehabilitation of stroke, treatment of heart failure, prevention of diabetes), 14 716 participants were randomised to physical activity interventions in 57 trials. No statistically detectable differences were evident between exercise and drug interventions in the secondary prevention of coronary heart disease and prediabetes. Physical activity interventions were more effective than drug treatment among patients with stroke (odds ratios, exercise vanticoagulants 0.09, 95% credible intervals 0.01 to 0.70 and exercise v antiplatelets 0.10, 0.01 to 0.62). Diuretics were more effective than exercise in heart failure (exercise v diuretics 4.11,1.17to 24.76). Inconsistency between direct and indirect comparisons was not significant.

Conclusions

Although limited in quantity, existing randomised trial evidence on exercise interventions suggests that exercise and many drug interventions are often potentially similar in terms of their mortality benefits in the secondary prevention of coronary heart disease, rehabilitation after stroke, treatment of heart failure, and prevention of diabetes.

Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases

Context

There are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.

Objective

Using meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and “others”).

Methods

We analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100%.

Results

Nominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100%. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3% for genetic studies in psychiatry to 86.4% for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and “true” effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.

Conclusion

The differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.

Evidence of surgical outcomes fluctuates over time: results from a cumulative meta-analysis of laparoscopic versus open appendectomy for acute appendicitis

Background

In surgical trials, complex variables such as equipment development and surgeons’ learning curve are involved. The evidence obtained in these trials can thus fluctuate over time. We explored the stability of the evidence obtained during surgery by conducting a cumulative meta-analysis of randomized controlled trials for open and laparoscopic appendectomy.

Methods

We conducted a cumulative meta-analysis of randomized controlled trials comparing laparoscopic appendectomy with open appendectomy for acute appendicitis, a topic with the greatest number of trials in the gastroenterological surgical field. We searched the MEDLINE (PubMed), EMBASE, and CINAHL databases up to September 2014 and reviewed the bibliographies. Outcomes were the incidence of intra-abdominal abscess, incidence of wound infection, operative time, and length of hospital stay. We used the 95 % confidence interval (95 % CI) of effect size for the significance test.

Results

Sixty-four trials were included in this analysis. Of the 51 trials addressing intra-abdominal abscesses, our cumulative meta-analysis of trials published up to and including 2001 demonstrated statistical significance in favor of open appendectomy (cumulative odds ratio [OR] 2.35, 95 % CI 1.30–4.25). The effect size in favor of open procedures began to disappear after 2001, leading to an insignificant result with an overall cumulative OR of 1.32 (95 % CI 0.84–2.10) when laparoscopic appendectomy was compared with open appendectomy.

Conclusions

The evidence regarding treatment effectiveness changed over time, after treatment effectiveness became significant in trials comparing laparoscopic and open appendectomy. Observing only the 95 % confidence interval of effect size from a meta-analysis may not provide conclusive results.

Electronic supplementary material

The online version of this article (doi:10.1186/s12876-016-0453-0) contains supplementary material, which is available to authorized users.

Systematic Reviews of Anesthesiologic Interventions Reported as Statistically Significant: Problems with Power, Precision, and Type 1 Error Protection

BACKGROUND

The GRADE Working Group assessment of the quality of evidence is being used increasingly to inform clinical decisions and guidelines. The assessment involves explicit consideration of all sources of uncertainty. One of these sources is imprecision or random error. Many published meta-analyses are underpowered and likely to be updated in the future. When data are sparse and there are repeated updates, the risk of random error is increased. Trial Sequential Analysis (TSA) is one of several methodologies that estimates this increased risk (and decreased precision) in meta-analyses. With nominally statistically significant meta-analyses of anesthesiologic interventions, we used TSA to estimate power and imprecision in the context of sparse data and repeated updates.

METHODS

We conducted a search to identify all systematic reviews with meta-analyses that investigated an intervention that may be implemented by an anesthesiologist during the perioperative period. We randomly selected 50 meta-analyses that reported a statistically significant dichotomous outcome in their abstract. We applied TSA to these meta-analyses by using 2 main TSA approaches: relative risk reduction 20% and relative risk reduction consistent with the conventional 95% confidence limit closest to null. We calculated the power achieved by each included meta-analysis, by using each TSA approach, and we calculated the proportion that maintained statistical significance when allowing for sparse data and repeated updates.

RESULTS

From 11,870 titles, we found 682 systematic reviews that investigated anesthesiologic interventions. In the 50 sampled meta-analyses, the median number of trials included was 8 (interquartile range [IQR], 5-14), the median number of participants was 964 (IQR, 523-1736), and the median number of participants with the outcome was 202 (IQR, 96-443). By using both of our main TSA approaches, only 12% (95% CI, 5%-25%) of the meta-analyses had power ≥ 80%, and only 32% (95% CI, 20%-47%) of the meta-analyses preserved the risk of type 1 error <5%.

CONCLUSIONS

Most nominally statistically significant meta-analyses of anesthesiologic interventions are underpowered, and many do not maintain their risk of type 1 error <5% if TSA monitoring boundaries are applied. Consideration of the effect of sparse data and repeated updates is needed when assessing the imprecision of meta-analyses of anesthesiologic interventions.

Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey

OBJECTIVE

To assess differences in estimated treatment effects for mortality between observational studies with routinely collected health data (RCD; that are published before trials are available) and subsequent evidence from randomized controlled trials on the same clinical question.

DESIGN

Meta-epidemiological survey.

DATA SOURCES

PubMed searched up to November 2014.

METHODS

Eligible RCD studies were published up to 2010 that used propensity scores to address confounding bias and reported comparative effects of interventions for mortality. The analysis included only RCD studies conducted before any trial was published on the same topic. The direction of treatment effects, confidence intervals, and effect sizes (odds ratios) were compared between RCD studies and randomized controlled trials. The relative odds ratio (that is, the summary odds ratio of trial(s) divided by the RCD study estimate) and the summary relative odds ratio were calculated across all pairs of RCD studies and trials. A summary relative odds ratio greater than one indicates that RCD studies gave more favorable mortality results.

RESULTS

The evaluation included 16 eligible RCD studies, and 36 subsequent published randomized controlled trials investigating the same clinical questions (with 17,275 patients and 835 deaths). Trials were published a median of three years after the corresponding RCD study. For five (31%) of the 16 clinical questions, the direction of treatment effects differed between RCD studies and trials. Confidence intervals in nine (56%) RCD studies did not include the RCT effect estimate. Overall, RCD studies showed significantly more favorable mortality estimates by 31% than subsequent trials (summary relative odds ratio 1.31 (95% confidence interval 1.03 to 1.65; I(2)=0%)).

CONCLUSIONS

Studies of routinely collected health data could give different answers from subsequent randomized controlled trials on the same clinical questions, and may substantially overestimate treatment effects. Caution is needed to prevent misguided clinical decision making.

Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies

Over the last decade, oxytocin (OT) has received focus in numerous studies associating intranasal administration of this peptide with various aspects of human social behavior. These studies in humans are inspired by animal research, especially in rodents, showing that central manipulations of the OT system affect behavioral phenotypes related to social cognition, including parental behavior, social bonding, and individual recognition. Taken together, these studies in humans appear to provide compelling, but sometimes bewildering, evidence for the role of OT in influencing a vast array of complex social cognitive processes in humans. In this article, we investigate to what extent the human intranasal OT literature lends support to the hypothesis that intranasal OT consistently influences a wide spectrum of social behavior in humans. We do this by considering statistical features of studies within this field, including factors like statistical power, prestudy odds, and bias. Our conclusion is that intranasal OT studies are generally underpowered and that there is a high probability that most of the published intranasal OT findings do not represent true effects. Thus, the remarkable reports that intranasal OT influences a large number of human social behaviors should be viewed with healthy skepticism, and we make recommendations to improve the reliability of human OT studies in the future.

Combining Antidepressants in Acute Treatment of Depression: A Meta-Analysis of 38 Studies Including 4511 Patients

OBJECTIVE

Combining antidepressants (ADs) for therapy of acute depression is frequently employed, but randomized studies have yielded conflicting results. We conducted a systematic review and meta-analysis aimed at determining efficacy and tolerability of combination therapy.

METHODS

MEDLINE, Embase, PsycINFO, and CENTRAL databases were systematically searched through March 2014 for controlled studies comparing combinations of ADs with AD monotherapy in adult patients suffering from acute depression. The prespecified primary outcome was standardized mean difference (SMD), secondary outcomes were response, remission, and dropouts.

RESULTS

Among 8688 articles screened, 38 studies were eligible, including 4511 patients. Combination treatment was statistically, significantly superior to monotherapy (SMD 0.29; 95% CI 0.16 to 0.42). During monotherapy, slightly fewer patients dropped out due to adverse events (OR 0.90; 95% CI 0.53 to 1.53). Studies were heterogeneous (I(2) = 63%), and there was indication of moderate publication bias (fail-safe N for an effect of 0.1:44), but results remained robust across prespecified secondary outcomes and subgroups, including analyses restricted to randomized controlled trials and low risk of bias studies. Meta-regression revealed an association of SMD with difference in imipramine-equivalent dose. Combining a reuptake inhibitor with an antagonist of presynaptic α2-autoreceptors was superior to other combinations.

CONCLUSION

Combining ADs seems to be superior to monotherapy with only slightly more patients dropping out. Combining a reuptake inhibitor with an antagonist of presynaptic α2-autoreceptors seems to be significantly more effective than other combinations. Overall, our search revealed a dearth of well-designed studies.

Performance of informative priors skeptical of large treatment effects in clinical trials: A simulation study

One of the main advantages of Bayesian analyses of clinical trials is their ability to formally incorporate skepticism about large treatment effects through the use of informative priors. We conducted a simulation study to assess the performance of informative normal, Student- t, and beta distributions in estimating relative risk (RR) or odds ratio (OR) for binary outcomes. Simulation scenarios varied the prior standard deviation (SD; level of skepticism of large treatment effects), outcome rate in the control group, true treatment effect, and sample size. We compared the priors with regards to bias, mean squared error (MSE), and coverage of 95% credible intervals. Simulation results show that the prior SD influenced the posterior to a greater degree than the particular distributional form of the prior. For RR, priors with a 95% interval of 0.50-2.0 performed well in terms of bias, MSE, and coverage under most scenarios. For OR, priors with a wider 95% interval of 0.23-4.35 had good performance. We recommend the use of informative priors that exclude implausibly large treatment effects in analyses of clinical trials, particularly for major outcomes such as mortality.