A test for publication bias in meta-analysis with sparse binary data

Erector spinae plane block for postoperative pain

BACKGROUND

Acute and chronic postoperative pain are important healthcare problems, which can be treated with a combination of opioids and regional anaesthesia. The erector spinae plane block (ESPB) is a new regional anaesthesia technique, which might be able to reduce opioid consumption and related side effects.

OBJECTIVES

To compare the analgesic effects and side effect profile of ESPB against no block, placebo block or other regional anaesthetic techniques.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase and Web of Science on 4 January 2021 and updated the search on 3 January 2022.

SELECTION CRITERIA

Randomised controlled trials (RCTs) investigating adults undergoing surgery with general anaesthesia were included. We included ESPB in comparison with no block, placebo blocks or other regional anaesthesia techniques irrespective of language, publication year, publication status or technique of regional anaesthesia used (ultrasound, landmarks or peripheral nerve stimulator). Quasi-RCTs, cluster-RCTs, cross-over trials and studies investigating co-interventions in either arm were excluded.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed all trials for inclusion and exclusion criteria, and risk of bias (RoB), and extracted data. We assessed risk of bias using the Cochrane RoB 2 tool, and we used GRADE to rate the certainty of evidence for the primary outcomes. The primary outcomes were postoperative pain at rest at 24 hours and block-related adverse events. Secondary outcomes were postoperative pain at rest (2, 48 hours) and during activity (2, 24 and 48 hours after surgery), chronic pain after three and six months, as well as cumulative oral morphine requirements at 2, 24 and 48 hours after surgery and rates of opioid-related side effects.

MAIN RESULTS

We identified 69 RCTs in the first search and included these in the systematic review. We included 64 RCTs (3973 participants) in the meta-analysis. The outcome postoperative pain was reported in 38 out of 64 studies; block-related adverse events were reported in 40 out of 64 studies. We assessed RoB as low in 44 (56%), some concerns in 24 (31%) and high in 10 (13%) of the study results. Overall, 57 studies reported one or both primary outcomes. Only one study reported results on chronic pain after surgery. In the updated literature search on 3 January 2022 we found 37 new studies and categorised these as awaiting classification. ESPB compared to no block There is probably a slight but not clinically relevant reduction in pain intensity at rest 24 hours after surgery in patients treated with ESPB compared to no block (visual analogue scale (VAS), 0 to 10 points) (mean difference (MD) -0.77 points, 95% confidence interval (CI) -1.08 to -0.46; 17 trials, 958 participants; moderate-certainty evidence). There may be no difference in block-related adverse events between the groups treated with ESPB and those receiving no block (no events in 18 trials reported, 1045 participants, low-certainty evidence). ESPB compared to placebo block ESPB probably has no effect on postoperative pain intensity at rest 24 hours after surgery compared to placebo block (MD -0.14 points, 95% CI -0.29 to 0.00; 8 trials, 499 participants; moderate-certainty evidence). There may be no difference in block-related adverse events between ESPB and placebo blocks (no events in 10 trials reported; 592 participants; low-certainty evidence). ESPB compared to other regional anaesthetic techniques Paravertebral block (PVB) ESPB may not have any additional effect on postoperative pain intensity at rest 24 hours after surgery compared to PVB (MD 0.23 points, 95% CI -0.06 to 0.52; 7 trials, 478 participants; low-certainty evidence). There is probably no difference in block-related adverse events (risk ratio (RR) 0.27, 95% CI 0.08 to 0.95; 7 trials, 522 participants; moderate-certainty evidence). Transversus abdominis plane block (TAPB) ESPB may not have any additional effect on postoperative pain intensity at rest 24 hours after surgery compared to TAPB (MD -0.16 points, 95% CI -0.46 to 0.14; 3 trials, 160 participants; low-certainty evidence). There may be no difference in block-related adverse events (RR 1.00, 95% CI 0.21 to 4.83; 4 trials, 202 participants; low-certainty evidence). Serratus anterior plane block (SAPB) The effect on postoperative pain could not be assessed because no studies reported this outcome. There may be no difference in block-related adverse events (RR 1.00, 95% CI 0.06 to 15.59; 2 trials, 110 participants; low-certainty evidence). Pectoralis plane block (PECSB) ESPB may not have any additional effect on postoperative pain intensity at rest 24 hours after surgery compared to PECSB (MD 0.24 points, 95% CI -0.11 to 0.58; 2 trials, 98 participants; low-certainty evidence). The effect on block-related adverse events could not be assessed. Quadratus lumborum block (QLB) Only one study reported on each of the primary outcomes. Intercostal nerve block (ICNB) ESPB may not have any additional effect on postoperative pain intensity at rest 24 hours after surgery compared to ICNB, but this is uncertain (MD -0.33 points, 95% CI -3.02 to 2.35; 2 trials, 131 participants; very low-certainty evidence). There may be no difference in block-related adverse events, but this is uncertain (RR 0.09, 95% CI 0.04 to 2.28; 3 trials, 181 participants; very low-certainty evidence). Epidural analgesia (EA) We are uncertain whether ESPB has an effect on postoperative pain intensity at rest 24 hours after surgery compared to EA (MD 1.20 points, 95% CI -2.52 to 4.93; 2 trials, 81 participants; very low-certainty evidence). A risk ratio for block-related adverse events was not estimable because only one study reported this outcome.

AUTHORS' CONCLUSIONS

ESPB in addition to standard care probably does not improve postoperative pain intensity 24 hours after surgery compared to no block. The number of block-related adverse events following ESPB was low. Further research is required to study the possibility of extending the duration of analgesia. We identified 37 new studies in the updated search and there are three ongoing studies, suggesting possible changes to the effect estimates and the certainty of the evidence in the future.

Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies

When comparing the performance of two or more competing tests, simulation studies commonly focus on statistical power. However, if the size of the tests being compared are either different from one another or from the nominal size, comparing tests based on power alone may be misleading. By analogy with diagnostic accuracy studies, we introduce relative positive and negative likelihood ratios to factor in both power and size in the comparison of multiple tests. We derive sample size formulas for a comparative simulation study. As an example, we compared the performance of six statistical tests for small-study effects in meta-analyses of randomized controlled trials: Begg's rank correlation, Egger's regression, Schwarzer's method for sparse data, the trim-and-fill method, the arcsine-Thompson test, and Lin and Chu's combined test. We illustrate that comparing power alone, or power adjusted or penalized for size, can be misleading, and how the proposed likelihood ratio approach enables accurate comparison of the trade-off between power and size between competing tests.

Exploratory systematic review and meta-analysis on period poverty

BACKGROUND

Period poverty is a global health and social issue that needs to be addressed. It has been reported that many females compromise their education, employment, and social commitments during their menstruation days due to a number of reasons, including lack of access to toilets or menstrual products.

AIM

To provide a comprehensive understanding on period poverty, including outcomes associated with menstruation.

METHODS

All observational and randomised clinical trials reporting menstruation challenges, menstrual poverty and menstrual products were included. Our search strategy included multiple electronic databases of PubMed, Web of Science, ScienceDirect, ProQuest and EMBASE. Studies published in a peer review journal in English between the 30^th of April 1980 and the 30^th of April 2022 were included. The Newcastle-Ottawa Scale was used to assess the risk of bias of the systematic included studies. Pooled odds ratios (ORs) together with 95% confidence intervals (CIs) are reported overall and for sub-groups.

RESULTS

A total of 80 studies were systematically selected, where 38 were included in the meta-analysis. Of the 38 studies, 28 focused on children and young girls (i.e., 10-24 years old) and 10 included participants with a wider age range of 15-49 years. The prevalence of using disposable sanitary pads was 45% (95%CI: 0.35-0.58). The prevalence of menstrual education pre-menarche was 68% (95%CI: 0.56-0.82). The prevalence of good menstrual hygiene management (MHM) was 39% (95%CI: 0.25-0.61). Women in rural areas (OR = 0.30, 95%CI: 0.13-0.69) were 0.70 times less likely to have good MHM practices than those living in urban areas.

CONCLUSION

There was a lack of evidence, especially from low- and middle- income countries. Further research to better understand the scope and prevalence of period poverty should be considered. This will enable the development of improved policies to increase access to menstrual products and medical support where necessary.

The hard numbers of tuberculosis epidemiology in wildlife: A meta-regression and systematic review

Tuberculosis (TB) is a widespread disease that crosses the human and animal health boundaries, with infection being reported in wildlife, from temperate and subtropical to arctic regions. Often, TB in wild species is closely associated with disease occurrence in livestock but the TB burden in wildlife remains poorly quantified on a global level. Through meta-regression and systematic review, this study aimed to summarize global information on TB prevalence in commonly infected wildlife species and to draw a global picture of the scientific knowledge accumulated in wildlife TB. For these purposes, a literature search was conducted through the Web of Science and Google Scholar. The 223 articles retrieved, concerning a 39-year period, were submitted to bibliometric analysis and 54 publications regarding three wildlife hosts fulfilled the criteria for meta-regression. Using a random-effects model, the worldwide pooled TB prevalence in wild boar is higher than for any other species and estimated as 21.98%, peaking in Spain (31.68%), Italy (23.84%) and Hungary (18.12%). The pooled prevalence of TB in red deer is estimated at 13.71%, with Austria (31.58%), Portugal (27.75%), New Zealand (19.26%) and Spain (12.08%) positioning on the top, while for European badger it was computed 11.75%, peaking in the UK (16.43%) and Ireland (22.87%). Despite these hard numbers, a declining trend in wildlife TB prevalence is apparent over the last decades. The overall heterogeneity calculated by multivariable regression ranged from 28.61% (wild boar) to 60.92% (red deer), indicating that other unexplored moderators could explain disease burden. The systematic review shows that the most prolific countries contributing to knowledge related with wildlife TB are settled in Europe and Mycobacterium bovis is the most reported pathogen (89.5%). This study provides insight into the global epidemiology of wildlife TB, ascertaining research gaps that need to be explored and informing how should surveillance be refined.

Using Nonword Repetition to Identify Language Impairment in Bilingual Children: A Meta-Analysis of Diagnostic Accuracy

Purpose Nonword repetition has been endorsed as a less biased method of assessment for children from culturally and linguistically diverse backgrounds, but there are currently no systematic reviews or meta-analyses on its use with bilingual children. The purpose of this study was to evaluate diagnostic accuracy of nonword repetition in the identification of language impairment (LI) in bilingual children. Method Using a key word search of peer-reviewed literature from several large electronic databases, as well as ancestral and forward searches, 13 studies were identified that met the eligibility criteria. Studies were evaluated on the basis of quality of evidence, design characteristics, and reported diagnostic accuracy. A meta-regression analysis, based on study results, was conducted to identify task characteristics that may be associated with better classification accuracy. Results Diagnostic accuracy across studies ranged from poor to good. Bilingual children with LI performed with more difficulty on nonword repetition tasks than those with typical language. Quasi-universal tasks, which account for the phonotactic constraints of multiple languages, exhibited better diagnostic accuracy and resulted in less misidentification of children with typical language than language-specific tasks. Conclusions Evidence suggests that nonword repetition may be a useful tool in the assessment and screening of LI in bilingual children, though it should be used in conjunction with other measures. Quasi-universal tasks demonstrate the potential to further reduce assessment bias, but extant research is limited.

Using clinical trial registries to inform Copas selection model for publication bias in meta-analysis

Prospective registration of study protocols in clinical trial registries is a useful way to minimize the risk of publication bias in meta-analysis, and several clinical trial registries are available nowadays. However, they are mainly used as a tool for searching studies and information submitted to the registries has not been utilized as efficiently as it could. In addressing publication bias in meta-analyses, sensitivity analysis with the Copas selection model is a more objective alternative to widely-used graphical methods such as the funnel-plot and the trim-and-fill method. Despite its ability to quantify the potential impact of publication bias, the Copas selection model relies on sensitivity analyses, in which some parameters are varied across a certain range. This may result in some difficulty in interpreting the results. In this paper, we propose an alternative inference procedure for the Copas selection model by utilizing information from clinical trial registries. Our method provides a simple and accurate way to estimate all unknown parameters of the Copas selection model. A simulation study revealed that our proposed method resulted in smaller biases and more accurate confidence intervals than existing methods. Furthermore, three published meta-analyses were re-analyzed to demonstrate how to implement the proposed method in practice.

Tests for publication bias are unreliable in case of heteroscedasticity

Regression based methods for the detection of publication bias in meta-analysis have been extensively evaluated in literature. When dealing with continuous outcomes, specific hidden factors (e.g., heteroscedasticity) may interfere with the test statistics. In this paper we investigate the influence of residual heteroscedasticity on the performance of four tests for publication bias: the Egger test, the Begg-Mazumdar test and two tests based on weighted regression. In the presence of heteroscedasticity, the Egger test and the weighted regression tests highly inflate the Type I error rate, while the Begg-Mazumdar test deflates the Type I error rate. Although all three tests already have low statistical power, heteroscedasticity typically reduces it further. Our results in combination with earlier discussions on publication bias tests lead us to conclude that application of these tests on continuous treatment effects is not warranted.

EMBRACE: An EM-based bias reduction approach through Copas-model estimation for quantifying the evidence of selective publishing in network meta-analysis

Systematic reviews and meta-analyses synthesize results from well-conducted studies to optimize healthcare decision-making. Network meta-analysis (NMA) is particularly useful for improving precision, drawing new comparisons, and ranking multiple interventions. However, recommendations can be misled if published results are a selective sample of what has been collected by trialists, particularly when publication status is related to the significance of the findings. Unfortunately, the missing-not-at-random nature of this problem and the numerous parameters involved in modeling NMAs pose unique computational challenges to quantifying and correcting for publication bias, such that sensitivity analysis is used in practice. Motivated by this important methodological gap, we developed a novel and stable expectation-maximization (EM) algorithm to correct for publication bias in the network setting. We validate the method through simulation studies and show that it achieves substantial bias reduction in small to moderately sized NMAs. We also calibrate the method against a Bayesian analysis of a published NMA on antiplatlet therapies for maintaining vascular patency.

Estimates of the global and continental burden of animal tuberculosis in key livestock species worldwide: A meta-analysis study

Zoonotic animal tuberculosis (TB) is a One Health paradigm infectious disease, caused by Mycobacterium tuberculosis complex bacteria, that affects different host species with varying levels of management. In most developed countries, official surveillance and control strategies support the longitudinal reporting of herd and/or animal prevalence. However, for under resourced countries without surveillance plans, this information may be obtained from cross-sectional studies only. The objective of this meta-analysis was to perform a worldwide estimate of the overall prevalence of animal TB in different livestock species whose importance in production systems varies according to the region of the world. The ISI's Web of Science and Google Scholar were searched combining keywords and related database-specific subject terms to identify relevant cohort or cross-sectional work published in this topic. A total of 443 articles were retrieved, screened, and a final set of 182 references included. Potential sources of variation were investigated using subgroup analyses and meta-regression. Prevalence estimates in five mammalian host groups were stratified according to host species, host characteristics, anatomical localization of lesions, sample size, geographical location, and diagnostic tests. The multivariable meta-regression analysis accounted for a range between 0% (farmed wild boar) and 68.71% (camelids) of the overall observed heterogeneity, indicating that the pondered predictors partially explain the observed variability. Differences in the overall prevalence of TB across hosts were small, with most groups showing values around 10%, except farmed wild boar (41%). The sample size emerged as an important moderator, with small size studies leading to the overestimation of prevalence. TB prevalence rates were very heterogeneous across continents and depended on the host, with lower values (below 10%) in Africa and Asia, while North America (33.6%, cattle), Europe (51%, goats), and South America (85.7%, pigs) exhibited higher rates, possibly related to greater densities of specific host groups managed on more intensive production systems. Stratification by diagnostic tests evidenced heterogeneous prevalence rates depending on the host group, possibly reflecting differences in test performance across different hosts. Results from this study highlight different TB burden scenarios, pinpointing host groups and diagnostics that should be prioritized in surveillance systems in different regions, thus providing policy-relevant information to catalyse TB control in settings with lower installed capacity and better resource allocation at the human-animal-environment interface.

A historical review of publication bias

Publication bias is a well-known threat to the validity of meta-analyses and, more broadly, the reproducibility of scientific findings. When policies and recommendations are predicated on an incomplete evidence base, it undermines the goals of evidence-based decision-making. Great strides have been made in the last 50 years to understand and address this problem, including calls for mandatory trial registration and the development of statistical methods to detect and correct for publication bias. We offer an historical account of seminal contributions by the evidence synthesis community, with an emphasis on the parallel development of graph-based and selection model approaches. We also draw attention to current innovations and opportunities for future methodological work.

A Bayesian approach to assessing small-study effects in meta-analysis of a binary outcome with controlled false positive rate

Publication bias threatens meta-analysis validity. It is often assessed via the funnel plot; an asymmetric plot implies small-study effects, and publication bias is one cause of the asymmetry. Egger's regression test is a widely used tool to quantitatively assess such asymmetry. It examines the association between the observed effect sizes and their sample SEs; a strong association indicates small-study effects. However, its false positive rates may be inflated if such an association intrinsically exists even if no small-study effects appear, particularly in meta-analyses of odds ratios (ORs). Various alternatives are available to address this problem. They usually replace Egger's regression predictor or response with different measures; consequently, they are powerful only in specific cases. We propose a Bayesian approach to assessing small-study effects in meta-analyses of ORs. It controls false positive rates by using latent "true" SEs, rather than sample SEs, in the Egger-type regression to avoid the intrinsic association between ORs and their SEs. Although "true" SEs are unknown in practice, they can be modeled under the Bayesian framework. We use simulated and real data to compare various methods. When ORs are away from 1, the proposed method may have high powers with controlled false positive rates, while Egger's test has seriously inflated false positive rates; nevertheless, in other situations, some other methods may be superior. In general, the proposed method may serve as an alternative to rule out potential confounding effects caused by the intrinsic association between ORs and their SEs in the assessment of small-study effects.

Risk of malignancies in patients with spondyloarthritis treated with biologics compared with those treated with non-biologics: a systematic review and meta-analysis

Background:

The aim of our study was to synthesize evidence on the occurrence of malignancy in spondyloarthritis (SpA), from randomized controlled trials (RCTs) comparing biologics with non-biologics and biologics to each other.

Methods:

We systematically searched Medline, Cochrane Library, EMBASE, Scopus and ClinicalTrials.gov from inception until 31 October 2018. RCTs with ⩾24-week follow-up were included. We extracted data using standardized forms and assessed the risk of bias using the Cochrane Risk of Bias Tool. We performed pair-wise meta-analyses and network meta-analyses to compare the risk of malignancy for each biologics class and SpA type. We reported the Peto odds ratio (OR) of any malignancy along with 95% confidence intervals (95% CI). Bayesian posterior probabilities comparing risk of malignancy of each biologic class with non-biologics were computed as supplementary measures.

Results:

Fifty-four trials were included; most (44/54) had follow-up <1 year. Among 14,245 patients, 63 developed a malignancy. While most Peto ORs were >1, they had wide 95% CI and p >0.05. The overall Peto OR comparing biologics with non-biologics was 1.42 (95% CI 0.80–2.53). Only interleukin-17 inhibitors in peripheral SpA had p <0.05 (Peto OR 2.77, 95% CI 1.07–7.13); the posterior probability that the risk was higher than non-biologics was 98%. Stratified analyses revealed no consistent trend by prior exposure to biologics, duration of follow-up, study quality, study-arm crossover, analytical approaches and type of malignancy.

Conclusions:

Our findings indicate no overall elevated risk of malignancy with biologics in SpA. As our meta-analyses are unable to conclude on the long-term risk, long-term pharmacovigilance of biologics in SpA may still be warranted.

Quantitative meta-analysis of heart rate variability finds reduced parasympathetic cardiac tone in women compared to men during laboratory-based social stress

Heart rate variability (HRV) is the inter-beat interval variation between consecutive heartbeats and an autonomic reflection of emotional regulatory abilities to flexibly respond to challenges, such as psychosocial stress. Whereas there are known sex differences in stress-induced hormonal and emotional responses, we identified a gap in our understanding of sex-specific autonomic cardiac control during stress. Thus, we assessed HRV prior to, during and after administration of a public speech task in healthy participants (n = 929) according to sex. Our meta-analysis found that during stress, women had lower HRV than men, with an overall Hedges' g of 0.29 (p < 0.0001) and 0.29 (p = 0.0003) for fixed and random effects models, respectively. We did not find significant heterogeneity or evidence of publication bias. Analyses of additional timepoints showed no baseline difference and marginally lower HRV in women during anticipation and recovery. Findings of the present meta-analysis confirm sex differences in stress-induced hyperarousal and form a justification for implementation of mechanistic studies evaluating gonadal hormones, their potent metabolites and pro-inflammatory cytokines as mediators of this relationship.

Hybrid test for publication bias in meta-analysis

Publication bias frequently appears in meta-analyses when the included studies' results (e.g., p-values) influence the studies' publication processes. Some unfavorable studies may be suppressed from publication, so the meta-analytic results may be biased toward an artificially favorable direction. Many statistical tests have been proposed to detect publication bias in recent two decades. However, they often make dramatically different assumptions about the cause of publication bias; therefore, they are usually powerful only in certain cases that support their particular assumptions, while their powers may be fairly low in many other cases. Although several simulation studies have been carried out to compare different tests' powers under various situations, it is typically infeasible to justify the exact mechanism of publication bias in a real-world meta-analysis and thus select the corresponding optimal publication bias test. We introduce a hybrid test for publication bias by synthesizing various tests and incorporating their benefits, so that it maintains relatively high powers across various mechanisms of publication bias. The superior performance of the proposed hybrid test is illustrated using simulation studies and three real-world meta-analyses with different effect sizes. It is compared with many existing methods, including the commonly used regression and rank tests, and the trim-and-fill method.

Higher Circulating Cortisol in the Follicular vs. Luteal Phase of the Menstrual Cycle: A Meta-Analysis

Although results of animal research show that interactions between stress and sex hormones are implicated in the development of affective disorders in women, translation of these findings to patients has been scarce. As a basic step toward advancing this field of research, we analyzed findings of studies which reported circulating cortisol levels in healthy women in the follicular vs. luteal phase of the menstrual cycle. We deemed this analysis critical not only to advance our understanding of basic physiology, but also as an important contrast to the findings of future studies evaluating stress and sex hormones in women with affective disorders. We hypothesized that cortisol levels would be lower in the follicular phase based on the proposition that changes in levels of potent GABAergic neurosteroids, including allopregnanolone, during the menstrual cycle dynamically change in the opposite direction relative to cortisol levels. Implementing strict inclusion criteria, we compiled results of high-quality studies involving 778 study participants to derive a standardized mean difference between circulating cortisol levels in the follicular vs. luteal phase of the menstrual cycle. In line with our hypothesis, our meta-analysis found that women in the follicular phase had higher cortisol levels than women in the luteal phase, with an overall Hedges' g of 0.13 (p < 0.01) for the random effects model. No significant between-study difference was detected, with the level of heterogeneity in the small range. Furthermore, there was no evidence of publication bias. As cortisol regulation is a delicate process, we review some of the basic mechanisms by which progesterone, its potent metabolites, and estradiol regulate cortisol output and circulation to contribute to the net effect of higher cortisol in the follicular phase.

Interpretation of endpoints in a network meta-analysis of new oral anticoagulants following total hip or total knee replacement surgery

New oral anticoagulant (NOAC) regimens [dabigatran 150 mg (D150) and 220 mg (D220), rivaroxaban 10 mg (R20), and apixaban 2.5 mg bid (A5)] were effective and safe compared to enoxaparin for the prevention of venous thromboembolism (VTE) following elective total knee (TKR) or hip replacement (THR) surgery. First a cluster analysis was used to identify homogeneous studies for the trial programs of each NOAC. Second, only studies reporting VTE and VTE-related death, major bleeding, and mortality were included. The odds ratio (OR) and 95% confidence interval (CI) were calculated for each NOAC regimen versus the comparator. Third, these data were used for the indirect comparison between NOACs. Cluster analysis identified duration of treatment (10 ± 5 and 34 ± 5 days) as the only homogeneous parameter across all NOAC programs (p>0.05) except for A5 and VTE over 10 ± 5 days (analysis not performed). The results of the calculated OR and 95% CI of the four NOAC regimens over 10 ± 5 and 34 ± 5 days showed inferiority of D150 and D220 compared to R10 for VTE (p<0.01, p<0.001). Comparisons of major bleeding and mortality were not different for all indirect comparisons. Despite the lack of standard definitions for VTE and bleeding outcomes, cluster analysis seems to be an appropriate tool to identify homogeneity across trial programs and to perform an indirect comparison for NOACs for prevention of VTE following TKR and THR surgery.

Methodological Standards for Meta-Analyses and Qualitative Systematic Reviews of Cardiac Prevention and Treatment Studies: A Scientific Statement From the American Heart Association

Meta-analyses are becoming increasingly popular, especially in the fields of cardiovascular disease prevention and treatment. They are often considered to be a reliable source of evidence for making healthcare decisions. Unfortunately, problems among meta-analyses such as the misapplication and misinterpretation of statistical methods and tests are long-standing and widespread. The purposes of this statement are to review key steps in the development of a meta-analysis and to provide recommendations that will be useful for carrying out meta-analyses and for readers and journal editors, who must interpret the findings and gauge methodological quality. To make the statement practical and accessible, detailed descriptions of statistical methods have been omitted. Based on a survey of cardiovascular meta-analyses, published literature on methodology, expert consultation, and consensus among the writing group, key recommendations are provided. Recommendations reinforce several current practices, including protocol registration; comprehensive search strategies; methods for data extraction and abstraction; methods for identifying, measuring, and dealing with heterogeneity; and statistical methods for pooling results. Other practices should be discontinued, including the use of levels of evidence and evidence hierarchies to gauge the value and impact of different study designs (including meta-analyses) and the use of structured tools to assess the quality of studies to be included in a meta-analysis. We also recommend choosing a pooling model for conventional meta-analyses (fixed effect or random effects) on the basis of clinical and methodological similarities among studies to be included, rather than the results of a test for statistical heterogeneity.

Methods for detecting, quantifying, and adjusting for dissemination bias in meta-analysis are described

OBJECTIVE

To systematically review methodological articles which focus on nonpublication of studies and to describe methods of detecting and/or quantifying and/or adjusting for dissemination in meta-analyses. To evaluate whether the methods have been applied to an empirical data set for which one can be reasonably confident that all studies conducted have been included.

STUDY DESIGN AND SETTING

We systematically searched Medline, the Cochrane Library, and Web of Science, for methodological articles that describe at least one method of detecting and/or quantifying and/or adjusting for dissemination bias in meta-analyses.

RESULTS

The literature search retrieved 2,224 records, of which we finally included 150 full-text articles. A great variety of methods to detect, quantify, or adjust for dissemination bias were described. Methods included graphical methods mainly based on funnel plot approaches, statistical methods, such as regression tests, selection models, sensitivity analyses, and a great number of more recent statistical approaches. Only few methods have been validated in empirical evaluations using unpublished studies obtained from regulators (Food and Drug Administration, European Medicines Agency).

CONCLUSION

We present an overview of existing methods to detect, quantify, or adjust for dissemination bias. It remains difficult to advise which method should be used as they are all limited and their validity has rarely been assessed. Therefore, a thorough literature search remains crucial in systematic reviews, and further steps to increase the availability of all research results need to be taken.

What type, or combination of exercise can improve preferred gait speed in older adults? A meta-analysis

BACKGROUND

Improved preferred gait speed in older adults is associated with increased survival rates. There are inconsistent findings in clinical trials regarding effects of exercise on preferred gait speed, and heterogeneity in interventions in the current reviews and meta-analyses.

OBJECTIVE

to determine the meta-effects of different types or combinations of exercise interventions from randomized controlled trials on improvement in preferred gait speed.

METHODS

DATA SOURCES

A literature search was performed; the following databases were searched for studies from 1990 up to 9 December 2013: PubMed, EMBASE, EBSCO (AMED, CINAHL, ERIC, Medline, PsycInfo, and SocINDEX), and the Cochrane Library.

STUDY ELIGIBILITY CRITERIA

Randomized controlled trials of exercise interventions for older adults ≥ 65 years, that provided quantitative data (mean/SD) on preferred gait speed at baseline and post-intervention, as a primary or secondary outcome measure in the published article were included. Studies were excluded when the PEDro score was ≤4, or if participants were selected for a specific neurological or neurodegenerative disease, Chronic Obstructive Pulmonary Disease, cardiovascular disease, recent lower limb fractures, lower limb joint replacements, or severe cognitive impairments. The meta-effect is presented in Forest plots with 95 % confidence

STUDY APPRAISAL AND SYNTHESIS METHODS

intervals and random weights assigned to each trial. Homogeneity and risk of publication bias were assessed.

RESULTS

Twenty-five studies were analysed in this meta-analysis. Data from six types or combinations of exercise interventions were pooled into sub-analyses. First, there is a significant positive meta-effect of resistance training progressed to 70-80 % of 1RM on preferred gait speed of 0.13 [CI 95 % 0.09-0.16] m/s. The difference between intervention- and control groups shows a substantial meaningful change (>0.1 m/s). Secondly, a significant positive meta-effect of interventions with a rhythmic component on preferred gait speed of 0.07 [CI 95 % 0.03-0.10] m/s was found. Thirdly, there is a small significant positive meta-effect of progressive resistance training, combined with balance-, and endurance training of 0.05 [CI 95 % 0.00-0.09] m/s. The other sub-analyses show non-significant small positive meta-affects.

CONCLUSIONS

Progressive resistance training with high intensities, is the most effective exercise modality for improving preferred gait speed. Sufficient muscle strength seems an important condition for improving preferred gait speed. The addition of balance-, and/or endurance training does not contribute to the significant positive effects of progressive resistance training. A promising component is exercise with a rhythmic component. Keeping time to music or rhythm possibly trains higher cognitive functions that are important for gait.

LIMITATIONS

The focus of the present meta-analysis was at avoiding as much heterogeneity in exercise interventions. However heterogeneity in the research populations could not be completely avoided, there are probably differences in health status within different studies.

A modified regression method to test publication bias in meta-analyses with binary outcomes

BACKGROUND

The tendency towards publication bias is greater for observational studies than for randomized clinical trials. Several statistical methods have been developed to test the publication bias. However, almost all existing methods exhibit rather low power or have inappropriate type I error rates.

METHODS

We propose a modified regression method, which used a smoothed variance to estimate the precision of a study, to test for publication bias in meta-analyses of observational studies. A comprehensive simulation study is carried out, and a real-world example is considered.

RESULTS

The simulation results indicate that the performance of tests varies with the number of included studies, level of heterogeneity, event rates, and sample size ratio between two groups. Neither the existing tests nor the newly developed method is particularly powerful in all simulation scenarios. However, our proposed method has a more robust performance across different settings. In the presence of heterogeneity, the arcsine-Thompson test is a suitable alternative, and Peters' test can be considered as a complementary method when mild or no heterogeneity is present.

CONCLUSIONS

Several factors should be taken into consideration when employing asymmetry tests for publication bias. Based on our simulation results, we provide a concise table to show the appropriate use of regression methods to test for publication bias based on our simulation results.

Presenting simulation results in a nested loop plot

Background

Statisticians investigate new methods in simulations to evaluate their properties for future real data applications. Results are often presented in a number of figures, e.g., Trellis plots. We had conducted a simulation study on six statistical methods for estimating the treatment effect in binary outcome meta-analyses, where selection bias (e.g., publication bias) was suspected because of apparent funnel plot asymmetry. We varied five simulation parameters: true treatment effect, extent of selection, event proportion in control group, heterogeneity parameter, and number of studies in meta-analysis. In combination, this yielded a total number of 768 scenarios. To present all results using Trellis plots, 12 figures were needed.

Methods

Choosing bias as criterion of interest, we present a ‘nested loop plot’, a diagram type that aims to have all simulation results in one plot. The idea was to bring all scenarios into a lexicographical order and arrange them consecutively on the horizontal axis of a plot, whereas the treatment effect estimate is presented on the vertical axis.

Results

The plot illustrates how parameters simultaneously influenced the estimate. It can be combined with a Trellis plot in a so-called hybrid plot. Nested loop plots may also be applied to other criteria such as the variance of estimation.

Conclusion

The nested loop plot, similar to a time series graph, summarizes all information about the results of a simulation study with respect to a chosen criterion in one picture and provides a suitable alternative or an addition to Trellis plots.

Electronic supplementary material

The online version of this article (doi:10.1186/1471-2288-14-129) contains supplementary material, which is available to authorized users.

Statistical methods for dealing with publication bias in meta-analysis

Publication bias is an inevitable problem in the systematic review and meta-analysis. It is also one of the main threats to the validity of meta-analysis. Although several statistical methods have been developed to detect and adjust for the publication bias since the beginning of 1980s, some of them are not well known and are not being used properly in both the statistical and clinical literature. In this paper, we provided a critical and extensive discussion on the methods for dealing with publication bias, including statistical principles, implementation, and software, as well as the advantages and limitations of these methods. We illustrated a practical application of these methods in a meta-analysis of continuous support for women during childbirth.

A test for reporting bias in trial networks: simulation and case studies

BACKGROUND

Networks of trials assessing several treatment options available for the same condition are increasingly considered. Randomized trial evidence may be missing because of reporting bias. We propose a test for reporting bias in trial networks.

METHODS

We test whether there is an excess of trials with statistically significant results across a network of trials. The observed number of trials with nominally statistically significant p-values across the network is compared with the expected number. The performance of the test (type I error rate and power) was assessed using simulation studies under different scenarios of selective reporting bias. Examples are provided for networks of antidepressant and antipsychotic trials, where reporting biases have been previously demonstrated by comparing published to Food and Drug Administration (FDA) data.

RESULTS

In simulations, the test maintained the type I error rate and was moderately powerful after adjustment for type I error rate, except when the between-trial variance was substantial. In all, a positive test result increased moderately or markedly the probability of reporting bias being present, while a negative test result was not very informative. In the two examples, the test gave a signal for an excess of statistically significant results in the network of published data but not in the network of FDA data.

CONCLUSION

The test could be useful to document an excess of significant findings in trial networks, providing a signal for potential publication bias or other selective analysis and outcome reporting biases.

How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study

OBJECTIVE

To investigate the impact of a higher publishing probability for statistically significant positive outcomes on the false-positive rate in meta-analysis.

DESIGN

Meta-analyses of different sizes (N=10, N=20, N=50 and N=100), levels of heterogeneity and levels of publication bias were simulated.

PRIMARY AND SECONDARY OUTCOME MEASURES

The type I error rate for the test of the mean effect size (ie, the rate at which the meta-analyses showed that the mean effect differed from 0 when it in fact equalled 0) was estimated. Additionally, the power and type I error rate of publication bias detection methods based on the funnel plot were estimated.

RESULTS

In the presence of a publication bias characterised by a higher probability of including statistically significant positive results, the meta-analyses frequently concluded that the mean effect size differed from zero when it actually equalled zero. The magnitude of the effect of publication bias increased with an increasing number of studies and between-study variability. A higher probability of including statistically significant positive outcomes introduced little asymmetry to the funnel plot. A publication bias of a sufficient magnitude to frequently overturn the meta-analytic conclusions was difficult to detect by publication bias tests based on the funnel plot. When statistically significant positive results were four times more likely to be included than other outcomes and a large between-study variability was present, more than 90% of the meta-analyses of 50 and 100 studies wrongly showed that the mean effect size differed from zero. In the same scenario, publication bias tests based on the funnel plot detected the bias at rates not exceeding 15%.

CONCLUSIONS

This study adds to the evidence that publication bias is a major threat to the validity of medical research and supports the usefulness of efforts to limit publication bias.

Testing for publication bias in diagnostic meta-analysis: a simulation study

The present study investigates the performance of several statistical tests to detect publication bias in diagnostic meta-analysis by means of simulation. While bivariate models should be used to pool data from primary studies in diagnostic meta-analysis, univariate measures of diagnostic accuracy are preferable for the purpose of detecting publication bias. In contrast to earlier research, which focused solely on the diagnostic odds ratio or its logarithm ( ln ω), the tests are combined with four different univariate measures of diagnostic accuracy. For each combination of test and univariate measure, both type I error rate and statistical power are examined under diverse conditions. The results indicate that tests based on linear regression or rank correlation cannot be recommended in diagnostic meta-analysis, because type I error rates are either inflated or power is too low, irrespective of the applied univariate measure. In contrast, the combination of trim and fill and ln ω has non-inflated or only slightly inflated type I error rates and medium to high power, even under extreme circumstances (at least when the number of studies per meta-analysis is large enough). Therefore, we recommend the application of trim and fill combined with ln ω to detect funnel plot asymmetry in diagnostic meta-analysis.

Meta-analysis and The Cochrane Collaboration: 20 years of the Cochrane Statistical Methods Group

The Statistical Methods Group has played a pivotal role in The Cochrane Collaboration over the past 20 years. The Statistical Methods Group has determined the direction of statistical methods used within Cochrane reviews, developed guidance for these methods, provided training, and continued to discuss and consider new and controversial issues in meta-analysis. The contribution of Statistical Methods Group members to the meta-analysis literature has been extensive and has helped to shape the wider meta-analysis landscape.In this paper, marking the 20th anniversary of The Cochrane Collaboration, we reflect on the history of the Statistical Methods Group, beginning in 1993 with the identification of aspects of statistical synthesis for which consensus was lacking about the best approach. We highlight some landmark methodological developments that Statistical Methods Group members have contributed to in the field of meta-analysis. We discuss how the Group implements and disseminates statistical methods within The Cochrane Collaboration. Finally, we consider the importance of robust statistical methodology for Cochrane systematic reviews, note research gaps, and reflect on the challenges that the Statistical Methods Group faces in its future direction.

Combining study outcome measures using dominance adjusted weights

Weighting of studies in meta-analysis is usually implemented by using the estimated inverse variances of treatment effect estimates. However, there is a possibility of one study dominating other studies in the estimation process by taking on a weight that is above some upper limit. We implement an estimator of the heterogeneity variance that takes advantage of dominance adjusted weights. The performance of this estimator is compared with that of the commonly used estimator in meta-analysis, the DerSimonian-Laird estimator. Two test procedures for the overall treatment effect are proposed that are based on the quadratic form associated with the proposed heterogeneity variance estimator. An example is given to illustrate the application of these procedures. Copyright © 2013 John Wiley & Sons, Ltd.

Study factors influencing ventricular enlargement in schizophrenia: a 20 year follow-up meta-analysis

A meta-analysis was performed on studies employing the ventricular-brain ratio to compare schizophrenic subjects to that of normal controls. This was a follow-up to a similar meta-analysis published in 1992 in which study-, in addition to clinical-, factors were found to contribute significantly to the reported difference between patients with schizophrenia and controls. Seventy-two (N=72) total studies were identified from the peer reviewed literature, 39 from the original meta-analysis, and 33 additional studies published since which met strict criteria for inclusion and analysis - thus representing ~30 years of schizophrenia ventricular enlargement research. Sample characteristics from schizophrenics and controls were coded for use as predictor variables against within sample VBR values as well as for between sample VBR differences. Additionally, a number of factors concerning how the studies were conducted and reported were also coded. Obtained data was subjected to unweighted univariate as well as multiple regression analyses. In particular, results indicated significant differences between schizophrenics and controls in ventricular size but also the influence of the diagnostic criteria used to define schizophrenia on the magnitude of the reported VBR. This suggests that differing factors of the diagnostic criteria may be sensitive to ventricular enlargement and might be worthy of further examination. Interestingly, we observed an inverse relationship between VBR difference and the number of co-authors on the study. This latter finding suggests that larger research groups report smaller VBR differences and may be more conservative or exacting in their research methodology. Analyses weighted by sample size provided identical conclusions. The effects of study factors such as these are helpful for understanding the variation in the size of the reported differences in VBR between patients and controls as well as for understanding the evolution of research on complex clinical syndromes employing neuroimaging morphometrics.

Detecting and adjusting for small-study effects in meta-analysis

Publication bias and related types of small-study effects threaten the validity of systematic reviews. The existence of small-study effects has been demonstrated in empirical studies. Small-study effects are graphically diagnosed by inspection of the funnel plot. Though observed funnel plot asymmetry cannot be easily linked to a specific reason, tests based on funnel plot asymmetry have been proposed. Beyond a vast range of funnel plot tests, there exist several methods for adjusting treatment effect estimates for these biases. In this article, we consider the trim-and-fill method, the Copas selection model, and more recent regression-based approaches. The methods are exemplified using a meta-analysis from the literature and compared in a simulation study, based on binary response data. They are also applied to a large set of meta-analyses. Some fundamental differences between the approaches are discussed. An assumption common to the trim-and-fill method and the Copas selection model is that the small-study effect is caused by selection. The trim-and-fill method corresponds to an unknown implicit model generated by the symmetry assumption, whereas the Copas selection model is a parametric statistical model. However, it requires a sensitivity analysis. Regression-based approaches are easier to implement and not based on a specific selection model. Both simulations and applications suggest that in the presence of strong selection both the trim-and-fill method and the Copas selection model may not fully eliminate bias, while regression-based approaches seem to be a promising alternative.

Chest compressions before defibrillation for out-of-hospital cardiac arrest: a meta-analysis of randomized controlled clinical trials

BACKGROUND

Current 2005 guidelines for advanced cardiac life support strongly recommend immediate defibrillation for out-of-hospital cardiac arrest. However, findings from experimental and clinical studies have indicated a potential advantage of pretreatment with chest compression-only cardiopulmonary resuscitation (CPR) prior to defibrillation in improving outcomes. The aim of this meta-analysis is to evaluate the beneficial effect of chest compression-first versus defibrillation-first on survival in patients with out-of-hospital cardiac arrest.

METHODS

Main outcome measures were survival to hospital discharge (primary endpoint), return of spontaneous circulation (ROSC), neurologic outcome and long-term survival. Randomized, controlled clinical trials that were published between January 1, 1950, and June 19, 2010, were identified by a computerized search using SCOPUS, MEDLINE, BIOS, EMBASE, the Cochrane Central Register of Controlled Trials, International Pharmaceutical Abstracts database, and Web of Science and supplemented by conference proceedings. Random effects models were used to calculate pooled odds ratios (ORs). A subgroup analysis was conducted to explore the effects of response interval greater than 5 min on outcomes.

RESULTS

A total of four trials enrolling 1503 subjects were integrated into this analysis. No difference was found between chest compression-first versus defibrillation-first in the rate of return of spontaneous circulation (OR 1.01 [0.82-1.26]; P = 0.979), survival to hospital discharge (OR 1.10 [0.70-1.70]; P = 0.686) or favorable neurologic outcomes (OR 1.02 [0.31-3.38]; P = 0.979). For 1-year survival, however, the OR point estimates favored chest compression first (OR 1.38 [0.95-2.02]; P = 0.092) but the 95% CI crossed 1.0, suggesting insufficient estimate precision. Similarly, for cases with prolonged response times (> 5 min) point estimates pointed toward superiority of chest compression first (OR 1.45 [0.66-3.20]; P = 0.353), but the 95% CI again crossed 1.0.

CONCLUSIONS

Current evidence does not support the notion that chest compression first prior to defibrillation improves the outcome of patients in out-of-hospital cardiac arrest. It appears that both treatments are equivalent. However, subgroup analyses indicate that chest compression first may be beneficial for cardiac arrests with a prolonged response time.

Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis

Statistical heterogeneity and small-study effects are 2 major issues affecting the validity of meta-analysis. In this article, we introduce the concept of a limit meta-analysis, which leads to shrunken, empirical Bayes estimates of study effects after allowing for small-study effects. This in turn leads to 3 model-based adjusted pooled treatment-effect estimators and associated confidence intervals. We show how visualizing our estimators using the radial plot indicates how they can be calculated using existing software. The concept of limit meta-analysis also gives rise to a new measure of heterogeneity, termed G(2), for heterogeneity that remains after small-study effects are accounted for. In a simulation study with binary data and small-study effects, we compared our proposed estimators with those currently used together with a recent proposal by Moreno and others. Our criteria were bias, mean squared error (MSE), variance, and coverage of 95% confidence intervals. Only the estimators arising from the limit meta-analysis produced approximately unbiased treatment-effect estimates in the presence of small-study effects, while the MSE was acceptably small, provided that the number of studies in the meta-analysis was not less than 10. These limit meta-analysis estimators were also relatively robust against heterogeneity and one of them had a relatively small coverage error.

Arcsine test for publication bias in meta-analyses with binary outcomes

In meta-analyses, it sometimes happens that smaller trials show different, often larger, treatment effects. One possible reason for such 'small study effects' is publication bias. This is said to occur when the chance of a smaller study being published is increased if it shows a stronger effect. Assuming no other small study effects, under the null hypothesis of no publication bias, there should be no association between effect size and effect precision (e.g. inverse standard error) among the trials in a meta-analysis.A number of tests for small study effects/publication bias have been developed. These use either a non-parametric test or a regression test for association between effect size and precision. However, when the outcome is binary, the effect is summarized by the log-risk ratio or log-odds ratio (log OR). Unfortunately, these measures are not independent of their estimated standard error. Consequently, established tests reject the null hypothesis too frequently. We propose new tests based on the arcsine transformation, which stabilizes the variance of binomial random variables. We report results of a simulation study under the Copas model (on the log OR scale) for publication bias, which evaluates tests so far proposed in the literature. This shows that: (i) the size of one of the new tests is comparable to those of the best existing tests, including those recently published; and (ii) among such tests it has slightly greater power, especially when the effect size is small and heterogeneity is present. Arcsine tests have additional advantages that they can include trials with zero events in both arms and that they can be very easily performed using the existing software for regression tests.

Recent developments in meta-analysis

The art and science of meta-analysis, the combination of results from multiple independent studies, is now more than a century old. In the last 30 years, however, as the need for medical research and clinical practice to be based on the totality of relevant and sound evidence has been increasingly recognized, the impact of meta-analysis has grown enormously. In this paper, we review highlights of recent developments in meta-analysis in medical research. We outline in particular how emphasis has been placed on (i) heterogeneity and random-effects analyses; (ii) special consideration in different areas of application; (iii) assessing bias within and across studies; and (iv) extension of ideas to complex evidence synthesis. We conclude the paper with some remarks on ongoing challenges and possible directions for the future.

Meta-analyses with binary outcomes: how many studies need to be omitted to detect a publication bias?

Meta- and pooled analyses are increasingly applied to aggregate the results of a number of studies, especially in health sciences. A typical difficulty is the presence of a publication bias. Usually Egger's regression test and funnel plots are applied to detect such a publication bias. A simulation study was conducted to investigate the quantity of null and negative results required to be omitted to detect a publication bias. In particular, the performance of Egger's test and funnel plots was considered in two scenarios with binary outcomes and expected odds ratios (OR) of 1 and 2, respectively. For both scenarios Egger's test detected only a small fraction of publication biases if few studies were deleted, corresponding to the results of a random deletion. Moreover, if a true null effect is present Egger's test is quite unlikely to detect a publication bias even if a considerable proportion of the null results are missing. Generally, the detection of a publication bias using Egger's test is only likely if both the "true" effect and the bias are large enough. Visual inspection of the funnel plots resulted in a higher fraction of detected publication biases in cases where a bias was present and in cases where studies were randomly deleted, revealing the arbitrariness of this method. Evidence indicates that standard methods for detection of a publication bias do not necessarily detect such a bias; thus, additional tests for publication bias need to be applied.

[Practical considerations on detection of publication bias]

The present review aims to answer 3 questions: does publication bias need to be assessed in meta-analyses?; what procedures, not requiring complex statistical approaches, can be applied to detect it?; and should other factors be taken into account when interpreting the procedures? The first question is easy to answer. Publication bias is a potential threat to the validity of the conclusions of meta-analyses. Therefore, both the MOOSE and QUOROM statements include publication bias in their guidelines; nevertheless, many meta-analyses do not use these statements (e.g., meta-analyses conducted by the Cochrane Collaboration), perhaps because they use a comprehensive search strategy. There are many methods to assess publication bias. The most frequently used are funnel plots or <<Christmas trees>>, <<trim and fill>> (which allow the effects of bias to be estimated), and methods based upon regression on plots, such as Egger's method and funnel plot regression. An advantage of these methods is that they can only be applied using published data. However, agreement between these methods in detecting bias is often poor. Therefore, application of more than one method to detect publication bias is recommended. To correctly interpret the results, the number of pooled studies should be more than 10 and the existence of heterogeneity in the pooled estimate must be taken into account.