Should studies with no events in both arms be excluded in evidence synthesis?

OBJECTIVES

In safety assessment, studies with no events are a frequent occurrence when conducting meta-analyses. The current approach in meta-analysis is to exclude double-zero studies from the synthesis. In this study, we compared the performance of excluding and including double-zero studies.

METHODS

A simulation with 5000 iterations was conducted based on the real-world dataset from Cochrane reviews. The true distribution of the rare events rather than normal distribution for the effects were used in the data generating mechanism to simulate aggregate meta-analysis data. We used Doi's inverse variance heterogeneity (IVhet) model for the meta-analyses with continuity correction (of 0.5) to include double-zero studies and used the odds ratio effect size. The performance of including versus excluding double-zero studies were then compared.

RESULTS

Generally, there was much larger mean squared error when double zero studies were excluded than when double-zero studies were included. The coverage when studies were excluded rapidly deteriorates as heterogeneity increased, while remained at or above the nominal level when double-zero studies were included. When there were very few double-zero studies, the performances was almost the same when including or excluding these studies. Subgroup analysis showed that, even for meta-analyses with unbalanced sample size across the two arms, including double-zero studies improved performance compared to when they were excluded.

CONCLUSIONS

Including double-zero studies in meta-analysis improved performance substantively when compared to excluding them, especially when the proportion of double-zero studies was large. Continuity correction with use of the IVhet model is therefore a good solution to deal with double-zero studies and should be considered in future meta-analyses.

Estimating risk and rate ratio in rare events meta-analysis with the Mantel–Haenszel estimator and assessing heterogeneity

Meta-analysis of binary outcome data faces often a situation where studies with a rare event are part of the set of studies to be considered. These studies have low occurrence of event counts to the extreme that no events occur in one or both groups to be compared. This raises issues how to estimate validly the summary risk or rate ratio across studies. A preferred choice is the Mantel–Haenszel estimator, which is still defined in the situation of zero studies unless all studies have zeros in one of the groups to be compared. For this situation, a modified Mantel–Haenszel estimator is suggested and shown to perform well by means of simulation work. Also, confidence interval estimation is discussed and evaluated in a simulation study. In a second part, heterogeneity of relative risk across studies is investigated with a new chi-square type statistic which is based on a conditional binomial distribution where the conditioning is on the event margin for each study. This is necessary as the conventional Q-statistic is undefined in the occurrence of zero studies. The null-distribution of the proposed Q-statistic is obtained by means of a parametric bootstrap as a chi-square approximation is not valid for rare events meta-analysis, as bootstrapping of the null-distribution shows. In addition, for the effect heterogeneity situation, confidence interval estimation is considered using a nonparametric bootstrap procedure. The proposed techniques are illustrated at hand of three meta-analytic data sets.

Exact inference for fixed-effects meta-analysis of proportions

Meta-analysis of proportions is conceptually simple: Faced with a binary outcome in multiple studies, we seek inference on some overall proportion of successes/failures. Under common effect models, exact inference has long been available, but is not when we more realistically allow for heterogeneity of the proportions. Instead a wide range of nonexact fixed-effects methods are used, the interpretation of some of which is challenging. In this paper, we present methods for exact statistical tests and confidence intervals for fixed-effects meta-analysis of proportions. These methods retain the interpretability of the underlying parameter of interest, and can be implemented in straightforward software. We also show how our inference on the overall proportion is compatible with exact inference on heterogeneity of proportions. An illustrative example from a recent kidney disease study shows how the method's performance can be assessed in practice.

Synthesis of evidence from zero-events studies: A comparison of one-stage framework methods

In evidence synthesis, dealing with zero-events studies is an important and complicated task that has generated broad discussion. Numerous methods provide valid solutions to synthesizing data from studies with zero-events, either based on a frequentist or a Bayesian framework. Among frequentist frameworks, the one-stage methods have their unique advantages to deal with zero-events studies, especially for double-arm-zero-events. In this article, we give a concise overview of the one-stage frequentist methods. We conducted simulation studies to compare the statistical properties of these methods to the two-stage frequentist method (continuity correction) for meta-analysis with zero-events studies when double-zero-events studies were included. Our simulation studies demonstrated that the generalized estimating equation with unstructured correlation and beta-binomial method had the best performance among the one-stage methods. The random intercepts generalized linear mixed model showed good performance in the absence of obvious between-study variance. Our results also showed that the continuity correction with inverse-variance heterogeneous (IVhet) analytic model based on the two-stage framework had good performance when the between-study variance was obvious and the group size was balanced for included studies. In summary, the one-stage framework has unique advantages to deal with studies with zero events and is not susceptive to group size ratio. It should be considered in future meta-analyses whenever possible.

A comparison of confidence distribution approaches for rare event meta-analysis

Meta-analysis of rare event data has recently received increasing attention due to the challenging issues rare events pose to traditional meta-analytic methods. One specific way to combine information and analyze rare event meta-analysis data utilizes confidence distributions (CDs). While several CD methods exist, no comparisons have been made to determine which method is best suited for homogeneous or heterogeneous meta-analyses with rare events. In this article, we review several CD methods: Fisher's classic P-value combination method, one that combines P-value functions, another that combines confidence intervals, and one that combines confidence log-likelihood functions. We compare these CD approaches, and we propose and compare variations of these methods to determine which method produces reliable results for homogeneous or heterogeneous rare event meta-analyses. We find that for homogeneous rare event data, most CD methods perform very well. On the other hand, for heterogeneous rare event data, there is a clear split in performance between some CD methods, with some performing very poorly and others performing reasonably well.

Accurate confidence intervals for risk difference in meta-analysis with rare events

BACKGROUND

Meta-analysis provides a useful statistical tool to effectively estimate treatment effect from multiple studies. When the outcome is binary and it is rare (e.g., safety data in clinical trials), the traditionally used methods may have unsatisfactory performance.

METHODS

We propose using importance sampling to compute confidence intervals for risk difference in meta-analysis with rare events. The proposed intervals are not exact, but they often have the coverage probabilities close to the nominal level. We compare the proposed accurate intervals with the existing intervals from the fixed- or random-effects models and the interval by Tian et al. (2009).

RESULTS

We conduct extensive simulation studies to compare them with regards to coverage probability and average length, when data are simulated under the homogeneity or heterogeneity assumption of study effects.

CONCLUSIONS

The proposed accurate interval based on the random-effects model for sample space ordering generally has satisfactory performance under the heterogeneity assumption, while the traditionally used interval based on the fixed-effects model works well when the studies are homogeneous.

Real-world Performance of Meta-analysis Methods for Double-Zero-Event Studies with Dichotomous Outcomes Using the Cochrane Database of Systematic Reviews

BACKGROUND

Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews.

METHODS

The agreement of five commonly used methods (i.e., the inverse-variance, Mantel-Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen's κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel-Haenszel methods.

RESULTS

When the proportion of DZS studies was lower than 50% in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50%, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel-Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel-Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel-Haenszel methods using the EMP correction.

CONCLUSIONS

To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.

An Efficient Numerical Algorithm for Exact Inference in Meta Analysis

The performance of commonly used asymptotic inference procedures for the random effects model used in meta analysis relies on the number of studies. When the number of studies is moderate or small, the exact inference procedure is more reliable than the asymptotic counterparts. However, the related numerical computation may be demanding and an obstacle of routine use of the exact method. In this paper, we proposed a novel numerical algorithm for constructing the exact 95% confidence interval of the location parameter in the random effects model. The algorithm is much faster than the naive method and may greatly facilitate the use of the more appropriate exact inference procedure in meta analysis. Numerical studies and real data examples are used to illustrate the advantage of the proposed method.

A Bayesian Meta-analysis Method for Estimating Risk Difference of Rare Events

Bayesian meta-analysis has been more frequently utilized for synthesizing safety and efficacy information to support landmark decision-making due to its flexibility of incorporating prior information and availability of computing software. However, when the outcome is binary and the events are rare, where event counts can be zero, conventional meta-analysis methods including Bayesian methods may not work well. Several methods have been proposed to tackle this issue but the prior knowledge of event rate was not utilized to increase precision of risk difference estimates. To better estimate risk differences, we propose a new Bayesian method, Beta prior BInomial model for Risk Differences (B-BIRD), which takes into account the prior information of rare events. B-BIRD is illustrated using a real data set of 48 clinical trials about a type 2 diabetes drug. In simulation studies, it performs well in low event rate settings.