Appropriateness of conducting and reporting random-effects meta-analysis in oncology

A random-effects model is often applied in meta-analysis when considerable heterogeneity among studies is observed due to the differences in patient characteristics, timeframe, treatment regimens, and other study characteristics. Since 2014, the journals Research Synthesis Methods and the Annals of Internal Medicine have published a few noteworthy papers that explained why the most widely used method for pooling heterogeneous studies-the DerSimonian-Laird (DL) estimator-can produce biased estimates with falsely high precision and recommended to use other several alternative methods. Nevertheless, more than half of studies (55.7%) published in top oncology-specific journals during 2015-2022 did not report any detailed method in the random-effects meta-analysis. Of the studies that did report the methodology used, the DL method was still the dominant one reported. Thus, while the authors recommend that Research Synthesis Methods and the Annals of Internal Medicine continue to increase the publication of its articles that report on specific methods for handling heterogeneity and use random-effects estimates that provide more accurate confidence limits than the DL estimator, other journals that publish meta-analyses in oncology (and presumably in other disease areas) are urged to do the same on a much larger scale than currently documented.

Jointly pooling aggregated effect sizes and their standard errors from studies with continuous clinical outcomes

The DerSimonian-Laird (DL) weighted average method for aggregated data meta-analysis has been widely used for the estimation of overall effect sizes. It is criticized for its underestimation of the standard error of the overall effect size in the presence of heterogeneous effect sizes. Due to this negative property, many alternative estimation approaches have been proposed in the literature. One of the earliest alternative approaches was developed by Hardy and Thompson (HT), who implemented a profile likelihood instead of the moment-based approach of DL. Others have further extended this likelihood approach and proposed higher-order likelihood inferences (e.g., Bartlett-type corrections). In addition, corrections factors for the estimated DL standard error, like the Hartung-Knapp-Sidik-Jonkman (HKSJ) adjustment, and the restricted maximum likelihood (REML) estimation have been suggested too. Although these improvements address the uncertainty in estimating the between-study variance better than the DL method, they all assume that the true within-study standard errors are known and equal to the observed standard errors of the effect sizes. Here, we will treat the observed standard errors as estimators for the within-study variability and we propose a bivariate likelihood approach that jointly estimates the overall effect size, the between-study variance, and the potentially heteroskedastic within-study variances. We study the performance of the proposed method by means of simulation, and compare it to DL (with and without HKSJ), HT, their higher-order likelihood methods, and REML. Our proposed approach seems to have better or similar coverages compared to the other approaches and it appears to be less biased in the case of heteroskedastic within-study variances when this heteroskedasticty is correlated with the effect size.

Assessing Heterogeneity in Random-Effects Meta-analysis

The random-effects model allows for the possibility that studies in a meta-analysis have heterogeneous effects. That is, observed study estimates vary not only due to random sampling error but also due to inherent differences in the way studies have been designed and conducted. In this chapter, we consider methods to estimate the heterogeneity variance parameter in a random-effects model, consider in more detail what this parameter represents and how the possible explanations for heterogeneity can be explored through statistical methods. Toward the end of this chapter, publication bias is discussed as an alternative explanation for why observed effect estimates might form some distribution other than what we might come to expect.

A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses

Studies combined in a meta-analysis often have differences in their design and conduct that can lead to heterogeneous results. A random-effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian-Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta-analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian-Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule-Mandel method has considerable positive bias in meta-analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta-analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta-analysis. The estimated summary effect of the meta-analysis and its confidence interval derived from the Hartung-Knapp-Sidik-Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.

An empirical comparison of heterogeneity variance estimators in 12 894 meta-analyses

Heterogeneity in meta-analysis is most commonly estimated using a moment-based approach described by DerSimonian and Laird. However, this method has been shown to produce biased estimates. Alternative methods to estimate heterogeneity include the restricted maximum likelihood approach and those proposed by Paule and Mandel, Sidik and Jonkman, and Hartung and Makambi. We compared the impact of these five methods on the results of 12,894 meta-analyses extracted from the Cochrane Database of Systematic Reviews. We compared the methods in terms of the following: (1) the extent of heterogeneity, expressed as an I(2) statistic; (2) the overall effect estimate; (3) the precision of the overall effect estimate; and (4) p-values testing the no effect hypothesis. Results suggest that, in some meta-analyses, I(2) estimates differ by more than 50% when different heterogeneity estimators are used. Conclusions naively based on statistical significance (at a 5% level) were discordant for at least one pair of estimators in 7.5% of meta-analyses, indicating that the choice of heterogeneity estimator could affect the conclusions of a meta-analysis. These findings imply that using a single estimate of heterogeneity may lead to non-robust results in some meta-analyses, and researchers should consider using alternatives to the DerSimonian and Laird method.