Shortcomings of an approximate confidence interval for moment-based estimators of the between-study variance in random-effects meta-analysis

Multistep estimators of the between-study covariance matrix under the multivariate random-effects model for meta-analysis

A wide variety of methods are available to estimate the between-study variance under the univariate random-effects model for meta-analysis. Some, but not all, of these estimators have been extended so that they can be used in the multivariate setting. We begin by extending the univariate generalised method of moments, which immediately provides a wider class of multivariate methods than was previously available. However, our main proposal is to use this new type of estimator to derive multivariate multistep estimators of the between-study covariance matrix. We then use the connection between the univariate multistep and Paule-Mandel estimators to motivate taking the limit, where the number of steps tends toward infinity. We illustrate our methodology using two contrasting examples and investigate its properties in a simulation study. We conclude that the proposed methodology is a fully viable alternative to existing estimation methods, is well suited to sensitivity analyses that explore the use of alternative estimators, and should be used instead of the existing DerSimonian and Laird-type moments based estimator in application areas where data are expected to be heterogeneous. However, multistep estimators do not seem to outperform the existing estimators when the data are more homogeneous. Advantages of the new multivariate multistep estimator include its semi-parametric nature and that it is computationally feasible in high dimensions. Our proposed estimation methods are also applicable for multivariate random-effects meta-regression, where study-level covariates are included in the model.

A Practical Approach in Refining Binary Outcome for Treatment Effect of COVID-19 According to Geographical Diversity

The recent COVID-19 pandemic has drawn attention to health and economics worldwide. Initially, diseases only ravage local populations, while a pandemic could aggravate global economic burdens. Lopinavir/Ritonavir is an anti-HIV drug that was used on small scale patients during SARS, but its effectiveness for COVID-19 treatment is still unclear. Previous studies or meta-analysis have retrieved clinical data of subgroup analysis to evaluate the efficacy and safety of Lopinavir/Ritonavir for the treatment of COVID-19 in a few affected regions. However, geographical diversity and small number of studies bias correction were not achieved in such subgroup analysis of published meta-analysis. The present study demonstrates a practical approach in refining the binary outcome for COVID-19 treatment of Lopinavir/Ritonavir according to geographical location diversity and small number of studies (less than or equal to five) for subgroup analysis. After performing practical approach, the risk of adverse event with LPV/RTV for treatment of COVID-19 becomes nonsignificant compared to previous meta-analysis. Furthermore, we also notice heterogeneity of random effect of meta-analysis may be declined after proposed adjustment. In conclusion, proposed practical approach is recommend for performing a subgroup analysis to avoid concentration in a single geographical location and small number of studies bias.

Study specific prediction intervals for random-effects meta-analysis: A tutorial: Prediction intervals in meta-analysis

The pooled estimate of the average effect is of primary interest when fitting the random-effects model for meta-analysis. However estimates of study specific effects, for example those displayed on forest plots, are also often of interest. In this tutorial, we present the case, with the accompanying statistical theory, for estimating the study specific true effects using so called 'empirical Bayes estimates' or 'Best Unbiased Linear Predictions' under the random-effects model. These estimates can be accompanied by prediction intervals that indicate a plausible range of study specific true effects. We coalesce and elucidate the available literature and illustrate the methodology using two published meta-analyses as examples. We also perform a simulation study that reveals that coverage probability of study specific prediction intervals are substantially too low if the between-study variance is small but not negligible. Researchers need to be aware of this defect when interpreting prediction intervals. We also show how empirical Bayes estimates, accompanied with study specific prediction intervals, can embellish forest plots. We hope that this tutorial will serve to provide a clear theoretical underpinning for this methodology and encourage its widespread adoption.

Jackknife empirical likelihood confidence intervals for assessing heterogeneity in meta-analysis of rare binary event data

In meta-analysis, the heterogeneity of effect sizes across component studies is typically described by a variance parameter in a random-effects (Re) model. In the literature, methods for constructing confidence intervals (CIs) for the parameter often assume that study-level effect sizes are normally distributed. However, this assumption might be violated in practice, especially in meta-analysis of rare binary events. We propose to use jackknife empirical likelihood (JEL), a nonparametric approach that uses jackknife pseudo-values, to construct CIs for the heterogeneity parameter. To compute jackknife pseudo-values, we employ a moment-based estimator and consider two commonly used weighing schemes (i.e., equal and inverse variance weights). We prove that with each scheme, the resulting log empirical likelihood ratio follows a chi-square distribution asymptotically. We further examine the performance of the proposed JEL methods and compare them with existing CIs through simulation studies and data examples that focus on data of rare binary events. Our numerical results suggest that the JEL method with equal weights compares favorably to alternatives, especially when (observed) effect sizes are non-normal and the number of component studies is large. Thus, it is worth serious consideration in statistical inference.

A new justification of the Hartung-Knapp method for random-effects meta-analysis based on weighted least squares regression

The Hartung-Knapp method for random-effects meta-analysis, that was also independently proposed by Sidik and Jonkman, is becoming advocated for general use. This method has previously been justified by taking all estimated variances as known and using a different pivotal quantity to the more conventional one when making inferences about the average effect. We provide a new conceptual framework for, and justification of, the Hartung-Knapp method. Specifically, we show that inferences from fitted random-effects models, using both the conventional and the Hartung-Knapp method, are equivalent to those from closely related intercept only weighted least squares regression models. This observation provides a new link between Hartung and Knapp's methodology for meta-analysis and standard linear models, where it can be seen that the Hartung-Knapp method can be justified by a linear model that makes a slightly weaker assumption than taking all variances as known. This provides intuition for why the Hartung-Knapp method has been found to perform better than the conventional one in simulation studies. Furthermore, our new findings give more credence to ad hoc adjustments of confidence intervals from the Hartung-Knapp method that ensure these are at least as wide as more conventional confidence intervals. The conceptual basis for the Hartung-Knapp method that we present here should replace the established one because it more clearly illustrates the potential benefit of using it.

Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the between-study variance in meta-analysis

The effect sizes of studies included in a meta‐analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so‐called between‐study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between‐study variance facilitates interpretation of the meta‐analytic results. Two methods that are recommended to be used for creating such a confidence interval are the Q‐profile and generalized Q‐statistic method that both make use of the Q‐statistic. These methods are exact if the assumptions underlying the random‐effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte‐Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta‐analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random‐effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.

When should meta-analysis avoid making hidden normality assumptions?

Meta-analysis is a widely used statistical technique. The simplicity of the calculations required when performing conventional meta-analyses belies the parametric nature of the assumptions that justify them. In particular, the normal distribution is extensively, and often implicitly, assumed. Here, we review how the normal distribution is used in meta-analysis. We discuss when the normal distribution is likely to be adequate and also when it should be avoided. We discuss alternative and more advanced methods that make less use of the normal distribution. We conclude that statistical methods that make fewer normality assumptions should be considered more often in practice. In general, statisticians and applied analysts should understand the assumptions made by their statistical analyses. They should also be able to defend these assumptions. Our hope is that this article will foster a greater appreciation of the extent to which assumptions involving the normal distribution are made in statistical methods for meta-analysis. We also hope that this article will stimulate further discussion and methodological work.

Multistep estimators of the between-study variance: The relationship with the Paule-Mandel estimator

A wide variety of estimators of the between-study variance are available in random-effects meta-analysis. Many, but not all, of these estimators are based on the method of moments. The DerSimonian-Laird estimator is widely used in applications, but the Paule-Mandel estimator is an alternative that is now recommended. Recently, DerSimonian and Kacker have developed two-step moment-based estimators of the between-study variance. We extend these two-step estimators so that multiple (more than two) steps are used. We establish the surprising result that the multistep estimator tends towards the Paule-Mandel estimator as the number of steps becomes large. Hence, the iterative scheme underlying our new multistep estimator provides a hitherto unknown relationship between two-step estimators and Paule-Mandel estimator. Our analysis suggests that two-step estimators are not necessarily distinct estimators in their own right; instead, they are quantities that are closely related to the usual iterative scheme that is used to calculate the Paule-Mandel estimate. The relationship that we establish between the multistep and Paule-Mandel estimator is another justification for the use of the latter estimator. Two-step and multistep estimators are perhaps best conceptualized as approximate Paule-Mandel estimators.