The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey

BACKGROUND

Statistical tests for funnel-plot asymmetry are common in meta-analyses. Inappropriate application can generate misleading inferences about publication bias. We aimed to measure, in a survey of meta-analyses, how frequently the application of these tests would be not meaningful or inappropriate.

METHODS

We evaluated all meta-analyses of binary outcomes with é 3 studies in the Cochrane Database of Systematic Reviews (2003, issue 2). A separate, restricted analysis was confined to the largest meta-analysis in each of the review articles. In each meta-analysis, we assessed whether criteria to apply asymmetry tests were met: no significant heterogeneity, I2 < 50%, é 10 studies (with statistically significant results in at least 1) and ratio of the maximal to minimal variance across studies > 4. We performed a correlation and 2 regression asymmetry tests and evaluated their concordance. Finally, we sampled 60 meta-analyses from print journals in 2005 that cited use of the standard regression test.

RESULTS

A total of 366 of 6873 (5%) and 98 of 846 meta-analyses (12%) in the wider and restricted Cochrane data set, respectively, would have qualified for use of asymmetry tests. Asymmetry test results were significant in 7%-18% of the meta-analyses. Concordance between the 3 tests was modest (estimated k 0.33-0.66). Of the 60 journal meta-analyses, 7 (12%) would qualify for asymmetry tests; all 11 claims for identification of publication bias were made in the face of large and significant heterogeneity.

INTERPRETATION

Statistical conditions for employing asymmetry tests for publication bias are absent from most meta-analyses; yet, in medical journals these tests are performed often and interpreted erroneously.