Empirical comparisons of meta-analysis methods for diagnostic studies: a meta-epidemiological study

OBJECTIVES

Several methods are commonly used for meta-analyses of diagnostic studies, such as the bivariate linear mixed model (LMM). It estimates the overall sensitivity, specificity, their correlation, diagnostic OR (DOR) and the area under the curve (AUC) of the summary receiver operating characteristic (ROC) estimates. Nevertheless, the bivariate LMM makes potentially unrealistic assumptions (ie, normality of within-study estimates), which could be avoided by the bivariate generalised linear mixed model (GLMM). This article aims at investigating the real-world performance of the bivariate LMM and GLMM using meta-analyses of diagnostic studies from the Cochrane Library.

METHODS

We compared the bivariate LMM and GLMM using the relative differences in the overall sensitivity and specificity, their 95% CI widths, between-study variances, and the correlation between the (logit) sensitivity and specificity. We also explored their relationships with the number of studies, number of subjects, overall sensitivity and overall specificity.

RESULTS

Among the extracted 1379 meta-analyses, point estimates of overall sensitivities and specificities by the bivariate LMM and GLMM were generally similar, but their CI widths could be noticeably different. The bivariate GLMM generally produced narrower CIs than the bivariate LMM when meta-analyses contained 2-5 studies. For meta-analyses with <100 subjects or the overall sensitivities or specificities close to 0% or 100%, the bivariate LMM could produce substantially different AUCs, DORs and DOR CI widths from the bivariate GLMM.

CONCLUSIONS

The variation of estimates calls into question the appropriateness of the normality assumption within individual studies required by the bivariate LMM. In cases of notable differences presented in these methods' results, the bivariate GLMM may be preferred.

Predictive P-score for treatment ranking in Bayesian network meta-analysis

Background

Network meta-analysis (NMA) is a widely used tool to compare multiple treatments by synthesizing different sources of evidence. Measures such as the surface under the cumulative ranking curve (SUCRA) and the P-score are increasingly used to quantify treatment ranking. They provide summary scores of treatments among the existing studies in an NMA. Clinicians are frequently interested in applying such evidence from the NMA to decision-making in the future. This prediction process needs to account for the heterogeneity between the existing studies in the NMA and a future study.

Methods

This article introduces the predictive P-score for informing treatment ranking in a future study via Bayesian models. Two NMAs were used to illustrate the proposed measure; the first assessed 4 treatment strategies for smoking cessation, and the second assessed treatments for all-grade treatment-related adverse events. For all treatments in both NMAs, we obtained their conventional frequentist P-scores, Bayesian P-scores, and predictive P-scores.

Results

In the two examples, the Bayesian P-scores were nearly identical to the corresponding frequentist P-scores for most treatments, while noticeable differences existed for some treatments, likely owing to the different assumptions made by the frequentist and Bayesian NMA models. Compared with the P-scores, the predictive P-scores generally had a trend to converge toward a common value of 0.5 due to the heterogeneity. The predictive P-scores’ numerical estimates and the associated plots of posterior distributions provided an intuitive way for clinicians to appraise treatments for new patients in a future study.

Conclusions

The proposed approach adapts the existing frequentist P-score to the Bayesian framework. The predictive P-score can help inform medical decision-making in future studies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01397-5.

Methodological assessment of systematic reviews and meta-analyses on COVID-19: A meta-epidemiological study

RATIONALE, AIMS, AND OBJECTIVES

COVID-19 has caused an ongoing public health crisis. Many systematic reviews and meta-analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID-19 research. This meta-epidemiological study aims to methodologically assess the current systematic reviews and meta-analyses on COVID-19.

METHODS

We searched in various databases for systematic reviews with meta-analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity.

RESULTS

We identified 295 systematic reviews on COVID-19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID-19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I² statistic was widely used for assessing heterogeneity (92.20%); many meta-analyses had high values of I² . Among the meta-analyses using the random-effects model, 75.82% did not report the methods for model implementation; among those meta-analyses reporting implementation methods, the DerSimonian-Laird method was the most used one.

CONCLUSIONS

The current systematic reviews and meta-analyses on COVID-19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID-19 strictly follow well-developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta-analysis models.

Prior Choices of Between-Study Heterogeneity in Contemporary Bayesian Network Meta-analyses: an Empirical Study

BACKGROUND

Network meta-analysis (NMA) is a popular tool to compare multiple treatments in medical research. It is frequently implemented via Bayesian methods. The prior choice of between-study heterogeneity is critical in Bayesian NMAs. This study evaluates the impact of different priors for heterogeneity on NMA results.

METHODS

We identified all NMAs with binary outcomes published in The BMJ, JAMA, and The Lancet during 2010-2018, and extracted information about their prior choices for heterogeneity. Our primary analyses focused on those with publicly available full data. We re-analyzed the NMAs using 3 commonly-used non-informative priors and empirical informative log-normal priors. We obtained the posterior median odds ratios and 95% credible intervals of all comparisons, assessed the correlation among different priors, and used Bland-Altman plots to evaluate their agreement. The kappa statistic was also used to evaluate the agreement among these priors regarding statistical significance.

RESULTS

Among the selected Bayesian NMAs, 52.3% did not specify the prior choice for heterogeneity, and 84.1% did not provide rationales. We re-analyzed 19 NMAs with full data available, involving 894 studies, 173 treatments, and 395,429 patients. The correlation among posterior median (log) odds ratios using different priors were generally very strong for NMAs with over 20 studies. The informative priors produced substantially narrower credible intervals than non-informative priors, especially for NMAs with few studies. Bland-Altman plots and kappa statistics indicated strong overall agreement, but this was not always the case for a specific NMA.

CONCLUSIONS

Priors should be routinely reported in Bayesian NMAs. Sensitivity analyses are recommended to examine the impact of priors, especially for NMAs with relatively small sample sizes. Informative priors may produce substantially narrower credible intervals for such NMAs.

Association of JNC-8 and SPRINT Systolic Blood Pressure Levels With Cognitive Function and Related Racial Disparity

Importance

The Eighth Joint National Committee (JNC-8) recommended treating systolic blood pressure (SBP) to a target below 150 mm Hg in older adults, whereas data from the Systolic Blood Pressure Intervention Trial (SPRINT) suggested that a SBP level of lower than 120 mm Hg decreases cardiovascular event rates. Target SBP guidelines have not addressed the potential that black patients may have greater morbidity and mortality from hypertension, especially with regard to cognitive outcomes. The association of these discordant SBP targets with cognition and differences by race have not been systematically evaluated in the same population.

Objectives

To assess the long-term outcomes of the various recommended SBP levels and to determine if racial differences exist based on long-term cognitive trajectories.

Design, Setting, and Participants

A total of 1657 cognitively intact older adults receiving treatment for hypertension were studied from 1997 to 2007 in the Health Aging and Body Composition study. Data analysis was conducted from October 1, 2016, to January 1, 2017.

Main Outcomes and Measures

Cognition was assessed using the Modified Mini-Mental State Examination (3MSE) 4 times and the Digit Symbol Substitution Test (DSST) 5 times. At each visit, participants were classified as having an SBP level of 120 mm Hg or lower, 121 to 139 mm Hg, 140 to 149 mm Hg, or 150 mm Hg or higher based on the mean SBP level of 2 seated readings. Mixed models assessed the association of SBP levels with 10-year cognitive trajectories. The impact of race was tested using a race interaction term.

Results

During the 10-year study period, among the 1657 individuals (908 women and 784 black patients; mean [SE] age, 73.7 [0.1] years), there was a differential decrease in 3MSE and DSST scores by the SBP levels, with the greatest decrease in the group with SBP levels of 150 mm Hg or higher (adjusted decrease was 3.7 for 3MSE and 6.2 for DSST) and the lowest decrease in the group with SBP levels of 120 mm Hg or lower (adjusted decrease was 3.0 for 3MSE and 5.0 for DSST) (P < .001 for both). Compared with white patients, black patients had a greater difference between the higher and lower SBP levels in the decrease in cognition; adjusted differences between the group with SBP levels of 150 mm Hg or higher and the group with SBP levels of 120 mm Hg or lower were -0.05 in white patients and -0.08 in black patients for 3MSE (P = .03) and -0.07 in white patients and -0.13 in black patients for DSST (P = .05).

Conclusions and Relevance

For patients 70 years of age or older receiving treatment for hypertension, a SPRINT SBP level of 120 mm Hg or lower was not associated with worsening cognitive outcome and may be superior to the JNC-8 target for cognition. Lower SBP treatment levels may result in improved cognition in black patients.