Confidence score: a data-driven measure for inclusive systematic reviews considering unpublished preprints

OBJECTIVES

COVID-19, since its emergence in December 2019, has globally impacted research. Over 360 000 COVID-19-related manuscripts have been published on PubMed and preprint servers like medRxiv and bioRxiv, with preprints comprising about 15% of all manuscripts. Yet, the role and impact of preprints on COVID-19 research and evidence synthesis remain uncertain.

MATERIALS AND METHODS

We propose a novel data-driven method for assigning weights to individual preprints in systematic reviews and meta-analyses. This weight termed the "confidence score" is obtained using the survival cure model, also known as the survival mixture model, which takes into account the time elapsed between posting and publication of a preprint, as well as metadata such as the number of first 2-week citations, sample size, and study type.

RESULTS

Using 146 preprints on COVID-19 therapeutics posted from the beginning of the pandemic through April 30, 2021, we validated the confidence scores, showing an area under the curve of 0.95 (95% CI, 0.92-0.98). Through a use case on the effectiveness of hydroxychloroquine, we demonstrated how these scores can be incorporated practically into meta-analyses to properly weigh preprints.

DISCUSSION

It is important to note that our method does not aim to replace existing measures of study quality but rather serves as a supplementary measure that overcomes some limitations of current approaches.

CONCLUSION

Our proposed confidence score has the potential to improve systematic reviews of evidence related to COVID-19 and other clinical conditions by providing a data-driven approach to including unpublished manuscripts.

DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models

Linear mixed models are commonly used in healthcare-based association analyses for analyzing multi-site data with heterogeneous site-specific random effects. Due to regulations for protecting patients’ privacy, sensitive individual patient data (IPD) typically cannot be shared across sites. We propose an algorithm for fitting distributed linear mixed models (DLMMs) without sharing IPD across sites. This algorithm achieves results identical to those achieved using pooled IPD from multiple sites (i.e., the same effect size and standard error estimates), hence demonstrating the lossless property. The algorithm requires each site to contribute minimal aggregated data in only one round of communication. We demonstrate the lossless property of the proposed DLMM algorithm by investigating the associations between demographic and clinical characteristics and length of hospital stay in COVID-19 patients using administrative claims from the UnitedHealth Group Clinical Discovery Database. We extend this association study by incorporating 120,609 COVID-19 patients from 11 collaborative data sources worldwide.

A lossless, one-shot and privacy-preserving distributed algorithm was revealed for fitting linear mixed models on multi-site data. The algorithm was applied to a study of 120,609 COVID-19 patients using only minimal aggregated data from each of 14 sites.

EMBRACE: An EM-based bias reduction approach through Copas-model estimation for quantifying the evidence of selective publishing in network meta-analysis

Systematic reviews and meta-analyses synthesize results from well-conducted studies to optimize healthcare decision-making. Network meta-analysis (NMA) is particularly useful for improving precision, drawing new comparisons, and ranking multiple interventions. However, recommendations can be misled if published results are a selective sample of what has been collected by trialists, particularly when publication status is related to the significance of the findings. Unfortunately, the missing-not-at-random nature of this problem and the numerous parameters involved in modeling NMAs pose unique computational challenges to quantifying and correcting for publication bias, such that sensitivity analysis is used in practice. Motivated by this important methodological gap, we developed a novel and stable expectation-maximization (EM) algorithm to correct for publication bias in the network setting. We validate the method through simulation studies and show that it achieves substantial bias reduction in small to moderately sized NMAs. We also calibrate the method against a Bayesian analysis of a published NMA on antiplatlet therapies for maintaining vascular patency.

A historical review of publication bias

Publication bias is a well-known threat to the validity of meta-analyses and, more broadly, the reproducibility of scientific findings. When policies and recommendations are predicated on an incomplete evidence base, it undermines the goals of evidence-based decision-making. Great strides have been made in the last 50 years to understand and address this problem, including calls for mandatory trial registration and the development of statistical methods to detect and correct for publication bias. We offer an historical account of seminal contributions by the evidence synthesis community, with an emphasis on the parallel development of graph-based and selection model approaches. We also draw attention to current innovations and opportunities for future methodological work.

Rapid network meta-analysis using data from Food and Drug Administration approval packages is feasible but with limitations

OBJECTIVE

To test rapid approaches that use Drugs@FDA (a public database of approved drugs) and ClinicalTrials.gov to identify trials and to compare these two sources with bibliographic databases as an evidence base for a systematic review and network meta-analysis (NMA).

STUDY DESIGN AND SETTING

We searched bibliographic databases, Drugs@FDA, and ClinicalTrials.gov for eligible trials on first-line glaucoma medications. We extracted data, assessed risk of bias, and examined the completeness and consistency of information provided by different sources. We fitted random-effects NMA models separately for trials identified from each source and for all unique trials from three sources.

RESULTS

We identified 138 unique trials including 29,394 participants on 15 first-line glaucoma medications. For a given trial, information reported was sometimes inconsistent across data sources. Journal articles provided the most information needed for a systematic review; trial registrations provided the least. Compared to an NMA including all unique trials, we were able to generate reasonably precise effect estimates and similar relative rankings for available interventions using trials from Drugs@FDA alone (but not ClinicalTrials.gov).

CONCLUSIONS

A rapid NMA approach using data from Drugs@FDA is feasible but has its own limitations. Reporting of trial design and results can be improved in both the drug approval packages and on ClinicalTrials.gov.