Evidence synthesis, digital scribes, and translational challenges for artificial intelligence in healthcare

Healthcare has well-known challenges with safety, quality, and effectiveness, and many see artificial intelligence (AI) as essential to any solution. Emerging applications include the automated synthesis of best-practice research evidence including systematic reviews, which would ultimately see all clinical trial data published in a computational form for immediate synthesis. Digital scribes embed themselves in the process of care to detect, record, and summarize events and conversations for the electronic record. However, three persistent translational challenges must be addressed before AI is widely deployed. First, little effort is spent replicating AI trials, exposing patients to risks of methodological error and biases. Next, there is little reporting of patient harms from trials. Finally, AI built using machine learning may perform less effectively in different clinical settings.

A landscape analysis of HIV cure-related clinical trials and observational studies in 2018

OBJECTIVES

The community-based organisation Treatment Action Group has established an online listing of HIV cure-related trials and observational studies derived from trial registries. Our objective was to use the listing as a basis for a landscape analysis of the current status of HIV cure-related clinical research.

METHODS

Trials and observational studies listed as of August 2018 formed the sample set. Survey questions were developed on trial development, trial design, recruitment, enrolment, study completion and dissemination plans. A survey was sent to the contact(s) for each study. Supplemental information was collected from clinicaltrials.gov. The full dataset was then analysed.

RESULTS

A total of 99 interventional trials and 29 observational studies were included. Diverse interventions are under evaluation, including combinations of experimental candidates. Current studies plan to enrol over 7000 participants. Projected completion dates for ~90% of the sample fell between the fourth quarter of 2018 and the end of 2020. Potential obstacles to enrolment that were reported included concerns over invasive procedures and lack of potential benefit to participants. Data on the sex and ethnicity of enrollees were limited but sufficient to note a significant under-representation of women.

CONCLUSIONS

A considerable amount of HIV cure-related clinical research is under way. The results from these studies, which should help shape the future of the field, will become available over the next 2-4 years. Diversity both geographically and in terms of enrollees remains limited, particularly in terms of the participation of women, a concern that could significantly affect the generalisability of the findings.

Selective reporting bias in randomised controlled trials from two network meta-analyses: comparison of clinical trial registrations and their respective publications

OBJECTIVE

To determine (i) the difference in the frequency of serious adverse events (SAEs) reported in trial registrations and their respective primary publications and (ii) the effect of adding SAE data from registries to a network meta-analysis (NMA) in changing the surface under the cumulative ranking (SUCRA) curve values of interventions.

DESIGN

Secondary analysis of primary publications from two NMAs.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included randomised trials published in English after 2005 that were included in two NMAs of pharmacological interventions for Alzheimer's disease and chronic obstructive pulmonary disease.

DATA EXTRACTION

Two reviewers independently searched multiple international trial registries for registration status and abstracted data from the included study publications and ClinicalTrials.gov.

RESULTS

Of the 203 randomised trials included, 140 (69.0%) were registered with a trial registry and 72 (35.5%) posted results in the registry. The proportion of registered trials increased over time (38.5% in 2005 vs 78.6% in 2014). Of the publications with results posted in a trial registry, 14 (19.4%) had inconsistent reporting of overall SAEs; 7 (10.4%) studies did not report SAEs in the publication but did in the registry. In the 134 randomised trials with a prespecified primary outcome in the registry, 19 studies (9.4%) had a change in the primary outcome in the publication. Adding SAEs reported in registries to the NMAs did not affect the ranking of interventions.

CONCLUSION

We identified inconsistent reporting of SAEs in randomised trials that were included in two NMAs. Findings highlight the importance of including trial registries in the grey literature search and verifying safety data before incorporating it into NMAs.

STUDY REGISTRATION

osf.io/mk6dr.

SAMURAI: Sensitivity analysis of a meta-analysis with unpublished but registered analytical investigations (software)

BACKGROUND

The non-availability of clinical trial results contributes to publication bias, diminishing the validity of systematic reviews and meta-analyses. Although clinical trial registries have been established to reduce non-publication, the results from over half of all trials registered in ClinicalTrials.gov remain unpublished even 30 months after completion. Our goals were i) to utilize information available in registries (specifically, the number and sample sizes of registered unpublished studies) to gauge the sensitivity of a meta-analysis estimate of the effect size and its confidence interval to the non-publication of studies and ii) to develop user-friendly open-source software to perform this quantitative sensitivity analysis.

METHODS

The open-source software, the R package SAMURAI, was developed using R functions available in the R package metafor. The utility of SAMURAI is illustrated with two worked examples.

RESULTS

Our open-source software SAMURAI, can handle meta-analytic datasets of clinical trials with two independent treatment arms. Both binary and continuous outcomes are supported. For each unpublished study, the dataset requires only the sample sizes of each treatment arm and the user predicted 'outlook' for the studies. The user can specify five outlooks ranging from 'very positive' (i.e., very favorable towards intervention) to 'very negative' (i.e., very favorable towards control).SAMURAI assumes that control arms of unpublished studies have effects similar to the effect across control arms of published studies. For each experimental arm of an unpublished study, utilizing the user-provided outlook, SAMURAI randomly generates an effect estimate using a probability distribution, which may be based on a summary effect across published trials. SAMURAI then calculates the estimated summary treatment effect with a random effects model (DerSimonian & Laird method), and outputs the result as a forest plot.

CONCLUSIONS

To our knowledge, SAMURAI is currently the only tool that allows systematic reviewers to incorporate information about sample sizes of treatment groups in registered but unpublished clinical trials in their assessment of the potential impact of publication bias on meta-analyses. SAMURAI produces forest plots for visualizing how inclusion of registered unpublished studies might change the results of a meta-analysis. We hope systematic reviewers will find SAMURAI to be a useful addition to their toolkit.