Improving the delivery of palliative care through predictive modeling and healthcare informatics

Harnessing health information technology to promote equitable care for patients with limited English proficiency and complex care needs

BACKGROUND

Patients with language barriers encounter healthcare disparities, which may be alleviated by leveraging interpreter skills to reduce cultural, language, and literacy barriers through improved bidirectional communication. Evidence supports the use of in-person interpreters, especially for interactions involving patients with complex care needs. Unfortunately, due to interpreter shortages and clinician underuse of interpreters, patients with language barriers frequently do not get the language services they need or are entitled to. Health information technologies (HIT), including artificial intelligence (AI), have the potential to streamline processes, prompt clinicians to utilize in-person interpreters, and support prioritization.

METHODS

From May 1, 2023, to June 21, 2024, a single-center stepped wedge cluster randomized trial will be conducted within 35 units of Saint Marys Hospital & Methodist Hospital at Mayo Clinic in Rochester, Minnesota. The units include medical, surgical, trauma, and mixed ICUs and hospital floors that admit acute medical and surgical care patients as well as the emergency department (ED). The transitions between study phases will be initiated at 60-day intervals resulting in a 12-month study period. Units in the control group will receive standard care and rely on clinician initiative to request interpreter services. In the intervention group, the study team will generate a daily list of adult inpatients with language barriers, order the list based on their complexity scores (from highest to lowest), and share it with interpreter services, who will send a secure chat message to the bedside nurse. This engagement will be triggered by a predictive machine-learning algorithm based on a palliative care score, supplemented by other predictors of complexity including length of stay and level of care as well as procedures, events, and clinical notes.

DISCUSSION

This pragmatic clinical trial approach will integrate a predictive machine-learning algorithm into a workflow process and evaluate the effectiveness of the intervention. We will compare the use of in-person interpreters and time to first interpreter use between the control and intervention groups.

TRIAL REGISTRATION

NCT05860777. May 16, 2023.

Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice

Integrating machine learning (ML) models into clinical practice presents a challenge of maintaining their efficacy over time. While existing literature offers valuable strategies for detecting declining model performance, there is a need to document the broader challenges and solutions associated with the real-world development and integration of model monitoring solutions. This work details the development and use of a platform for monitoring the performance of a production-level ML model operating in Mayo Clinic. In this paper, we aimed to provide a series of considerations and guidelines necessary for integrating such a platform into a team's technical infrastructure and workflow. We have documented our experiences with this integration process, discussed the broader challenges encountered with real-world implementation and maintenance, and included the source code for the platform. Our monitoring platform was built as an R shiny application, developed and implemented over the course of 6 months. The platform has been used and maintained for 2 years and is still in use as of July 2023. The considerations necessary for the implementation of the monitoring platform center around 4 pillars: feasibility (what resources can be used for platform development?); design (through what statistics or models will the model be monitored, and how will these results be efficiently displayed to the end user?); implementation (how will this platform be built, and where will it exist within the IT ecosystem?); and policy (based on monitoring feedback, when and what actions will be taken to fix problems, and how will these problems be translated to clinical staff?). While much of the literature surrounding ML performance monitoring emphasizes methodological approaches for capturing changes in performance, there remains a battery of other challenges and considerations that must be addressed for successful real-world implementation.

Around the EQUATOR with Clin-STAR: Prediction modeling opportunities and challenges in aging research

The 2015 Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement was published to improve reporting transparency for prediction modeling studies. The objective of this review is to highlight methodologic challenges that aging-focused researchers will encounter when designing and reporting studies involving prediction models for older adults and provide guidance for addressing these challenges. In following the 22-item TRIPOD checklist, researchers must consider the representativeness of cohorts used (e.g., whether older adults with frailty, cognitive impairment, and social isolation were included), strategies for incorporating common geriatric predictors (e.g., age, comorbidities, functional status, and frailty), methods for handling missing data and competing risk of death, and assessment of model performance heterogeneity across important subgroups (e.g., age, sex, race, and ethnicity). We provide guidance to help aging-focused researchers develop, validate, and report models that can inform and improve patient care, which we label "TRIPOD-65."

Applications of Artificial Intelligence in Prostate Cancer Care: A Path to Enhanced Efficiency and Outcomes

The landscape of prostate cancer care has rapidly evolved. We have transitioned from the use of conventional imaging, radical surgeries, and single-agent androgen deprivation therapy to an era of advanced imaging, precision diagnostics, genomics, and targeted treatment options. Concurrently, the emergence of large language models (LLMs) has dramatically transformed the paradigm for artificial intelligence (AI). This convergence of advancements in prostate cancer management and AI provides a compelling rationale to comprehensively review the current state of AI applications in prostate cancer care. Here, we review the advancements in AI-driven applications across the continuum of the journey of a patient with prostate cancer from early interception to survivorship care. We subsequently discuss the role of AI in prostate cancer drug discovery, clinical trials, and clinical practice guidelines. In the localized disease setting, deep learning models demonstrated impressive performance in detecting and grading prostate cancer using imaging and pathology data. For biochemically recurrent diseases, machine learning approaches are being tested for improved risk stratification and treatment decisions. In advanced prostate cancer, deep learning can potentially improve prognostication and assist in clinical decision making. Furthermore, LLMs are poised to revolutionize information summarization and extraction, clinical trial design and operations, drug development, evidence synthesis, and clinical practice guidelines. Synergistic integration of multimodal data integration and human-AI integration are emerging as a key strategy to unlock the full potential of AI in prostate cancer care.

Predictive Models for Palliative Care Needs of Advanced Cancer Patients Receiving Chemotherapy

CONTEXT

Early palliative care is recommended within eight-week of diagnosing advanced cancer. Although guidelines suggest routine screening to identify cancer patients who could benefit from palliative care, implementing screening can be challenging due to understaffing and time constraints.

OBJECTIVES

To develop and evaluate machine learning models for predicting specialist palliative care needs in advanced cancer patients undergoing chemotherapy, and to investigate if predictive models could substitute screening tools.

METHODS

We conducted a retrospective cohort study using supervised machine learning. The study included patients aged 18 or older, diagnosed with metastatic or stage IV cancer, who underwent chemotherapy and distress screening at a designated cancer hospital in Japan from April 1, 2018, to March 31, 2023. Specialist palliative care needs were assessed based on distress screening scores and expert evaluations. Data sources were hospital's cancer registry, health claims database, and nursing admission records. The predictive model was developed using XGBoost, a machine learning algorithm.

RESULTS

Out of the 1878 included patients, 561 were analyzed. Among them, 114 (20.3%) exhibited needs for specialist palliative care. After under-sampling to address data imbalance, the models achieved an Area Under the Curve (AUC) of 0.89 with 95.8% sensitivity and a specificity of 71.9%. After feature selection, the model retained five variables, including the patient-reported pain score, and showcased an 0.82 AUC.

CONCLUSION

Our models could forecast specialist palliative care needs for advanced cancer patients on chemotherapy. Using five variables as predictors could replace screening tools and has the potential to contribute to earlier palliative care.

Using artificial intelligence to promote equitable care for inpatients with language barriers and complex medical needs: clinical stakeholder perspectives

OBJECTIVES

Inpatients with language barriers and complex medical needs suffer disparities in quality of care, safety, and health outcomes. Although in-person interpreters are particularly beneficial for these patients, they are underused. We plan to use machine learning predictive analytics to reliably identify patients with language barriers and complex medical needs to prioritize them for in-person interpreters.

MATERIALS AND METHODS

This qualitative study used stakeholder engagement through semi-structured interviews to understand the perceived risks and benefits of artificial intelligence (AI) in this domain. Stakeholders included clinicians, interpreters, and personnel involved in caring for these patients or for organizing interpreters. Data were coded and analyzed using NVIVO software.

RESULTS

We completed 49 interviews. Key perceived risks included concerns about transparency, accuracy, redundancy, privacy, perceived stigmatization among patients, alert fatigue, and supply-demand issues. Key perceived benefits included increased awareness of in-person interpreters, improved standard of care and prioritization for interpreter utilization; a streamlined process for accessing interpreters, empowered clinicians, and potential to overcome clinician bias.

DISCUSSION

This is the first study that elicits stakeholder perspectives on the use of AI with the goal of improved clinical care for patients with language barriers. Perceived benefits and risks related to the use of AI in this domain, overlapped with known hazards and values of AI but some benefits were unique for addressing challenges with providing interpreter services to patients with language barriers.

CONCLUSION

Artificial intelligence to identify and prioritize patients for interpreter services has the potential to improve standard of care and address healthcare disparities among patients with language barriers.

Exploring the Potential of Chatbots in Critical Care Nephrology

The exponential growth of artificial intelligence (AI) has allowed for its integration into multiple sectors, including, notably, healthcare. Chatbots have emerged as a pivotal resource for improving patient outcomes and assisting healthcare practitioners through various AI-based technologies. In critical care, kidney-related conditions play a significant role in determining patient outcomes. This article examines the potential for integrating chatbots into the workflows of critical care nephrology to optimize patient care. We detail their specific applications in critical care nephrology, such as managing acute kidney injury, alert systems, and continuous renal replacement therapy (CRRT); facilitating discussions around palliative care; and bolstering collaboration within a multidisciplinary team. Chatbots have the potential to augment real-time data availability, evaluate renal health, identify potential risk factors, build predictive models, and monitor patient progress. Moreover, they provide a platform for enhancing communication and education for both patients and healthcare providers, paving the way for enriched knowledge and honed professional skills. However, it is vital to recognize the inherent challenges and limitations when using chatbots in this domain. Here, we provide an in-depth exploration of the concerns tied to chatbots' accuracy, dependability, data protection and security, transparency, potential algorithmic biases, and ethical implications in critical care nephrology. While human discernment and intervention are indispensable, especially in complex medical scenarios or intricate situations, the sustained advancements in AI signal that the integration of precision-engineered chatbot algorithms within critical care nephrology has considerable potential to elevate patient care and pivotal outcome metrics in the future.

Effect of an Artificial Intelligence Decision Support Tool on Palliative Care Referral in Hospitalized Patients: A Randomized Clinical Trial

CONTEXT

Palliative care services are commonly provided to hospitalized patients, but accurately predicting who needs them remains a challenge.

OBJECTIVES

To assess the effectiveness on clinical outcomes of an artificial intelligence (AI)/machine learning (ML) decision support tool for predicting patient need for palliative care services in the hospital.

METHODS

The study design was a pragmatic, cluster-randomized, stepped-wedge clinical trial in 12 nursing units at two hospitals over a 15-month period between August 19, 2019, and November 17, 2020. Eligible patients were randomly assigned to either a medical service consultation recommendation triggered by an AI/ML tool predicting the need for palliative care services or usual care. The primary outcome was palliative care consultation note. Secondary outcomes included: hospital readmissions, length of stay, transfer to intensive care and palliative care consultation note by unit.

RESULTS

A total of 3183 patient hospitalizations were enrolled. Of eligible patients, A total of 2544 patients were randomized to the decision support tool (1212; 48%) and usual care (1332; 52%). Of these, 1717 patients (67%) were retained for analyses. Patients randomized to the intervention had a statistically significant higher incidence rate of palliative care consultation compared to the control group (IRR, 1.44 [95% CI, 1.11-1.92]). Exploratory evidence suggested that the decision support tool group reduced 60-day and 90-day hospital readmissions (OR, 0.75 [95% CI, 0.57, 0.97]) and (OR, 0.72 [95% CI, 0.55-0.93]) respectively.

CONCLUSION

A decision support tool integrated into palliative care practice and leveraging AI/ML demonstrated an increased palliative care consultation rate among hospitalized patients and reductions in hospitalizations.

Data Science Implementation Trends in Nursing Practice: A Review of the 2021 Literature

OBJECTIVES

The goal of this work was to provide a review of the implementation of data science-driven applications focused on structural or outcome-related nurse-sensitive indicators in the literature in 2021. By conducting this review, we aim to inform readers of trends in the nursing indicators being addressed, the patient populations and settings of focus, and lessons and challenges identified during the implementation of these tools.

METHODS

We conducted a rigorous descriptive review of the literature to identify relevant research published in 2021. We extracted data on model development, implementation-related strategies and measures, lessons learned, and challenges and stakeholder involvement. We also assessed whether reports of data science application implementations currently follow the guidelines of the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by AI (DECIDE-AI) framework.

RESULTS

Of 4,943 articles found in PubMed (NLM) and CINAHL (EBSCOhost), 11 were included in the final review and data extraction. Systems leveraging data science were developed for adult patient populations and were primarily deployed in hospital settings. The clinical domains targeted included mortality/deterioration, utilization/resource allocation, and hospital-acquired infections/COVID-19. The composition of development teams and types of stakeholders involved varied. Research teams more frequently reported on implementation methods than implementation results. Most studies provided lessons learned that could help inform future implementations of data science systems in health care.

CONCLUSION

In 2021, very few studies report on the implementation of data science-driven applications focused on structural- or outcome-related nurse-sensitive indicators. This gap in the sharing of implementation strategies needs to be addressed in order for these systems to be successfully adopted in health care settings.

Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm

Importance

Goal-concordant care is an ongoing challenge in hospital settings. Identification of high mortality risk within 30 days may call attention to the need to have serious illness conversations, including the documentation of patient goals of care.

Objective

To examine goals of care discussions (GOCDs) in a community hospital setting with patients identified as having a high risk of mortality by a machine learning mortality prediction algorithm.

Design, Setting, and Participants

This cohort study took place at community hospitals within 1 health care system. Participants included adult patients with a high risk of 30-day mortality who were admitted to 1 of 4 hospitals between January 2 and July 15, 2021. Patient encounters of inpatients in the intervention hospital where physicians were notified of the computed high risk mortality score were compared with patient encounters of inpatients in 3 community hospitals without the intervention (ie, matched control).

Intervention

Physicians of patients with a high risk of mortality within 30 days received notification and were encouraged to arrange for GOCDs.

Main Outcomes and Measures

The primary outcome was the percentage change of documented GOCDs prior to discharge. Propensity-score matching was completed on a preintervention and postintervention period using age, sex, race, COVID-19 status, and machine learning-predicted mortality risk scores. A difference-in-difference analysis validated the results.

Results

Overall, 537 patients were included in this study with 201 in the preintervention period (94 in the intervention group; 104 in the control group) and 336 patients in the postintervention period. The intervention and control groups included 168 patients per group and were well-balanced in age (mean [SD], 79.3 [9.60] vs 79.6 [9.21] years; standardized mean difference [SMD], 0.03), sex (female, 85 [51%] vs 85 [51%]; SMD, 0), race (White patients, 145 [86%] vs 144 [86%]; SMD 0.006), and Charlson comorbidities (median [range], 8.00 [2.00-15.0] vs 9.00 [2.00 to 19.0]; SMD, 0.34). Patients in the intervention group from preintervention to postintervention period were associated with being 5 times more likely to have documented GOCDs (OR, 5.11 [95% CI, 1.93 to 13.42]; P = .001) by discharge compared with matched controls, and GOCD occurred significantly earlier in the hospitalization in the intervention patients as compared with matched controls (median, 4 [95% CI, 3 to 6] days vs 16 [95% CI, 15 to not applicable] days; P < .001). Similar findings were observed for Black patient and White patient subgroups.

Conclusions and Relevance

In this cohort study, patients whose physicians had knowledge of high-risk predictions from machine learning mortality algorithms were associated with being 5 times more likely to have documented GOCDs than matched controls. Additional external validation is needed to determine if similar interventions would be helpful at other institutions.

The use of natural language processing in palliative care research: A scoping review

BACKGROUND

Natural language processing has been increasingly used in palliative care research over the last 5 years for its versatility and accuracy.

AIM

To evaluate and characterize natural language processing use in palliative care research, including the most commonly used natural language processing software and computational methods, data sources, trends in natural language processing use over time, and palliative care topics addressed.

DESIGN

A scoping review using the framework by Arksey and O'Malley and the updated recommendations proposed by Levac et al. was conducted.

SOURCES

PubMed, Web of Science, Embase, Scopus, and IEEE Xplore databases were searched for palliative care studies that utilized natural language processing tools. Data on study characteristics and natural language processing instruments used were collected and relevant palliative care topics were identified.

RESULTS

197 relevant references were identified. Of these, 82 were included after full-text review. Studies were published in 48 different journals from 2007 to 2022. The average sample size was 21,541 (median 435). Thirty-two different natural language processing software and 33 machine-learning methods were identified. Nine main sources for data processing and 15 main palliative care topics across the included studies were identified. The most frequent topic was mortality and prognosis prediction. We also identified a trend where natural language processing was frequently used in analyzing clinical serious illness conversations extracted from audio recordings.

CONCLUSIONS

We found 82 papers on palliative care using natural language processing methods for a wide-range of topics and sources of data that could expand the use of this methodology. We encourage researchers to consider incorporating this cutting-edge research methodology in future studies to improve published palliative care data.

Artificial Intelligence Implementation in Healthcare: A Theory-Based Scoping Review of Barriers and Facilitators

There is a large proliferation of complex data-driven artificial intelligence (AI) applications in many aspects of our daily lives, but their implementation in healthcare is still limited. This scoping review takes a theoretical approach to examine the barriers and facilitators based on empirical data from existing implementations. We searched the major databases of relevant scientific publications for articles related to AI in clinical settings, published between 2015 and 2021. Based on the theoretical constructs of the Consolidated Framework for Implementation Research (CFIR), we used a deductive, followed by an inductive, approach to extract facilitators and barriers. After screening 2784 studies, 19 studies were included in this review. Most of the cited facilitators were related to engagement with and management of the implementation process, while the most cited barriers dealt with the intervention's generalizability and interoperability with existing systems, as well as the inner settings' data quality and availability. We noted per-study imbalances related to the reporting of the theoretic domains. Our findings suggest a greater need for implementation science expertise in AI implementation projects, to improve both the implementation process and the quality of scientific reporting.

Palliative Care Exposure Relative to Predicted Risk of Six-Month Mortality in Hospitalized Adults

CONTEXT

The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care, prognostic discussions, and palliative care interventions remains unknown.

OBJECTIVES

To develop and validate a real-time predictive model for 180 day mortality using routinely available clinical and laboratory admission data and determine if palliative care exposure varies with predicted mortality risk.

METHODS

Adult admissions between October 1, 2013 and October.1, 2017 were included for the model derivation. A separate cohort was collected between January 1, 2018 and July 31, 2020 for validation. Patients were followed for 180 days from discharge, and logistic regression with selected variables was used to estimate patients' risk for mortality.

RESULTS

In the model derivation cohort, 7963 events of 180 day mortality (4.5% event rate) were observed. Median age was 53.0 (IQR 24.0-66.0) with 92,734 females (52.5%). Variables with strongest association with 180 day mortality included: Braden Score (OR 0.83; 95% CI 0.82-0.84); admission Do Not Resuscitate orders (OR 2.61; 95% CI 2.43-2.79); admission service and admission status. The model yielded excellent discriminatory ability in both the derivation (c-statistic 0.873; 95% CI 0.870-0.877; Brier score 0.04) and validation cohorts (c-statistic 0.844; 95% CI 0.840-0.847; Brier score 0.072). Inpatient palliative care consultations increased from 3% of minimal-risk encounters to 41% of high-risk encounters (P < 0.01).

CONCLUSION

We developed and temporally validated a predictive mortality model for adults from a large retrospective cohort, which helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.

Development and Structure of an Accurate Machine Learning Algorithm to Predict Inpatient Mortality and Hospice Outcomes in the Coronavirus Disease 2019 Era

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic has challenged the accuracy and racial biases present in traditional mortality scores. An accurate prognostic model that can be applied to hospitalized patients irrespective of race or COVID-19 status may benefit patient care.

RESEARCH DESIGN

This cohort study utilized historical and ongoing electronic health record features to develop and validate a deep-learning model applied on the second day of admission predicting a composite outcome of in-hospital mortality, discharge to hospice, or death within 30 days of admission. Model features included patient demographics, diagnoses, procedures, inpatient medications, laboratory values, vital signs, and substance use history. Conventional performance metrics were assessed, and subgroup analysis was performed based on race, COVID-19 status, and intensive care unit admission.

SUBJECTS

A total of 35,521 patients hospitalized between April 2020 and October 2020 at a single health care system including a tertiary academic referral center and 9 community hospitals.

RESULTS

Of 35,521 patients, including 9831 non-White patients and 2020 COVID-19 patients, 2838 (8.0%) met the composite outcome. Patients who experienced the composite outcome were older (73 vs. 61 y old) with similar sex and race distributions between groups. The model achieved an area under the receiver operating characteristic curve of 0.89 (95% confidence interval: 0.88, 0.91) and an average positive predictive value of 0.46 (0.40, 0.52). Model performance did not differ significantly in White (0.89) and non-White (0.90) subgroups or when grouping by COVID-19 status and intensive care unit admission.

CONCLUSION

A deep-learning model using large-volume, structured electronic health record data can effectively predict short-term mortality or hospice outcomes on the second day of admission in the general inpatient population without significant racial bias.

Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation

Objective

The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).

Materials and Methods

We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.

Results

Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.

Discussion

Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.

Conclusions

An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

Improving time to palliative care review with predictive modeling in an inpatient adult population: study protocol for a stepped-wedge, pragmatic randomized controlled trial

Background

Palliative care is a medical specialty centered on improving the quality of life (QOL) of patients with complex or life-threatening illnesses. The need for palliative care is increasing and with that the rigorous testing of triage tools that can be used quickly and reliably to identify patients that may benefit from palliative care.

Methods

To that aim, we will conduct a two-armed stepped-wedge cluster randomized trial rolled out to two inpatient hospitals to evaluate whether a machine learning algorithm accurately identifies patients who may benefit from a comprehensive review by a palliative care specialist and decreases time to receiving a palliative care consult in hospital. This is a single-center study which will be conducted from August 2019 to November 2020 at Saint Mary’s Hospital & Methodist Hospital both within Mayo Clinic Rochester in Minnesota. Clusters will be nursing units which will be chosen to be a mix of complex patients from Cardiology, Critical Care, and Oncology and had previously established relationships with palliative medicine. The stepped wedge design will have 12 units allocated to a design matrix of 5 treatment wedges. Each wedge will last 75 days resulting in a study period of 12 months of recruitment unless otherwise specified. Data will be analyzed with Bayesian hierarchical models with credible intervals denoting statistical significance.

Discussion

This intervention offers a pragmatic approach to delivering specialty palliative care to hospital patients in need using machine learning, thereby leading to high value care and improved outcomes. It is not enough for AI to be utilized by simply publishing research showing predictive performance; clinical trials demonstrating better outcomes are critically needed. Furthermore, the deployment of an AI algorithm is a complex process that requires multiple teams with varying skill sets. To evaluate a deployed AI, a pragmatic clinical trial can accommodate the difficulties of clinical practice while retaining scientific rigor.

Trial registration

ClinicalTrials.gov NCT03976297. Registered on 6 June 2019, prior to trial start.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05546-5.