Developing and Implementing Predictive Models in a Learning Healthcare System: Traditional and Artificial Intelligence Approaches in the Veterans Health Administration

Adaption and National Validation of a Tool for Predicting Mortality from Other Causes Among Men with Nonmetastatic Prostate Cancer

BACKGROUND

An electronic health record-based tool could improve accuracy and eliminate bias in provider estimation of the risk of death from other causes among men with nonmetastatic cancer.

OBJECTIVE

To recalibrate and validate the Veterans Aging Cohort Study Charlson Comorbidity Index (VACS-CCI) to predict non-prostate cancer mortality (non-PCM) and to compare it with a tool predicting prostate cancer mortality (PCM).

DESIGN, SETTING, AND PARTICIPANTS

An observational cohort of men with biopsy-confirmed nonmetastatic prostate cancer, enrolled from 2001 to 2018 in the national US Veterans Health Administration (VA), was divided by the year of diagnosis into the development (2001-2006 and 2008-2018) and validation (2007) sets.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Mortality (all cause, non-PCM, and PCM) was evaluated. Accuracy was assessed using calibration curves and C statistic in the development, validation, and combined sets; overall; and by age (<65 and 65+ yr), race (White and Black), Hispanic ethnicity, and treatment groups.

RESULTS AND LIMITATIONS

Among 107 370 individuals, we observed 24 977 deaths (86% non-PCM). The median age was 65 yr, 4947 were Black, and 5010 were Hispanic. Compared with CCI and age alone (C statistic 0.67, 95% confidence interval [CI] 0.67-0.68), VACS-CCI demonstrated improved validated discrimination (C statistic 0.75, 95% CI 0.74-0.75 for non-PCM). The prostate cancer mortality tool also discriminated well in validation (C statistic 0.81, 95% CI 0.78-0.83). Both were well calibrated overall and within subgroups. Owing to missing data, 18 009/125 379 (14%) were excluded, and VACS-CCI should be validated outside the VA prior to outside application.

CONCLUSIONS

VACS-CCI is ready for implementation within the VA. Electronic health record-assisted calculation is feasible, improves accuracy over age and CCI alone, and could mitigate inaccuracy and bias in provider estimation.

PATIENT SUMMARY

Veterans Aging Cohort Study Charlson Comorbidity Index is ready for application within the Veterans Health Administration. Electronic health record-assisted calculation is feasible, improves accuracy over age and Charlson Comorbidity Index alone, and might help mitigate inaccuracy and bias in provider estimation of the risk of non-prostate cancer mortality.

Progress with the Learning Health System 2.0: a rapid review of Learning Health Systems' responses to pandemics and climate change

BACKGROUND

Pandemics and climate change each challenge health systems through increasing numbers and new types of patients. To adapt to these challenges, leading health systems have embraced a Learning Health System (LHS) approach, aiming to increase the efficiency with which data is translated into actionable knowledge. This rapid review sought to determine how these health systems have used LHS frameworks to both address the challenges posed by the COVID-19 pandemic and climate change, and to prepare for future disturbances, and thus transition towards the LHS2.0.

METHODS

Three databases (Embase, Scopus, and PubMed) were searched for peer-reviewed literature published in English in the five years to March 2023. Publications were included if they described a real-world LHS's response to one or more of the following: the COVID-19 pandemic, future pandemics, current climate events, future climate change events. Data were extracted and thematically analyzed using the five dimensions of the Institute of Medicine/Zurynski-Braithwaite's LHS framework: Science and Informatics, Patient-Clinician Partnerships, Continuous Learning Culture, Incentives, and Structure and Governance.

RESULTS

The search yielded 182 unique publications, four of which reported on LHSs and climate change. Backward citation tracking yielded 13 additional pandemic-related publications. None of the climate change-related papers met the inclusion criteria. Thirty-two publications were included after full-text review. Most were case studies (n = 12, 38%), narrative descriptions (n = 9, 28%) or empirical studies (n = 9, 28%). Science and Informatics (n = 31, 97%), Continuous Learning Culture (n = 26, 81%), Structure and Governance (n = 23, 72%) were the most frequently discussed LHS dimensions. Incentives (n = 21, 66%) and Patient-Clinician Partnerships (n = 18, 56%) received less attention. Twenty-nine papers (91%) discussed benefits or opportunities created by pandemics to furthering the development of an LHS, compared to 22 papers (69%) that discussed challenges.

CONCLUSIONS

An LHS 2.0 approach appears well-suited to responding to the rapidly changing and uncertain conditions of a pandemic, and, by extension, to preparing health systems for the effects of climate change. LHSs that embrace a continuous learning culture can inform patient care, public policy, and public messaging, and those that wisely use IT systems for decision-making can more readily enact surveillance systems for future pandemics and climate change-related events.

TRIAL REGISTRATION

PROSPERO pre-registration: CRD42023408896.

Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions

BACKGROUND

The field of implementation science was developed to address the significant time delay between establishing an evidence-based practice and its widespread use. Although implementation science has contributed much toward bridging this gap, the evidence-to-practice chasm remains a challenge. There are some key aspects of implementation science in which advances are needed, including speed and assessing causality and mechanisms. The increasing availability of artificial intelligence applications offers opportunities to help address specific issues faced by the field of implementation science and expand its methods.

MAIN TEXT

This paper discusses the many ways artificial intelligence can address key challenges in applying implementation science methods while also considering potential pitfalls to the use of artificial intelligence. We answer the questions of "why" the field of implementation science should consider artificial intelligence, for "what" (the purpose and methods), and the "what" (consequences and challenges). We describe specific ways artificial intelligence can address implementation science challenges related to (1) speed, (2) sustainability, (3) equity, (4) generalizability, (5) assessing context and context-outcome relationships, and (6) assessing causality and mechanisms. Examples are provided from global health systems, public health, and precision health that illustrate both potential advantages and hazards of integrating artificial intelligence applications into implementation science methods. We conclude by providing recommendations and resources for implementation researchers and practitioners to leverage artificial intelligence in their work responsibly.

CONCLUSIONS

Artificial intelligence holds promise to advance implementation science methods ("why") and accelerate its goals of closing the evidence-to-practice gap ("purpose"). However, evaluation of artificial intelligence's potential unintended consequences must be considered and proactively monitored. Given the technical nature of artificial intelligence applications as well as their potential impact on the field, transdisciplinary collaboration is needed and may suggest the need for a subset of implementation scientists cross-trained in both fields to ensure artificial intelligence is used optimally and ethically.

Impact of the COVID-19 Pandemic on the Provision of Dialysis Service and Mortality in Veterans Receiving Maintenance Hemodialysis in the VA: An Interrupted Time-Series Analysis

INTRODUCTION

According to the US Renal Data System (USRDS), patients with end-stage kidney disease (ESKD) on maintenance dialysis had higher mortality during early COVID-19 pandemic. Less is known about the effect of the pandemic on the delivery of outpatient maintenance hemodialysis and its impact on death. We examined the effect of pandemic-related disruption on the delivery of dialysis treatment and mortality in patients with ESKD receiving maintenance hemodialysis in the Veterans Health Administration (VHA) facilities, the largest integrated national healthcare system in the USA.

METHODS

Using national VHA electronic health records data, we identified 7,302 Veterans with ESKD who received outpatient maintenance hemodialysis in VHA healthcare facilities during the COVID-19 pandemic (February 1, 2020, to December 31, 2021). We estimated the average change in the number of hemodialysis treatments received and deaths per 1,000 patients per month during the pandemic by conducting interrupted time-series analyses. We used seasonal autoregressive moving average (SARMA) models, in which February 2020 was used as the conditional intercept and months thereafter as conditional slope. The models were adjusted for seasonal variations and trends in rates during the pre-pandemic period (January 1, 2007, to January 31, 2020).

RESULTS

The number (95% CI) of hemodialysis treatments received per 1,000 patients per month during the pre-pandemic and pandemic periods were 12,670 (12,525-12,796) and 12,865 (12,729-13,002), respectively. Respective all-cause mortality rates (95% CI) were 17.1 (16.7-17.5) and 19.6 (18.5-20.7) per 1,000 patients per month. Findings from SARMA models demonstrate that there was no reduction in the dialysis treatments delivered during the pandemic (rate ratio: 0.999; 95% CI: 0.998-1.001), but there was a 2.3% (95% CI: 1.5-3.1%) increase in mortality. During the pandemic, the non-COVID hospitalization rate was 146 (95% CI: 143-149) per 1,000 patients per month, which was lower than the pre-pandemic rate of 175 (95% CI: 173-176). In contrast, there was evidence of higher use of telephone encounters during the pandemic (3,023; 95% CI: 2,957-3,089), compared with the pre-pandemic rate (1,282; 95% CI: 1,241-1,324).

CONCLUSIONS

We found no evidence that there was a disruption in the delivery of outpatient maintenance hemodialysis treatment in VHA facilities during the COVID-19 pandemic and that the modest rise in deaths during the pandemic is unlikely to be due to missed dialysis.

SMART choice (knee) tool: a patient-focused predictive model to predict improvement in health-related quality of life after total knee arthroplasty

BACKGROUND

Current predictive tools for TKA focus on clinicians rather than patients as the intended user. The purpose of this study was to develop a patient-focused model to predict health-related quality of life outcomes at 1-year post-TKA.

METHODS

Patients who underwent primary TKA for osteoarthritis from a tertiary institutional registry after January 2006 were analysed. The primary outcome was improvement after TKA defined by the minimal clinically important difference in utility score at 1-year post-surgery. Potential predictors included demographic information, comorbidities, lifestyle factors, and patient-reported outcome measures. Four models were developed, including both conventional statistics and machine learning (artificial intelligence) methods: logistic regression, classification tree, extreme gradient boosted trees, and random forest models. Models were evaluated using discrimination and calibration metrics.

RESULTS

A total of 3755 patients were included in the study. The logistic regression model performed the best with respect to both discrimination (AUC = 0.712) and calibration (intercept = -0.083, slope = 1.123, Brier score = 0.202). Less than 2% (n = 52) of the data were missing and therefore removed for complete case analysis. The final model used age (categorical), sex, baseline utility score, and baseline Veterans-RAND 12 responses as predictors.

CONCLUSION

The logistic regression model performed better than machine learning algorithms with respect to AUC and calibration plot. The logistic regression model was well calibrated enough to stratify patients into risk deciles based on their likelihood of improvement after surgery. Further research is required to evaluate the performance of predictive tools through pragmatic clinical trials.

LEVEL OF EVIDENCE

Level II, decision analysis.