Implementation and Continuous Monitoring of an Electronic Health Record Embedded Readmissions Clinical Decision Support Tool

Enhancing Clinical Decision Support in Nephrology: Addressing Algorithmic Bias Through Artificial Intelligence Governance

There has been a steady rise in the use of clinical decision support (CDS) tools to guide nephrology as well as general clinical care. Through guidance set by federal agencies and concerns raised by clinical investigators, there has been an equal rise in understanding whether such tools exhibit algorithmic bias leading to unfairness. This has spurred the more fundamental question of whether sensitive variables such as race should be included in CDS tools. In order to properly answer this question, it is necessary to understand how algorithmic bias arises. We break down 3 sources of bias encountered when using electronic health record data to develop CDS tools: (1) use of proxy variables, (2) observability concerns and (3) underlying heterogeneity. We discuss how answering the question of whether to include sensitive variables like race often hinges more on qualitative considerations than on quantitative analysis, dependent on the function that the sensitive variable serves. Based on our experience with our own institution's CDS governance group, we show how health system-based governance committees play a central role in guiding these difficult and important considerations. Ultimately, our goal is to foster a community practice of model development and governance teams that emphasizes consciousness about sensitive variables and prioritizes equity.

Exploring the Feasibility of an Electronic Tool for Predicting Retention in HIV Care: Provider Perspectives

Retention in care for people living with HIV (PLWH) is important for individual and population health. Preemptive identification of PLWH at high risk of lapsing in care may improve retention efforts. We surveyed providers at nine institutions throughout Chicago about their perspectives on using an electronic health record (EHR) tool to predict the risk of lapsing in care. Sixty-three percent (20/32) of providers reported currently assessing patients' risk for lapsing in care, and 91% (29/32) reported willingness to implement an EHR tool. When compared to those with other job roles, prescribers agreed (vs. neutral) that the tool would be less biased than personal judgment (OR 13.33, 95% CI 1.05, 169.56). Prescribers were also more likely to identify community health workers as persons who should deliver these interventions (OR 10.50, 95% CI 1.02, 108.58). Transportation, housing, substance use, and employment information were factors that providers wanted to be included in an EHR-based tool. Social workers were significantly more likely to indicate the inclusion of employment information as important (OR 10.50, 95% CI 1.11, 98.87) when compared to other participants. Acceptability of an EHR tool was high; future research should investigate barriers and evaluate the effectiveness of such a tool.

Effective Care Transitions: Reducing Readmissions to Improve Patient Care and Outcomes

PURPOSE/OBJECTIVES

Care transitions from one setting to another are vulnerable spaces where patients are susceptible to complications. Health systems, accountable care organizations, and payers recognize that care transition interventions are necessary to reduce unnecessary cost and utilization and improve patient outcomes following a hospitalization. Multiple care transition models exist, with varying degrees of intensity and success. This article describes a quality improvement project for a care transition model that incorporates key elements from the American Case Management Association's Transitions of Care Standards and the Transitional Care Management services as outlined by the Centers for Medicare & Medicaid Services.

PRIMARY PRACTICE SETTING

A collaboratively developed care transition model was implemented between a health system population health management office and a primary care organization.

FINDINGS/CONCLUSIONS

An effective care transitions model is stronger with collaboration among core members of a patient's care team, including a nurse care manager and a primary care provider. Ongoing quality improvement is necessary to gain efficiencies and effectiveness of such a model.

IMPLICATIONS FOR CASE MANAGEMENT PRACTICE

Care managers are integral in coordinating effective transitions. Care management practice includes transition of care standards that are associated with improved outcomes for patients at high risk for readmission. Interventions inclusive of medication reconciliation, identification and addressing of health-related social needs, review of discharge instructions, and coordinated follow-up are important factors that impact patient outcomes. Patients and their health system care teams benefit from the role of a care manager when there is a collaborative, coordinated, and timely approach to hospital follow-up.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare

OBJECTIVE

The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion.

MATERIALS AND METHODS

Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution.

RESULTS

An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center.

DISCUSSION

By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution.

CONCLUSIONS

We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Impact of a hospital service for adults with chronic childhood-onset disease: A propensity weighted analysis

BACKGROUND

Young adults with chronic childhood-onset diseases (CCOD) transitioning care from pediatrics to adult care are at high risk for readmission after hospital discharge. At our institution, we have implemented an inpatient service, the Med-Peds (MP) line, to improve transitions to adult care and reduce hospital utilization by young adults with CCOD.

OBJECTIVE

This study aimed to assess the effect of the MP line on length of stay (LOS) and 30-day readmission rates compared to other inpatient services.

METHODS

This was an observational, retrospective cohort analysis of patients admitted to the MP line compared to other hospital service lines over a 2-year period. To avoid potential confounding by indication for admission to the MP line, propensity score weighting methods were used.

RESULTS

The MP line cared for 302 patients with CCOD from June 2019 to July 2021. Compared to other service lines, there was a 33% reduction in relative risk of 30-day readmission (26.9% compared to 40.3%, risk ratio = 0.67, 95% confidence interval [CI] 0.55-0.81). LOS was 10% longer for the MP line (event time ratio (ETR): 1.10 95% CI 1.0-1.21) with median LOS 4.8 versus 4.5 days. Patients with sickle cell disease had less of a reduction in 30-day readmissions and longer LOS.

CONCLUSION

Hospitalization for young adults with CCOD on a MP service line was associated with lower 30-day readmission rates and longer LOS than hospitalization on other services. Further research is needed to assess which components of the line most contribute to decreased utilization.

Electronic health record-based readmission risk model performance for patients undergoing outpatient parenteral antibiotic therapy (OPAT)

BACKGROUND

Outpatient Parenteral Antibiotic Therapy (OPAT) provides coordinated services to deliver parenteral antibiotics outside of the acute care setting. However, the reduction in monitoring and supervision may impact the risks of readmission to the hospital. While identifying those at greatest risk of hospital readmission through use of computer decision support systems could aid in its prevention, validation of such tools in this patient population is lacking.

OBJECTIVE

The primary aim of this study is to determine the ability of the electronic health record-embedded EPIC Unplanned Readmission Model 1 to predict all-cause 30-day hospital unplanned readmissions in discharged patients receiving OPAT through the Duke University Heath System (DUHS) OPAT program. We then explored the impact of OPAT-specific variables on model performance.

METHODS

This retrospective cohort study included patients ≥ 18 years of age discharged to home or skilled nursing facility between July 1, 2019 -February 1, 2020 with OPAT care initiated inpatient and coordinated by the DUHS OPAT program and with at least one Epic readmission score during the index hospitalization. Those with a planned duration of OPAT < 7 days, receiving OPAT administered in a long-term acute care facility (LTAC), or ongoing renal replacement therapy were excluded. The relationship between the primary outcome (unplanned readmission during 30-day post-index discharge) and Epic readmission scores during the index admission (discharge and maximum) was examined using multivariable logistic regression models adjusted for additional predictors. The performance of the models was assessed with the scaled Brier score for overall model performance, the area under the receiver operating characteristics curve (C-index) for discrimination ability, calibration plot for calibration, and Hosmer-Lemeshow goodness-of-fit test for model fit.

RESULTS

The models incorporating maximum or discharge Epic readmission scores showed poor discrimination ability (C-index 0.51, 95% CI 0.45 to 0.58 for both models) in predicting 30-day unplanned readmission in the Duke OPAT cohort. Incorporating additional OPAT-specific variables did not improve the discrimination ability (C-index 0.55, 95% CI 0.49 to 0.62 for the max score; 0.56, 95% CI 0.49 to 0.62 for the discharge score). Although models for predicting 30-day unplanned OPAT-related readmission performed slightly better, discrimination ability was still poor (C-index 0.54, 95% CI 0.45 to 0.62 for both models).

CONCLUSION

EPIC Unplanned Readmission Model 1 scores were not useful in predicting either all-cause or OPAT-related 30-day unplanned readmission in the DUHS OPAT cohort. Further research is required to assess other predictors that can distinguish patients with higher risks of 30-day unplanned readmission in the DUHS OPAT patients.

Toward Predicting 30-Day Readmission Among Oncology Patients: Identifying Timely and Actionable Risk Factors

PURPOSE

Predicting 30-day readmission risk is paramount to improving the quality of patient care. In this study, we compare sets of patient-, provider-, and community-level variables that are available at two different points of a patient's inpatient encounter (first 48 hours and the full encounter) to train readmission prediction models and identify possible targets for appropriate interventions that can potentially reduce avoidable readmissions.

METHODS

Using electronic health record data from a retrospective cohort of 2,460 oncology patients and a comprehensive machine learning analysis pipeline, we trained and tested models predicting 30-day readmission on the basis of data available within the first 48 hours of admission and from the entire hospital encounter.

RESULTS

Leveraging all features, the light gradient boosting model produced higher, but comparable performance (area under receiver operating characteristic curve [AUROC]: 0.711) with the Epic model (AUROC: 0.697). Given features in the first 48 hours, the random forest model produces higher AUROC (0.684) than the Epic model (AUROC: 0.676). Both models flagged patients with a similar distribution of race and sex; however, our light gradient boosting and random forest models were more inclusive, flagging more patients among younger age groups. The Epic models were more sensitive to identifying patients with an average lower zip income. Our 48-hour models were powered by novel features at various levels: patient (weight change over 365 days, depression symptoms, laboratory values, and cancer type), hospital (winter discharge and hospital admission type), and community (zip income and marital status of partner).

CONCLUSION

We developed and validated models comparable with the existing Epic 30-day readmission models with several novel actionable insights that could create service interventions deployed by the case management or discharge planning teams that may decrease readmission rates over time.

Who Gets (and Who Should Get) a Serious Illness Conversation in the Hospital? An Analysis of Readmission Risk Score in an Electronic Health Record

Serious Illness Conversations (SICs) explore patients' prognostic awareness, hopes, and worries, and can help establish priorities for their care during and after hospitalization. While identifying patients who benefit from an SIC remains a challenge, this task may be facilitated by use of validated prediction scores available in most commercial electronic health records (EHRs), such as Epic's Readmission Risk Score (RRS). We identified the RRS on admission for all hospital encounters from October 2018 to August 2019 and measured the area under the receiver operating characteristic (AUROC) curve to determine whether RRS could accurately discriminate post discharge 6-month mortality. For encounters with standardized SIC documentation matched in a 1:3 ratio to controls by sex and age (±5 years), we constructed a multivariable, paired logistic regression model and measured the odds of SIC documentation per every 10% absolute increase in RRS. RRS was predictive of 6-month mortality with acceptable discrimination (AUROC .71) and was significantly associated with SIC documentation (adjusted OR 1.42, 95% CI 1.24-1.63). An RRS >28% used to identify patients with post discharge 6-month mortality had a high specificity (89.0%) and negative predictive value (NPV) (97.0%), but low sensitivity (25.2%) and positive predictive value (PPV) (7.9%). RRS may serve as a practical EHR-based screen to exclude patients not requiring an SIC, thereby leaving a smaller cohort to be further evaluated for SIC needs using other validated tools and clinical assessment.

Implementation Experience with a 30-Day Hospital Readmission Risk Score in a Large, Integrated Health System: A Retrospective Study

BACKGROUND

Driven by quality outcomes and economic incentives, predicting 30-day hospital readmissions remains important for healthcare systems. The Cleveland Clinic Health System (CCHS) implemented an internally validated readmission risk score in the electronic medical record (EMR).

OBJECTIVE

We evaluated the predictive accuracy of the readmission risk score across CCHS hospitals, across primary discharge diagnosis categories, between surgical/medical specialties, and by race and ethnicity.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Adult patients discharged from a CCHS hospital April 2017-September 2020.

MAIN MEASURES

Data was obtained from the CCHS EMR and billing databases. All patients discharged from a CCHS hospital were included except those from Oncology and Labor/Delivery, patients with hospice orders, or patients who died during admission. Discharges were categorized as surgical if from a surgical department or surgery was performed. Primary discharge diagnoses were classified per Agency for Healthcare Research and Quality Clinical Classifications Software Level 1 categories. Discrimination performance predicting 30-day readmission is reported using the c-statistic.

RESULTS

The final cohort included 600,872 discharges from 11 Northeast Ohio and Florida CCHS hospitals. The readmission risk score for the cohort had a c-statistic of 0.6875 with consistent yearly performance. The c-statistic for hospital sites ranged from 0.6762, CI [0.6634, 0.6876], to 0.7023, CI [0.6903, 0.7132]. Medical and surgical discharges showed consistent performance with c-statistics of 0.6923, CI [0.6807, 0.7045], and 0.6802, CI [0.6681, 0.6925], respectively. Primary discharge diagnosis showed variation, with lower performance for congenital anomalies and neoplasms. COVID-19 had a c-statistic of 0.6387. Subgroup analyses showed c-statistics of > 0.65 across race and ethnicity categories.

CONCLUSIONS

The CCHS readmission risk score showed good performance across diverse hospitals, across diagnosis categories, between surgical/medical specialties, and by patient race and ethnicity categories for 3 years after implementation, including during COVID-19. Evaluating clinical decision-making tools post-implementation is crucial to determine their continued relevance, identify opportunities to improve performance, and guide their appropriate use.

Combining adult with pediatric patient data to develop a clinical decision support tool intended for children: leveraging machine learning to model heterogeneity

Background

Clinical decision support (CDS) tools built using adult data do not typically perform well for children. We explored how best to leverage adult data to improve the performance of such tools. This study assesses whether it is better to build CDS tools for children using data from children alone or to use combined data from both adults and children.

Methods

Retrospective cohort using data from 2017 to 2020. Participants include all individuals (adults and children) receiving an elective surgery at a large academic medical center that provides adult and pediatric services. We predicted need for mechanical ventilation or admission to the intensive care unit (ICU). Predictor variables included demographic, clinical, and service utilization factors known prior to surgery. We compared predictive models built using machine learning to regression-based methods that used a pediatric or combined adult-pediatric cohort. We compared model performance based on Area Under the Receiver Operator Characteristic.

Results

While we found that adults and children have different risk factors, machine learning methods are able to appropriately model the underlying heterogeneity of each population and produce equally accurate predictive models whether using data only from pediatric patients or combined data from both children and adults. Results from regression-based methods were improved by the use of pediatric-specific data.

Conclusions

CDS tools for children can successfully use combined data from adults and children if the model accounts for underlying heterogeneity, as in machine learning models.

Implementation of a telehealth videoconference to improve hospital-to-skilled nursing care transitions: Preliminary data

BACKGROUND

Transition-related patient safety errors are high among patients discharged from hospitals to skilled nursing facilities (SNFs), and interventions are needed to improve communication between hospitals and SNF providers. Our objective was to describe the implementation of a pilot telehealth videoconference program modeled after Extension for Community Health Outcomes-Care Transitions and examine patient safety errors and readmissions.

METHODS

A multidisciplinary telehealth videoconference program was implemented at two academic hospitals for patients discharged to participating SNFs. Process measures, patient safety errors, and hospital readmissions were evaluated retrospectively for patients discussed at weekly conferences between July 2019-January 2020. Results were mapped to the constructs of the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) model. Descriptive statistics were reported for the conference process measures, patient and index hospitalization characteristics, and patient safety errors. The primary clinical outcome was all-cause 30-day readmissions. An intention-to-treat (ITT) analysis was conducted using logistic regression models fit to compare the probability of 30-day hospital readmission in patients discharged to participating SNFs across 7 months prior to after telehealth project implementation.

RESULTS

There were 263 patients (67% of eligible patients) discussed during 26 telehealth videoconferences. Mean discussion time per patient was 7.7 min and median prep time per patient was 24.2 min for the hospital pharmacist and 10.3 min for the hospital clinician. A total of 327 patient safety errors were uncovered, mostly related to communication (54%) and medications (43%). Differences in slopes (program period vs. pre-implementation) of the probability of readmission across the two time periods were not statistically significant (OR 0.95, [95% CI 0.75, 1.19]).

CONCLUSIONS

A pilot care innovations telehealth videoconference between hospital-based and SNF provider teams was successfully implemented within a large health system and enhanced care transitions by optimizing error-prone transitions. Future work is needed to understand process flow within nursing homes and its impact on clinical outcomes.

Inpatient pharmacists using a readmission risk model in supporting discharge medication reconciliation to reduce unplanned hospital readmissions: a quality improvement intervention

Introduction

Reducing unplanned hospital readmissions is an important priority for all hospitals and health systems. Hospital discharge can be complicated by discrepancies in the medication reconciliation and/or prescribing processes. Clinical pharmacist involvement in the medication reconciliation process at discharge can help prevent these discrepancies and possibly reduce unplanned hospital readmissions.

Methods

We report the results of our quality improvement intervention at Duke University Hospital, in which pharmacists were involved in the discharge medication reconciliation process on select high-risk general medicine patients over 2 years (2018–2020). Pharmacists performed traditional discharge medication reconciliation which included a review of medications for clinical appropriateness and affordability. A total of 1569 patients were identified as high risk for hospital readmission using the Epic readmission risk model and had a clinical pharmacist review the discharge medication reconciliation.

Results

This intervention was associated with a significantly lower 7-day readmission rate in patients who scored high risk for readmission and received pharmacist support in discharge medication reconciliation versus those patients who did not receive pharmacist support (5.8% vs 7.6%). There was no effect on readmission rates of 14 or 30 days. The clinical pharmacists had at least one intervention on 67% of patients reviewed and averaged 1.75 interventions per patient.

Conclusion

This quality improvement study showed that having clinical pharmacists intervene in the discharge medication reconciliation process in patients identified as high risk for readmission is associated with lower unplanned readmission rates at 7 days. The interventions by pharmacists were significant and well received by ordering providers. This study highlights the important role of a clinical pharmacist in the discharge medication reconciliation process.

Human Factors Considerations in Transitions in Care Clinical Decision Support System Implementation Studies

Objective: Review transitions in care clinical decision support system (CDSS) implementation studies and describe human factors considerations in users, design, alert types, intervention timing, and implementation outcomes. Methods: Literature review in PubMed guided by subject matter experts. Results: Twelve articles were included. Targeted users included physicians, nurses, pharmacists, or interdisciplinary teams. Alerts were deployed via email, cloud-based software, or the EHR in inpatient and/or outpatient settings. Outcome measures varied across articles, with mixed performance. There were six readmissions-focused, two prescribing, one laboratory, two prescribing and laboratory, and one discharge disposition CDSS. Few articles reported statistically significant differences in outcomes, and many reported alert fatigue. Discussion and Conclusion: Despite the increasing prevalence of CDSS for transitions in care, few articles describe implementation processes and outcomes, and evidence of clinical practice improvement is mixed. Future studies should utilize implementation science frameworks and incorporate appropriate implementation outcomes in addition to traditional clinical outcomes like readmission rates.