Implementation of Artificial Intelligence-Based Clinical Decision Support to Reduce Hospital Readmissions at a Regional Hospital

A comprehensive overview of barriers and strategies for AI implementation in healthcare: Mixed-method design

Implementation of artificial intelligence systems for healthcare is challenging. Understanding the barriers and implementation strategies can impact their adoption and allows for better anticipation and planning. This study's objective was to create a detailed inventory of barriers to and strategies for AI implementation in healthcare to support advancements in methods and implementation processes in healthcare. A sequential explanatory mixed method design was used. Firstly, scoping reviews and systematic literature reviews were identified using PubMed. Selected studies included empirical cases of AI implementation and use in clinical practice. As the reviews were deemed insufficient to fulfil the aim of the study, data collection shifted to the primary studies included in those reviews. The primary studies were screened by title and abstract, and thereafter read in full text. Then, data on barriers to and strategies for AI implementation were extracted from the included articles, thematically coded by inductive analysis, and summarized. Subsequently, a direct qualitative content analysis of 69 interviews with healthcare leaders and healthcare professionals confirmed and added results from the literature review. Thirty-eight empirical cases from the six identified scoping and literature reviews met the inclusion and exclusion criteria. Barriers to and strategies for AI implementation were grouped under three phases of implementation (planning, implementing, and sustaining the use) and were categorized into eleven concepts; Leadership, Buy-in, Change management, Engagement, Workflow, Finance and human resources, Legal, Training, Data, Evaluation and monitoring, Maintenance. Ethics emerged as a twelfth concept through qualitative analysis of the interviews. This study illustrates the inherent challenges and useful strategies in implementing AI in healthcare practice. Future research should explore various aspects of leadership, collaboration and contracts among key stakeholders, legal strategies surrounding clinicians' liability, solutions to ethical dilemmas, infrastructure for efficient integration of AI in workflows, and define decision points in the implementation process.

Artificial intelligence and machine learning: Definition of terms and current concepts in critical care research

With increasing computing power, artificial intelligence (AI) and machine learning (ML) have prospered, which facilitate the analysis of large datasets, especially those found in critical care. It is important to define these terminologies, to inform a standardized approach to critical care research. This manuscript hopes to clarify these terms with examples from medical literature. Three major components that are required for a successful ML implementation: (i) reliable dataset, (ii) ML algorithm, and (iii) unbiased model evaluation, are discussed. A reliable dataset can be structured or unstructured with limited noise, outliers, and missing values. ML, a subset of AI, is typically focused on supervised or unsupervised learning tasks in which the output is based on inputs and derived from iterative pattern recognition algorithms, while AI is the overall ability of a machine to "think" or mimic human behavior; and to analyze data free from human influence. Even with successful implementation, advanced AI and ML algorithms have faced challenges in adoption into practice, mainly due to their lack of interpretability, which hinders trust, buy-in, and engagement from clinicians. Consequently, traditional algorithms, such as linear and logistic regression, that may have reduced predictive power but are highly interpretable, continue to be widely used.

Recommendations to promote fairness and inclusion in biomedical AI research and clinical use

OBJECTIVE

Understanding and quantifying biases when designing and implementing actionable approaches to increase fairness and inclusion is critical for artificial intelligence (AI) in biomedical applications.

METHODS

In this Special Communication, we discuss how bias is introduced at different stages of the development and use of AI applications in biomedical sciences and health care. We describe various AI applications and their implications for fairness and inclusion in sections on 1) Bias in Data Source Landscapes, 2) Algorithmic Fairness, 3) Uncertainty in AI Predictions, 4) Explainable AI for Fairness and Equity, and 5) Sociological/Ethnographic Issues in Data and Results Representation.

RESULTS

We provide recommendations to address biases when developing and using AI in clinical applications.

CONCLUSION

These recommendations can be applied to informatics research and practice to foster more equitable and inclusive health care systems and research discoveries.

How artificial intelligence could transform emergency care

Artificial intelligence (AI) in healthcare is the ability of a computer to perform tasks typically associated with clinical care (e.g. medical decision-making and documentation). AI will soon be integrated into an increasing number of healthcare applications, including elements of emergency department (ED) care. Here, we describe the basics of AI, various categories of its functions (including machine learning and natural language processing) and review emerging and potential future use-cases for emergency care. For example, AI-assisted symptom checkers could help direct patients to the appropriate setting, models could assist in assigning triage levels, and ambient AI systems could document clinical encounters. AI could also help provide focused summaries of charts, summarize encounters for hand-offs, and create discharge instructions with an appropriate language and reading level. Additional use cases include medical decision making for decision rules, real-time models that predict clinical deterioration or sepsis, and efficient extraction of unstructured data for coding, billing, research, and quality initiatives. We discuss the potential transformative benefits of AI, as well as the concerns regarding its use (e.g. privacy, data accuracy, and the potential for changing the doctor-patient relationship).

A Neurosurgical Readmissions Reduction Program in an Academic Hospital Leveraging Machine Learning, Workflow Analysis, and Simulation

BACKGROUND

 Predicting 30-day hospital readmissions is crucial for improving patient outcomes, optimizing resource allocation, and achieving financial savings. Existing studies reporting the development of machine learning (ML) models predictive of neurosurgical readmissions do not report factors related to clinical implementation.

OBJECTIVES

 Train individual predictive models with good performance (area under the receiver operating characteristic curve or AUROC > 0.8), identify potential interventions through semi-structured interviews, and demonstrate estimated clinical and financial impact of these models.

METHODS

 Electronic health records were utilized with five ML methodologies: gradient boosting, decision tree, random forest, ridge logistic regression, and linear support vector machine. Variables of interest were determined by domain experts and literature. The dataset was split divided 80% for training and validation and 20% for testing randomly. Clinical workflow analysis was conducted using semi-structured interviews to identify possible intervention points. Calibrated agent-based models (ABMs), based on a previous study with interventions, were applied to simulate reductions of the 30-day readmission rate and financial costs.

RESULTS

 The dataset covered 12,334 neurosurgical intensive care unit (NSICU) admissions (11,029 patients); 1,903 spine surgery admissions (1,641 patients), and 2,208 traumatic brain injury (TBI) admissions (2,185 patients), with readmission rate of 13.13, 13.93, and 23.73%, respectively. The random forest model for NSICU achieved best performance with an AUROC score of 0.89, capturing potential patients effectively. Six interventions were identified through 12 semi-structured interviews targeting preoperative, inpatient stay, discharge phases, and follow-up phases. Calibrated ABMs simulated median readmission reduction rates and resulted in 13.13 to 10.12% (NSICU), 13.90 to 10.98% (spine surgery), and 23.64 to 21.20% (TBI). Approximately $1,300,614.28 in saving resulted from potential interventions.

CONCLUSION

 This study reports the successful development and simulation of an ML-based approach for predicting and reducing 30-day hospital readmissions in neurosurgery. The intervention shows feasibility in improving patient outcomes and reducing financial losses.

Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis

BACKGROUND

The adoption of predictive algorithms in health care comes with the potential for algorithmic bias, which could exacerbate existing disparities. Fairness metrics have been proposed to measure algorithmic bias, but their application to real-world tasks is limited.

OBJECTIVE

This study aims to evaluate the algorithmic bias associated with the application of common 30-day hospital readmission models and assess the usefulness and interpretability of selected fairness metrics.

METHODS

We used 10.6 million adult inpatient discharges from Maryland and Florida from 2016 to 2019 in this retrospective study. Models predicting 30-day hospital readmissions were evaluated: LACE Index, modified HOSPITAL score, and modified Centers for Medicare & Medicaid Services (CMS) readmission measure, which were applied as-is (using existing coefficients) and retrained (recalibrated with 50% of the data). Predictive performances and bias measures were evaluated for all, between Black and White populations, and between low- and other-income groups. Bias measures included the parity of false negative rate (FNR), false positive rate (FPR), 0-1 loss, and generalized entropy index. Racial bias represented by FNR and FPR differences was stratified to explore shifts in algorithmic bias in different populations.

RESULTS

The retrained CMS model demonstrated the best predictive performance (area under the curve: 0.74 in Maryland and 0.68-0.70 in Florida), and the modified HOSPITAL score demonstrated the best calibration (Brier score: 0.16-0.19 in Maryland and 0.19-0.21 in Florida). Calibration was better in White (compared to Black) populations and other-income (compared to low-income) groups, and the area under the curve was higher or similar in the Black (compared to White) populations. The retrained CMS and modified HOSPITAL score had the lowest racial and income bias in Maryland. In Florida, both of these models overall had the lowest income bias and the modified HOSPITAL score showed the lowest racial bias. In both states, the White and higher-income populations showed a higher FNR, while the Black and low-income populations resulted in a higher FPR and a higher 0-1 loss. When stratified by hospital and population composition, these models demonstrated heterogeneous algorithmic bias in different contexts and populations.

CONCLUSIONS

Caution must be taken when interpreting fairness measures' face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect health care use patterns and gaps in care. Simply relying on the statistical notions of bias could obscure or underplay the causes of health disparity. The imperfect health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness measures can serve as a useful routine assessment to detect disparate model performances but are insufficient to inform mechanisms or policy changes. However, such an assessment is an important first step toward data-driven improvement to address existing health disparities.

A multinational study on artificial intelligence adoption: Clinical implementers' perspectives

BACKGROUND

Despite substantial progress in AI research for healthcare, translating research achievements to AI systems in clinical settings is challenging and, in many cases, unsatisfactory. As a result, many AI investments have stalled at the prototype level, never reaching clinical settings.

OBJECTIVE

To improve the chances of future AI implementation projects succeeding, we analyzed the experiences of clinical AI system implementers to better understand the challenges and success factors in their implementations.

METHODS

Thirty-seven implementers of clinical AI from European and North and South American countries were interviewed. Semi-structured interviews were transcribed and analyzed qualitatively with the framework method, identifying the success factors and the reasons for challenges as well as documenting proposals from implementers to improve AI adoption in clinical settings.

RESULTS

We gathered the implementers' requirements for facilitating AI adoption in the clinical setting. The main findings include 1) the lesser importance of AI explainability in favor of proper clinical validation studies, 2) the need to actively involve clinical practitioners, and not only clinical researchers, in the inception of AI research projects, 3) the need for better information structures and processes to manage data access and the ethical approval of AI projects, 4) the need for better support for regulatory compliance and avoidance of duplications in data management approval bodies, 5) the need to increase both clinicians' and citizens' literacy as respects the benefits and limitations of AI, and 6) the need for better funding schemes to support the implementation, embedding, and validation of AI in the clinical workflow, beyond pilots.

CONCLUSION

Participants in the interviews are positive about the future of AI in clinical settings. At the same time, they proposenumerous measures to transfer research advancesinto implementations that will benefit healthcare personnel. Transferring AI research into benefits for healthcare workers and patients requires adjustments in regulations, data access procedures, education, funding schemes, and validation of AI systems.

Comparison of Acute Health Care Utilization Between Patients Receiving In-Center Hemodialysis and the General Population: A Population-Based Matched Cohort Study From Ontario, Canada

Background

Patients receiving maintenance hemodialysis have multiple comorbidities and are at high risk of presenting to the hospital. However, the incidence and cost of acute health care utilization in the in-center hemodialysis population and how this compares with other populations is poorly understood.

Objective

To determine the rate, pattern, and cost of emergency department visits and hospitalizations in patients receiving in-center hemodialysis compared with a matched general population.

Design

Population-based matched cohort study.

Setting

We used linked administrative health care databases from Ontario, Canada.

Patients

We included 25 379 patients (incident and prevalent) receiving in-center hemodialysis between January 1, 2010, and December 31, 2018. Patients were matched on birth date (±2 years), sex, and cohort entry date using a 1:4 ratio to 101 516 individuals from the general population.

Measurements

Our primary outcomes were emergency department visits (allowing for multiple visits per individual) and hospital admissions from the emergency department. We also assessed all-cause hospitalizations, all-cause readmissions within 30 days of discharge from the original hospitalization, length of stay for hospital admissions (including multiple visits per individual), and the financial cost of these admissions.

Methods

We presented the rate, percentage, median (25th, 75th percentiles), and incidence rate per 1000 person-years for emergency department visits and hospitalizations. Individual-level health care costs for emergency department visits and all-cause hospitalization were estimated using resource intensity weights multiplied by the cost per weighted case.

Results

Patients receiving in-center hemodialysis had substantially more comorbidities (eg, diabetes) than the matched general population. Eighty percent (n = 20 309) of patients receiving in-center hemodialysis had at least 1 emergency department visit compared with 56% (n = 56 452) of individuals in the matched general population, over a median follow-up of 1.8 years (25th, 75th percentiles: 0.7, 3.6) and 5.2 (2.5, 8.4) years, respectively. The incidence rate of emergency department visits, allowing for multiple visits per individual, was 2274 per 1000 person-years (95% confidence interval [CI]: 2263, 2286) for patients receiving in-center hemodialysis, which was almost 5 times as high as the matched general population (471 per 1000 person-years; 95% CI: 469, 473). The rate of hospital admissions from the emergency department and the rate of all-cause hospital admissions in the in-center hemodialysis population was more than 7 times as high as the matched general population (hospital admissions from the emergency department: 786 vs 101 per 1000 person-years; all-cause hospital admissions: 1056 vs 139 per 1000 person-years). The median number of all-cause hospitalization days per patient year was 4.0 (0, 16.5) in the in-center hemodialysis population compared with 0 (0, 0.5) in the matched general population. The cost per patient-year for emergency department visits in the in-center hemodialysis population was approximately 5.5 times as high as the matched general population while the cost of hospitalizations in the in-center hemodialysis population was approximately 11 times as high as the matched general population (emergency department visits: CAN$ 1153 vs CAN$ 209; hospitalizations: CAN$ 21 151 vs CAN$ 1873 [all costs in 2023 CAN$]).

Limitations

External generalizability and we could not determine whether emergency department visits and hospitalizations were preventable.

Conclusions

Patients receiving in-center hemodialysis have high acute health care utilization. These results improve our understanding of the burden of disease and the associated costs in the in-center hemodialysis population, highlight the need to improve acute outcomes, and can aid health care capacity planning. Additional research is needed to address the risk of hospitalization after controlling for patient comorbidities.

Trial registration

This is not applicable as this is a population-based matched cohort study and not a clinical trial.

Augmenting a Transitional Care Model With Artificial Intelligence Decreased Readmissions

OBJECTIVES

Evaluate if augmenting a transitions of care delivery model with insights from artificial intelligence (AI) that applied clinical and exogenous social determinants of health data would reduce rehospitalization in older adults.

DESIGN

Retrospective case-control study.

SETTING AND PARTICIPANTS

Adult patients discharged from integrated health system between November 1, 2019, and February 31, 2020, and enrolled in a rehospitalization reduction transitional care management program.

INTERVENTION

An AI algorithm utilizing multiple data sources including clinical, socioeconomic, and behavioral data was developed to predict patients at highest risk for readmitting within 30 days and provide care navigators five care recommendations to prevent rehospitalization.

METHODS

Adjusted incidence of rehospitalization was estimated with Poisson regression and compared between transitional care management enrollees that used AI insights and matched enrollees for whom AI insights were not used.

RESULTS

Analyses included 6371 hospital encounters between November 2019 and February 2020 across 12 hospitals. Of the encounters 29.3% were identified by AI as being medium-high risk for re-hospitalizing within 30 days, for which AI provided transitional care recommendations to the transitional care management team. The navigation team completed 40.2% of AI recommendations for these high-risk older adults. These patients had overall 21.0% less adjusted incidence of 30-day rehospitalization compared with matched control encounters, or 69 fewer rehospitalizations per 1000 encounters (95% CI 0.65‒0.95).

CONCLUSIONS AND IMPLICATIONS

Coordinating a patient's care continuum is critical for safe and effective transition of care. This study found that augmenting an existing transition of care navigation program with patient insights from AI reduced rehospitalization more than without AI insights. Augmenting transitional care with insights from AI could be a cost-effective intervention to improve transitional care outcomes and reduce unnecessary rehospitalization. Future studies should examine cost-effectiveness of augmenting transitional care models of care with AI when hospitals and post-acute providers partner with AI companies.

Digital Health Technologies for Post-Discharge Care after Heart Failure Hospitalisation to Relieve Symptoms and Improve Clinical Outcomes

The prevention of recurrent heart failure (HF) hospitalisations is of particular importance, as each such successive event may increase the risk of death. Effective care planning during the vulnerable phase after discharge is crucial for symptom control and improving patient prognosis. Many clinical trials have focused on telemedicine interventions in HF, with varying effects on the primary endpoints. However, the evidence of the effectiveness of telemedicine solutions in cardiology is growing. The scope of this review is to present complementary telemedicine modalities that can support outpatient care of patients recently hospitalised due to worsening HF. Remote disease management models, such as video (tele) consultations, structured telephone support, and remote monitoring of vital signs, were presented as core components of telecare. Invasive and non-invasive monitoring of volume status was described as an important step forward to prevent congestion-the main cause of clinical decompensation. The idea of virtual wards, combining these facilities with in-person visits, strengthens the opportunity for education and enhancement to promote more intensive self-care. Electronic platforms provide coordination of tasks within multidisciplinary teams and structured data that can be effectively used to develop predictive algorithms based on advanced digital science, such as artificial intelligence. The rapid progress in informatics, telematics, and device technologies provides a wide range of possibilities for further development in this area. However, there are still existing gaps regarding the use of telemedicine solutions in HF patients, and future randomised telemedicine trials and real-life registries are still definitely needed.

Artificial Intelligence Implementation in Healthcare: A Theory-Based Scoping Review of Barriers and Facilitators

There is a large proliferation of complex data-driven artificial intelligence (AI) applications in many aspects of our daily lives, but their implementation in healthcare is still limited. This scoping review takes a theoretical approach to examine the barriers and facilitators based on empirical data from existing implementations. We searched the major databases of relevant scientific publications for articles related to AI in clinical settings, published between 2015 and 2021. Based on the theoretical constructs of the Consolidated Framework for Implementation Research (CFIR), we used a deductive, followed by an inductive, approach to extract facilitators and barriers. After screening 2784 studies, 19 studies were included in this review. Most of the cited facilitators were related to engagement with and management of the implementation process, while the most cited barriers dealt with the intervention's generalizability and interoperability with existing systems, as well as the inner settings' data quality and availability. We noted per-study imbalances related to the reporting of the theoretic domains. Our findings suggest a greater need for implementation science expertise in AI implementation projects, to improve both the implementation process and the quality of scientific reporting.

Performance Characteristics of a Machine-Learning Tool to Predict 7-Day Hospital Readmissions

OBJECTIVES

To develop an institutional machine-learning (ML) tool that utilizes demographic, socioeconomic, and medical information to stratify risk for 7-day readmission after hospital discharge; assess the validity and reliability of the tool; and demonstrate its discriminatory capacity to predict readmissions.

PATIENTS AND METHODS

We performed a combined single-center, cross-sectional, and prospective study of pediatric hospitalists assessing the face and content validity of the developed readmission ML tool. The cross-sectional analyses used data from questionnaire Likert scale responses regarding face and content validity. Prospectively, we compared the discriminatory capacity of provider readmission risk versus the ML tool to predict 7-day readmissions assessed via area under the receiver operating characteristic curve analyses.

RESULTS

Overall, 80% (15 of 20) of hospitalists reported being somewhat to very confident with their ability to accurately predict readmission risk; 53% reported that an ML tool would influence clinical decision-making (face validity). The ML tool variable exhibiting the highest content validity was history of previous 7-day readmission. Prospective provider assessment of risk of 413 discharges showed minimal agreement with the ML tool (κ = 0.104 [95% confidence interval 0.028-0.179]). Both provider gestalt and ML calculations poorly predicted 7-day readmissions (area under the receiver operating characteristic curve: 0.67 vs 0.52; P = .11).

CONCLUSIONS

An ML tool for predicting 7-day hospital readmissions after discharge from the general pediatric ward had limited face and content validity among pediatric hospitalists. Both provider and ML-based determinations of readmission risk were of limited discriminatory value. Before incorporating similar tools into real-time discharge planning, model calibration efforts are needed.

Leveraging Advanced Data Analytics to Predict the Risk of All-Cause Seven-Day Emergency Readmissions

Introduction

Emergency readmissions have been a long-time, multifaceted, unsolved problem. Developing a predictive model calibrated with hospital-specific Electronic Health Record (EHR) data could give higher prediction accuracy and insights into high-risk patients for readmission. Thus, we need to proactively introduce the necessary interventions. This study aims to investigate the relationship between features that consider significant predictors of at-risk patients for seven-day readmission through logistic regression in addition to developing several machine learning models to test the predictability of those attributes using EHR data in a Saudi Arabia-specific ED context.

Methods

Univariate and multivariate logistic regression has been used to identify the most statistically significant features that contributed to classifying readmitted and not readmitted patients. Seven different machine learning models were trained and tested, and a comparison between the best-performing model was conducted in terms of five performance metrics. To construct the prediction model and internally validate it, the processed dataset was split into two sets: 70% for the training set and 30% for the test set or validation set.

Results

XGBoost achieved the highest accuracy (64%) in predicting early seven-day readmissions. Catboost was the second-best predictive model at 61%. XGBoost achieved the highest specificity at 70%, and all the models had a sensitivity of 57% except for XGBoost and Catboost at 32% and 38%, respectively. All predictive attributes, patient age, length of stay (LOS) in minutes, visit time (AM), marital status (married), number of medications, and number of abnormal lab results were significant predictors of early seven-day readmissions while marital status and number of vital-sign instabilities at discharge were not statistically significant predictors of seven-day readmission.

Conclusion

Although XGBoost and Catboost showed good accuracy, none of the models achieved good discriminative ability in terms of sensitivity and specificity. Thus, none can be clinically used for predicting early seven-day readmission. More predictive variables need to be fed into the model, specifically predictors approximate to the day of discharge, in order to optimize the model’s performance.

Artificial Intelligence Applications in Health Care Practice: A Scoping Review (Preprint)

Background

Artificial intelligence (AI) is often heralded as a potential disruptor that will transform the practice of medicine. The amount of data collected and available in health care, coupled with advances in computational power, has contributed to advances in AI and an exponential growth of publications. However, the development of AI applications does not guarantee their adoption into routine practice. There is a risk that despite the resources invested, benefits for patients, staff, and society will not be realized if AI implementation is not better understood.

Objective

The aim of this study was to explore how the implementation of AI in health care practice has been described and researched in the literature by answering 3 questions: What are the characteristics of research on implementation of AI in practice? What types and applications of AI systems are described? What characteristics of the implementation process for AI systems are discernible?

Methods

A scoping review was conducted of MEDLINE (PubMed), Scopus, Web of Science, CINAHL, and PsycINFO databases to identify empirical studies of AI implementation in health care since 2011, in addition to snowball sampling of selected reference lists. Using Rayyan software, we screened titles and abstracts and selected full-text articles. Data from the included articles were charted and summarized.

Results

Of the 9218 records retrieved, 45 (0.49%) articles were included. The articles cover diverse clinical settings and disciplines; most (32/45, 71%) were published recently, were from high-income countries (33/45, 73%), and were intended for care providers (25/45, 56%). AI systems are predominantly intended for clinical care, particularly clinical care pertaining to patient-provider encounters. More than half (24/45, 53%) possess no action autonomy but rather support human decision-making. The focus of most research was on establishing the effectiveness of interventions (16/45, 35%) or related to technical and computational aspects of AI systems (11/45, 24%). Focus on the specifics of implementation processes does not yet seem to be a priority in research, and the use of frameworks to guide implementation is rare.

Conclusions

Our current empirical knowledge derives from implementations of AI systems with low action autonomy and approaches common to implementations of other types of information systems. To develop a specific and empirically based implementation framework, further research is needed on the more disruptive types of AI systems being implemented in routine care and on aspects unique to AI implementation in health care, such as building trust, addressing transparency issues, developing explainable and interpretable solutions, and addressing ethical concerns around privacy and data protection.

Application of a Machine Learning-based Decision Support Tool to Improve an Injury Surveillance System Workflow

Abstract Background Emergency department (ED)-based injury surveillance systems across many countries face resourcing challenges related to manual validation and coding of data. Objective This paper describes the evaluation of a machine learning-based Decision Support Tool (DST) to assist injury surveillance departments in the validation, coding and use of their data, comparing outcomes in coding time and accuracy pre- and post-implementation. Methods Manually coded injury surveillance data has been used to develop, train and iteratively refine a machine learning-based classifier to enable semi-automated coding of injury narrative data. This paper describes a trial implementation of the machine learning-based DST in the Queensland Injury Surveillance Unit (QISU) workflow using a major pediatric hospital's emergency department data comparing outcomes in coding time and accuracy pre- and post-implementation. Results The study found a 10% reduction in manual coding time after the DST was introduced. The Kappa statistics analysis in both DST-assisted and unassisted data shows increases in accuracy across three data fields; injury intent (85.4% unassisted vs. 94.5% assisted), external cause (88.8% unassisted vs. 91.8% assisted) and injury factor (89.3% unassisted vs. 92.9% assisted). The classifier was also used to produce a timely report monitoring injury patterns during the COVID-19 pandemic. Hence, it has the potential for near real-time surveillance of emerging hazards to inform public health responses. Conclusions The integration of the DST into the injury surveillance workflow shows benefits as it facilitates timely reporting and acts as a DST in the manual coding process.

Risk stratification of syncope: Current syncope guidelines and beyond

Syncope is an alarming event carrying the possibility of serious outcomes, including sudden cardiac death (SCD). Therefore, immediate risk stratification should be applied whenever syncope occurs, especially in the Emergency Department, where most dramatic presentations occur. It has long been known that short- and long-term syncope prognosis is affected not only by its mechanism but also by presence of concomitant conditions, especially cardiovascular disease. Over the last two decades, several syncope prediction tools have been developed to refine patient stratification and triage patients who need expert in-hospital care from those who may receive nonurgent expert care in the community. However, despite promising results, prognostic tools for syncope remain challenging and often poorly effective. Current European Society of Cardiology syncope guidelines recommend an initial syncope workup based on detailed patient's history, physical examination supine and standing blood pressure, resting ECG, and laboratory tests, including cardiac biomarkers, where appropriate. Subsequent risk stratification based on screening of features aims to identify three groups: high-, intermediate- and low-risk. The first should immediately be hospitalized and appropriately investigated; intermediate group, with recurrent or medium-risk events, requires systematic evaluation by syncope experts; low-risk group, sporadic reflex syncope, merits education about its benign nature, and discharge. Thus, initial syncope risk stratification is crucial as it determines how and by whom syncope patients are managed. This review summarizes the crucial elements of syncope risk stratification, pros and cons of proposed risk evaluation scores, major challenges in initial syncope management, and how risk stratification impacts management of high-risk/recurrent syncope.

Human Factors Considerations in Transitions in Care Clinical Decision Support System Implementation Studies

Objective: Review transitions in care clinical decision support system (CDSS) implementation studies and describe human factors considerations in users, design, alert types, intervention timing, and implementation outcomes. Methods: Literature review in PubMed guided by subject matter experts. Results: Twelve articles were included. Targeted users included physicians, nurses, pharmacists, or interdisciplinary teams. Alerts were deployed via email, cloud-based software, or the EHR in inpatient and/or outpatient settings. Outcome measures varied across articles, with mixed performance. There were six readmissions-focused, two prescribing, one laboratory, two prescribing and laboratory, and one discharge disposition CDSS. Few articles reported statistically significant differences in outcomes, and many reported alert fatigue. Discussion and Conclusion: Despite the increasing prevalence of CDSS for transitions in care, few articles describe implementation processes and outcomes, and evidence of clinical practice improvement is mixed. Future studies should utilize implementation science frameworks and incorporate appropriate implementation outcomes in addition to traditional clinical outcomes like readmission rates.

Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation

BACKGROUND

Health care organizations are integrating a variety of machine learning (ML)-based clinical decision support (CDS) tools into their operations, but practitioners lack clear guidance regarding how to implement these tools so that they assist end users in their work.

PURPOSE

We designed this study to identify how health care organizations can facilitate collaborative development of ML-based CDS tools to enhance their value for health care delivery in real-world settings.

METHODOLOGY/APPROACH

We utilized qualitative methods, including 37 interviews in a large, multispecialty health system that developed and implemented two operational ML-based CDS tools in two of its hospital sites. We performed thematic analyses to inform presentation of an explanatory framework and recommendations.

RESULTS

We found that ML-based CDS tool development and implementation into clinical workflows proceeded in four phases: iterative solution coidentification, iterative coengagement, iterative coapplication, and iterative corefinement. Each phase is characterized by a collaborative back-and-forth process between the technology's developers and users, through which both users' activities and the technology itself are transformed.

CONCLUSION

Health care organizations that anticipate iterative collaboration to be an integral aspect of their ML-based CDS tools' development and implementation process may have more success in deploying ML-based CDS tools that assist end users in their work than organizations that expect a traditional technology innovation process.

PRACTICE IMPLICATIONS

Managers developing and implementing ML-based CDS tools should frame the work as a collaborative learning opportunity for both users and the technology itself and should solicit constructive feedback from users on potential changes to the technology, in addition to potential changes to user workflows, in an ongoing, iterative manner.

Augmented intelligence to predict 30-day mortality in patients with cancer

Aim: An augmented intelligence tool to predict short-term mortality risk among patients with cancer could help identify those in need of actionable interventions or palliative care services. Patients & methods: An algorithm to predict 30-day mortality risk was developed using socioeconomic and clinical data from patients in a large community hematology/oncology practice. Patients were scored weekly; algorithm performance was assessed using dates of death in patients' electronic health records. Results: For patients scored as highest risk for 30-day mortality, the event rate was 4.9% (vs 0.7% in patients scored as low risk; a 7.4-times greater risk). Conclusion: The development and validation of a decision tool to accurately identify patients with cancer who are at risk for short-term mortality is feasible.

Effect of a Real-Time Risk Score on 30-day Readmission Reduction in Singapore

OBJECTIVE

To develop a risk score for the real-time prediction of readmissions for patients using patient specific information captured in electronic medical records (EMR) in Singapore to enable the prospective identification of high-risk patients for enrolment in timely interventions.

METHODS

Machine-learning models were built to estimate the probability of a patient being readmitted within 30 days of discharge. EMR of 25,472 patients discharged from the medicine department at Ng Teng Fong General Hospital between January 2016 and December 2016 were extracted retrospectively for training and internal validation of the models. We developed and implemented a real-time 30-day readmission risk score generation in the EMR system, which enabled the flagging of high-risk patients to care providers in the hospital. Based on the daily high-risk patient list, the various interfaces and flow sheets in the EMR were configured according to the information needs of the various stakeholders such as the inpatient medical, nursing, case management, emergency department, and postdischarge care teams.

RESULTS

Overall, the machine-learning models achieved good performance with area under the receiver operating characteristic ranging from 0.77 to 0.81. The models were used to proactively identify and attend to patients who are at risk of readmission before an actual readmission occurs. This approach successfully reduced the 30-day readmission rate for patients admitted to the medicine department from 11.7% in 2017 to 10.1% in 2019 (p < 0.01) after risk adjustment.

CONCLUSION

Machine-learning models can be deployed in the EMR system to provide real-time forecasts for a more comprehensive outlook in the aspects of decision-making and care provision.