Advancing In-Hospital Clinical Deterioration Prediction Models

Application of machine learning in predicting survival outcomes involving real-world data: a scoping review

BACKGROUND

Despite the interest in machine learning (ML) algorithms for analyzing real-world data (RWD) in healthcare, the use of ML in predicting time-to-event data, a common scenario in clinical practice, is less explored. ML models are capable of algorithmically learning from large, complex datasets and can offer advantages in predicting time-to-event data. We reviewed the recent applications of ML for survival analysis using RWD in healthcare.

METHODS

PUBMED and EMBASE were searched from database inception through March 2023 to identify peer-reviewed English-language studies of ML models for predicting time-to-event outcomes using the RWD. Two reviewers extracted information on the data source, patient population, survival outcome, ML algorithms, and the Area Under the Curve (AUC).

RESULTS

Of 257 citations, 28 publications were included. Random survival forests (N = 16, 57%) and neural networks (N = 11, 39%) were the most popular ML algorithms. There was variability across AUC for these ML models (median 0.789, range 0.6-0.950). ML algorithms were predominately considered for predicting overall survival in oncology (N = 12, 43%). ML survival models were often used to predict disease prognosis or clinical events (N = 27, 96%) in the oncology, while less were used for treatment outcomes (N = 1, 4%).

CONCLUSIONS

The ML algorithms, random survival forests and neural networks, are mainly used for RWD to predict survival outcomes such as disease prognosis or clinical events in the oncology. This review shows that more opportunities remain to apply these ML algorithms to inform treatment decision-making in clinical practice. More methodological work is also needed to ensure the utility and applicability of ML models in survival outcomes.

ECMO PAL: using deep neural networks for survival prediction in venoarterial extracorporeal membrane oxygenation

PURPOSE

Venoarterial extracorporeal membrane oxygenation (VA-ECMO) is a complex and high-risk life support modality used in severe cardiorespiratory failure. ECMO survival scores are used clinically for patient prognostication and outcomes risk adjustment. This study aims to create the first artificial intelligence (AI)-driven ECMO survival score to predict in-hospital mortality based on a large international patient cohort.

METHODS

A deep neural network, ECMO Predictive Algorithm (ECMO PAL) was trained on a retrospective cohort of 18,167 patients from the international Extracorporeal Life Support Organisation (ELSO) registry (2017-2020), and performance was measured using fivefold cross-validation. External validation was performed on all adult registry patients from 2021 (N = 5015) and compared against existing prognostication scores: SAVE, Modified SAVE, and ECMO ACCEPTS for predicting in-hospital mortality.

RESULTS

Mean age was 56.8 ± 15.1 years, with 66.7% of patients being male and 50.2% having a pre-ECMO cardiac arrest. Cross-validation demonstrated an inhospital mortality sensitivity and precision of 82.1 ± 0.2% and 77.6 ± 0.2%, respectively. Validation accuracy was only 2.8% lower than training accuracy, reducing from 75.5% to 72.7% [99% confidence interval (CI) 71.1-74.3%]. ECMO PAL accuracy outperformed the ECMO ACCEPTS (54.7%), SAVE (61.1%), and Modified SAVE (62%) scores.

CONCLUSIONS

ECMO PAL is the first AI-powered ECMO survival score trained and validated on large international patient cohorts. ECMO PAL demonstrated high generalisability across ECMO regions and outperformed existing, widely used scores. Beyond ECMO, this study highlights how large international registry data can be leveraged for AI prognostication for complex critical care therapies.

Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature review

BACKGROUND AND OBJECTIVE

Early identification of patients at risk of deterioration can prevent life-threatening adverse events and shorten length of stay. Although there are numerous models applied to predict patient clinical deterioration, most are based on vital signs and have methodological shortcomings that are not able to provide accurate estimates of deterioration risk. The aim of this systematic review is to examine the effectiveness, challenges, and limitations of using machine learning (ML) techniques to predict patient clinical deterioration in hospital settings.

METHODS

A systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and meta-Analyses (PRISMA) guidelines using EMBASE, MEDLINE Complete, CINAHL Complete, and IEEExplore databases. Citation searching was carried out for studies that met inclusion criteria. Two reviewers used the inclusion/exclusion criteria to independently screen studies and extract data. To address any discrepancies in the screening process, the two reviewers discussed their findings and a third reviewer was consulted as needed to reach a consensus. Studies focusing on use of ML in predicting patient clinical deterioration that were published from inception to July 2022 were included.

RESULTS

A total of 29 primary studies that evaluated ML models to predict patient clinical deterioration were identified. After reviewing these studies, we found that 15 types of ML techniques have been employed to predict patient clinical deterioration. While six studies used a single technique exclusively, several others utilised a combination of classical techniques, unsupervised and supervised learning, as well as other novel techniques. Depending on which ML model was applied and the type of input features, ML models predicted outcomes with an area under the curve from 0.55 to 0.99.

CONCLUSIONS

Numerous ML methods have been employed to automate the identification of patient deterioration. Despite these advancements, there is still a need for further investigation to examine the application and effectiveness of these methods in real-world situations.

Nursing science as a federally-recognized STEM degree: A call to action for the United States with global implications

Nursing science contributes to advancements in patient care, public health, and innovation within numerous scientific domains. Despite commonality with United States Department of Education definitions of a science, technology, engineering, and mathematics (STEM) educational programs, nursing continues to be excluded from Department of Homeland Security STEM classification. This exclusion prevents societal recognition of nursing as a science and limits attraction of clinicians and nurse scientists born outside of the United States due to omission from various federal visa provisions the Department of Homeland Security classification provides. We evaluated existing Department of Homeland Security STEM-classified educational programs and identified methodological and content congruency among STEM-classified programs and nursing. We provide clear evidence that nursing contributes impactful STEM research; and argue that inclusion is critical for advancement of the profession and the potential to mitigate the faculty shortage. Beyond evaluation of nursing as a STEM field, we offer a policy-focused solution for development and diversification of the nursing workforce.

Informatics and data science perspective on Future of Nursing 2020-2030: Charting a pathway to health equity

The Future of Nursing 2020 to 2030 report explicitly addresses the need for integration of nursing expertise in designing, generating, analyzing, and applying data to support initiatives focused on social determinants of health (SDOH) and health equity. The metrics necessary to enable and evaluate progress on all recommendations require harnessing existing data sources and developing new ones, as well as transforming and integrating data into information systems to facilitate communication, information sharing, and decision making among the key stakeholders. We examine the recommendations of the 2021 report through an interdisciplinary lens that integrates nursing, biomedical informatics, and data science by addressing three critical questions: (a) what data are needed?, (b) what infrastructure and processes are needed to transform data into information?, and (c) what information systems are needed to "level up" nurse-led interventions from the micro-level to the meso- and macro-levels to address social determinants of health and advance health equity?

Early Prediction of All-Cause Clinical Deterioration in General Wards Patients: Development and Validation of a Biomarker-Based Machine Learning Model Derived From Rapid Response Team Activations

OBJECTIVE

The aim of the study is to evaluate the performance of a biomarker-based machine learning (ML) model (not including vital signs) derived from reviewed rapid response team (RRT) activations in predicting all-cause deterioration in general wards patients.

DESIGN

This is a retrospective single-institution study. All consecutive adult patients' cases on noncritical wards identified by RRT calls occurring at least 24 hours after patient admission, between April 2018 and June 2020, were included. The cases were reviewed and labeled for clinical deterioration by a multidisciplinary expert consensus panel. A supervised learning approach was adopted based on a set of biomarkers and demographic data available in the patient's electronic medical record (EMR).

SETTING

The setting is a 250-bed tertiary university hospital with a basic EMR, with adult (>18 y) patients on general wards.

PATIENTS

The study analyzed the cases of 514 patients for which the RRT was activated. Rapid response teams were extracted from the hospital telephone log data. Two hundred eighteen clinical deterioration cases were identified in these patients after expert chart review and complemented by 146 "nonevent" cases to build the training and validation data set.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The best performance was achieved with the random forests algorithm, with a maximal area under the receiver operating curve of 0.90 and F1 score of 0.85 obtained at prediction time T0-6h, slightly decreasing but still acceptable (area under the receiver operating curve, >0.8; F1 score, >0.75) at T0-42h. The system outperformed most classical track-and-trigger systems both in terms of prediction performance and prediction horizon.

CONCLUSIONS

In hospitals with a basic EMR, a biomarker-based ML model could be used to predict clinical deterioration in general wards patients earlier than classical track-and-trigger systems, thus enabling appropriate clinical interventions for patient safety and improved outcomes.

Machine learning techniques for mortality prediction in emergency departments: a systematic review

OBJECTIVES

This systematic review aimed to assess the performance and clinical feasibility of machine learning (ML) algorithms in prediction of in-hospital mortality for medical patients using vital signs at emergency departments (EDs).

DESIGN

A systematic review was performed.

SETTING

The databases including Medline (PubMed), Scopus and Embase (Ovid) were searched between 2010 and 2021, to extract published articles in English, describing ML-based models utilising vital sign variables to predict in-hospital mortality for patients admitted at EDs. Critical appraisal and data extraction for systematic reviews of prediction modelling studies checklist was used for study planning and data extraction. The risk of bias for included papers was assessed using the prediction risk of bias assessment tool.

PARTICIPANTS

Admitted patients to the ED.

MAIN OUTCOME MEASURE

In-hospital mortality.

RESULTS

Fifteen articles were included in the final review. We found that eight models including logistic regression, decision tree, K-nearest neighbours, support vector machine, gradient boosting, random forest, artificial neural networks and deep neural networks have been applied in this domain. Most studies failed to report essential main analysis steps such as data preprocessing and handling missing values. Fourteen included studies had a high risk of bias in the statistical analysis part, which could lead to poor performance in practice. Although the main aim of all studies was developing a predictive model for mortality, nine articles did not provide a time horizon for the prediction.

CONCLUSION

This review provided an updated overview of the state-of-the-art and revealed research gaps; based on these, we provide eight recommendations for future studies to make the use of ML more feasible in practice. By following these recommendations, we expect to see more robust ML models applied in the future to help clinicians identify patient deterioration earlier.

Validation of an Automatic Tagging System for Identifying Respiratory and Hemodynamic Deterioration Events in the Intensive Care Unit

Objectives

Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers.

Methods

This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts’ eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system.

Results

The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers.

Conclusions

We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.

Value of laboratory results in addition to vital signs in a machine learning algorithm to predict in-hospital cardiac arrest: A single-center retrospective cohort study

BACKGROUND

Although machine learning-based prediction models for in-hospital cardiac arrest (IHCA) have been widely investigated, it is unknown whether a model based on vital signs alone (Vitals-Only model) can perform similarly to a model that considers both vital signs and laboratory results (Vitals+Labs model).

METHODS

All adult patients hospitalized in a tertiary care hospital in Japan between October 2011 and October 2018 were included in this study. Random forest models with/without laboratory results (Vitals+Labs model and Vitals-Only model, respectively) were trained and tested using chronologically divided datasets. Both models use patient demographics and eight-hourly vital signs collected within the previous 48 hours. The primary and secondary outcomes were the occurrence of IHCA in the next 8 and 24 hours, respectively. The area under the receiver operating characteristic curve (AUC) was used as a comparative measure. Sensitivity analyses were performed under multiple statistical assumptions.

RESULTS

Of 141,111 admitted patients (training data: 83,064, test data: 58,047), 338 had an IHCA (training data: 217, test data: 121) during the study period. The Vitals-Only model and Vitals+Labs model performed comparably when predicting IHCA within the next 8 hours (Vitals-Only model vs Vitals+Labs model, AUC = 0.862 [95% confidence interval (CI): 0.855-0.868] vs 0.872 [95% CI: 0.867-0.878]) and 24 hours (Vitals-Only model vs Vitals+Labs model, AUC = 0.830 [95% CI: 0.825-0.835] vs 0.837 [95% CI: 0.830-0.844]). Both models performed similarly well on medical, surgical, and ward patient data, but did not perform well for intensive care unit patients.

CONCLUSIONS

In this single-center study, the machine learning model predicted IHCAs with good discrimination. The addition of laboratory values to vital signs did not significantly improve its overall performance.

Nurses' judgments of patient risk of deterioration at change-of-shift handoff: Agreement between nurses and comparison with early warning scores

BACKGROUND

Nurses begin forming judgments regarding patients' clinical stability during change-of-shift handoffs.

OBJECTIVES

To examine the agreement between incoming and outgoing nurses' judgments of deterioration risk following handoff and compare these judgments to commonly used early warning scores (MEWS, NEWS, ViEWS).

METHODS

Following handoffs on three medical/surgical units, nurses completed the Patient Acuity Rating. Nurse ratings were compared with computed early warning scores based on clinical data. In follow-up interviews, nurses were invited to describe their experiences of using the rating scale.

RESULTS

Sixty-two nurses carried out 444 handoffs for 158 patients. While the agreement between incoming and outgoing nurses was fair, correlations with early warning scores were low. Nurses struggled with predicting risk and used their impressions of differential risk across all the patients to whom they had been assigned to arrive at their ratings.

CONCLUSION

Nurses shared information that influenced their clinical judgments at handoff; not all of these cues may necessarily be captured in early warning scores.