Prediction of early unplanned intensive care unit readmission in a UK tertiary care hospital: a cross-sectional machine learning approach

Predicting intubation for intensive care units patients: A deep learning approach to improve patient management

OBJECTIVE

For patients in the Intensive Care Unit (ICU), the timing of intubation has a significant association with patients' outcomes. However, accurate prediction of the timing of intubation remains an unsolved challenge due to the noisy, sparse, heterogeneous, and unbalanced nature of ICU data. In this study, our objective is to develop a workflow for pre-processing ICU data and to develop a customized deep learning model to predict the need for intubation.

METHODS

To improve the prediction accuracy, we transform the intubation prediction task into a time series classification task. We carefully design a sequence of data pre-processing steps to handle the multimodal noisy data. Firstly, we discretize the sequential data and address missing data using interpolation. Next, we employ a sampling strategy to address data imbalance and standardize the data to facilitate faster model convergence. Furthermore, we employ the feature selection technique and propose an ensemble model to combine features learned by different deep learning models.

RESULTS

The performance is evaluated on Medical Information Mart for Intensive Care (MIMIC)-III, an ICU dataset. Our proposed Deep Feature Fusion method achieves an area under the curve (AUC) of the receiver operating curve (ROC) of 0.8953, surpassing the performance of other deep learning and traditional machine learning models.

CONCLUSION

Our proposed Deep Feature Fusion method proves to be a viable approach for predicting intubation and outperforms other deep learning and classical machine learning models. The study confirms that high-frequency time-varying indicators, particularly Mean Blood Pressure (MeanBP) and peripheral oxygen saturation (SpO2), are significant risk factors for predicting intubation.

Predictors of in-ICU length of stay among congenital heart defect patients using artificial intelligence model: A pilot study

Objective

This study aims to develop a predictive model using artificial intelligence to estimate the ICU length of stay (LOS) for Congenital Heart Defects (CHD) patients after surgery, improving care planning and resource management.

Design

We analyze clinical data from 2240 CHD surgery patients to create and validate the predictive model. Twenty AI models are developed and evaluated for accuracy and reliability.

Setting

The study is conducted in a Brazilian hospital's Cardiovascular Surgery Department, focusing on transplants and cardiopulmonary surgeries.

Participants

Retrospective analysis is conducted on data from 2240 consecutive CHD patients undergoing surgery.

Interventions

Ninety-three pre and intraoperative variables are used as ICU LOS predictors.

Measurements and main results

Utilizing regression and clustering methodologies for ICU LOS (ICU Length of Stay) estimation, the Light Gradient Boosting Machine, using regression, achieved a Mean Squared Error (MSE) of 15.4, 11.8, and 15.2 days for training, testing, and unseen data. Key predictors included metrics such as "Mechanical Ventilation Duration", "Weight on Surgery Date", and "Vasoactive-Inotropic Score". Meanwhile, the clustering model, Cat Boost Classifier, attained an accuracy of 0.6917 and AUC of 0.8559 with similar key predictors.

Conclusions

Patients with higher ventilation times, vasoactive-inotropic scores, anoxia time, cardiopulmonary bypass time, and lower weight, height, BMI, age, hematocrit, and presurgical oxygen saturation have longer ICU stays, aligning with existing literature.

A comparative study of explainable ensemble learning and logistic regression for predicting in-hospital mortality in the emergency department

This study addresses the challenges associated with emergency department (ED) overcrowding and emphasizes the need for efficient risk stratification tools to identify high-risk patients for early intervention. While several scoring systems, often based on logistic regression (LR) models, have been proposed to indicate patient illness severity, this study aims to compare the predictive performance of ensemble learning (EL) models with LR for in-hospital mortality in the ED. A cross-sectional single-center study was conducted at the ED of Imam Reza Hospital in northeast Iran from March 2016 to March 2017. The study included adult patients with one to three levels of emergency severity index. EL models using Bagging, AdaBoost, random forests (RF), Stacking and extreme gradient boosting (XGB) algorithms, along with an LR model, were constructed. The training and validation visits from the ED were randomly divided into 80% and 20%, respectively. After training the proposed models using tenfold cross-validation, their predictive performance was evaluated. Model performance was compared using the Brier score (BS), The area under the receiver operating characteristics curve (AUROC), The area and precision-recall curve (AUCPR), Hosmer-Lemeshow (H-L) goodness-of-fit test, precision, sensitivity, accuracy, F1-score, and Matthews correlation coefficient (MCC). The study included 2025 unique patients admitted to the hospital's ED, with a total percentage of hospital deaths at approximately 19%. In the training group and the validation group, 274 of 1476 (18.6%) and 152 of 728 (20.8%) patients died during hospitalization, respectively. According to the evaluation of the presented framework, EL models, particularly Bagging, predicted in-hospital mortality with the highest AUROC (0.839, CI (0.802-0.875)) and AUCPR = 0.64 comparable in terms of discrimination power with LR (AUROC (0.826, CI (0.787-0.864)) and AUCPR = 0.61). XGB achieved the highest precision (0.83), sensitivity (0.831), accuracy (0.842), F1-score (0.833), and the highest MCC (0.48). Additionally, the most accurate models in the unbalanced dataset belonged to RF with the lowest BS (0.128). Although all studied models overestimate mortality risk and have insufficient calibration (P > 0.05), stacking demonstrated relatively good agreement between predicted and actual mortality. EL models are not superior to LR in predicting in-hospital mortality in the ED. Both EL and LR models can be considered as screening tools to identify patients at risk of mortality.

External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems

OBJECTIVES

Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population.

DESIGN

PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs.

SETTING

The general wards of two large hospitals, one an academic medical center and the other a community hospital.

SUBJECTS

The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE's performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861-0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275-0.320) at the first hospital; AUROC 0.875 (0.851-0.902), AUPRC 0.339 (0.281-0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions.

CONCLUSIONS

Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.

Assessing the effects of therapeutic combinations on SARS-CoV-2 infected patient outcomes: A big data approach

BACKGROUND

The COVID-19 pandemic has demonstrated the need for efficient and comprehensive, simultaneous assessment of multiple combined novel therapies for viral infection across the range of illness severity. Randomized Controlled Trials (RCT) are the gold standard by which efficacy of therapeutic agents is demonstrated. However, they rarely are designed to assess treatment combinations across all relevant subgroups. A big data approach to analyzing real-world impacts of therapies may confirm or supplement RCT evidence to further assess effectiveness of therapeutic options for rapidly evolving diseases such as COVID-19.

METHODS

Gradient Boosted Decision Tree, Deep and Convolutional Neural Network classifiers were implemented and trained on the National COVID Cohort Collaborative (N3C) data repository to predict the patients' outcome of death or discharge. Models leveraged the patients' characteristics, the severity of COVID-19 at diagnosis, and the calculated proportion of days on different treatment combinations after diagnosis as features to predict the outcome. Then, the most accurate model is utilized by eXplainable Artificial Intelligence (XAI) algorithms to provide insights about the learned treatment combination impacts on the model's final outcome prediction.

RESULTS

Gradient Boosted Decision Tree classifiers present the highest prediction accuracy in identifying patient outcomes with area under the receiver operator characteristic curve of 0.90 and accuracy of 0.81 for the outcomes of death or sufficient improvement to be discharged. The resulting model predicts the treatment combinations of anticoagulants and steroids are associated with the highest probability of improvement, followed by combined anticoagulants and targeted antivirals. In contrast, monotherapies of single drugs, including use of anticoagulants without steroid or antivirals are associated with poorer outcomes.

CONCLUSIONS

This machine learning model by accurately predicting the mortality provides insights about the treatment combinations associated with clinical improvement in COVID-19 patients. Analysis of the model's components suggests benefit to treatment with combination of steroids, antivirals, and anticoagulant medication. The approach also provides a framework for simultaneously evaluating multiple real-world therapeutic combinations in future research studies.

The risk assessment tool for intensive care unit readmission: A systematic review and meta-analysis

OBJECTIVE

To review and evaluate existing risk assessment tools for intensive care unitreadmission.

METHODS

Nine electronic databases (Medline, CINAHL, Web of Science, Cochrane Library, Embase, Sino Med, CNKI, VIP, and Wan fang) were systematically searched from their inception to September 2022. Two authors independently extracted data from the literature included. Meta-analysis was performed under the bivariate modeling and summary receiver operating characteristic curve method.

RESULTS

A total of 29 studies were included in this review, among which 11 were quantitatively Meta-analyzed. The results showed Stability and Workload Index for Transfer: Sensitivity = 0.55, Specificity = 0.65, Area under curve = 0.63. And Early warning score: Sensitivity = 0.78, Specificity = 0.83, Area under curve = 0.88. The remaining tools included scores, nomograms, machine learning models, and deep learning models. These studies, with varying reports on thresholds, case selection, data preprocessing, and model performance, have a high risk of bias.

CONCLUSION

We cannot identify a tool that can be used directly in intensive care unit readmission risk assessment. Scores based on early warning score are moderately accurate in predicting readmission, but there is heterogeneity and publication bias that requires model adjustment for local factors such as resources, demographics, and case mix. Machine learning models present a promising modeling technique but have a high methodological bias and require further validation.

IMPLICATIONS FOR CLINICAL PRACTICE

Using reliable risk assessment tools is essential for the early identification of unplanned intensive care unit readmission risk in critically ill patients.  A reliable risk assessment tool must be developed, which is the focus of further research.

Iterated cross validation method for prediction of survival in diffuse large B-cell lymphoma for small size dataset

Efforts have been made to improve the risk stratification model for patients with diffuse large B-cell lymphoma (DLBCL). This study aimed to evaluate the disease prognosis using machine learning models with iterated cross validation (CV) method. A total of 122 patients with pathologically confirmed DLBCL and receiving rituximab-containing chemotherapy were enrolled. Contributions of clinical, laboratory, and metabolic imaging parameters from fluorine-18 fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) scans to the prognosis were evaluated using five regression models, namely logistic regression, random forest, support vector classifier (SVC), deep neural network (DNN), and fuzzy neural network models. Binary classification predictions for 3-year progression free survival (PFS) and 3-year overall survival (OS) were conducted. The 10-iterated fivefold CV with shuffling process was conducted to predict the capability of learning machines. The median PFS and OS were 41.0 and 43.6 months, respectively. Two indicators were found to be independent predictors for prognosis: international prognostic index and total metabolic tumor volume (MTVsum) from FDG PET/CT. For PFS, SVC and DNN (both with accuracy 71%) have the best predictive results, of which outperformed other algorithms. For OS, the DNN has the best predictive result (accuracy 76%). Using clinical and metabolic parameters as input variables, the machine learning methods with iterated CV method add the predictive values for PFS and OS evaluation in DLBCL patients.

Predicting readmission to the cardiovascular intensive care unit using recurrent neural networks

If a patient can be discharged from an intensive care unit (ICU) is usually decided by the treating physicians based on their clinical experience. However, nowadays limited capacities and growing socioeconomic burden of our health systems increase the pressure to discharge patients as early as possible, which may lead to higher readmission rates and potentially fatal consequences for the patients. Therefore, here we present a long short-term memory-based deep learning model (LSTM) trained on time series data from Medical Information Mart for Intensive Care (MIMIC-III) dataset to assist physicians in making decisions if patients can be safely discharged from cardiovascular ICUs. To underline the strengths of our LSTM we compare its performance with a logistic regression model, a random forest, extra trees, a feedforward neural network and with an already known, more complex LSTM as well as an LSTM combined with a convolutional neural network. The results of our evaluation show that our LSTM outperforms most of the above models in terms of area under receiver operating characteristic curve. Moreover, our LSTM shows the best performance with respect to the area under precision-recall curve. The deep learning solution presented in this article can help physicians decide on patient discharge from the ICU. This may not only help to increase the quality of patient care, but may also help to reduce costs and to optimize ICU resources. Further, the presented LSTM-based approach may help to improve existing and develop new medical machine learning prediction models.

Predictive Modeling for Readmission to Intensive Care: A Systematic Review

To evaluate the methodologic rigor and predictive performance of models predicting ICU readmission; to understand the characteristics of ideal prediction models; and to elucidate relationships between appropriate triage decisions and patient outcomes.

DATA SOURCES

PubMed, Web of Science, Cochrane, and Embase.

STUDY SELECTION

Primary literature that reported the development or validation of ICU readmission prediction models within from 2010 to 2021.

DATA EXTRACTION

Relevant study information was extracted independently by two authors using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies checklist. Bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool. Data sources, modeling methodology, definition of outcomes, performance, and risk of bias were critically evaluated to elucidate relevant relationships.

DATA SYNTHESIS

Thirty-three articles describing models were included. Six studies had a high overall risk of bias due to improper inclusion criteria or omission of critical analysis details. Four other studies had an unclear overall risk of bias due to lack of detail describing the analysis. Overall, the most common (50% of studies) source of bias was the filtering of candidate predictors via univariate analysis. The poorest performing models used existing clinical risk or acuity scores such as Acute Physiologic Assessment and Chronic Health Evaluation II, Sequential Organ Failure Assessment, or Stability and Workload Index for Transfer as the sole predictor. The higher-performing ICU readmission prediction models used homogenous patient populations, specifically defined outcomes, and routinely collected predictors that were analyzed over time.

CONCLUSIONS

Models predicting ICU readmission can achieve performance advantages by using longitudinal time series modeling, homogenous patient populations, and predictor variables tailored to those populations.

Identifying and evaluating barriers for the implementation of machine learning in the intensive care unit

BACKGROUND

Despite apparent promise and the availability of numerous examples in the literature, machine learning models are rarely used in practice in ICU units. This mismatch suggests that there are poorly understood barriers preventing uptake, which we aim to identify.

METHODS

We begin with a qualitative study with 29 interviews of 40 Intensive Care Unit-, hospital- and MedTech company staff members. As a follow-up to the study, we attempt to quantify some of the technical issues raised. To perform experiments we selected two models based on criteria such as medical relevance. Using these models we measure the loss of performance in predictive models due to drift over time, change of available patient features, scarceness of data, and deploying a model in a different context to the one it was built in.

RESULTS

The qualitative study confirms our assumptions on the potential of AI-driven analytics for patient care, as well as showing the prevalence and type of technical blocking factors that are responsible for its slow uptake. The experiments confirm that each of these issues can cause important loss of predictive model performance, depending on the model and the issue.

CONCLUSIONS

Based on the qualitative study and quantitative experiments we conclude that more research on practical solutions to enable AI-driven innovation in Intensive Care Units is needed. Furthermore, the general poor situation with respect to public, usable implementations of predictive models would appear to limit the possibilities for both the scientific repeatability of the underlying research and the transfer of this research into practice.

Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines

Background

Intensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Widely used post hoc explanation methods also have inherent limitations. Few studies are evaluating inherently interpretable ML models for health care and involve clinicians in inspecting the trained model.

Methods

An inherently interpretable model for the prediction of 3 day ICU readmission was developed. We used explainable boosting machines that learn modular risk functions and which have already been shown to be suitable for the health care domain. We created a retrospective cohort of 15,589 ICU stays and 169 variables collected between 2006 and 2019 from the University Hospital Münster. A team of physicians inspected the model, checked the plausibility of each risk function, and removed problematic ones. We collected qualitative feedback during this process and analyzed the reasons for removing risk functions. The performance of the final explainable boosting machine was compared with a validated clinical score and three commonly used ML models. External validation was performed on the widely used Medical Information Mart for Intensive Care version IV database.

Results

The developed explainable boosting machine used 67 features and showed an area under the precision-recall curve of 0.119 ± 0.020 and an area under the receiver operating characteristic curve of 0.680 ± 0.025. It performed on par with state-of-the-art gradient boosting machines (0.123 ± 0.016, 0.665 ± 0.036) and outperformed the Simplified Acute Physiology Score II (0.084 ± 0.025, 0.607 ± 0.019), logistic regression (0.092 ± 0.026, 0.587 ± 0.016), and recurrent neural networks (0.095 ± 0.008, 0.594 ± 0.027). External validation confirmed that explainable boosting machines (0.221 ± 0.023, 0.760 ± 0.010) performed similarly to gradient boosting machines (0.232 ± 0.029, 0.772 ± 0.018). Evaluation of the model inspection showed that explainable boosting machines can be useful to detect and remove problematic risk functions.

Conclusions

We developed an inherently interpretable ML model for 3 day ICU readmission prediction that reached the state-of-the-art performance of black box models. Our results suggest that for low- to medium-dimensional datasets that are common in health care, it is feasible to develop ML models that allow a high level of human control without sacrificing performance.

Analyzing adverse drug reaction using statistical and machine learning methods: A systematic review

BACKGROUND

Adverse drug reactions (ADRs) are unintended negative drug-induced responses. Determining the association between drugs and ADRs is crucial, and several methods have been proposed to demonstrate this association. This systematic review aimed to examine the analytical tools by considering original articles that utilized statistical and machine learning methods for detecting ADRs.

METHODS

A systematic literature review was conducted based on articles published between 2015 and 2020. The keywords used were statistical, machine learning, and deep learning methods for detecting ADR signals. The study was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) guidelines.

RESULTS

We reviewed 72 articles, of which 51 and 21 addressed statistical and machine learning methods, respectively. Electronic medical record (EMR) data were exclusively analyzed using the regression method. For FDA Adverse Event Reporting System (FAERS) data, components of the disproportionality method were preferable. DrugBank was the most used database for machine learning. Other methods accounted for the highest and supervised methods accounted for the second highest.

CONCLUSIONS

Using the 72 main articles, this review provides guidelines on which databases are frequently utilized and which analysis methods can be connected. For statistical analysis, >90% of the cases were analyzed by disproportionate or regression analysis with each spontaneous reporting system (SRS) data or electronic medical record (EMR) data; for machine learning research, however, there was a strong tendency to analyze various data combinations. Only half of the DrugBank database was occupied, and the k-nearest neighbor method accounted for the greatest proportion.

Comparison of Four Machine Learning Techniques for Prediction of Intensive Care Unit Length of Stay in Heart Transplantation Patients

Background

Post-operative heart transplantation patients often require admission to an intensive care unit (ICU). Early prediction of the ICU length of stay (ICU-LOS) of these patients is of great significance and can guide treatment while reducing the mortality rate among patients. However, conventional linear models have tended to perform worse than non-linear models.

Materials and Methods

We collected the clinical data of 365 patients from Wuhan Union Hospital who underwent heart transplantation surgery between April 2017 and August 2020. The patients were randomly divided into training data (N = 256) and test data (N = 109) groups. 84 clinical features were collected for each patient. Features were validated using the Least Absolute Shrinkage and Selection Operator (LASSO) regression’s fivefold cross-validation method. We obtained Shapley Additive explanations (SHAP) values by executing package “shap” to interpret model predictions. Four machine learning models and logistic regression algorithms were developed. The area under the receiver operating characteristic curve (AUC-ROC) was used to compare the prediction performance of different models. Finally, for the convenience of clinicians, an online web-server was established and can be freely accessed via the website https://wuhanunion.shinyapps.io/PredictICUStay/.

Results

In this study, 365 consecutive patients undergoing heart transplantation surgery for moderate (NYHA grade 3) or severe (NYHA grade 4) heart failure were collected in Wuhan Union Hospital from 2017 to 2020. The median age of the recipient patients was 47.2 years, while the median age of the donors was 35.58 years. 330 (90.4%) of the donor patients were men, and the average surgery duration was 260.06 min. Among this cohort, 47 (12.9%) had renal complications, 25 (6.8%) had hepatic complications, 11 (3%) had undergone chest re-exploration and 19 (5.2%) had undergone extracorporeal membrane oxygenation (ECMO). The following six important clinical features were selected using LASSO regression, and according to the result of SHAP, the rank of importance was (1) the use of extracorporeal membrane oxygenation (ECMO); (2) donor age; (3) the use of an intra-aortic balloon pump (IABP); (4) length of surgery; (5) high creatinine (Cr); and (6) the use of continuous renal replacement therapy (CRRT). The eXtreme Gradient Boosting (XGBoost) algorithm presented significantly better predictive performance (AUC-ROC = 0.88) than other models [Accuracy: 0.87; sensitivity: 0.98; specificity: 0.51; positive predictive value (PPV): 0.86; negative predictive value (NPV): 0.93].

Conclusion

Using the XGBoost classifier with heart transplantation patients can provide an accurate prediction of ICU-LOS, which will not only improve the accuracy of clinical decision-making but also contribute to the allocation and management of medical resources; it is also a real-world example of precision medicine in hospitals.

A Machine Learning Based Discharge Prediction of Cardiovascular Diseases Patients in Intensive Care Units

This paper targets a major challenge of how to effectively allocate medical resources in intensive care units (ICUs). We trained multiple regression models using the Medical Information Mart for Intensive Care III (MIMIC III) database recorded in the period between 2001 and 2012. The training and validation dataset included pneumonia, sepsis, congestive heart failure, hypotension, chest pain, coronary artery disease, fever, respiratory failure, acute coronary syndrome, shortness of breath, seizure and transient ischemic attack, and aortic stenosis patients’ recorded data. Then we tested the models on the unseen data of patients diagnosed with coronary artery disease, congestive heart failure or acute coronary syndrome. We included the admission characteristics, clinical prescriptions, physiological measurements, and discharge characteristics of those patients. We assessed the models’ performance using mean residuals and running times as metrics. We ran multiple experiments to study the data partition’s impact on the learning phase. The total running time of our best-evaluated model is 123,450.9 mS. The best model gives an average accuracy of 98%, highlighting the location of discharge, initial diagnosis, location of admission, drug therapy, length of stay and internal transfers as the most influencing patterns to decide a patient’s readiness for discharge.

Prediction of unplanned 30-day readmission for ICU patients with heart failure

BACKGROUND

Intensive Care Unit (ICU) readmissions in patients with heart failure (HF) result in a significant risk of death and financial burden for patients and healthcare systems. Prediction of at-risk patients for readmission allows for targeted interventions that reduce morbidity and mortality.

METHODS AND RESULTS

We presented a process mining/deep learning approach for the prediction of unplanned 30-day readmission of ICU patients with HF. A patient's health records can be understood as a sequence of observations called event logs; used to discover a process model. Time information was extracted using the DREAM (Decay Replay Mining) algorithm. Demographic information and severity scores upon admission were then combined with the time information and fed to a neural network (NN) model to further enhance the prediction efficiency. Additionally, several machine learning (ML) algorithms were developed to be used as the baseline models for the comparison of the results.

RESULTS

By using the Medical Information Mart for Intensive Care III (MIMIC-III) dataset of 3411 ICU patients with HF, our proposed model yielded an area under the receiver operating characteristics (AUROC) of 0.930, 95% confidence interval of [0.898-0.960], the precision of 0.886, sensitivity of 0.805, accuracy of 0.841, and F-score of 0.800 which were far better than the results of the best baseline model and the existing literature.

CONCLUSIONS

The proposed approach was capable of modeling the time-related variables and incorporating the medical history of patients from prior hospital visits for prediction. Thus, our approach significantly improved the outcome prediction compared to that of other ML-based models and health calculators.

Domain Adaptation Using Convolutional Autoencoder and Gradient Boosting for Adverse Events Prediction in the Intensive Care Unit

More than 5 million patients have admitted annually to intensive care units (ICUs) in the United States. The leading causes of mortality are cardiovascular failures, multi-organ failures, and sepsis. Data-driven techniques have been used in the analysis of patient data to predict adverse events, such as ICU mortality and ICU readmission. These models often make use of temporal or static features from a single ICU database to make predictions on subsequent adverse events. To explore the potential of domain adaptation, we propose a method of data analysis using gradient boosting and convolutional autoencoder (CAE) to predict significant adverse events in the ICU, such as ICU mortality and ICU readmission. We demonstrate our results from a retrospective data analysis using patient records from a publicly available database called Multi-parameter Intelligent Monitoring in Intensive Care-II (MIMIC-II) and a local database from Children's Healthcare of Atlanta (CHOA). We demonstrate that after adopting novel data imputation on patient ICU data, gradient boosting is effective in both the mortality prediction task and the ICU readmission prediction task. In addition, we use gradient boosting to identify top-ranking temporal and non-temporal features in both prediction tasks. We discuss the relationship between these features and the specific prediction task. Lastly, we indicate that CAE might not be effective in feature extraction on one dataset, but domain adaptation with CAE feature extraction across two datasets shows promising results.

The Potential Cost-Effectiveness of a Machine Learning Tool That Can Prevent Untimely Intensive Care Unit Discharge

OBJECTIVES

The machine learning prediction model Pacmed Critical (PC), currently under development, may guide intensivists in their decision-making process on the most appropriate time to discharge a patient from the intensive care unit (ICU). Given the financial pressure on healthcare budgets, this study assessed whether PC has the potential to be cost-effective compared with standard care, without the use of PC, for Dutch patients in the ICU from a societal perspective.

METHODS

A 1-year, 7-state Markov model reflecting the ICU care pathway and incorporating the PC decision tool was developed. A hypothetical cohort of 1000 adult Dutch patients admitted in the ICU was entered in the model. We used the literature, expert opinion, and data from Amsterdam University Medical Center for model parameters. The uncertainty surrounding the incremental cost-effectiveness ratio was assessed using deterministic and probabilistic sensitivity analyses and scenario analyses.

RESULTS

PC was a cost-effective strategy with an incremental cost-effectiveness ratio of €18 507 per quality-adjusted life-year. PC remained cost-effective over standard care in multiple scenarios and sensitivity analyses. The likelihood that PC will be cost-effective was 71% at a willingness-to-pay threshold of €30 000 per quality-adjusted life-year. The key driver of the results was the parameter "reduction in ICU length of stay."

CONCLUSIONS

We showed that PC has the potential to be cost-effective for Dutch ICUs in a time horizon of 1 year. This study is one of the first cost-effectiveness analyses of a machine learning device. Further research is needed to validate the effectiveness of PC, thereby focusing on the key parameter "reduction in ICU length of stay" and potential spill-over effects.

Electronic health record machine learning model predicts trauma inpatient mortality in real time: A validation study

INTRODUCTION

Patient outcome prediction models are underused in clinical practice because of lack of integration with real-time patient data. The electronic health record (EHR) has the ability to use machine learning (ML) to develop predictive models. While an EHR ML model has been developed to predict clinical deterioration, it has yet to be validated for use in trauma. We hypothesized that the Epic Deterioration Index (EDI) would predict mortality and unplanned intensive care unit (ICU) admission in trauma patients.

METHODS

A retrospective analysis of a trauma registry was used to identify patients admitted to a level 1 trauma center for >24 hours from October 2019 to July 2020. We evaluated the performance of the EDI, which is constructed from 125 objective patient measures within the EHR, in predicting mortality and unplanned ICU admissions. We performed a 5 to 1 match on age because it is a major component of EDI, then examined the area under the receiver operating characteristic curve (AUROC), and benchmarked it against Injury Severity Score (ISS) and new injury severity score (NISS).

RESULTS

The study cohort consisted of 1,325 patients admitted with a mean age of 52.5 years and 91% following blunt injury. The in-hospital mortality rate was 2%, and unplanned ICU admission rate was 2.6%. In predicting mortality, the maximum EDI within 24 hours of admission had an AUROC of 0.98 compared with 0.89 of ISS and 0.91 of NISS. For unplanned ICU admission, the EDI slope within 24 hours of ICU admission had a modest performance with an AUROC of 0.66.

CONCLUSION

Epic Deterioration Index appears to perform strongly in predicting in-patient mortality similarly to ISS and NISS. In addition, it can be used to predict unplanned ICU admissions. This study helps validate the use of this real-time EHR ML-based tool, suggesting that EDI should be incorporated into the daily care of trauma patients.

LEVEL OF EVIDENCE

Prognostic, level III.

Gender-sensitive word embeddings for healthcare

OBJECTIVE

To analyze gender bias in clinical trials, to design an algorithm that mitigates the effects of biases of gender representation on natural-language (NLP) systems trained on text drawn from clinical trials, and to evaluate its performance.

MATERIALS AND METHODS

We analyze gender bias in clinical trials described by 16 772 PubMed abstracts (2008-2018). We present a method to augment word embeddings, the core building block of NLP-centric representations, by weighting abstracts by the number of women participants in the trial. We evaluate the resulting gender-sensitive embeddings performance on several clinical prediction tasks: comorbidity classification, hospital length of stay prediction, and intensive care unit (ICU) readmission prediction.

RESULTS

For female patients, the gender-sensitive model area under the receiver-operator characteristic (AUROC) is 0.86 versus the baseline of 0.81 for comorbidity classification, mean absolute error 4.59 versus the baseline of 4.66 for length of stay prediction, and AUROC 0.69 versus 0.67 for ICU readmission. All results are statistically significant.

DISCUSSION

Women have been underrepresented in clinical trials. Thus, using the broad clinical trials literature as training data for statistical language models could result in biased models, with deficits in knowledge about women. The method presented enables gender-sensitive use of publications as training data for word embeddings. In experiments, the gender-sensitive embeddings show better performance than baseline embeddings for the clinical tasks studied. The results highlight opportunities for recognizing and addressing gender and other representational biases in the clinical trials literature.

CONCLUSION

Addressing representational biases in data for training NLP embeddings can lead to better results on downstream tasks for underrepresented populations.

Application of machine learning to predict the outcome of pediatric traumatic brain injury

PURPOSE

Traumatic brain injury (TBI) generally causes mortality and disability, particularly in children. Machine learning (ML) is a computer algorithm, applied as a clinical prediction tool. The present study aims to assess the predictability of ML for the functional outcomes of pediatric TBI.

METHODS

A retrospective cohort study was performed targeting children with TBI who were admitted to the trauma center of southern Thailand between January 2009 and July 2020. The patient was excluded if he/she (1) did not undergo a CT scan of the brain, (2) died within the first 24 h, (3) had unavailable complete medical records during admission, or (4) was unable to provide updated outcomes. Clinical and radiologic characteristics were collected such as vital signs, Glasgow coma scale score, and characteristics of intracranial injuries. The functional outcome was assessed using the King's Outcome Scale for Childhood Head Injury, which was thus dichotomized into favourable outcomes and unfavourable outcomes: good recovery and moderate disability were categorized as the former, whereas death, vegetative state, and severe disability were categorized as the latter. The prognostic factors were estimated using traditional binary logistic regression. By data splitting, 70% of data were used for training the ML models and the remaining 30% were used for testing the ML models. The supervised algorithms including support vector machines, neural networks, random forest, logistic regression, naive Bayes and k-nearest neighbor were performed for training of the ML models. Therefore, the ML models were tested for the predictive performances by the testing datasets.

RESULTS

There were 828 patients in the cohort. The median age was 72 months (interquartile range 104.7 months, range 2-179 months). Road traffic accident was the most common mechanism of injury, accounting for 68.7%. At hospital discharge, favourable outcomes were achieved in 97.0% of patients, while the mortality rate was 2.2%. Glasgow coma scale score, hypotension, pupillary light reflex, and subarachnoid haemorrhage were associated with TBI outcomes following traditional binary logistic regression; hence, the 4 prognostic factors were used for building ML models and testing performance. The support vector machine model had the best performance for predicting pediatric TBI outcomes: sensitivity 0.95, specificity 0.60, positive predicted value 0.99, negative predictive value 1.0; accuracy 0.94, and area under the receiver operating characteristic curve 0.78.

CONCLUSION

The ML algorithms of the present study have a high sensitivity; therefore they have the potential to be screening tools for predicting functional outcomes and counselling prognosis in general practice of pediatric TBIs.

Explainable Machine Learning on AmsterdamUMCdb for ICU Discharge Decision Support: Uniting Intensivists and Data Scientists

Unexpected ICU readmission is associated with longer length of stay and increased mortality. To prevent ICU readmission and death after ICU discharge, our team of intensivists and data scientists aimed to use AmsterdamUMCdb to develop an explainable machine learning–based real-time bedside decision support tool.

Machine Learning, Deep Learning, and Closed Loop Devices-Anesthesia Delivery

With the tremendous volume of data captured during surgeries and procedures, critical care, and pain management, the field of anesthesiology is uniquely suited for the application of machine learning, neural networks, and closed loop technologies. In the past several years, this area has expanded immensely in both interest and clinical applications. This article provides an overview of the basic tenets of machine learning, neural networks, and closed loop devices, with emphasis on the clinical applications of these technologies.

A panoramic view of proteomics and multiomics in precision health

Health is often qualitatively defined as a status free from disease and its quantitative definition requires finding the boundary separating health from pathological conditions. Since many complex diseases have a strong genetic component, substantial efforts have been made to sequence large-scale personal genomes; however, we are not yet able to effectively quantify health status from personal genomes. Since mutational impacts are ultimately manifested at the protein level, we envision that introducing a panoramic proteomic view of complex diseases will allow us to mechanistically understand the molecular etiologies of human diseases. In this perspective article, we will highlight key proteomic approaches to identify pathogenic mutations and map their convergent pathways underlying disease pathogenesis and the integration of omics data at multiple levels to define the borderline between health and disease.

Predicting Recurrence for Patients With Ischemic Cerebrovascular Events Based on Process Discovery and Transfer Learning

The recurrence of Ischemic cerebrovascular events (ICE) often results in a high rate of mortality and disability. However, due to the lack of labeled follow-up data in hospitals, prediction methods using traditional machine learning are usually not available or reliable. Therefore, we propose a new framework for predicting the long-term recurrence risk in patients with ICE after discharge from hospitals based on process mining and transfer learning, to point out high-risk patients for intervention. First, process models are discovered from clinical guidelines for analyzing the similarity of ICE population data collected by different medical institutions, and the control flow found are taken as added characteristics of patients. Then we use the in-hospital data (target domain) and the national stroke screening data (source domain), to develop risk prediction models applying instance filter and weight-based transfer learning method. To verify our method, 205 cases from a tertiary hospital and 2954 cases from the screening cohort (2015-2017) are tested. Experimental results show that our framework can improve the performance of three instance-based transfer algorithms. This study provides a comprehensive and efficient approach for applying transfer learning, to alleviate the limitation of insufficient labeled follow-up data in hospitals.

A Locally Optimized Data-Driven Tool to Predict Sepsis-Associated Vasopressor Use in the ICU

OBJECTIVES

To train a model to predict vasopressor use in ICU patients with sepsis and optimize external performance across hospital systems using domain adaptation, a transfer learning approach.

DESIGN

Observational cohort study.

SETTING

Two academic medical centers from January 2014 to June 2017.

PATIENTS

Data were analyzed from 14,512 patients (9,423 at the development site and 5,089 at the validation site) who were admitted to an ICU and met Center for Medicare and Medicaid Services definition of severe sepsis either before or during the ICU stay. Patients were excluded if they never developed sepsis, if the ICU length of stay was less than 8 hours or more than 20 days or if they developed shock up to the first 4 hours of ICU admission.

MEASUREMENTS AND MAIN RESULTS

Forty retrospectively collected features from the electronic medical records of adult ICU patients at the development site (four hospitals) were used as inputs for a neural network Weibull-Cox survival model to derive a prediction tool for future need of vasopressors. Domain adaptation updated parameters to optimize model performance in the validation site (two hospitals), a different healthcare system over 2,000 miles away. The cohorts at both sites were randomly split into training and testing sets (80% and 20%, respectively). When applied to the test set in the development site, the model predicted vasopressor use 4-24 hours in advance with an area under the receiver operator characteristic curve, specificity, and positive predictive value ranging from 0.80 to 0.81, 56.2% to 61.8%, and 5.6% to 12.1%, respectively. Domain adaptation improved performance of the model to predict vasopressor use within 4 hours at the validation site (area under the receiver operator characteristic curve 0.81 [CI, 0.80-0.81] from 0.77 [CI, 0.76-0.77], p < 0.01; specificity 59.7% [CI, 58.9-62.5%] from 49.9% [CI, 49.5-50.7%], p < 0.01; positive predictive value 8.9% [CI, 8.5-9.4%] from 7.3 [7.1-7.4%], p < 0.01).

CONCLUSIONS

Domain adaptation improved performance of a model predicting sepsis-associated vasopressor use during external validation.

[Prediction of intensive care unit readmission for critically ill patients based on ensemble learning]

OBJECTIVE

To develop machine learning models for predicting intensive care unit (ICU) readmission using ensemble learning algorithms.

METHODS

A publicly accessible American ICU database, medical information mart for intensive care (MIMIC)-Ⅲ as the data source was used, and the patients were selected by the inclusion and exclusion criteria. A set of variables that had the predictive ability of outcome including demographics, vital signs, laboratory tests, and comorbidities of patients were extracted from the dataset. We built the ICU readmission prediction models based on ensemble learning methods including random forest, adaptive boosting (AdaBoost), and gradient boosting decision tree (GBDT), and compared the prediction performance of the machine learning models with a conventional Logistic regression model. Five-fold cross validation was used to train and validate the prediction models. Average sensitivity, positive prediction value, negative prediction value, false positive rate, false negative rate, area under the receiver operating characteristic curve (AUROC) and Brier score were used as performance measures. After constructing the prediction models, top 10 predictive variables based on importance ranking were identified by the model with the best discrimination.

RESULTS

Among these ICU readmission prediction models, GBDT (AUROC=0.858) had better performance than random forest (AUROC=0.827), and was slightly superior to AdaBoost (AUROC=0.851) in terms of AUROC. Compared with Logistic regression (AUROC=0.810), the discrimination of the three ensemble learning models was much better. The feature importance provided by GBDT showed that the top ranking variables included vital signs and laboratory tests. The patients with ICU readmission had higher mean arterial pressure, systolic blood pressure, diastolic blood pressure, and heart rate than the patients without ICU readmission. Meanwhile, the patients readmitted to ICU experienced lower urine output and higher serum creatinine. Overall, the patients having repeated admissions during their hospitalization showed worse heart function and renal function compared with the patients without ICU readmission.

CONCLUSION

The ensemble learning based ICU readmission prediction models had better performance than Logistic regression model. Such ensemble learning models have the potential to aid ICU physicians in identifying those patients with high risk of ICU readmission and thus help improve overall clinical outcomes.

Machine learning in patient flow: a review

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

Application of Machine Learning in Intensive Care Unit (ICU) Settings Using MIMIC Dataset: Systematic Review

Modern Intensive Care Units (ICUs) provide continuous monitoring of critically ill patients susceptible to many complications affecting morbidity and mortality. ICU settings require a high staff-to-patient ratio and generates a sheer volume of data. For clinicians, the real-time interpretation of data and decision-making is a challenging task. Machine Learning (ML) techniques in ICUs are making headway in the early detection of high-risk events due to increased processing power and freely available datasets such as the Medical Information Mart for Intensive Care (MIMIC). We conducted a systematic literature review to evaluate the effectiveness of applying ML in the ICU settings using the MIMIC dataset. A total of 322 articles were reviewed and a quantitative descriptive analysis was performed on 61 qualified articles that applied ML techniques in ICU settings using MIMIC data. We assembled the qualified articles to provide insights into the areas of application, clinical variables used, and treatment outcomes that can pave the way for further adoption of this promising technology and possible use in routine clinical decision-making. The lessons learned from our review can provide guidance to researchers on application of ML techniques to increase their rate of adoption in healthcare.

Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: A scoping review

OBJECTIVE

The study sought to describe the prevalence and nature of clinical expert involvement in the development, evaluation, and implementation of clinical decision support systems (CDSSs) that utilize machine learning to analyze electronic health record data to assist nurses and physicians in prognostic and treatment decision making (ie, predictive CDSSs) in the hospital.

MATERIALS AND METHODS

A systematic search of PubMed, CINAHL, and IEEE Xplore and hand-searching of relevant conference proceedings were conducted to identify eligible articles. Empirical studies of predictive CDSSs using electronic health record data for nurses or physicians in the hospital setting published in the last 5 years in peer-reviewed journals or conference proceedings were eligible for synthesis. Data from eligible studies regarding clinician involvement, stage in system design, predictive CDSS intention, and target clinician were charted and summarized.

RESULTS

Eighty studies met eligibility criteria. Clinical expert involvement was most prevalent at the beginning and late stages of system design. Most articles (95%) described developing and evaluating machine learning models, 28% of which described involving clinical experts, with nearly half functioning to verify the clinical correctness or relevance of the model (47%).

DISCUSSION

Involvement of clinical experts in predictive CDSS design should be explicitly reported in publications and evaluated for the potential to overcome predictive CDSS adoption challenges.

CONCLUSIONS

If present, clinical expert involvement is most prevalent when predictive CDSS specifications are made or when system implementations are evaluated. However, clinical experts are less prevalent in developmental stages to verify clinical correctness, select model features, preprocess data, or serve as a gold standard.

Intensive care digital health response to emerging infectious disease outbreaks such as COVID-19

The COVID-19 pandemic has required intensive care units to rapidly adjust and adapt their existing practices. Although there has a focus on expanding critical care infrastructure, equipment and workforce, plans have not emphasised the need to increase digital capabilities. The objective of this report was to recognise key areas of digital health related to the COVID-19 response. We identified and explored six focus areas relevant to intensive care, including using digital solutions to increase critical care capacity, developing surge capacity within an electronic health record, maintenance and downtime planning, training considerations and the role of data analytics. This article forms the basis of a framework for the intensive care digital health response to COVID-19 and other emerging infectious disease outbreaks.

Predicting adult neuroscience intensive care unit admission from emergency department triage using a retrospective, tabular-free text machine learning approach

Early admission to the neurosciences intensive care unit (NSICU) is associated with improved patient outcomes. Natural language processing offers new possibilities for mining free text in electronic health record data. We sought to develop a machine learning model using both tabular and free text data to identify patients requiring NSICU admission shortly after arrival to the emergency department (ED). We conducted a single-center, retrospective cohort study of adult patients at the Mount Sinai Hospital, an academic medical center in New York City. All patients presenting to our institutional ED between January 2014 and December 2018 were included. Structured (tabular) demographic, clinical, bed movement record data, and free text data from triage notes were extracted from our institutional data warehouse. A machine learning model was trained to predict likelihood of NSICU admission at 30 min from arrival to the ED. We identified 412,858 patients presenting to the ED over the study period, of whom 1900 (0.5%) were admitted to the NSICU. The daily median number of ED presentations was 231 (IQR 200–256) and the median time from ED presentation to the decision for NSICU admission was 169 min (IQR 80–324). A model trained only with text data had an area under the receiver-operating curve (AUC) of 0.90 (95% confidence interval (CI) 0.87–0.91). A structured data-only model had an AUC of 0.92 (95% CI 0.91–0.94). A combined model trained on structured and text data had an AUC of 0.93 (95% CI 0.92–0.95). At a false positive rate of 1:100 (99% specificity), the combined model was 58% sensitive for identifying NSICU admission. A machine learning model using structured and free text data can predict NSICU admission soon after ED arrival. This may potentially improve ED and NSICU resource allocation. Further studies should validate our findings.

Dynamic survival prediction in intensive care units from heterogeneous time series without the need for variable selection or curation

Extensive monitoring in intensive care units (ICUs) generates large quantities of data which contain numerous trends that are difficult for clinicians to systematically evaluate. Current approaches to such heterogeneity in electronic health records (EHRs) discard pertinent information. We present a deep learning pipeline that uses all uncurated chart, lab, and output events for prediction of in-hospital mortality without variable selection. Over 21,000 ICU patients and tens of thousands of variables derived from the MIMIC-III database were used to train and validate our model. Recordings in the first few hours of a patient's stay were found to be strongly predictive of mortality, outperforming models using SAPS II and OASIS scores, AUROC 0.72 and 0.76 at 24 h respectively, within just 12 h of ICU admission. Our model achieves a very strong predictive performance of AUROC 0.85 (95% CI 0.83-0.86) after 48 h. Predictive performance increases over the first 48 h, but suffers from diminishing returns, providing rationale for time-limited trials of critical care and suggesting that the timing of decision making can be optimised and individualised.

Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study

Background

Congenital heart disease accounts for almost a third of all major congenital anomalies. Congenital heart defects have a significant impact on morbidity, mortality and health costs for children and adults. Research regarding the risk of pre-surgical mortality is scarce.

Objectives

Our goal is to generate a predictive model calculator adapted to the regional reality focused on individual mortality prediction among patients with congenital heart disease undergoing cardiac surgery.

Methods

Two thousand two hundred forty CHD consecutive patients’ data from InCor’s heart surgery program was used to develop and validate the preoperative risk-of-death prediction model of congenital patients undergoing heart surgery. There were six artificial intelligence models most cited in medical references used in this study: Multilayer Perceptron (MLP), Random Forest (RF), Extra Trees (ET), Stochastic Gradient Boosting (SGB), Ada Boost Classification (ABC) and Bag Decision Trees (BDT).

Results

The top performing areas under the curve were achieved using Random Forest (0.902). Most influential predictors included previous admission to ICU, diagnostic group, patient's height, hypoplastic left heart syndrome, body mass, arterial oxygen saturation, and pulmonary atresia. These combined predictor variables represent 67.8% of importance for the risk of mortality in the Random Forest algorithm.

Conclusions

The representativeness of “hospital death” is greater in patients up to 66 cm in height and body mass index below 13.0 for InCor’s patients. The proportion of “hospital death” declines with the increased arterial oxygen saturation index. Patients with prior hospitalization before surgery had higher “hospital death” rates than who did not required such intervention. The diagnoses groups having the higher fatal outcomes probability are aligned with the international literature. A web application is presented where researchers and providers can calculate predicted mortality based on the CgntSCORE on any web browser or smartphone.

Implementation of Artificial Intelligence-Based Clinical Decision Support to Reduce Hospital Readmissions at a Regional Hospital

BACKGROUND

Hospital readmissions are a key quality metric, which has been tied to reimbursement. One strategy to reduce readmissions is to direct resources to patients at the highest risk of readmission. This strategy necessitates a robust predictive model coupled with effective, patient-centered interventions.

OBJECTIVE

The aim of this study was to reduce unplanned hospital readmissions through the use of artificial intelligence-based clinical decision support.

METHODS

A commercially vended artificial intelligence tool was implemented at a regional hospital in La Crosse, Wisconsin between November 2018 and April 2019. The tool assessed all patients admitted to general care units for risk of readmission and generated recommendations for interventions intended to decrease readmission risk. Similar hospitals were used as controls. Change in readmission rate was assessed by comparing the 6-month intervention period to the same months of the previous calendar year in exposure and control hospitals.

RESULTS

Among 2,460 hospitalizations assessed using the tool, 611 were designated by the tool as high risk. Sensitivity and specificity for risk assignment were 65% and 89%, respectively. Over 6 months following implementation, readmission rates decreased from 11.4% during the comparison period to 8.1% (p < 0.001). After accounting for the 0.5% decrease in readmission rates (from 9.3 to 8.8%) at control hospitals, the relative reduction in readmission rate was 25% (p < 0.001). Among patients designated as high risk, the number needed to treat to avoid one readmission was 11.

CONCLUSION

We observed a decrease in hospital readmission after implementing artificial intelligence-based clinical decision support. Our experience suggests that use of artificial intelligence to identify patients at the highest risk for readmission can reduce quality gaps when coupled with patient-centered interventions.

A Tool to Predict Readmission to the Intensive Care Unit in Surgical Critical Care Patients-The RISC Score

BACKGROUND

Readmission to the Intensive Care Unit (ICU) is associated with a high risk of in-hospital mortality and higher health care costs. Previously published tools to predict ICU readmission in surgical ICU patients have important limitations that restrict their clinical implementation. We sought to develop a clinically intuitive score that can be implemented to predict readmission to the ICU after surgery or trauma. We designed the score to emphasize modifiable predictors.

METHODS

In this retrospective cohort study, we included surgical patients requiring critical care between June 2015 and January 2019 at Beth Israel Deaconess Medical Center, Harvard Medical School, MA, USA. We used logistic regression to fit a prognostic model for ICU readmission from a priori defined, widely available candidate predictors. The score performance was compared with existing prediction instruments.

RESULTS

Of 7,126 patients, 168 (2.4%) were readmitted to the ICU during the same hospitalization. The final score included 8 variables addressing demographical factors, surgical factors, physiological parameters, ICU treatment and the acuity of illness. The maximum score achievable was 13 points. Potentially modifiable predictors included the inability to ambulate at ICU discharge, substantial positive fluid balance (>5 liters), severe anemia (hemoglobin <7 mg/dl), hyperglycemia (>180 mg/dl), and long ICU length of stay (>5 days). The score yielded an area under the receiver operating characteristic curve of 0.78 (95% CI 0.74-0.82) and significantly outperformed previously published scores. The performance of the underlying model was confirmed by leave-one-out cross-validation.

CONCLUSION

The RISC-score is a clinically intuitive prediction instrument that helps identify surgical ICU patients at high risk for ICU readmission. The simplicity of the score facilitates its clinical implementation across surgical divisions.

Demonstrating the consequences of learning missingness patterns in early warning systems for preventative health care: A novel simulation and solution

When using tree-based methods to develop predictive analytics and early warning systems for preventive healthcare, it is important to use an appropriate imputation method to prevent learning the missingness pattern. To demonstrate this, we developed a novel simulation that generated synthetic electronic health record data using a variational autoencoder with a custom loss function, which took into account the high missing rate of electronic health data. We showed that when tree-based methods learn missingness patterns (correlated with adverse events) in electronic health record data, this leads to decreased performance if the system is used in a new setting that has different missingness patterns. Performance is worst in this scenario when the missing rate between those with and without an adverse event is the greatest. We found that randomized and Bayesian regression imputation methods mitigate the issue of learning the missingness pattern for tree-based methods. We used this information to build a novel early warning system for predicting patient deterioration in general wards and telemetry units: PICTURE (Predicting Intensive Care Transfers and other UnfoReseen Events). To develop, tune, and test PICTURE, we used labs and vital signs from electronic health records of adult patients over four years (n = 133,089 encounters). We analyzed primary outcomes of unplanned intensive care unit transfer, emergency vasoactive medication administration, cardiac arrest, and death. We compared PICTURE with existing early warning systems and logistic regression at multiple levels of granularity. When analyzing PICTURE on the testing set using all observations within a hospital encounter (event rate = 3.4%), PICTURE had an area under the receiver operating characteristic curve (AUROC) of 0.83 and an adjusted (event rate = 4%) area under the precision-recall curve (AUPR) of 0.27, while the next best tested method-regularized logistic regression-had an AUROC of 0.80 and an adjusted AUPR of 0.22. To ensure system interpretability, we applied a state-of-the-art prediction explainer that provided a ranked list of features contributing most to the prediction. Though it is currently difficult to compare machine learning-based early warning systems, a rudimentary comparison with published scores demonstrated that PICTURE is on par with state-of-the-art machine learning systems. To facilitate more robust comparisons and development of early warning systems in the future, we have released our variational autoencoder's code and weights so researchers can (a) test their models on data similar to our institution and (b) make their own synthetic datasets.

Application of electronic medical record-derived analytics in critical care: Rothman Index predicts mortality and readmissions in surgical intensive care unit patients

INTRODUCTION

The Rothman Index (RI) is an objective measurement of a patient's overall condition, automatically generated from 26 variables including vital signs, laboratory data, cardiac rhythms, and nursing assessments. The purpose of this study was to assess the validity of RI scores in predicting surgical ICU (SICU) readmission rates and mortality.

METHODS

We conducted a single-center retrospective analysis of surgical patients who were transferred from the SICU to the surgical floor from December 2014 to December 2016. Data included demographics, length of stay (LOS), mortality, and RI at multiple pretransfer and post-transfer time points.

RESULTS

A total of 1,445 SICU patients were transferred to the surgical floor; 79 patients (5.5%) were readmitted within 48 hours of transfer. Mean age was 52 years, and 67% were male. Compared to controls, patients readmitted to the SICU within 48 hours experienced higher LOS (29 vs. 11 days, p < 0.05) as well as higher mortality (2.5% vs. 0.6%, p < 0.05). Patients requiring readmission also had a lower RI at 72, 48, and 24 hours before transfer as well as at 24 and 48 hours after transfer (p < 0.05 for all). Rothman Index scores were categorized into higher-risk (<40), medium-risk (40-65), and lower-risk groups (>65); RI scores at 24 hours before transfer were inversely proportional to overall mortality (RI < 40 = 2.5%, RI 40-65 = 0.3%, and RI > 65 = 0%; p < 0.05) and SICU readmission rates (RI < 40 = 9%, RI 40-65 = 5.2%, and RI > 65 = 2.8%; p < 0.05). Patients transferred with RI scores greater than 83 did not require SICU readmission within 48 hours.

CONCLUSION

Surgical ICU patients requiring readmission within 48 hours of transfer have a significantly higher mortality and longer LOS compared to those who do not. Patients requiring readmission also have significantly lower pretransfer and post-transfer RI scores compared to those who do not. Rothman Index scores may be used as a clinical tool for evaluating patients before transfer from the SICU. Prospective studies are warranted to further validate use of this technology.

LEVEL OF EVIDENCE

Retrospective database review, level III.

Artificial Intelligence in Anesthesiology: Current Techniques, Clinical Applications, and Limitations

Artificial intelligence has been advancing in fields including anesthesiology. This scoping review of the intersection of artificial intelligence and anesthesia research identified and summarized six themes of applications of artificial intelligence in anesthesiology: (1) depth of anesthesia monitoring, (2) control of anesthesia, (3) event and risk prediction, (4) ultrasound guidance, (5) pain management, and (6) operating room logistics. Based on papers identified in the review, several topics within artificial intelligence were described and summarized: (1) machine learning (including supervised, unsupervised, and reinforcement learning), (2) techniques in artificial intelligence (e.g., classical machine learning, neural networks and deep learning, Bayesian methods), and (3) major applied fields in artificial intelligence.The implications of artificial intelligence for the practicing anesthesiologist are discussed as are its limitations and the role of clinicians in further developing artificial intelligence for use in clinical care. Artificial intelligence has the potential to impact the practice of anesthesiology in aspects ranging from perioperative support to critical care delivery to outpatient pain management.

Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

OBJECTIVE

We aimed to explore the added value of common machine learning (ML) algorithms for prediction of outcome for moderate and severe traumatic brain injury.

STUDY DESIGN AND SETTING

We performed logistic regression (LR), lasso regression, and ridge regression with key baseline predictors in the IMPACT-II database (15 studies, n = 11,022). ML algorithms included support vector machines, random forests, gradient boosting machines, and artificial neural networks and were trained using the same predictors. To assess generalizability of predictions, we performed internal, internal-external, and external validation on the recent CENTER-TBI study (patients with Glasgow Coma Scale <13, n = 1,554). Both calibration (calibration slope/intercept) and discrimination (area under the curve) was quantified.

RESULTS

In the IMPACT-II database, 3,332/11,022 (30%) died and 5,233(48%) had unfavorable outcome (Glasgow Outcome Scale less than 4). In the CENTER-TBI study, 348/1,554(29%) died and 651(54%) had unfavorable outcome. Discrimination and calibration varied widely between the studies and less so between the studied algorithms. The mean area under the curve was 0.82 for mortality and 0.77 for unfavorable outcomes in the CENTER-TBI study.

CONCLUSION

ML algorithms may not outperform traditional regression approaches in a low-dimensional setting for outcome prediction after moderate or severe traumatic brain injury. Similar to regression-based prediction models, ML algorithms should be rigorously validated to ensure applicability to new populations.

Artificial Intelligence and Machine Learning in Anesthesiology

Commercial applications of artificial intelligence and machine learning have made remarkable progress recently, particularly in areas such as image recognition, natural speech processing, language translation, textual analysis, and self-learning. Progress had historically languished in these areas, such that these skills had come to seem ineffably bound to intelligence. However, these commercial advances have performed best at single-task applications in which imperfect outputs and occasional frank errors can be tolerated.The practice of anesthesiology is different. It embodies a requirement for high reliability, and a pressured cycle of interpretation, physical action, and response rather than any single cognitive act. This review covers the basics of what is meant by artificial intelligence and machine learning for the practicing anesthesiologist, describing how decision-making behaviors can emerge from simple equations. Relevant clinical questions are introduced to illustrate how machine learning might help solve them-perhaps bringing anesthesiology into an era of machine-assisted discovery.

Temporal convolutional networks allow early prediction of events in critical care

OBJECTIVE

Clinical interventions and death in the intensive care unit (ICU) depend on complex patterns in patients' longitudinal data. We aim to anticipate these events earlier and more consistently so that staff can consider preemptive action.

MATERIALS AND METHODS

We use a temporal convolutional network to encode longitudinal data and a feedforward neural network to encode demographic data from 4713 ICU admissions in 2014-2018. For each hour of each admission, we predict events in the subsequent 1-6 hours. We compare performance with other models including a recurrent neural network.

RESULTS

Our model performed similarly to the recurrent neural network for some events and outperformed it for others. This performance increase was more evident in a sensitivity analysis where the prediction timeframe was varied. Average positive predictive value (95% CI) was 0.786 (0.781-0.790) and 0.738 (0.732-0.743) for up- and down-titrating FiO2, 0.574 (0.519-0.625) for extubation, 0.139 (0.117-0.162) for intubation, 0.533 (0.492-0.572) for starting noradrenaline, 0.441 (0.433-0.448) for fluid challenge, and 0.315 (0.282-0.352) for death.

DISCUSSION

Events were better predicted where their important determinants were captured in structured electronic health data, and where they occurred in homogeneous circumstances. We produce partial dependence plots that show our model learns clinically-plausible associations between its inputs and predictions.

CONCLUSION

Temporal convolutional networks improve prediction of clinical events when used to represent longitudinal ICU data.

Readmissions to General ICUs in a Geographic Area of Poland Are Seemingly Associated with Better Outcomes

BACKGROUND

Various factors can contribute to high mortality rates in intensive care units (ICUs). Here, we intended to define a population of patients readmitted to general ICUs in Poland and to identify independent predictors of ICU readmission.

METHODS

Data derived from adult ICU admissions from the Silesian region of Poland were analyzed. First-time ICU readmissions (≤30 days from ICU discharge after index admissions) were compared with first-time ICU admissions. Pre-admission and admission variables that independently influenced the need for ICU readmission were identified.

RESULTS

Among the 21,495 ICU admissions, 839 were first-time readmissions (3.9%). Patients readmitted to the ICU had lower mean APACHE II (21.2 ± 8.0 vs. 23.2 ± 8.8, p < 0.001) and TISS-28 scores (33.7 ± 7.4 vs. 35.2 ± 7.8, p < 0.001) in the initial 24 h following ICU admission, compared to first-time admissions. ICU readmissions were associated with lower mortality vs. first-time admissions (39.2% vs. 44.3%, p = 0.004). Independent predictors for ICU readmission included the admission from a surgical ward (among admission sources), chronic respiratory failure, cachexia, previous stroke, chronic neurological diseases (among co-morbidities), and multiple trauma or infection (among primary reasons for ICU admission).

CONCLUSIONS

High mortality associated with first-time ICU admissions is associated with a lower mortality rate during ICU readmissions.

Prediction of ICU Readmissions Using Data at Patient Discharge

Unplanned readmissions to ICU contribute to high health care costs and poor patient outcomes. 6-7% of all ICU cases see a readmission within 72 hours. Machine learning models on electronic health record data can help identify these cases, providing more information about short and long-term risks to clinicians at the time of ICU discharge. While time-toevent techniques have been used in clinical care, models that identify risks over time using higher-dimensional, non-linear machine learning models need to be developed to present changes in risk with non-linear techniques. This work identifies risks of ICU readmissions at 24 hours, 72 hours, 7 days, 30 days, and bounceback readmissions in the same hospital admission with an AUROC for 72 hours of 0.76 and for bounceback of 0.84.

Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory

BACKGROUND

Unplanned readmission of a hospitalized patient is an indicator of patients' exposure to risk and an avoidable waste of medical resources. In addition to hospital readmission, intensive care unit (ICU) readmission brings further financial risk, along with morbidity and mortality risks. Identification of high-risk patients who are likely to be readmitted can provide significant benefits for both patients and medical providers. The emergence of machine learning solutions to detect hidden patterns in complex, multi-dimensional datasets provides unparalleled opportunities for developing an efficient discharge decision-making support system for physicians and ICU specialists.

METHODS AND FINDINGS

We used supervised machine learning approaches for ICU readmission prediction. We used machine learning methods on comprehensive, longitudinal clinical data from the MIMIC-III to predict the ICU readmission of patients within 30 days of their discharge. We incorporate multiple types of features including chart events, demographic, and ICD-9 embeddings. We have utilized recent machine learning techniques such as Recurrent Neural Networks (RNN) with Long Short-Term Memory (LSTM), by this we have been able to incorporate the multivariate features of EHRs and capture sudden fluctuations in chart event features (e.g. glucose and heart rate). We show that our LSTM-based solution can better capture high volatility and unstable status in ICU patients, an important factor in ICU readmission. Our machine learning models identify ICU readmissions at a higher sensitivity rate of 0.742 (95% CI, 0.718-0.766) and an improved Area Under the Curve of 0.791 (95% CI, 0.782-0.800) compared with traditional methods. We perform in-depth deep learning performance analysis, as well as the analysis of each feature contribution to the predictive model.

CONCLUSION

Our manuscript highlights the ability of machine learning models to improve our ICU decision-making accuracy and is a real-world example of precision medicine in hospitals. These data-driven solutions hold the potential for substantial clinical impact by augmenting clinical decision-making for physicians and ICU specialists. We anticipate that machine learning models will improve patient counseling, hospital administration, allocation of healthcare resources and ultimately individualized clinical care.

Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs

OBJECTIVE

Sepsis remains a costly and prevalent syndrome in hospitals; however, machine learning systems can increase timely sepsis detection using electronic health records. This study validates a gradient boosted ensemble machine learning tool for sepsis detection and prediction, and compares its performance to existing methods.

MATERIALS AND METHODS

Retrospective data was drawn from databases at the University of California, San Francisco (UCSF) Medical Center and the Beth Israel Deaconess Medical Center (BIDMC). Adult patient encounters without sepsis on admission, and with at least one recording of each of six vital signs (SpO₂, heart rate, respiratory rate, temperature, systolic and diastolic blood pressure) were included. We compared the performance of the machine learning algorithm (MLA) to that of commonly used scoring systems. Area under the receiver operating characteristic (AUROC) curve was our primary measure of accuracy. MLA performance was measured at sepsis onset, and at 24 and 48 h prior to sepsis onset.

RESULTS

The MLA achieved an AUROC of 0.88, 0.84, and 0.83 for sepsis onset and 24 and 48 h prior to onset, respectively. These values were superior to those of SIRS (0.66), MEWS (0.61), SOFA (0.72), and qSOFA (0.60) at time of onset. When trained on UCSF data and tested on BIDMC data, sepsis onset AUROC was 0.89.

DISCUSSION AND CONCLUSION

The MLA predicts sepsis up to 48 h in advance and identifies sepsis onset more accurately than commonly used tools, maintaining high performance for sepsis detection when trained and tested on separate datasets.

Towards a decision support tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from MIMIC-III and Bristol, UK

OBJECTIVE

The primary objective is to develop an automated method for detecting patients that are ready for discharge from intensive care.

DESIGN

We used two datasets of routinely collected patient data to test and improve on a set of previously proposed discharge criteria.

SETTING

Bristol Royal Infirmary general intensive care unit (GICU).

PATIENTS

Two cohorts derived from historical datasets: 1870 intensive care patients from GICU in Bristol, and 7592 from Medical Information Mart for Intensive Care (MIMIC)-III.

RESULTS

In both cohorts few successfully discharged patients met all of the discharge criteria. Both a random forest and a logistic classifier, trained using multiple-source cross-validation, demonstrated improved performance over the original criteria and generalised well between the cohorts. The classifiers showed good agreement on which features were most predictive of readiness-for-discharge, and these were generally consistent with clinical experience. By weighting the discharge criteria according to feature importance from the logistic model we showed improved performance over the original criteria, while retaining good interpretability.

CONCLUSIONS

Our findings indicate the feasibility of the proposed approach to ready-for-discharge classification, which could complement other risk models of specific adverse outcomes in a future decision support system. Avenues for improvement to produce a clinically useful tool are identified.

Emergency department triage prediction of clinical outcomes using machine learning models

Background

Development of emergency department (ED) triage systems that accurately differentiate and prioritize critically ill from stable patients remains challenging. We used machine learning models to predict clinical outcomes, and then compared their performance with that of a conventional approach—the Emergency Severity Index (ESI).

Methods

Using National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, from 2007 through 2015, we identified all adult patients (aged ≥ 18 years). In the randomly sampled training set (70%), using routinely available triage data as predictors (e.g., demographics, triage vital signs, chief complaints, comorbidities), we developed four machine learning models: Lasso regression, random forest, gradient boosted decision tree, and deep neural network. As the reference model, we constructed a logistic regression model using the five-level ESI data. The clinical outcomes were critical care (admission to intensive care unit or in-hospital death) and hospitalization (direct hospital admission or transfer). In the test set (the remaining 30%), we measured the predictive performance, including area under the receiver-operating-characteristics curve (AUC) and net benefit (decision curves) for each model.

Results

Of 135,470 eligible ED visits, 2.1% had critical care outcome and 16.2% had hospitalization outcome. In the critical care outcome prediction, all four machine learning models outperformed the reference model (e.g., AUC, 0.86 [95%CI 0.85–0.87] in the deep neural network vs 0.74 [95%CI 0.72–0.75] in the reference model), with less under-triaged patients in ESI triage levels 3 to 5 (urgent to non-urgent). Likewise, in the hospitalization outcome prediction, all machine learning models outperformed the reference model (e.g., AUC, 0.82 [95%CI 0.82–0.83] in the deep neural network vs 0.69 [95%CI 0.68–0.69] in the reference model) with less over-triages in ESI triage levels 1 to 3 (immediate to urgent). In the decision curve analysis, all machine learning models consistently achieved a greater net benefit—a larger number of appropriate triages considering a trade-off with over-triages—across the range of clinical thresholds.

Conclusions

Compared to the conventional approach, the machine learning models demonstrated a superior performance to predict critical care and hospitalization outcomes. The application of modern machine learning models may enhance clinicians’ triage decision making, thereby achieving better clinical care and optimal resource utilization.

Electronic supplementary material

The online version of this article (10.1186/s13054-019-2351-7) contains supplementary material, which is available to authorized users.