Prediction of ICU Readmissions Using Data at Patient Discharge

Performance of advanced machine learning algorithms overlogistic regression in predicting hospital readmissions: A meta-analysis

Objectives

Machine learning algorithms are being increasingly used for predicting hospital readmissions. This meta-analysis evaluated the performance of logistic regression (LR) and machine learning (ML) models for the prediction of 30-day hospital readmission among patients in the US.

Methods

Electronic databases (i.e., Medline, PubMed, and Embase) were searched from January 2015 to December 2019. Only studies in the English language were included. Two reviewers performed studies screening, quality appraisal, and data collection. The quality of the studies was assessed using the Quality in Prognosis Studies (QUIPS) tool. Model performance was evaluated using the Area Under the Curve (AUC). A random-effects meta-analysis was performed using STATA 16.

Results

Nine studies were included based on the selection criteria. The most common ML techniques were tree-based methods such as boosting and random forest. Most of the studies had a low risk of bias (8/9). The AUC was greater with ML to predict 30-day all-cause hospital readmission compared with LR [Mean Difference (MD): 0.03; 95% Confidence Interval (CI) 0.01-0.05]. Subgroup analyses found that deep-learning methods had a better performance compared with LR (MD 0.06; 95% CI, 0.04-0.09), followed by neural networks (MD: 0.03; 95% CI, 0.03-0.03), while the AUCs of the tree-based (MD: 0.02; 95% CI -0.00-0.04) and kernel-based (MD: 0.02; 95% CI 0.02 (-0.13-0.16) methods were no different compared to LR. More than half of the studies evaluated heart failure-related rehospitalization (N = 5). For the readmission prediction among heart failure patients, ML performed better compared with LR, with a mean difference in AUC of 0.04 (95% CI, 0.01-0.07). The leave-one-out sensitivity analysis confirmed the robustness of the findings.

Conclusion

Multiple ML methods were used to predict 30-day all-cause hospital readmission. Performance varied across the ML methods, with deep-learning methods showing the best performance over the LR.

Knowledge Graph Embeddings for ICU readmission prediction

BACKGROUND

Intensive Care Unit (ICU) readmissions represent both a health risk for patients,with increased mortality rates and overall health deterioration, and a financial burden for healthcare facilities. As healthcare became more data-driven with the introduction of Electronic Health Records (EHR), machine learning methods have been applied to predict ICU readmission risk. However, these methods disregard the meaning and relationships of data objects and work blindly over clinical data without taking into account scientific knowledge and context. Ontologies and Knowledge Graphs can help bridge this gap between data and scientific context, as they are computational artefacts that represent the entities of a domain and their relationships to each other in a formalized way.

METHODS AND RESULTS

We have developed an approach that enriches EHR data with semantic annotations to ontologies to build a Knowledge Graph. A patient's ICU stay is represented by Knowledge Graph embeddings in a contextualized manner, which are used by machine learning models to predict 30-days ICU readmissions. This approach is based on several contributions: (1) an enrichment of the MIMIC-III dataset with patient-oriented annotations to various biomedical ontologies; (2) a Knowledge Graph that defines patient data with biomedical ontologies; (3) a predictive model of ICU readmission risk that uses Knowledge Graph embeddings; (4) a variant of the predictive model that targets different time points during an ICU stay. Our predictive approaches outperformed both a baseline and state-of-the-art works achieving a mean Area Under the Receiver Operating Characteristic Curve of 0.827 and an Area Under the Precision-Recall Curve of 0.691. The application of this novel approach to help clinicians decide whether a patient can be discharged has the potential to prevent the readmission of [Formula: see text] of Intensive Care Unit patients, without unnecessarily prolonging the stay of those who would not require it.

CONCLUSION

The coupling of semantic annotation and Knowledge Graph embeddings affords two clear advantages: they consider scientific context and they are able to build representations of EHR information of different types in a common format. This work demonstrates the potential for impact that integrating ontologies and Knowledge Graphs into clinical machine learning applications can have.

Identifying and evaluating barriers for the implementation of machine learning in the intensive care unit

BACKGROUND

Despite apparent promise and the availability of numerous examples in the literature, machine learning models are rarely used in practice in ICU units. This mismatch suggests that there are poorly understood barriers preventing uptake, which we aim to identify.

METHODS

We begin with a qualitative study with 29 interviews of 40 Intensive Care Unit-, hospital- and MedTech company staff members. As a follow-up to the study, we attempt to quantify some of the technical issues raised. To perform experiments we selected two models based on criteria such as medical relevance. Using these models we measure the loss of performance in predictive models due to drift over time, change of available patient features, scarceness of data, and deploying a model in a different context to the one it was built in.

RESULTS

The qualitative study confirms our assumptions on the potential of AI-driven analytics for patient care, as well as showing the prevalence and type of technical blocking factors that are responsible for its slow uptake. The experiments confirm that each of these issues can cause important loss of predictive model performance, depending on the model and the issue.

CONCLUSIONS

Based on the qualitative study and quantitative experiments we conclude that more research on practical solutions to enable AI-driven innovation in Intensive Care Units is needed. Furthermore, the general poor situation with respect to public, usable implementations of predictive models would appear to limit the possibilities for both the scientific repeatability of the underlying research and the transfer of this research into practice.

Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines

Background

Intensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Widely used post hoc explanation methods also have inherent limitations. Few studies are evaluating inherently interpretable ML models for health care and involve clinicians in inspecting the trained model.

Methods

An inherently interpretable model for the prediction of 3 day ICU readmission was developed. We used explainable boosting machines that learn modular risk functions and which have already been shown to be suitable for the health care domain. We created a retrospective cohort of 15,589 ICU stays and 169 variables collected between 2006 and 2019 from the University Hospital Münster. A team of physicians inspected the model, checked the plausibility of each risk function, and removed problematic ones. We collected qualitative feedback during this process and analyzed the reasons for removing risk functions. The performance of the final explainable boosting machine was compared with a validated clinical score and three commonly used ML models. External validation was performed on the widely used Medical Information Mart for Intensive Care version IV database.

Results

The developed explainable boosting machine used 67 features and showed an area under the precision-recall curve of 0.119 ± 0.020 and an area under the receiver operating characteristic curve of 0.680 ± 0.025. It performed on par with state-of-the-art gradient boosting machines (0.123 ± 0.016, 0.665 ± 0.036) and outperformed the Simplified Acute Physiology Score II (0.084 ± 0.025, 0.607 ± 0.019), logistic regression (0.092 ± 0.026, 0.587 ± 0.016), and recurrent neural networks (0.095 ± 0.008, 0.594 ± 0.027). External validation confirmed that explainable boosting machines (0.221 ± 0.023, 0.760 ± 0.010) performed similarly to gradient boosting machines (0.232 ± 0.029, 0.772 ± 0.018). Evaluation of the model inspection showed that explainable boosting machines can be useful to detect and remove problematic risk functions.

Conclusions

We developed an inherently interpretable ML model for 3 day ICU readmission prediction that reached the state-of-the-art performance of black box models. Our results suggest that for low- to medium-dimensional datasets that are common in health care, it is feasible to develop ML models that allow a high level of human control without sacrificing performance.

Unplanned Readmissions to the Intensive Care Unit Among Geriatric Trauma Patients

BACKGROUND

Unplanned readmission/bounceback to the intensive care unit (ICUBB) is a prevalent issue in the medical community. The geriatric population is incompletely studied in regard to ICUBB. We sought to determine if ICUBB in older patients was associated with higher risk of mortality. We hypothesized that, of those who were older, those with ICUBB would have higher mortality compared to those with no ICUBB. Further, we hypothesized that of those with ICUBB, older age would lead to higher mortality.

METHODS

The Pennsylvania Trauma Outcome Study database was retrospectively queried from 2003 to 2018 for all trauma patients of age ≥40 years. Those with advance directives were excluded. Adjusted analysis in the form of logistic regressions controlling for demographic and injury covariates and clustering by facility were used to assess the adjusted impact of ICUBB and age on mortality.

RESULTS

363,778 patients were aged ≥40 years. When comparing mortalities between the age 40 and 49 years group and those in older groups, a dramatic increase in mortality was observed between those in each respective age category with ICUBB vs non-ICUBB. This trend was most prominent in those in the 90+ years age group (ICUBB: AOR: 34.78, P < .001; non-ICUBB: AOR: 9.08, P < .001). A second model only including patients who had ICUBB found that patients of age ≥65 years had significantly higher odds of mortality (AOR: 4.10, P < .001) when compared to their younger counterparts (age <65 years).

DISCUSSION

An ICUBB seems to exacerbate mortality rates as age increases. This profound increase in mortality calls for strategies to be developed, especially in the older population, to attempt to mitigate the factors leading to ICUBB.

Application of machine learning in predicting hospital readmissions: a scoping review of the literature

Background

Advances in machine learning (ML) provide great opportunities in the prediction of hospital readmission. This review synthesizes the literature on ML methods and their performance for predicting hospital readmission in the US.

Methods

This review was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews (PRISMA-ScR) Statement. The extraction of items was also guided by the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Electronic databases PUBMED, MEDLINE, and EMBASE were systematically searched from January 1, 2015, through December 10, 2019. The articles were imported into COVIDENCE online software for title/abstract screening and full-text eligibility. Observational studies using ML techniques for hospital readmissions among US patients were eligible for inclusion. Articles without a full text available in the English language were excluded. A qualitative synthesis included study characteristics, ML algorithms utilized, and model validation, and quantitative analysis assessed model performance. Model performances in terms of Area Under the Curve (AUC) were analyzed using R software. Quality in Prognosis Studies (QUIPS) tool was used to assess the quality of the reviewed studies.

Results

Of 522 citations reviewed, 43 studies met the inclusion criteria. A majority of the studies used electronic health records (24, 56%), followed by population-based data sources (15, 35%) and administrative claims data (4, 9%). The most common algorithms were tree-based methods (23, 53%), neural network (NN) (14, 33%), regularized logistic regression (12, 28%), and support vector machine (SVM) (10, 23%). Most of these studies (37, 85%) were of high quality. A majority of these studies (28, 65%) reported ML algorithms with an AUC above 0.70. There was a range of variability within AUC reported by these studies with a median of 0.68 (IQR: 0.64–0.76; range: 0.50–0.90).

Conclusions

The ML algorithms involving tree-based methods, NN, regularized logistic regression, and SVM are commonly used to predict hospital readmission in the US. Further research is needed to compare the performance of ML algorithms for hospital readmission prediction.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01284-z.

Recurrent Neural Networks for Early Detection of Heart Failure From Longitudinal Electronic Health Record Data: Implications for Temporal Modeling With Respect to Time Before Diagnosis, Data Density, Data Quantity, and Data Type

BACKGROUND

We determined the impact of data volume and diversity and training conditions on recurrent neural network methods compared with traditional machine learning methods.

METHODS AND RESULTS

Using longitudinal electronic health record data, we assessed the relative performance of machine learning models trained to detect a future diagnosis of heart failure in primary care patients. Model performance was assessed in relation to data parameters defined by the combination of different data domains (data diversity), the number of patient records in the training data set (data quantity), the number of encounters per patient (data density), the prediction window length, and the observation window length (ie, the time period before the prediction window that is the source of features for prediction). Data on 4370 incident heart failure cases and 30 132 group-matched controls were used. Recurrent neural network model performance was superior under a variety of conditions that included (1) when data were less diverse (eg, a single data domain like medication or vital signs) given the same training size; (2) as data quantity increased; (3) as density increased; (4) as the observation window length increased; and (5) as the prediction window length decreased. When all data domains were used, the performance of recurrent neural network models increased in relation to the quantity of data used (ie, up to 100% of the data). When data are sparse (ie, fewer features or low dimension), model performance is lower, but a much smaller training set size is required to achieve optimal performance compared with conditions where data are more diverse and includes more features.

CONCLUSIONS

Recurrent neural networks are effective for predicting a future diagnosis of heart failure given sufficient training set size. Model performance appears to continue to improve in direct relation to training set size.