1
|
Hegselmann S, Ertmer C, Volkert T, Gottschalk A, Dugas M, Varghese J. Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines. Front Med (Lausanne) 2022; 9:960296. [PMID: 36082270 PMCID: PMC9445989 DOI: 10.3389/fmed.2022.960296] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Accepted: 08/03/2022] [Indexed: 11/17/2022] Open
Abstract
Background Intensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Widely used post hoc explanation methods also have inherent limitations. Few studies are evaluating inherently interpretable ML models for health care and involve clinicians in inspecting the trained model. Methods An inherently interpretable model for the prediction of 3 day ICU readmission was developed. We used explainable boosting machines that learn modular risk functions and which have already been shown to be suitable for the health care domain. We created a retrospective cohort of 15,589 ICU stays and 169 variables collected between 2006 and 2019 from the University Hospital Münster. A team of physicians inspected the model, checked the plausibility of each risk function, and removed problematic ones. We collected qualitative feedback during this process and analyzed the reasons for removing risk functions. The performance of the final explainable boosting machine was compared with a validated clinical score and three commonly used ML models. External validation was performed on the widely used Medical Information Mart for Intensive Care version IV database. Results The developed explainable boosting machine used 67 features and showed an area under the precision-recall curve of 0.119 ± 0.020 and an area under the receiver operating characteristic curve of 0.680 ± 0.025. It performed on par with state-of-the-art gradient boosting machines (0.123 ± 0.016, 0.665 ± 0.036) and outperformed the Simplified Acute Physiology Score II (0.084 ± 0.025, 0.607 ± 0.019), logistic regression (0.092 ± 0.026, 0.587 ± 0.016), and recurrent neural networks (0.095 ± 0.008, 0.594 ± 0.027). External validation confirmed that explainable boosting machines (0.221 ± 0.023, 0.760 ± 0.010) performed similarly to gradient boosting machines (0.232 ± 0.029, 0.772 ± 0.018). Evaluation of the model inspection showed that explainable boosting machines can be useful to detect and remove problematic risk functions. Conclusions We developed an inherently interpretable ML model for 3 day ICU readmission prediction that reached the state-of-the-art performance of black box models. Our results suggest that for low- to medium-dimensional datasets that are common in health care, it is feasible to develop ML models that allow a high level of human control without sacrificing performance.
Collapse
Affiliation(s)
- Stefan Hegselmann
- Institute of Medical Informatics, University of Münster, Münster, Germany
| | - Christian Ertmer
- Department of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, Germany
| | - Thomas Volkert
- Department of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, Germany
| | - Antje Gottschalk
- Department of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, Germany
| | - Martin Dugas
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Julian Varghese
- Institute of Medical Informatics, University of Münster, Münster, Germany
| |
Collapse
|
2
|
Xingwei W, Huan C, Mengting L, Lv Q, Jiaying Z, Enwu L, Jiuqun Z, Rongsheng T. A machine learning-based risk warning platform for potentially inappropriate prescriptions for elderly patients with cardiovascular disease. Front Pharmacol 2022; 13:804566. [PMID: 36034817 PMCID: PMC9402906 DOI: 10.3389/fphar.2022.804566] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Accepted: 07/11/2022] [Indexed: 11/17/2022] Open
Abstract
Potentially inappropriate prescribing (PIP), including potentially inappropriate medications (PIMs) and potential prescribing omissions (PPOs), is a major risk factor for adverse drug reactions (ADRs). Establishing a risk warning model for PIP to screen high-risk patients and implementing targeted interventions would significantly reduce the occurrence of PIP and adverse drug events. Elderly patients with cardiovascular disease hospitalized at the Sichuan Provincial People’s Hospital were included in the study. Information about PIP, PIM, and PPO was obtained by reviewing patient prescriptions according to the STOPP/START criteria (2nd edition). Data were divided into a training set and test set at a ratio of 8:2. Five sampling methods, three feature screening methods, and eighteen machine learning algorithms were used to handle data and establish risk warning models. A 10-fold cross-validation method was employed for internal validation in the training set, and the bootstrap method was used for external validation in the test set. The performances were assessed by area under the receiver operating characteristic curve (AUC), and the risk warning platform was developed based on the best models. The contributions of features were interpreted using SHapley Additive ExPlanation (SHAP). A total of 404 patients were included in the study (318 [78.7%] with PIP; 112 [27.7%] with PIM; and 273 [67.6%] with PPO). After data sampling and feature selection, 15 datasets were obtained and 270 risk warning models were built based on them to predict PIP, PPO, and PIM, respectively. External validation showed that the AUCs of the best model for PIP, PPO, and PIM were 0.8341, 0.7007, and 0.7061, respectively. The results suggested that angina, number of medications, number of diseases, and age were the key factors in the PIP risk warning model. The risk warning platform was established to predict PIP, PIM, and PPO, which has acceptable accuracy, prediction performance, and potential clinical application perspective.
Collapse
Affiliation(s)
- Wu Xingwei
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| | - Chang Huan
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Li Mengting
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Qin Lv
- Department of Pulmonary and Critical Care Medicine, Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital, Chengdu, China
| | - Zhang Jiaying
- Department of Western Pharmacy, Chengdu First People’s Hospital, Chengdu, China
| | - Long Enwu
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| | - Zhu Jiuqun
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
- *Correspondence: Zhu Jiuqun, ; Tong Rongsheng,
| | - Tong Rongsheng
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People’s Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
- *Correspondence: Zhu Jiuqun, ; Tong Rongsheng,
| |
Collapse
|
3
|
Xingwei W, Huan C, Mengting L, Lv Q, Jiaying Z, Enwu L, Jiuqun Z, Rongsheng T. A machine learning-based risk warning platform for potentially inappropriate prescriptions for elderly patients with cardiovascular disease. Front Pharmacol 2022. [PMID: 36034817 DOI: 10.3389/fphar.2022.804566.ecollection] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/05/2023] Open
Abstract
Potentially inappropriate prescribing (PIP), including potentially inappropriate medications (PIMs) and potential prescribing omissions (PPOs), is a major risk factor for adverse drug reactions (ADRs). Establishing a risk warning model for PIP to screen high-risk patients and implementing targeted interventions would significantly reduce the occurrence of PIP and adverse drug events. Elderly patients with cardiovascular disease hospitalized at the Sichuan Provincial People's Hospital were included in the study. Information about PIP, PIM, and PPO was obtained by reviewing patient prescriptions according to the STOPP/START criteria (2nd edition). Data were divided into a training set and test set at a ratio of 8:2. Five sampling methods, three feature screening methods, and eighteen machine learning algorithms were used to handle data and establish risk warning models. A 10-fold cross-validation method was employed for internal validation in the training set, and the bootstrap method was used for external validation in the test set. The performances were assessed by area under the receiver operating characteristic curve (AUC), and the risk warning platform was developed based on the best models. The contributions of features were interpreted using SHapley Additive ExPlanation (SHAP). A total of 404 patients were included in the study (318 [78.7%] with PIP; 112 [27.7%] with PIM; and 273 [67.6%] with PPO). After data sampling and feature selection, 15 datasets were obtained and 270 risk warning models were built based on them to predict PIP, PPO, and PIM, respectively. External validation showed that the AUCs of the best model for PIP, PPO, and PIM were 0.8341, 0.7007, and 0.7061, respectively. The results suggested that angina, number of medications, number of diseases, and age were the key factors in the PIP risk warning model. The risk warning platform was established to predict PIP, PIM, and PPO, which has acceptable accuracy, prediction performance, and potential clinical application perspective.
Collapse
Affiliation(s)
- Wu Xingwei
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| | - Chang Huan
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Li Mengting
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Qin Lv
- Department of Pulmonary and Critical Care Medicine, Sichuan Academy of Medical Sciences and Sichuan Provincial People's Hospital, Chengdu, China
| | - Zhang Jiaying
- Department of Western Pharmacy, Chengdu First People's Hospital, Chengdu, China
| | - Long Enwu
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| | - Zhu Jiuqun
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| | - Tong Rongsheng
- Personalized Drug Therapy Key Laboratory of Sichuan Province, Department of Pharmacy, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, Sichuan, China
| |
Collapse
|