1
|
Chang SC, Grunkemeier GL, Goldman JD, Wang M, McKelvey PA, Hadlock J, Wei Q, Diaz GA. A simplified pneumonia severity index (PSI) for clinical outcome prediction in COVID-19. PLoS One 2024; 19:e0303899. [PMID: 38771892 PMCID: PMC11108185 DOI: 10.1371/journal.pone.0303899] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Accepted: 05/02/2024] [Indexed: 05/23/2024] Open
Abstract
BACKGROUND The Pneumonia Score Index (PSI) was developed to estimate the risk of dying within 30 days of presentation for community-acquired pneumonia patients and is a strong predictor of 30-day mortality after COVID-19. However, three of its required 20 variables (skilled nursing home, altered mental status and pleural effusion) are not discreetly available in the electronic medical record (EMR), resulting in manual chart review for these 3 factors. The goal of this study is to compare a simplified 17-factor version (PSI-17) to the original (denoted PSI-20) in terms of prediction of 30-day mortality in COVID-19. METHODS In this retrospective cohort study, the hospitalized patients with confirmed SARS-CoV-2 infection between 2/28/20-5/28/20 were identified to compare the predictive performance between PSI-17 and PSI-20. Correlation was assessed between PSI-17 and PSI-20, and logistic regressions were performed for 30-day mortality. The predictive abilities were compared by discrimination, calibration, and overall performance. RESULTS Based on 1,138 COVID-19 patients, the correlation between PSI-17 and PSI-20 was 0.95. Univariate logistic regression showed that PSI-17 had performance similar to PSI-20, based on AUC, ICI and Brier Score. After adjusting for confounding variables by multivariable logistic regression, PSI-17 and PSI-20 had AUCs (95% CI) of 0.85 (0.83-0.88) and 0.86 (0.84-0.89), respectively, indicating no significant difference in AUC at significance level of 0.05. CONCLUSION PSI-17 and PSI-20 are equally effective predictors of 30-day mortality in terms of several performance metrics. PSI-17 can be obtained without the manual chart review, which allows for automated risk calculations within an EMR. PSI-17 can be easily obtained and may be a comparable alternative to PSI-20.
Collapse
Affiliation(s)
- Shu-Ching Chang
- Providence St. Joseph Health, Portland, Oregon, United States of America
| | - Gary L. Grunkemeier
- Division of Cardiothoracic Surgery, Oregon Health & Science University, Portland, OR, United States of America
| | - Jason D. Goldman
- Division of Infectious Diseases, Swedish Medical Center, Seattle, WA, United States of America
- Swedish Center for Research and Innovation, Swedish Medical Center, Seattle, WA, United States of America
- Division of Allergy and Infectious Diseases, University of Washington, Seattle, WA, United States of America
| | - Mansen Wang
- ClinChoice, Portland, OR, United States of America
| | - Paul A. McKelvey
- Providence Heart Institute, Providence St. Joseph Health, Portland, Oregon, United States of America
| | - Jennifer Hadlock
- Institute for Systems Biology, Seattle, Washington, United States of America
| | - Qi Wei
- Institute for Systems Biology, Seattle, Washington, United States of America
| | - George A. Diaz
- Division of Medicine, Section of Infectious Diseases, Providence Regional Medical Center Everett, Everett, WA, United States of America
- Washington State University Elson S. Floyd College of Medicine, Spokane, WA, United States of America
- Providence Research Network, Renton, WA, United States of America
| |
Collapse
|
2
|
Jeon ET, Lee HJ, Park TY, Jin KN, Ryu B, Lee HW, Kim DH. Machine learning-based prediction of in-ICU mortality in pneumonia patients. Sci Rep 2023; 13:11527. [PMID: 37460837 DOI: 10.1038/s41598-023-38765-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Accepted: 07/14/2023] [Indexed: 07/20/2023] Open
Abstract
Conventional severity-of-illness scoring systems have shown suboptimal performance for predicting in-intensive care unit (ICU) mortality in patients with severe pneumonia. This study aimed to develop and validate machine learning (ML) models for mortality prediction in patients with severe pneumonia. This retrospective study evaluated patients admitted to the ICU for severe pneumonia between January 2016 and December 2021. The predictive performance was analyzed by comparing the area under the receiver operating characteristic curve (AU-ROC) of ML models to that of conventional severity-of-illness scoring systems. Three ML models were evaluated: (1) logistic regression with L2 regularization, (2) gradient-boosted decision tree (LightGBM), and (3) multilayer perceptron (MLP). Among the 816 pneumonia patients included, 223 (27.3%) patients died. All ML models significantly outperformed the Simplified Acute Physiology Score II (AU-ROC: 0.650 [0.584-0.716] vs 0.820 [0.771-0.869] for logistic regression vs 0.827 [0.777-0.876] for LightGBM 0.838 [0.791-0.884] for MLP; P < 0.001). In the analysis for NRI, the LightGBM and MLP models showed superior reclassification compared with the logistic regression model in predicting in-ICU mortality in all length of stay in the ICU subgroups; all age subgroups; all subgroups with any APACHE II score, PaO2/FiO2 ratio < 200; all subgroups with or without history of respiratory disease; with or without history of CVA or dementia; treatment with mechanical ventilation, and use of inotropic agents. In conclusion, the ML models have excellent performance in predicting in-ICU mortality in patients with severe pneumonia. Moreover, this study highlights the potential advantages of selecting individual ML models for predicting in-ICU mortality in different subgroups.
Collapse
Affiliation(s)
- Eun-Tae Jeon
- Department of Radiology, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea
| | - Hyo Jin Lee
- Division of Respiratory and Critical Care, Department of Internal Medicine, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea
| | - Tae Yun Park
- Division of Respiratory and Critical Care, Department of Internal Medicine, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea
| | - Kwang Nam Jin
- Department of Radiology, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea
| | - Borim Ryu
- Center for Data Science, Biomedical Research Institute, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, Seoul, South Korea
| | - Hyun Woo Lee
- Division of Respiratory and Critical Care, Department of Internal Medicine, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea.
| | - Dong Hyun Kim
- Department of Radiology, Seoul National University College of Medicine, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, 5 Gil 20, Boramae-Road, Dongjak-gu, Seoul, South Korea.
| |
Collapse
|
3
|
Cai W, Xu J, Chen Y, Wu X, Zeng Y, Yu F. Performance of Machine Learning Algorithms for Predicting Disease Activity in Inflammatory Bowel Disease. Inflammation 2023:10.1007/s10753-023-01827-0. [PMID: 37171693 DOI: 10.1007/s10753-023-01827-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Revised: 04/17/2023] [Accepted: 04/24/2023] [Indexed: 05/13/2023]
Abstract
This study aimed to explore the effectiveness of predicting disease activity in patients with inflammatory bowel disease (IBD), using machine learning (ML) models. A retrospective research was undertaken on IBD patients who were admitted into the First Affiliated Hospital of Wenzhou Medical University between September 2011 and September 2019. At first, data were randomly split into a 3:1 ratio of training to test set. The least absolute shrinkage and selection operator (LASSO) algorithm was applied to reduce the dimension of variables. These variables were used to generate seven ML algorithms, namely random forests (RFs), adaptive boosting (AdaBoost), K-nearest neighbors (KNNs), support vector machines (SVMs), naïve Bayes (NB), ridge regression, and eXtreme gradient boosting (XGBoost) to train to predict disease activity in IBD patients. SHapley Additive exPlanation (SHAP) analysis was performed to rank variable importance. A total of 876 participants with IBD, consisting of 275 ulcerative colitis (UC) and 601 Crohn's disease (CD), were retrospectively enrolled in the study. Thirty-three variables were obtained from the clinical characteristics and laboratory tests of the participants. Finally, after LASSO analysis, 11 and 5 variables were screened out to construct ML models for CD and UC, respectively. All seven ML models performed well in predicting disease activity in the CD and UC test sets. Among these ML models, SVM was more effective in predicting disease activity in the CD group, whose AUC reached 0.975, sensitivity 0.947, specificity 0.920, and accuracy 0.933. AdaBoost performed best for the UC group, with an AUC of 0.911, sensitivity 0.844, specificity 0.875, and accuracy 0.855. ML algorithms were available and capable of predicting disease activity in IBD patients. Based on clinical and laboratory variables, ML algorithms demonstrate great promise in guiding physicians' decision-making.
Collapse
Affiliation(s)
- Weimin Cai
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Jun Xu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Yihan Chen
- Department of Gastroenterology and Hepatology, Wenzhou Central Hospital, Wenzhou, 325000, China
| | - Xiao Wu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Yuan Zeng
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Fujun Yu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China.
| |
Collapse
|
4
|
Jones BE, Ying J, Nevers MR, Alba PR, Patterson OV, Peterson KS, Rutter E, Christensen MA, Stern S, Jones MM, Gundlapalli A, Dean NC, Samore MC, Greene T. Trends in Illness Severity, Hospitalization, and Mortality for Community-Onset Pneumonia at 118 US Veterans Affairs Medical Centers. J Gen Intern Med 2022; 37:3839-3847. [PMID: 35266121 PMCID: PMC8906522 DOI: 10.1007/s11606-022-07413-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Accepted: 01/13/2022] [Indexed: 12/16/2022]
Abstract
BACKGROUND Deaths from pneumonia were decreasing globally prior to the COVID-19 pandemic, but it is unclear whether this was due to changes in patient populations, illness severity, diagnosis, hospitalization thresholds, or treatment. Using clinical data from the electronic health record among a national cohort of patients initially diagnosed with pneumonia, we examined temporal trends in severity of illness, hospitalization, and short- and long-term deaths. DESIGN Retrospective cohort PARTICIPANTS: All patients >18 years presenting to emergency departments (EDs) at 118 VA Medical Centers between 1/1/2006 and 12/31/2016 with an initial clinical diagnosis of pneumonia and confirmed by chest imaging report. EXPOSURES Year of encounter. MAIN MEASURES Hospitalization and 30-day and 90-day mortality. Illness severity was defined as the probability of each outcome predicted by machine learning predictive models using age, sex, comorbidities, vital signs, and laboratory data from encounters during years 2006-2007, and similar models trained on encounters from years 2015 to 2016. We estimated the changes in hospitalizations and 30-day and 90-day mortality between the first and the last 2 years of the study period accounted for by illness severity using time covariate decompositions with model estimates. RESULTS Among 196,899 encounters across the study period, hospitalization decreased from 71 to 63%, 30-day mortality 10 to 7%, 90-day mortality 16 to 12%, and 1-year mortality 29 to 24%. Comorbidity risk increased, but illness severity decreased. Decreases in illness severity accounted for 21-31% of the decrease in hospitalizations, and 45-47%, 32-24%, and 17-19% of the decrease in 30-day, 90-day, and 1-year mortality. Findings were similar among underrepresented patients and those with only hospital discharge diagnosis codes. CONCLUSIONS Outcomes for community-onset pneumonia have improved across the VA healthcare system after accounting for illness severity, despite an increase in cases and comorbidity burden.
Collapse
Affiliation(s)
- Barbara E Jones
- Division of Pulmonary & Critical Care, University of Utah, 50 North Medical Drive, Salt Lake City, UT, 84132, USA.
- VA Salt Lake City Healthcare System, Salt Lake City, USA.
| | - Jian Ying
- Division of Epidemiology, University of Utah, Salt Lake City, USA
| | | | - Patrick R Alba
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, VA Informatics and Computing Infrastructure, University of Utah, Salt Lake City, USA
| | - Olga V Patterson
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, VA Informatics and Computing Infrastructure, University of Utah, Salt Lake City, USA
| | - Kelly S Peterson
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, Veterans Health Administration Office of Analytics and Performance Integration, University of Utah, Salt Lake City, USA
| | - Elizabeth Rutter
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Emergency Medicine, University of Utah, Salt Lake City, USA
| | - Matthew A Christensen
- Division of Allergy, Pulmonary, & Critical Care Medicine, Vanderbilt University Medical Center, Nashville, USA
| | - Sarah Stern
- Division of Pulmonary & Critical Care, University of Utah, 50 North Medical Drive, Salt Lake City, UT, 84132, USA
- VA Salt Lake City Healthcare System, Salt Lake City, USA
| | - Makoto M Jones
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, University of Utah, Salt Lake City, USA
| | - Adi Gundlapalli
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, University of Utah, Salt Lake City, USA
| | - Nathan C Dean
- Division of Pulmonary & Critical Care, University of Utah, 50 North Medical Drive, Salt Lake City, UT, 84132, USA
| | - Matthew C Samore
- VA Salt Lake City Healthcare System, Salt Lake City, USA
- Division of Epidemiology, University of Utah, Salt Lake City, USA
| | - Tome Greene
- Department of Population Health Sciences, University of Utah, Salt Lake City, USA
| |
Collapse
|
5
|
Leiner J, Pellissier V, König S, Hohenstein S, Ueberham L, Nachtigall I, Meier-Hellmann A, Kuhlen R, Hindricks G, Bollmann A. Machine learning-derived prediction of in-hospital mortality in patients with severe acute respiratory infection: analysis of claims data from the German-wide Helios hospital network. Respir Res 2022; 23:264. [PMID: 36151525 PMCID: PMC9502925 DOI: 10.1186/s12931-022-02180-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Accepted: 09/05/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Severe acute respiratory infections (SARI) are the most common infectious causes of death. Previous work regarding mortality prediction models for SARI using machine learning (ML) algorithms that can be useful for both individual risk stratification and quality of care assessment is scarce. We aimed to develop reliable models for mortality prediction in SARI patients utilizing ML algorithms and compare its performances with a classic regression analysis approach. METHODS Administrative data (dataset randomly split 75%/25% for model training/testing) from years 2016-2019 of 86 German Helios hospitals was retrospectively analyzed. Inpatient SARI cases were defined by ICD-codes J09-J22. Three ML algorithms were evaluated and its performance compared to generalized linear models (GLM) by computing receiver operating characteristic area under the curve (AUC) and area under the precision-recall curve (AUPRC). RESULTS The dataset contained 241,988 inpatient SARI cases (75 years or older: 49%; male 56.2%). In-hospital mortality was 11.6%. AUC and AUPRC in the testing dataset were 0.83 and 0.372 for GLM, 0.831 and 0.384 for random forest (RF), 0.834 and 0.382 for single layer neural network (NNET) and 0.834 and 0.389 for extreme gradient boosting (XGBoost). Statistical comparison of ROC AUCs revealed a better performance of NNET and XGBoost as compared to GLM. CONCLUSION ML algorithms for predicting in-hospital mortality were trained and tested on a large real-world administrative dataset of SARI patients and showed good discriminatory performances. Broad application of our models in clinical routine practice can contribute to patients' risk assessment and quality management.
Collapse
Affiliation(s)
- Johannes Leiner
- Department of Electrophysiology, Heart Center Leipzig at University of Leipzig, Leipzig, Germany. .,Real World Evidence and Health Technology Assessment, Helios Health Institute, Berlin, Germany.
| | - Vincent Pellissier
- Real World Evidence and Health Technology Assessment, Helios Health Institute, Berlin, Germany
| | - Sebastian König
- Department of Electrophysiology, Heart Center Leipzig at University of Leipzig, Leipzig, Germany.,Real World Evidence and Health Technology Assessment, Helios Health Institute, Berlin, Germany
| | - Sven Hohenstein
- Real World Evidence and Health Technology Assessment, Helios Health Institute, Berlin, Germany
| | - Laura Ueberham
- Clinic for Cardiology, University Hospital Leipzig, Leipzig, Germany
| | - Irit Nachtigall
- Department of Infectious Diseases and Infection Prevention, Helios Hospital Emil-von-Behring, Berlin, Germany.,Institute of Hygiene and Environmental Medicine, Charité - Universitaetsmedizin Berlin, Berlin, Germany
| | | | | | - Gerhard Hindricks
- Department of Electrophysiology, Heart Center Leipzig at University of Leipzig, Leipzig, Germany
| | - Andreas Bollmann
- Department of Electrophysiology, Heart Center Leipzig at University of Leipzig, Leipzig, Germany.,Real World Evidence and Health Technology Assessment, Helios Health Institute, Berlin, Germany
| |
Collapse
|
6
|
Xu Z, Guo K, Chu W, Lou J, Chen C. Performance of Machine Learning Algorithms for Predicting Adverse Outcomes in Community-Acquired Pneumonia. Front Bioeng Biotechnol 2022; 10:903426. [PMID: 35845426 PMCID: PMC9278327 DOI: 10.3389/fbioe.2022.903426] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Accepted: 05/16/2022] [Indexed: 12/31/2022] Open
Abstract
Background: The ability to assess adverse outcomes in patients with community-acquired pneumonia (CAP) could improve clinical decision-making to enhance clinical practice, but the studies remain insufficient, and similarly, few machine learning (ML) models have been developed. Objective: We aimed to explore the effectiveness of predicting adverse outcomes in CAP through ML models. Methods: A total of 2,302 adults with CAP who were prospectively recruited between January 2012 and March 2015 across three cities in South America were extracted from DryadData. After a 70:30 training set: test set split of the data, nine ML algorithms were executed and their diagnostic accuracy was measured mainly by the area under the curve (AUC). The nine ML algorithms included decision trees, random forests, extreme gradient boosting (XGBoost), support vector machines, Naïve Bayes, K-nearest neighbors, ridge regression, logistic regression without regularization, and neural networks. The adverse outcomes included hospital admission, mortality, ICU admission, and one-year post-enrollment status. Results: The XGBoost algorithm had the best performance in predicting hospital admission. Its AUC reached 0.921, and accuracy, precision, recall, and F1-score were better than those of other models. In the prediction of ICU admission, a model trained with the XGBoost algorithm showed the best performance with AUC 0.801. XGBoost algorithm also did a good job at predicting one-year post-enrollment status. The results of AUC, accuracy, precision, recall, and F1-score indicated the algorithm had high accuracy and precision. In addition, the best performance was seen by the neural network algorithm when predicting death (AUC 0.831). Conclusions: ML algorithms, particularly the XGBoost algorithm, were feasible and effective in predicting adverse outcomes of CAP patients. The ML models based on available common clinical features had great potential to guide individual treatment and subsequent clinical decisions.
Collapse
Affiliation(s)
- Zhixiao Xu
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Kun Guo
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Weiwei Chu
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Jingwen Lou
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Chengshui Chen
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China.,The Interventional Pulmonary Key Laboratory of Zhejiang Province, Wenzhou, China
| |
Collapse
|