1
|
Zeng J, Zhang D, Lin S, Su X, Wang P, Zhao Y, Zheng Z. Comparative analysis of machine learning vs. traditional modeling approaches for predicting in-hospital mortality after cardiac surgery: temporal and spatial external validation based on a nationwide cardiac surgery registry. EUROPEAN HEART JOURNAL. QUALITY OF CARE & CLINICAL OUTCOMES 2024; 10:121-131. [PMID: 37218710 DOI: 10.1093/ehjqcco/qcad028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Revised: 05/12/2023] [Accepted: 05/21/2023] [Indexed: 05/24/2023]
Abstract
AIMS Preoperative risk assessment is crucial for cardiac surgery. Although previous studies suggested machine learning (ML) may improve in-hospital mortality predictions after cardiac surgery compared to traditional modeling approaches, the validity is doubted due to lacking external validation, limited sample sizes, and inadequate modeling considerations. We aimed to assess predictive performance between ML and traditional modelling approaches, while addressing these major limitations. METHODS AND RESULTS Adult cardiac surgery cases (n = 168 565) between 2013 and 2018 in the Chinese Cardiac Surgery Registry were used to develop, validate, and compare various ML vs. logistic regression (LR) models. The dataset was split for temporal (2013-2017 for training, 2018 for testing) and spatial (geographically-stratified random selection of 83 centers for training, 22 for testing) experiments, respectively. Model performances were evaluated in testing sets for discrimination and calibration. The overall in-hospital mortality was 1.9%. In the temporal testing set (n = 32 184), the best-performing ML model demonstrated a similar area under the receiver operating characteristic curve (AUC) of 0.797 (95% CI 0.779-0.815) to the LR model (AUC 0.791 [95% CI 0.775-0.808]; P = 0.12). In the spatial experiment (n = 28 323), the best ML model showed a statistically better but modest performance improvement (AUC 0.732 [95% CI 0.710-0.754]) than LR (AUC 0.713 [95% CI 0.691-0.737]; P = 0.002). Varying feature selection methods had relatively smaller effects on ML models. Most ML and LR models were significantly miscalibrated. CONCLUSION ML provided only marginal improvements over traditional modelling approaches in predicting cardiac surgery mortality with routine preoperative variables, which calls for more judicious use of ML in practice.
Collapse
Affiliation(s)
- Juntong Zeng
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
| | - Danwei Zhang
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
- Department of Cardiac Surgery, Fujian Children's Hospital (Fujian Branch of Shanghai Children's Medical Center), College of Clinical Medicine for Obstetrics & Gynecology and Pediatrics, Fujian Medical University, 966 Hengyu Road, Jinan, Fuzhou, 350014, People's Republic of China
| | - Shen Lin
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
- Department of Cardiovascular Surgery, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
| | - Xiaoting Su
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
| | - Peng Wang
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
| | - Yan Zhao
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
| | - Zhe Zheng
- National Clinical Research Center of Cardiovascular Diseases, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- State Key Laboratory of Cardiovascular Disease, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Chinese Academy of Medical Sciences and Peking Union Medical College, 9 Dongdansantiao, Dongcheng, Beijing, 100730, People's Republic of China
- Department of Cardiovascular Surgery, Fuwai Hospital, National Center for Cardiovascular Diseases, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
- Key Laboratory of Coronary Heart Disease Risk Prediction and Precision Therapy, Chinese Academy of Medical Sciences and Peking Union Medical College, 167 Beilishi Road, Xicheng, Beijing, 100037, People's Republic of China
| |
Collapse
|
2
|
Dong T, Sinha S, Zhai B, Fudulu DP, Chan J, Narayan P, Judge A, Caputo M, Dimagli A, Benedetto U, Angelini GD. Cardiac surgery risk prediction using ensemble machine learning to incorporate legacy risk scores: A benchmarking study. Digit Health 2023; 9:20552076231187605. [PMID: 37492033 PMCID: PMC10363892 DOI: 10.1177/20552076231187605] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2023] [Accepted: 06/23/2023] [Indexed: 07/27/2023] Open
Abstract
Objective The introduction of new clinical risk scores (e.g. European System for Cardiac Operative Risk Evaluation (EuroSCORE) II) superseding original scores (e.g. EuroSCORE I) with different variable sets typically result in disparate datasets due to high levels of missingness for new score variables prior to time of adoption. Little is known about the use of ensemble learning to incorporate disparate data from legacy scores. We tested the hypothesised that Homogenenous and Heterogeneous Machine Learning (ML) ensembles will have better performance than ensembles of Dynamic Model Averaging (DMA) for combining knowledge from EuroSCORE I legacy data with EuroSCORE II data to predict cardiac surgery risk. Methods Using the National Adult Cardiac Surgery Audit dataset, we trained 12 different base learner models, based on two different variable sets from either EuroSCORE I (LogES) or EuroScore II (ES II), partitioned by the time of score adoption (1996-2016 or 2012-2016) and evaluated on holdout set (2017-2019). These base learner models were ensembled using nine different combinations of six ML algorithms to produce homogeneous or heterogeneous ensembles. Performance was assessed using a consensus metric. Results Xgboost homogenous ensemble (HE) was the highest performing model (clinical effectiveness metric (CEM) 0.725) with area under the curve (AUC) (0.8327; 95% confidence interval (CI) 0.8323-0.8329) followed by Random Forest HE (CEM 0.723; AUC 0.8325; 95%CI 0.8320-0.8326). Across different heterogenous ensembles, significantly better performance was obtained by combining siloed datasets across time (CEM 0.720) than building ensembles of either 1996-2011 (t-test adjusted, p = 1.67×10-6) or 2012-2019 (t-test adjusted, p = 1.35×10-193) datasets alone. Conclusions Both homogenous and heterogenous ML ensembles performed significantly better than DMA ensemble of Bayesian Update models. Time-dependent ensemble combination of variables, having differing qualities according to time of score adoption, enabled previously siloed data to be combined, leading to increased power, clinical interpretability of variables and usage of data.
Collapse
Affiliation(s)
- Tim Dong
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Shubhra Sinha
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Ben Zhai
- School of Computing Science, Northumbria University, Newcastle upon Tyne, UK
| | - Daniel P Fudulu
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Jeremy Chan
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Pradeep Narayan
- Department of Cardiac Surgery, Rabindranath Tagore International Institute of Cardiac Sciences, Kolkata, India
| | - Andy Judge
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Massimo Caputo
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Arnaldo Dimagli
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Umberto Benedetto
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Gianni D Angelini
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| |
Collapse
|
3
|
Bi S, Chen S, Li J, Gu J. Machine learning-based prediction of in-hospital mortality for post cardiovascular surgery patients admitting to intensive care unit: a retrospective observational cohort study based on a large multi-center critical care database. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 226:107115. [PMID: 36126435 DOI: 10.1016/j.cmpb.2022.107115] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/11/2021] [Revised: 07/15/2022] [Accepted: 09/04/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVES The acute physiology and chronic health evaluation-IV model (APACHE-IV), and the sequential organ failure assessment (SOFA) score are two traditional severity assessment systems that can be applied to cardiac surgery patients admitted to intensive care units (ICUs). However, the performance of machine learning approaches in post cardiovascular surgery (PCS) patients admitted to the ICU remains unknown. METHODS The clinical data of adult subjects were collected from the eICU database. Seven models were constructed based on the training set (70% random sample) for predicting hospital mortality, including two traditional models based on APACHE-IV and SOFA scores and five machine learning models. We measured the models' performance in the remaining 30% of the sample by computing AUC-ROC values, prospective prediction results, and decision curves and compared the models with net reclassification improvement. RESULTS This study included 5860 PCS patients. The AUC-ROC value of the Xgboost model significantly outperformed the APACHE-IV and SOFA scores (0.12 [0.06-0.17] p < 0.01, 0.18 [0.1-0.26] p < 0.01 respectively). The use of ML models would also gain more clinical net benefits than traditional models based on decision curve analysis. There was a significant improvement in integrated discrimination when comparing the backward stepwise linear regression model with the APACHE-IV model (0.11 [0.05, 0.16], p < 0.01) and SOFA model (0.12 [0.06, 0.17], p < 0.01). CONCLUSIONS In conclusion, the predictive ability of ML models was better than that of traditional models. The present study suggested that developing advanced prognosis prediction tools could support clinical decision-making in the ICU for PCS patients.
Collapse
Affiliation(s)
- Siwei Bi
- Department of Burn and Plastic Surgery, West China Hospital, Sichuan University, Chengdu 610041, China
| | - Shanshan Chen
- West China School of Medicine, Sichuan University, Chengdu 610041, China
| | - Jingyi Li
- West China School of Medicine, Sichuan University, Chengdu 610041, China
| | - Jun Gu
- Department of Cardiovascular Surgery, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China.
| |
Collapse
|
4
|
Yu Y, Peng C, Zhang Z, Shen K, Zhang Y, Xiao J, Xi W, Wang P, Rao J, Jin Z, Wang Z. Machine Learning Methods for Predicting Long-Term Mortality in Patients After Cardiac Surgery. Front Cardiovasc Med 2022; 9:831390. [PMID: 35592400 PMCID: PMC9110683 DOI: 10.3389/fcvm.2022.831390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2021] [Accepted: 03/21/2022] [Indexed: 11/21/2022] Open
Abstract
Objective: This study aims to construct and validate several machine learning (ML) algorithms to predict long-term mortality and identify risk factors in unselected patients post-cardiac surgery. Methods The Medical Information Mart for Intensive Care (MIMIC-III) database was used to perform a retrospective administrative database study. Candidate predictors consisted of the demographics, comorbidity, vital signs, laboratory test results, scoring systems, and treatment information on the first day of ICU admission. Four-year mortality was set as the study outcome. We used the ML methods of logistic regression (LR), artificial neural network (NNET), naïve bayes (NB), gradient boosting machine (GBM), adapting boosting (Ada), random forest (RF), bagged trees (BT), and eXtreme Gradient Boosting (XGB). The prognostic capacity and clinical utility of these ML models were compared using the area under the receiver operating characteristic curves (AUC), calibration curves, and decision curve analysis (DCA). Results Of 7,368 patients in MIMIC-III included in the final cohort, a total of 1,337 (18.15%) patients died during a 4-year follow-up. Among 65 variables extracted from the database, a total of 25 predictors were selected using recursive feature elimination and included in the subsequent analysis. The Ada model performed best among eight models in both discriminatory ability with the highest AUC of 0.801 and goodness of fit (visualized by calibration curve). Moreover, the DCA shows that the net benefit of the RF, Ada, and BT models surpassed that of other ML models for almost all threshold probability values. Additionally, through the Ada technique, we determined that red blood cell distribution width (RDW), blood urea nitrogen (BUN), SAPS II, anion gap (AG), age, urine output, chloride, creatinine, congestive heart failure, and SOFA were the Top 10 predictors in the feature importance rankings. Conclusions The Ada model performs best in predicting 4-year mortality after cardiac surgery among the eight ML models, which might have significant application in the development of early warning systems for patients following operations.
Collapse
Affiliation(s)
- Yue Yu
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Chi Peng
- Department of Health Statistics, Naval Medical University, Shanghai, China
| | - Zhiyuan Zhang
- Department of Cardiothoracic Surgery, No. 988 Hospital of Joint Logistic Support Force of PLA, Zhengzhou, China
| | - Kejia Shen
- Department of Personnel Administration, Second Affiliated Hospital of Naval Medical University, Shanghai, China
| | - Yufeng Zhang
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Jian Xiao
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Wang Xi
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Pei Wang
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Jin Rao
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
| | - Zhichao Jin
- Department of Health Statistics, Naval Medical University, Shanghai, China
- *Correspondence: Zhichao Jin
| | - Zhinong Wang
- Department of Cardiothoracic Surgery, Changzheng Hospital, Naval Medical University, Shanghai, China
- Zhinong Wang
| |
Collapse
|
5
|
Agwar FD, Tekleab AM. Heart surgery by the locals in resource-limited settings: The experience from Ethiopia. JTCVS OPEN 2022; 9:98-105. [PMID: 36003472 PMCID: PMC9390689 DOI: 10.1016/j.xjon.2022.01.004] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/11/2021] [Accepted: 01/12/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND In developing countries, despite its demand is high, heart surgery is not always accessible to the neediest patients. We aimed to describe the early outcomes of heart surgeries that were performed by a local cardiac surgical team in Addis Ababa, Ethiopia. METHODS Data were collected through chart abstraction of patients who underwent heart surgery from the period of June 2017 to July 2021 by the same local cardiac surgical team at 3 centers in Addis Ababa, Ethiopia. Data were analyzed using the Statistical Package for the Social Sciences for Windows version 20.0. RESULTS A total of 290 patients who underwent heart surgery during the specified period were included in the study. Of the total, 192 patients underwent valve surgery (177 were patients with rheumatic valvular disease and 15 were valve surgeries with other causes) with a 30-day mortality rate of 9 (4.7%), 33 patients underwent coronary artery bypass graft with a 30-day mortality rate of 3 (9.1%), 58 patients underwent repair for congenital heart diseases with no 30-day mortality. Specifically, button Bentall was done for 1 patient; maze procedure was done for 2 patients along with mitral valve surgery, and a total of 7 out of 290 (2.4%) underwent redo heart surgery. The overall procedure-related mortality was 4.1%. CONCLUSIONS In addition to operating on a large number of cardiac patients, the local cardiac surgical team was able to do complex surgical procedures such as button Bentall, left maze procedure, redo valve surgeries, and coronary artery bypass graft surgery in a resource-limited setup. The overall patient outcome was comparable to reports from other centers.
Collapse
Affiliation(s)
| | - Atnafu Mekonnen Tekleab
- Department of Pediatrics and Child Health, St Paul's Hospital Millennium Medical College, Addis Ababa, Ethiopia
| |
Collapse
|
6
|
Rellum SR, Schuurmans J, van der Ven WH, Eberl S, Driessen AHG, Vlaar APJ, Veelo DP. Machine learning methods for perioperative anesthetic management in cardiac surgery patients: a scoping review. J Thorac Dis 2022; 13:6976-6993. [PMID: 35070381 PMCID: PMC8743411 DOI: 10.21037/jtd-21-765] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2021] [Accepted: 08/27/2021] [Indexed: 12/27/2022]
Abstract
Background Machine learning (ML) is developing fast with promising prospects within medicine and already has several applications in perioperative care. We conducted a scoping review to examine the extent and potential limitations of ML implementation in perioperative anesthetic care, specifically in cardiac surgery patients. Methods We mapped the current literature by searching three databases: MEDLINE (Ovid), EMBASE (Ovid), and Cochrane Library. Articles were eligible if they reported on perioperative ML use in the field of cardiac surgery with relevance to anesthetic practices. Data on the applicability of ML and comparability to conventional statistical methods were extracted. Results Forty-six articles on ML relevant to the work of the anesthesiologist in cardiac surgery were identified. Three main categories emerged: (I) event and risk prediction, (II) hemodynamic monitoring, and (III) automation of echocardiography. Prediction models based on ML tend to behave similarly to conventional statistical methods. Using dynamic hemodynamic or ultrasound data in ML models, however, shifts the potential to promising results. Conclusions ML in cardiac surgery is increasingly used in perioperative anesthetic management. The majority is used for prediction purposes similar to conventional clinical scores. Remarkable ML model performances are achieved when using real-time dynamic parameters. However, beneficial clinical outcomes of ML integration have yet to be determined. Nonetheless, the first steps introducing ML in perioperative anesthetic care for cardiac surgery have been taken.
Collapse
Affiliation(s)
- Santino R Rellum
- Department of Anesthesiology, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands.,Department of Intensive Care, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Jaap Schuurmans
- Department of Anesthesiology, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands.,Department of Intensive Care, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Ward H van der Ven
- Department of Anesthesiology, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands.,Department of Intensive Care, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Susanne Eberl
- Department of Anesthesiology, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Antoine H G Driessen
- Department of Cardiothoracic Surgery, Heart Center, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Alexander P J Vlaar
- Department of Intensive Care, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| | - Denise P Veelo
- Department of Anesthesiology, Amsterdam UMC, Location AMC, Amsterdam, The Netherlands
| |
Collapse
|
7
|
Benedetto U, Sinha S, Lyon M, Dimagli A, Gaunt TR, Angelini G, Sterne J. Can machine learning improve mortality prediction following cardiac surgery? Eur J Cardiothorac Surg 2021; 58:1130-1136. [PMID: 32810233 DOI: 10.1093/ejcts/ezaa229] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/25/2020] [Revised: 05/20/2020] [Accepted: 05/26/2020] [Indexed: 01/07/2023] Open
Abstract
OBJECTIVES Interest in the clinical usefulness of machine learning for risk prediction has bloomed recently. Cardiac surgery patients are at high risk of complications and therefore presurgical risk assessment is of crucial relevance. We aimed to compare the performance of machine learning algorithms over traditional logistic regression (LR) model to predict in-hospital mortality following cardiac surgery. METHODS A single-centre data set of prospectively collected information from patients undergoing adult cardiac surgery from 1996 to 2017 was split into 70% training set and 30% testing set. Prediction models were developed using neural network, random forest, naive Bayes and retrained LR based on features included in the EuroSCORE. Discrimination was assessed using area under the receiver operating characteristic curve, and calibration analysis was undertaken using the calibration belt method. Model calibration drift was assessed by comparing Goodness of fit χ2 statistics observed in 2 equal bins from the testing sample ordered by procedure date. RESULTS A total of 28 761 cardiac procedures were performed during the study period. The in-hospital mortality rate was 2.7%. Retrained LR [area under the receiver operating characteristic curve 0.80; 95% confidence interval (CI) 0.77-0.83] and random forest model (0.80; 95% CI 0.76-0.83) showed the best discrimination. All models showed significant miscalibration. Retrained LR proved to have the weakest calibration drift. CONCLUSIONS Our findings do not support the hypothesis that machine learning methods provide advantage over LR model in predicting operative mortality after cardiac surgery.
Collapse
Affiliation(s)
- Umberto Benedetto
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK.,NIHR Bristol Biomedical Research Centre, University of Bristol, University Hospitals Bristol NHS Foundation Trust, Bristol, UK
| | - Shubhra Sinha
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Matt Lyon
- NIHR Bristol Biomedical Research Centre, University of Bristol, University Hospitals Bristol NHS Foundation Trust, Bristol, UK.,Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.,MRC Integrative Epidemiology Unit, University of Bristol, Bristol, UK
| | - Arnaldo Dimagli
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK
| | - Tom R Gaunt
- NIHR Bristol Biomedical Research Centre, University of Bristol, University Hospitals Bristol NHS Foundation Trust, Bristol, UK.,Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.,MRC Integrative Epidemiology Unit, University of Bristol, Bristol, UK
| | - Gianni Angelini
- Translational Health Sciences, Bristol Heart Institute, University of Bristol, Bristol, UK.,NIHR Bristol Biomedical Research Centre, University of Bristol, University Hospitals Bristol NHS Foundation Trust, Bristol, UK
| | - Jonathan Sterne
- NIHR Bristol Biomedical Research Centre, University of Bristol, University Hospitals Bristol NHS Foundation Trust, Bristol, UK.,Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.,MRC Integrative Epidemiology Unit, University of Bristol, Bristol, UK
| |
Collapse
|
8
|
Zhenya Q, Zhang Z. A hybrid cost-sensitive ensemble for heart disease prediction. BMC Med Inform Decis Mak 2021; 21:73. [PMID: 33632225 PMCID: PMC7905907 DOI: 10.1186/s12911-021-01436-7] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2020] [Accepted: 02/11/2021] [Indexed: 11/23/2022] Open
Abstract
BACKGROUND Heart disease is the primary cause of morbidity and mortality in the world. It includes numerous problems and symptoms. The diagnosis of heart disease is difficult because there are too many factors to analyze. What's more, the misclassification cost could be very high. METHODS A cost-sensitive ensemble method was proposed to improve the efficiency of diagnosis and reduce the misclassification cost. The proposed method contains five heterogeneous classifiers: random forest, logistic regression, support vector machine, extreme learning machine and k-nearest neighbor. T-test was used to investigate if the performance of the ensemble was better than individual classifiers and the contribution of Relief algorithm. RESULTS The best performance was achieved by the proposed method according to ten-fold cross validation. The statistical tests demonstrated that the performance of the proposed ensemble was significantly superior to individual classifiers, and the efficiency of classification was distinctively improved by Relief algorithm. CONCLUSIONS The proposed ensemble gained significantly better results compared with individual classifiers and previous studies, which implies that it can be used as a promising alternative tool in medical decision making for heart disease diagnosis.
Collapse
Affiliation(s)
- Qi Zhenya
- College of Management and Economics, Tianjin University, Nankai District, Tianjin, 300072 People’s Republic of China
| | - Zuoru Zhang
- School of Mathematical Science, Hebei Normal University, Yuhua District, Shijiazhuang, 050024 People’s Republic of China
| |
Collapse
|
9
|
Validation and quality measurements for STS, EuroSCORE II and a regional risk model in Brazilian patients. PLoS One 2020; 15:e0238737. [PMID: 32911513 PMCID: PMC7482975 DOI: 10.1371/journal.pone.0238737] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2020] [Accepted: 08/22/2020] [Indexed: 12/13/2022] Open
Abstract
OBJECTIVES The objectives of this study were to describe a novel statewide registry for cardiac surgery in Brazil (REPLICCAR), to compare a regional risk model (SPScore) with EuroSCORE II and STS, and to understand where quality improvement and safety initiatives can be implemented. METHODS A total of 11 sites in the state of São Paulo, Brazil, formed an online registry platform to capture information on risk factors and outcomes after cardiac surgery procedures for all consecutive patients. EuroSCORE II and STS values were calculated for each patient. An SPScore model was designed and compared with EuroSCORE II and STS to predict 30-day outcomes: death, reoperation, readmission, and any morbidity. RESULTS A total of 5222 patients were enrolled in this study between November 2013 and December 2017. The observed 30-day mortality rate was 7.6%. Most patients were older, overweight, and classified as New York Heart Association (NYHA) functional class III; 14.5% of the patient population had a positive diagnosis of rheumatic heart disease, 10.9% had insulin-dependent diabetes, and 19 individuals had a positive diagnosis of Chagas disease. When evaluating the prediction performance, we found that SPScore outperformed EuroSCORE II and STS in the prediction of mortality (0.90 vs. 0.76 and 0.77), reoperation (0.84 vs. 0.60 and 0.56), readmission (0.84 vs. 0.55 and 0.51), and any morbidity (0.80 vs. 0.65 and 0.64), respectively (p<0.001). CONCLUSIONS The REPLICCAR registry might stimulate the creation of other cardiac surgery registries in developing countries, ultimately improving the regional quality of care provided to patients.
Collapse
|
10
|
Chang Junior J, Binuesa F, Caneo LF, Turquetto ALR, Arita ECTC, Barbosa AC, Fernandes AMDS, Trindade EM, Jatene FB, Dossou PE, Jatene MB. Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study. PLoS One 2020; 15:e0238199. [PMID: 32886688 PMCID: PMC7473591 DOI: 10.1371/journal.pone.0238199] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2020] [Accepted: 08/11/2020] [Indexed: 12/29/2022] Open
Abstract
BACKGROUND Congenital heart disease accounts for almost a third of all major congenital anomalies. Congenital heart defects have a significant impact on morbidity, mortality and health costs for children and adults. Research regarding the risk of pre-surgical mortality is scarce. OBJECTIVES Our goal is to generate a predictive model calculator adapted to the regional reality focused on individual mortality prediction among patients with congenital heart disease undergoing cardiac surgery. METHODS Two thousand two hundred forty CHD consecutive patients' data from InCor's heart surgery program was used to develop and validate the preoperative risk-of-death prediction model of congenital patients undergoing heart surgery. There were six artificial intelligence models most cited in medical references used in this study: Multilayer Perceptron (MLP), Random Forest (RF), Extra Trees (ET), Stochastic Gradient Boosting (SGB), Ada Boost Classification (ABC) and Bag Decision Trees (BDT). RESULTS The top performing areas under the curve were achieved using Random Forest (0.902). Most influential predictors included previous admission to ICU, diagnostic group, patient's height, hypoplastic left heart syndrome, body mass, arterial oxygen saturation, and pulmonary atresia. These combined predictor variables represent 67.8% of importance for the risk of mortality in the Random Forest algorithm. CONCLUSIONS The representativeness of "hospital death" is greater in patients up to 66 cm in height and body mass index below 13.0 for InCor's patients. The proportion of "hospital death" declines with the increased arterial oxygen saturation index. Patients with prior hospitalization before surgery had higher "hospital death" rates than who did not required such intervention. The diagnoses groups having the higher fatal outcomes probability are aligned with the international literature. A web application is presented where researchers and providers can calculate predicted mortality based on the CgntSCORE on any web browser or smartphone.
Collapse
Affiliation(s)
- João Chang Junior
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
- Fundação Armando Alvares Penteado–FAAP, São Paulo, Brazil
- Escola Superior de Engenharia e Gestão–ESEG, São Paulo, Brazil
| | - Fábio Binuesa
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| | - Luiz Fernando Caneo
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| | - Aida Luiza Ribeiro Turquetto
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
- Health Technology Assessment Center of Clinics Hospital–NATS-HCFMUSP, São Paulo, Brazil
| | | | - Aline Cristina Barbosa
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| | - Alfredo Manoel da Silva Fernandes
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| | - Evelinda Marramon Trindade
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
- Health Technology Assessment Center of Clinics Hospital–NATS-HCFMUSP, São Paulo, Brazil
- São Paulo State Health Secretariat–SES-SP, São Paulo, Brazil
| | - Fábio Biscegli Jatene
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| | - Paul-Eric Dossou
- Institut Catholique des Arts et Métiers–Icam, Paris-Sénart, France
| | - Marcelo Biscegli Jatene
- Department of Cardiovascular Surgery—Pediatric Cardiac Unit, Heart Institute of University of São Paulo Medical School—HCFMUSP—InCor, São Paulo, Brazil
| |
Collapse
|
11
|
Shahian DM, Lippmann RP. Commentary: Machine learning and cardiac surgery risk prediction. J Thorac Cardiovasc Surg 2020; 163:2090-2092. [PMID: 32951875 DOI: 10.1016/j.jtcvs.2020.08.058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/18/2020] [Revised: 08/18/2020] [Accepted: 08/18/2020] [Indexed: 10/23/2022]
Affiliation(s)
- David M Shahian
- Division of Cardiac Surgery, Department of Surgery, and Center for Quality and Safety, Massachusetts General Hospital, Boston, Mass.
| | - Richard P Lippmann
- Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Mass
| |
Collapse
|
12
|
Benedetto U, Dimagli A, Sinha S, Cocomello L, Gibbison B, Caputo M, Gaunt T, Lyon M, Holmes C, Angelini GD. Machine learning improves mortality risk prediction after cardiac surgery: Systematic review and meta-analysis. J Thorac Cardiovasc Surg 2020; 163:2075-2087.e9. [PMID: 32900480 DOI: 10.1016/j.jtcvs.2020.07.105] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/23/2020] [Revised: 07/16/2020] [Accepted: 07/30/2020] [Indexed: 02/01/2023]
Abstract
BACKGROUND Interest in the usefulness of machine learning (ML) methods for outcomes prediction has continued to increase in recent years. However, the advantage of advanced ML model over traditional logistic regression (LR) remains controversial. We performed a systematic review and meta-analysis of studies comparing the discrimination accuracy between ML models versus LR in predicting operative mortality following cardiac surgery. METHODS The present systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement. Discrimination ability was assessed using the C-statistic. Pooled C-statistics and its 95% credibility interval for ML models and LR were obtained were obtained using a Bayesian framework. Pooled estimates for ML models and LR were compared to inform on difference between the 2 approaches. RESULTS We identified 459 published citations of which 15 studies met inclusion criteria and were used for the quantitative and qualitative analysis. When the best ML model from individual study was used, meta-analytic estimates showed that ML were associated with a significantly higher C-statistic (ML, 0.88; 95% credibility interval, 0.83-0.93 vs LR, 0.81; 95% credibility interval, 0.77-0.85; P = .03). When individual ML algorithms were instead selected, we found a nonsignificant trend toward better prediction with each of ML algorithms. We found no evidence of publication bias (P = .70). CONCLUSIONS The present findings suggest that when compared with LR, ML models provide better discrimination in mortality prediction after cardiac surgery. However, the magnitude and clinical influence of such an improvement remains uncertain.
Collapse
Affiliation(s)
- Umberto Benedetto
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom.
| | - Arnaldo Dimagli
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| | - Shubhra Sinha
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| | - Lucia Cocomello
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| | - Ben Gibbison
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| | - Massimo Caputo
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| | - Tom Gaunt
- Population Health Sciences, University of Bristol, London, United Kingdom
| | - Matt Lyon
- Population Health Sciences, University of Bristol, London, United Kingdom
| | - Chris Holmes
- Department of Statistics, University of Oxford, Oxford, United Kingdom
| | - Gianni D Angelini
- Department of Translational Health Sciences, Bristol Heart Institute, University of Bristol, London, United Kingdom
| |
Collapse
|
13
|
Antunes MJ. Commentary: Repairing the rheumatic mitral valve in the young—definitively worthwhile! JTCVS OPEN 2020; 1:31-32. [PMID: 36003199 PMCID: PMC9390539 DOI: 10.1016/j.xjon.2020.02.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/25/2020] [Revised: 02/25/2020] [Accepted: 02/26/2020] [Indexed: 11/18/2022]
Affiliation(s)
- Manuel J. Antunes
- Address for reprints: Manuel J. Antunes, MD, PhD, DSc, Faculty of Medicine, University of Coimbra, 3000-075 Coimbra, Portugal.
| |
Collapse
|