Predictive performance of six mortality risk scores and the development of a novel model in a prospective cohort of patients undergoing valve surgery secondary to rheumatic fever

Comparative analysis of machine learning vs. traditional modeling approaches for predicting in-hospital mortality after cardiac surgery: temporal and spatial external validation based on a nationwide cardiac surgery registry

AIMS

Preoperative risk assessment is crucial for cardiac surgery. Although previous studies suggested machine learning (ML) may improve in-hospital mortality predictions after cardiac surgery compared to traditional modeling approaches, the validity is doubted due to lacking external validation, limited sample sizes, and inadequate modeling considerations. We aimed to assess predictive performance between ML and traditional modelling approaches, while addressing these major limitations.

METHODS AND RESULTS

Adult cardiac surgery cases (n = 168 565) between 2013 and 2018 in the Chinese Cardiac Surgery Registry were used to develop, validate, and compare various ML vs. logistic regression (LR) models. The dataset was split for temporal (2013-2017 for training, 2018 for testing) and spatial (geographically-stratified random selection of 83 centers for training, 22 for testing) experiments, respectively. Model performances were evaluated in testing sets for discrimination and calibration. The overall in-hospital mortality was 1.9%. In the temporal testing set (n = 32 184), the best-performing ML model demonstrated a similar area under the receiver operating characteristic curve (AUC) of 0.797 (95% CI 0.779-0.815) to the LR model (AUC 0.791 [95% CI 0.775-0.808]; P = 0.12). In the spatial experiment (n = 28 323), the best ML model showed a statistically better but modest performance improvement (AUC 0.732 [95% CI 0.710-0.754]) than LR (AUC 0.713 [95% CI 0.691-0.737]; P = 0.002). Varying feature selection methods had relatively smaller effects on ML models. Most ML and LR models were significantly miscalibrated.

CONCLUSION

ML provided only marginal improvements over traditional modelling approaches in predicting cardiac surgery mortality with routine preoperative variables, which calls for more judicious use of ML in practice.

Cardiac surgery risk prediction using ensemble machine learning to incorporate legacy risk scores: A benchmarking study

Objective

The introduction of new clinical risk scores (e.g. European System for Cardiac Operative Risk Evaluation (EuroSCORE) II) superseding original scores (e.g. EuroSCORE I) with different variable sets typically result in disparate datasets due to high levels of missingness for new score variables prior to time of adoption. Little is known about the use of ensemble learning to incorporate disparate data from legacy scores. We tested the hypothesised that Homogenenous and Heterogeneous Machine Learning (ML) ensembles will have better performance than ensembles of Dynamic Model Averaging (DMA) for combining knowledge from EuroSCORE I legacy data with EuroSCORE II data to predict cardiac surgery risk.

Methods

Using the National Adult Cardiac Surgery Audit dataset, we trained 12 different base learner models, based on two different variable sets from either EuroSCORE I (LogES) or EuroScore II (ES II), partitioned by the time of score adoption (1996-2016 or 2012-2016) and evaluated on holdout set (2017-2019). These base learner models were ensembled using nine different combinations of six ML algorithms to produce homogeneous or heterogeneous ensembles. Performance was assessed using a consensus metric.

Results

Xgboost homogenous ensemble (HE) was the highest performing model (clinical effectiveness metric (CEM) 0.725) with area under the curve (AUC) (0.8327; 95% confidence interval (CI) 0.8323-0.8329) followed by Random Forest HE (CEM 0.723; AUC 0.8325; 95%CI 0.8320-0.8326). Across different heterogenous ensembles, significantly better performance was obtained by combining siloed datasets across time (CEM 0.720) than building ensembles of either 1996-2011 (t-test adjusted, p = 1.67×10-6) or 2012-2019 (t-test adjusted, p = 1.35×10-193) datasets alone.

Conclusions

Both homogenous and heterogenous ML ensembles performed significantly better than DMA ensemble of Bayesian Update models. Time-dependent ensemble combination of variables, having differing qualities according to time of score adoption, enabled previously siloed data to be combined, leading to increased power, clinical interpretability of variables and usage of data.

Machine learning-based prediction of in-hospital mortality for post cardiovascular surgery patients admitting to intensive care unit: a retrospective observational cohort study based on a large multi-center critical care database

BACKGROUND AND OBJECTIVES

The acute physiology and chronic health evaluation-IV model (APACHE-IV), and the sequential organ failure assessment (SOFA) score are two traditional severity assessment systems that can be applied to cardiac surgery patients admitted to intensive care units (ICUs). However, the performance of machine learning approaches in post cardiovascular surgery (PCS) patients admitted to the ICU remains unknown.

METHODS

The clinical data of adult subjects were collected from the eICU database. Seven models were constructed based on the training set (70% random sample) for predicting hospital mortality, including two traditional models based on APACHE-IV and SOFA scores and five machine learning models. We measured the models' performance in the remaining 30% of the sample by computing AUC-ROC values, prospective prediction results, and decision curves and compared the models with net reclassification improvement.

RESULTS

This study included 5860 PCS patients. The AUC-ROC value of the Xgboost model significantly outperformed the APACHE-IV and SOFA scores (0.12 [0.06-0.17] p < 0.01, 0.18 [0.1-0.26] p < 0.01 respectively). The use of ML models would also gain more clinical net benefits than traditional models based on decision curve analysis. There was a significant improvement in integrated discrimination when comparing the backward stepwise linear regression model with the APACHE-IV model (0.11 [0.05, 0.16], p < 0.01) and SOFA model (0.12 [0.06, 0.17], p < 0.01).

CONCLUSIONS

In conclusion, the predictive ability of ML models was better than that of traditional models. The present study suggested that developing advanced prognosis prediction tools could support clinical decision-making in the ICU for PCS patients.

Machine Learning Methods for Predicting Long-Term Mortality in Patients After Cardiac Surgery

Objective:

This study aims to construct and validate several machine learning (ML) algorithms to predict long-term mortality and identify risk factors in unselected patients post-cardiac surgery.

Methods

The Medical Information Mart for Intensive Care (MIMIC-III) database was used to perform a retrospective administrative database study. Candidate predictors consisted of the demographics, comorbidity, vital signs, laboratory test results, scoring systems, and treatment information on the first day of ICU admission. Four-year mortality was set as the study outcome. We used the ML methods of logistic regression (LR), artificial neural network (NNET), naïve bayes (NB), gradient boosting machine (GBM), adapting boosting (Ada), random forest (RF), bagged trees (BT), and eXtreme Gradient Boosting (XGB). The prognostic capacity and clinical utility of these ML models were compared using the area under the receiver operating characteristic curves (AUC), calibration curves, and decision curve analysis (DCA).

Results

Of 7,368 patients in MIMIC-III included in the final cohort, a total of 1,337 (18.15%) patients died during a 4-year follow-up. Among 65 variables extracted from the database, a total of 25 predictors were selected using recursive feature elimination and included in the subsequent analysis. The Ada model performed best among eight models in both discriminatory ability with the highest AUC of 0.801 and goodness of fit (visualized by calibration curve). Moreover, the DCA shows that the net benefit of the RF, Ada, and BT models surpassed that of other ML models for almost all threshold probability values. Additionally, through the Ada technique, we determined that red blood cell distribution width (RDW), blood urea nitrogen (BUN), SAPS II, anion gap (AG), age, urine output, chloride, creatinine, congestive heart failure, and SOFA were the Top 10 predictors in the feature importance rankings.

Conclusions

The Ada model performs best in predicting 4-year mortality after cardiac surgery among the eight ML models, which might have significant application in the development of early warning systems for patients following operations.

Heart surgery by the locals in resource-limited settings: The experience from Ethiopia

BACKGROUND

In developing countries, despite its demand is high, heart surgery is not always accessible to the neediest patients. We aimed to describe the early outcomes of heart surgeries that were performed by a local cardiac surgical team in Addis Ababa, Ethiopia.

METHODS

Data were collected through chart abstraction of patients who underwent heart surgery from the period of June 2017 to July 2021 by the same local cardiac surgical team at 3 centers in Addis Ababa, Ethiopia. Data were analyzed using the Statistical Package for the Social Sciences for Windows version 20.0.

RESULTS

A total of 290 patients who underwent heart surgery during the specified period were included in the study. Of the total, 192 patients underwent valve surgery (177 were patients with rheumatic valvular disease and 15 were valve surgeries with other causes) with a 30-day mortality rate of 9 (4.7%), 33 patients underwent coronary artery bypass graft with a 30-day mortality rate of 3 (9.1%), 58 patients underwent repair for congenital heart diseases with no 30-day mortality. Specifically, button Bentall was done for 1 patient; maze procedure was done for 2 patients along with mitral valve surgery, and a total of 7 out of 290 (2.4%) underwent redo heart surgery. The overall procedure-related mortality was 4.1%.

CONCLUSIONS

In addition to operating on a large number of cardiac patients, the local cardiac surgical team was able to do complex surgical procedures such as button Bentall, left maze procedure, redo valve surgeries, and coronary artery bypass graft surgery in a resource-limited setup. The overall patient outcome was comparable to reports from other centers.

Machine learning methods for perioperative anesthetic management in cardiac surgery patients: a scoping review

Background

Machine learning (ML) is developing fast with promising prospects within medicine and already has several applications in perioperative care. We conducted a scoping review to examine the extent and potential limitations of ML implementation in perioperative anesthetic care, specifically in cardiac surgery patients.

Methods

We mapped the current literature by searching three databases: MEDLINE (Ovid), EMBASE (Ovid), and Cochrane Library. Articles were eligible if they reported on perioperative ML use in the field of cardiac surgery with relevance to anesthetic practices. Data on the applicability of ML and comparability to conventional statistical methods were extracted.

Results

Forty-six articles on ML relevant to the work of the anesthesiologist in cardiac surgery were identified. Three main categories emerged: (I) event and risk prediction, (II) hemodynamic monitoring, and (III) automation of echocardiography. Prediction models based on ML tend to behave similarly to conventional statistical methods. Using dynamic hemodynamic or ultrasound data in ML models, however, shifts the potential to promising results.

Conclusions

ML in cardiac surgery is increasingly used in perioperative anesthetic management. The majority is used for prediction purposes similar to conventional clinical scores. Remarkable ML model performances are achieved when using real-time dynamic parameters. However, beneficial clinical outcomes of ML integration have yet to be determined. Nonetheless, the first steps introducing ML in perioperative anesthetic care for cardiac surgery have been taken.

Can machine learning improve mortality prediction following cardiac surgery?

OBJECTIVES

Interest in the clinical usefulness of machine learning for risk prediction has bloomed recently. Cardiac surgery patients are at high risk of complications and therefore presurgical risk assessment is of crucial relevance. We aimed to compare the performance of machine learning algorithms over traditional logistic regression (LR) model to predict in-hospital mortality following cardiac surgery.

METHODS

A single-centre data set of prospectively collected information from patients undergoing adult cardiac surgery from 1996 to 2017 was split into 70% training set and 30% testing set. Prediction models were developed using neural network, random forest, naive Bayes and retrained LR based on features included in the EuroSCORE. Discrimination was assessed using area under the receiver operating characteristic curve, and calibration analysis was undertaken using the calibration belt method. Model calibration drift was assessed by comparing Goodness of fit χ2 statistics observed in 2 equal bins from the testing sample ordered by procedure date.

RESULTS

A total of 28 761 cardiac procedures were performed during the study period. The in-hospital mortality rate was 2.7%. Retrained LR [area under the receiver operating characteristic curve 0.80; 95% confidence interval (CI) 0.77-0.83] and random forest model (0.80; 95% CI 0.76-0.83) showed the best discrimination. All models showed significant miscalibration. Retrained LR proved to have the weakest calibration drift.

CONCLUSIONS

Our findings do not support the hypothesis that machine learning methods provide advantage over LR model in predicting operative mortality after cardiac surgery.

A hybrid cost-sensitive ensemble for heart disease prediction

BACKGROUND

Heart disease is the primary cause of morbidity and mortality in the world. It includes numerous problems and symptoms. The diagnosis of heart disease is difficult because there are too many factors to analyze. What's more, the misclassification cost could be very high.

METHODS

A cost-sensitive ensemble method was proposed to improve the efficiency of diagnosis and reduce the misclassification cost. The proposed method contains five heterogeneous classifiers: random forest, logistic regression, support vector machine, extreme learning machine and k-nearest neighbor. T-test was used to investigate if the performance of the ensemble was better than individual classifiers and the contribution of Relief algorithm.

RESULTS

The best performance was achieved by the proposed method according to ten-fold cross validation. The statistical tests demonstrated that the performance of the proposed ensemble was significantly superior to individual classifiers, and the efficiency of classification was distinctively improved by Relief algorithm.

CONCLUSIONS

The proposed ensemble gained significantly better results compared with individual classifiers and previous studies, which implies that it can be used as a promising alternative tool in medical decision making for heart disease diagnosis.

Validation and quality measurements for STS, EuroSCORE II and a regional risk model in Brazilian patients

OBJECTIVES

The objectives of this study were to describe a novel statewide registry for cardiac surgery in Brazil (REPLICCAR), to compare a regional risk model (SPScore) with EuroSCORE II and STS, and to understand where quality improvement and safety initiatives can be implemented.

METHODS

A total of 11 sites in the state of São Paulo, Brazil, formed an online registry platform to capture information on risk factors and outcomes after cardiac surgery procedures for all consecutive patients. EuroSCORE II and STS values were calculated for each patient. An SPScore model was designed and compared with EuroSCORE II and STS to predict 30-day outcomes: death, reoperation, readmission, and any morbidity.

RESULTS

A total of 5222 patients were enrolled in this study between November 2013 and December 2017. The observed 30-day mortality rate was 7.6%. Most patients were older, overweight, and classified as New York Heart Association (NYHA) functional class III; 14.5% of the patient population had a positive diagnosis of rheumatic heart disease, 10.9% had insulin-dependent diabetes, and 19 individuals had a positive diagnosis of Chagas disease. When evaluating the prediction performance, we found that SPScore outperformed EuroSCORE II and STS in the prediction of mortality (0.90 vs. 0.76 and 0.77), reoperation (0.84 vs. 0.60 and 0.56), readmission (0.84 vs. 0.55 and 0.51), and any morbidity (0.80 vs. 0.65 and 0.64), respectively (p<0.001).

CONCLUSIONS

The REPLICCAR registry might stimulate the creation of other cardiac surgery registries in developing countries, ultimately improving the regional quality of care provided to patients.

Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study

BACKGROUND

Congenital heart disease accounts for almost a third of all major congenital anomalies. Congenital heart defects have a significant impact on morbidity, mortality and health costs for children and adults. Research regarding the risk of pre-surgical mortality is scarce.

OBJECTIVES

Our goal is to generate a predictive model calculator adapted to the regional reality focused on individual mortality prediction among patients with congenital heart disease undergoing cardiac surgery.

METHODS

Two thousand two hundred forty CHD consecutive patients' data from InCor's heart surgery program was used to develop and validate the preoperative risk-of-death prediction model of congenital patients undergoing heart surgery. There were six artificial intelligence models most cited in medical references used in this study: Multilayer Perceptron (MLP), Random Forest (RF), Extra Trees (ET), Stochastic Gradient Boosting (SGB), Ada Boost Classification (ABC) and Bag Decision Trees (BDT).

RESULTS

The top performing areas under the curve were achieved using Random Forest (0.902). Most influential predictors included previous admission to ICU, diagnostic group, patient's height, hypoplastic left heart syndrome, body mass, arterial oxygen saturation, and pulmonary atresia. These combined predictor variables represent 67.8% of importance for the risk of mortality in the Random Forest algorithm.

CONCLUSIONS

The representativeness of "hospital death" is greater in patients up to 66 cm in height and body mass index below 13.0 for InCor's patients. The proportion of "hospital death" declines with the increased arterial oxygen saturation index. Patients with prior hospitalization before surgery had higher "hospital death" rates than who did not required such intervention. The diagnoses groups having the higher fatal outcomes probability are aligned with the international literature. A web application is presented where researchers and providers can calculate predicted mortality based on the CgntSCORE on any web browser or smartphone.

Machine learning improves mortality risk prediction after cardiac surgery: Systematic review and meta-analysis

BACKGROUND

Interest in the usefulness of machine learning (ML) methods for outcomes prediction has continued to increase in recent years. However, the advantage of advanced ML model over traditional logistic regression (LR) remains controversial. We performed a systematic review and meta-analysis of studies comparing the discrimination accuracy between ML models versus LR in predicting operative mortality following cardiac surgery.

METHODS

The present systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement. Discrimination ability was assessed using the C-statistic. Pooled C-statistics and its 95% credibility interval for ML models and LR were obtained were obtained using a Bayesian framework. Pooled estimates for ML models and LR were compared to inform on difference between the 2 approaches.

RESULTS

We identified 459 published citations of which 15 studies met inclusion criteria and were used for the quantitative and qualitative analysis. When the best ML model from individual study was used, meta-analytic estimates showed that ML were associated with a significantly higher C-statistic (ML, 0.88; 95% credibility interval, 0.83-0.93 vs LR, 0.81; 95% credibility interval, 0.77-0.85; P = .03). When individual ML algorithms were instead selected, we found a nonsignificant trend toward better prediction with each of ML algorithms. We found no evidence of publication bias (P = .70).

CONCLUSIONS

The present findings suggest that when compared with LR, ML models provide better discrimination in mortality prediction after cardiac surgery. However, the magnitude and clinical influence of such an improvement remains uncertain.