Commentary: Dabblers: Beware of hidden dangers in machine-learning comparisons

Machine learning using institution-specific multi-modal electronic health records improves mortality risk prediction for cardiac surgery patients

Background

The Society of Thoracic Surgeons risk scores are widely used to assess risk of morbidity and mortality in specific cardiac surgeries but may not perform optimally in all patients. In a cohort of patients undergoing cardiac surgery, we developed a data-driven, institution-specific machine learning-based model inferred from multi-modal electronic health records and compared the performance with the Society of Thoracic Surgeons models.

Methods

All adult patients undergoing cardiac surgery between 2011 and 2016 were included. Routine electronic health record administrative, demographic, clinical, hemodynamic, laboratory, pharmacological, and procedural data features were extracted. The outcome was postoperative mortality. The database was randomly split into training (development) and test (evaluation) cohorts. Models developed using 4 classification algorithms were compared using 6 evaluation metrics. The performance of the final model was compared with the Society of Thoracic Surgeons models for 7 index surgical procedures.

Results

A total of 6392 patients were included and described by 4016 features. Overall mortality was 3.0% (n = 193). The XGBoost algorithm using only features with no missing data (336 features) yielded the best-performing predictor. When applied to the test set, the predictor performed well (F-measure = 0.775; precision = 0.756; recall = 0.795; accuracy = 0.986; area under the receiver operating characteristic curve = 0.978; area under the precision-recall curve = 0.804). eXtreme Gradient Boosting consistently demonstrated improved performance over the Society of Thoracic Surgeons models when evaluated on index procedures within the test set.

Conclusions

Machine learning models using institution-specific multi-modal electronic health records may improve performance in predicting mortality for individual patients undergoing cardiac surgery compared with the standard-of-care, population-derived Society of Thoracic Surgeons models. Institution-specific models may provide insights complementary to population-derived risk predictions to aid patient-level decision making.

Prediction of operative mortality for patients undergoing cardiac surgical procedures without established risk scores

OBJECTIVE

Current cardiac surgery risk models do not address a substantial fraction of procedures. We sought to create models to predict the risk of operative mortality for an expanded set of cases.

METHODS

Four supervised machine learning models were trained using preoperative variables present in the Society of Thoracic Surgeons (STS) data set of the Massachusetts General Hospital to predict and classify operative mortality in procedures without STS risk scores. A total of 424 (5.5%) mortality events occurred out of 7745 cases. Models included logistic regression with elastic net regularization (LogReg), support vector machine, random forest (RF), and extreme gradient boosted trees (XGBoost). Model discrimination was assessed via area under the receiver operating characteristic curve (AUC), and calibration was assessed via calibration slope and expected-to-observed event ratio. External validation was performed using STS data sets from Brigham and Women's Hospital (BWH) and the Johns Hopkins Hospital (JHH).

RESULTS

Models performed comparably with the highest mean AUC of 0.83 (RF) and expected-to-observed event ratio of 1.00. On external validation, the AUC was 0.81 in BWH (RF) and 0.79 in JHH (LogReg/RF). Models trained and applied on the same institution's data achieved AUCs of 0.81 (BWH: LogReg/RF/XGBoost) and 0.82 (JHH: LogReg/RF/XGBoost).

CONCLUSIONS

Machine learning models trained on preoperative patient data can predict operative mortality at a high level of accuracy for cardiac surgical procedures without established risk scores. Such procedures comprise 23% of all cardiac surgical procedures nationwide. This work also highlights the value of using local institutional data to train new prediction models that account for institution-specific practices.

Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores

OBJECTIVE

In the left ventricular assist device domain, the receiver operating characteristic is a commonly applied metric of performance of classifiers. However, the receiver operating characteristic can provide a distorted view of classifiers' ability to predict short-term mortality due to the overwhelmingly greater proportion of patients who survive, that is, imbalanced data. This study illustrates the ambiguity of the receiver operating characteristic in evaluating 2 classifiers of 90-day left ventricular assist device mortality and introduces the precision recall curve as a supplemental metric that is more representative of left ventricular assist device classifiers in predicting the minority class.

METHODS

This study compared the receiver operating characteristic and precision recall curve for 2 classifiers for 90-day left ventricular assist device mortality, HeartMate Risk Score and Random Forest for 800 patients (test group) recorded in the Interagency Registry for Mechanically Assisted Circulatory Support who received a continuous-flow left ventricular assist device between 2006 and 2016 (mean age, 59 years; 146 female vs 654 male patients), in whom 90-day mortality rate is only 8%.

RESULTS

The receiver operating characteristic indicates similar performance of Random Forest and HeartMate Risk Score classifiers with respect to area under the curve of 0.77 and Random Forest 0.63, respectively. This is in contrast to their precision recall curve with area under the curve of 0.43 versus 0.16 for Random Forest and HeartMate Risk Score, respectively. The precision recall curve for HeartMate Risk Score showed the precision rapidly decreased to only 10% with slightly increasing sensitivity.

CONCLUSIONS

The receiver operating characteristic can portray an overly optimistic performance of a classifier or risk score when applied to imbalanced data. The precision recall curve provides better insight about the performance of a classifier by focusing on the minority class.

A novel predictive model for poor in-hospital outcomes in patients with acute kidney injury after cardiac surgery

OBJECTIVE

Patients with cardiac surgery-associated acute kidney injury are at risk of renal replacement therapy and in-hospital death. We aimed to develop and validate a novel predictive model for poor in-hospital outcomes among patients with cardiac surgery-associated acute kidney injury.

METHODS

A total of 196 patients diagnosed with cardiac surgery-associated acute kidney injury were enrolled in this study as the training cohort, and 32 blood cytokines were measured. Least absolute shrinkage and selection operator regression and random forest quantile-classifier were performed to identify the key blood predictors for in-hospital composite outcomes (requiring renal replacement therapy or in-hospital death). The logistic regression model incorporating the selected predictors was validated internally using bootstrapping and externally in an independent cohort (n = 52).

RESULTS

A change in serum creatinine (delta serum creatinine) and interleukin 16 and interleukin 8 were selected as key predictors for composite outcomes. The logistic regression model incorporating interleukin 16, interleukin 8, and delta serum creatinine yielded the optimal performance, with decent discrimination (area under the receiver operating characteristic curve: 0.947; area under the precision-recall curve: 0.809) and excellent calibration (Brier score: 0.056, Hosmer-Lemeshow test P = .651). Application of the model in the validation cohort yielded good discrimination. A nomogram was generated for clinical use, and decision curve analysis demonstrated that the new model adds more net benefit than delta serum creatinine.

CONCLUSIONS

We developed and validated a promising predictive model for in-hospital composite outcomes among patients with cardiac surgery-associated acute kidney injury and demonstrated interleukin-16 and interleukin-8 as useful predictors to improve risk stratification for poor in-hospital outcomes among those with cardiac surgery-associated acute kidney injury.

Machine Learning to Predict Outcomes and Cost by Phase of Care After Coronary Artery Bypass Grafting

BACKGROUND

Machine learning may enhance prediction of outcomes after coronary artery bypass grafting (CABG). We sought to develop and validate a dynamic machine learning model to predict CABG outcomes at clinically relevant pre- and postoperative time points.

METHODS

The Society of Thoracic Surgeons (STS) registry data elements from 2086 isolated CABG patients were divided into training and testing datasets and input into Extreme Gradient Boosting decision-tree machine learning algorithms. Two prediction models were developed based on data from preoperative (80 parameters) and postoperative (125 parameters) phases of care. Outcomes included operative mortality, major morbidity or mortality, high cost, and 30-day readmission. Machine learning and STS model performance were assessed using accuracy and the area under the precision-recall curve (AUC-PR).

RESULTS

Preoperative machine learning models predicted mortality (accuracy, 98%; AUC-PR = 0.16; F1 = 0.24), major morbidity or mortality (accuracy, 75%; AUC-PR = 0.33; F1 = 0.42), high cost (accuracy, 83%; AUC-PR = 0.51; F1 = 0.52), and 30-day readmission (accuracy, 70%; AUC-PR = 0.47; F1 = 0.49) with high accuracy. Preoperative machine learning models performed similarly to the STS for prediction of mortality (STS AUC-PR = 0.11; P = .409) and outperformed STS for prediction of mortality or major morbidity (STS AUC-PR = 0.28; P < .001). Addition of intraoperative parameters further improved machine learning model performance for major morbidity or mortality (AUC-PR = 0.39; P < .01) and high cost (AUC-PR = 0.64; P < .01), with cross-clamp and bypass times emerging as important additive predictive parameters.

CONCLUSIONS

Machine learning can predict mortality, major morbidity, high cost, and readmission after isolated CABG. Prediction based on the phase of care allows for dynamic risk assessment through the hospital course, which may benefit quality assessment and clinical decision-making.