Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery

A machine learning approach for predicting textbook outcome after cytoreductive surgery and hyperthermic intraperitoneal chemotherapy

INTRODUCTION

Peritoneal carcinomatosis is considered a late-stage manifestation of neoplastic diseases. Cytoreductive surgery with hyperthermic intraperitoneal chemotherapy (CRS-HIPEC) can be an effective treatment for these patients. However, the procedure is associated with significant morbidity. Our aim was to develop a machine learning model to predict the probability of achieving textbook outcome (TO) after CRS-HIPEC using only preoperatively known variables.

METHODS

Adult patients with peritoneal carcinomatosis and who underwent CRS-HIPEC were included from a large, single-center, prospectively maintained dataset (2001-2020). TO was defined as a hospital length of stay ≤14 days and no postoperative adverse events including any complications, reoperation, readmission, and mortality within 90 days. Four models (logistic regression, neural network, random forest, and XGBoost) were trained, validated, and a user-friendly risk calculator was then developed.

RESULTS

A total of 1954 CRS-HIPEC procedures for peritoneal carcinomatosis were included. Overall, 13% (n = 258) achieved TO following CRS-HIPEC procedure. XGBoost and logistic regression had the highest area under the curve (AUC) (0.76) after model optimization, followed by random forest (AUC 0.75) and neural network (AUC 0.74). The top preoperative variables associated with achieving a TO were lower peritoneal cancer index scores, not undergoing proctectomy, splenectomy, or partial colectomy and being asymptomatic from peritoneal metastases prior to surgery.

CONCLUSION

This is a data-driven study to predict the probability of achieving TO after CRS-HIPEC. The proposed pipeline has the potential to not only identify patients for whom surgery is not associated with prohibitive risk, but also aid surgeons in communicating this risk to patients.

Machine learning improves prediction of postoperative outcomes after gastrointestinal surgery: a systematic review and meta-analysis

BACKGROUND

Machine learning (ML) approaches have become increasingly popular in predicting surgical outcomes. However, it is unknown whether they are superior to traditional statistical methods such as logistic regression (LR). This study aimed to perform a systematic review and meta-analysis to compare the performance of ML vs LR models in predicting postoperative outcomes for patients undergoing gastrointestinal (GI) surgery.

METHODS

A systematic search of Embase, MEDLINE, Cochrane, Web of Science, and Google Scholar was performed through December 2022. The primary outcome was the discriminatory performance of ML vs LR models as measured by the area under the receiver operating characteristic curve (AUC). A meta-analysis was then performed using a random effects model.

RESULTS

A total of 62 LR models and 143 ML models were included across 38 studies. On average, the best-performing ML models had a significantly higher AUC than the LR models (ΔAUC, 0.07; 95% CI, 0.04-0.09; P < .001). Similarly, on average, the best-performing ML models had a significantly higher logit (AUC) than the LR models (Δlogit [AUC], 0.41; 95% CI, 0.23-0.58; P < .001). Approximately half of studies (44%) were found to have a low risk of bias. Upon a subset analysis of only low-risk studies, the difference in logit (AUC) remained significant (ML vs LR, Δlogit [AUC], 0.40; 95% CI, 0.14-0.66; P = .009).

CONCLUSION

We found a significant improvement in discriminatory ability when using ML over LR algorithms in predicting postoperative outcomes for patients undergoing GI surgery. Subsequent efforts should establish standardized protocols for both developing and reporting studies using ML models and explore the practical implementation of these models.

Machine Learning - Based Bleeding Risk Predictions in Atrial Fibrillation Patients on Direct Oral Anticoagulants

Importance

Accurately predicting major bleeding events in non-valvular atrial fibrillation (AF) patients on direct oral anticoagulants (DOACs) is crucial for personalized treatment and improving patient outcomes, especially with emerging alternatives like left atrial appendage closure devices. The left atrial appendage closure devices reduce stroke risk comparably but with significantly fewer non-procedural bleeding events.

Objective

To evaluate the performance of machine learning (ML) risk models in predicting clinically significant bleeding events requiring hospitalization and hemorrhagic stroke in non-valvular AF patients on DOACs compared to conventional bleeding risk scores (HAS-BLED, ORBIT, and ATRIA) at the index visit to a cardiologist for AF management.

Design

Prognostic modeling with retrospective cohort study design using electronic health record (EHR) data, with clinical follow-up at one-, two-, and five-years.

Setting

University of Pittsburgh Medical Center (UPMC) system.

Participants

24,468 non-valvular AF patients aged ≥18 years treated with DOACs, excluding those with prior history of significant bleeding, other indications for DOACs, on warfarin or contraindicated to DOACs.

Exposures

DOAC therapy for non-valvular AF.

Main Outcomes and Measures

The primary endpoint was clinically significant bleeding requiring hospitalization within one year of index visit. The models incorporated demographic, clinical, and laboratory variables available in the EHR at the index visit.

Results

Among 24,468 patients, 553 (2.3%) had bleeding events within one year, 829 (3.5%) within two years, and 1,292 (5.8%) within five years of index visit. We evaluated multivariate logistic regression and ML models including random forest, classification trees, k-nearest neighbor, naive Bayes, and extreme gradient boosting (XGBoost) which modestly outperformed HAS-BLED, ATRIA, and ORBIT scores in predicting clinically significant bleeding at 1-year follow-up. The best performing model (random forest) showed area under the curve (AUC-ROC) 0.76 (0.70-0.81), G-Mean score of 0.67, net reclassification index 0.14 compared to 0.57 (0.50-0.63), G-Mean score of 0.57 for HASBLED score, p-value for difference <0.001. The ML models had improved performance compared to conventional risk across time-points of 2-year and 5-years and within the subgroup of hemorrhagic stroke. SHAP analysis identified novel risk factors including measures from body mass index, cholesterol profile, and insurance type beyond those used in conventional risk scores.

Conclusions and Relevance

Our findings demonstrate the superior performance of ML models compared to conventional bleeding risk scores and identify novel risk factors highlighting the potential for personalized bleeding risk assessment in AF patients on DOACs.

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

PURPOSE

Preoperative prediction of postoperative complications (PCs) in inpatients with cancer is challenging. We developed an explainable machine learning (ML) model to predict PCs in a heterogenous population of inpatients with cancer undergoing same-hospitalization major operations.

METHODS

Consecutive inpatients who underwent same-hospitalization operations from December 2017 to June 2021 at a single institution were retrospectively reviewed. The ML model was developed and tested using electronic health record (EHR) data to predict 30-day PCs for patients with Clavien-Dindo grade 3 or higher (CD 3+) per the CD classification system. Model performance was assessed using area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), and calibration plots. Model explanation was performed using the Shapley additive explanations (SHAP) method at cohort and individual operation levels.

RESULTS

A total of 988 operations in 827 inpatients were included. The ML model was trained using 788 operations and tested using a holdout set of 200 operations. The CD 3+ complication rates were 28.6% and 27.5% in the training and holdout test sets, respectively. Training and holdout test sets' model performance in predicting CD 3+ complications yielded an AUROC of 0.77 and 0.73 and an AUPRC of 0.56 and 0.52, respectively. Calibration plots demonstrated good reliability. The SHAP method identified features and the contributions of the features to the risk of PCs.

CONCLUSION

We trained and tested an explainable ML model to predict the risk of developing PCs in patients with cancer. Using patient-specific EHR data, the ML model accurately discriminated the risk of developing CD 3+ complications and displayed top features at the individual operation and cohort level.

Explainable artificial intelligence model for mortality risk prediction in the intensive care unit: a derivation and validation study

BACKGROUND

The lack of transparency is a prevalent issue among the current machine-learning (ML) algorithms utilized for predicting mortality risk. Herein, we aimed to improve transparency by utilizing the latest ML explicable technology, SHapley Additive exPlanation (SHAP), to develop a predictive model for critically ill patients.

METHODS

We extracted data from the Medical Information Mart for Intensive Care IV database, encompassing all intensive care unit admissions. We employed nine different methods to develop the models. The most accurate model, with the highest area under the receiver operating characteristic curve, was selected as the optimal model. Additionally, we used SHAP to explain the workings of the ML model.

RESULTS

The study included 21 395 critically ill patients, with a median age of 68 years (interquartile range, 56-79 years), and most patients were male (56.9%). The cohort was randomly split into a training set (N = 16 046) and a validation set (N = 5349). Among the nine models developed, the Random Forest model had the highest accuracy (87.62%) and the best area under the receiver operating characteristic curve value (0.89). The SHAP summary analysis showed that Glasgow Coma Scale, urine output, and blood urea nitrogen were the top three risk factors for outcome prediction. Furthermore, SHAP dependency analysis and SHAP force analysis were used to interpret the Random Forest model at the factor level and individual level, respectively.

CONCLUSION

A transparent ML model for predicting outcomes in critically ill patients using SHAP methodology is feasible and effective. SHAP values significantly improve the explainability of ML models.

Prediction of Early Recurrence Following CRS/HIPEC in Patients With Disseminated Appendiceal Cancer

INTRODUCTION

In patients with disseminated appendiceal cancer (dAC) who underwent cytoreductive surgery (CRS) with hyperthermic intraperitoneal chemotherapy (HIPEC), characterizing and predicting those who will develop early recurrence could provide a framework for personalizing follow-up. This study aims to: (1) characterize patients with dAC that are at risk for recurrence within 2 y following of CRS ± HIPEC (early recurrence; ER), (2) utilize automated machine learning (AutoML) to predict at-risk patients, and (3) identifying factors that are influential for prediction.

METHODS

A 12-institution cohort of patients with dAC treated with CRS ± HIPEC between 2000 and 2017 was used to train predictive models using H2O.ai's AutoML. Patients with early recurrence (ER) were compared to those who did not have recurrence or presented with recurrence after 2 y (control; C). However, 75% of the data was used for training and 25% for validation, and models were 5-fold cross-validated.

RESULTS

A total of 949 patients were included, with 337 ER patients (35.5%). Patients with ER had higher markers of inflammation, worse disease burden with poor response, and received greater intraoperative fluids/blood products. The highest performing AutoML model was a Stacked Ensemble (area under the curve = 0.78, area under the curve precision recall = 0.66, positive predictive value = 85%, and negative predictive value = 63%). Prediction was influenced by blood markers, operative course, and factors associated with worse disease burden.

CONCLUSIONS

In this multi-institutional cohort of dAC patients that underwent CRS ± HIPEC, AutoML performed well in predicting patients with ER. Variables suggestive of poor tumor biology were the most influential for prediction. Our work provides a framework for identifying patients with ER that might benefit from shorter interval surveillance early after surgery.

Quantifying the impact of AI recommendations with explanations on prescription decision making

The influence of AI recommendations on physician behaviour remains poorly characterised. We assess how clinicians' decisions may be influenced by additional information more broadly, and how this influence can be modified by either the source of the information (human peers or AI) and the presence or absence of an AI explanation (XAI, here using simple feature importance). We used a modified between-subjects design where intensive care doctors (N = 86) were presented on a computer for each of 16 trials with a patient case and prompted to prescribe continuous values for two drugs. We used a multi-factorial experimental design with four arms, where each clinician experienced all four arms on different subsets of our 24 patients. The four arms were (i) baseline (control), (ii) peer human clinician scenario showing what doses had been prescribed by other doctors, (iii) AI suggestion and (iv) XAI suggestion. We found that additional information (peer, AI or XAI) had a strong influence on prescriptions (significantly for AI, not so for peers) but simple XAI did not have higher influence than AI alone. There was no correlation between attitudes to AI or clinical experience on the AI-supported decisions and nor was there correlation between what doctors self-reported about how useful they found the XAI and whether the XAI actually influenced their prescriptions. Our findings suggest that the marginal impact of simple XAI was low in this setting and we also cast doubt on the utility of self-reports as a valid metric for assessing XAI in clinical experts.

An Innovative Solution Based on TSCA-ViT for Osteosarcoma Diagnosis in Resource-Limited Settings

Identifying and managing osteosarcoma pose significant challenges, especially in resource-constrained developing nations. Advanced diagnostic methods involve isolating the nucleus from cancer cells for comprehensive analysis. However, two main challenges persist: mitigating image noise during the capture and transmission of cellular sections, and providing an efficient, accurate, and cost-effective solution for cell nucleus segmentation. To tackle these issues, we introduce the Twin-Self and Cross-Attention Vision Transformer (TSCA-ViT). This pioneering AI-based system employs a directed filtering algorithm for noise reduction and features an innovative transformer architecture with a twin attention mechanism for effective segmentation. The model also incorporates cross-attention-enabled skip connections to augment spatial information. We evaluated our method on a dataset of 1000 osteosarcoma pathology slide images from the Second People's Hospital of Huaihua, achieving a remarkable average precision of 97.7%. This performance surpasses traditional methodologies. Furthermore, TSCA-ViT offers enhanced computational efficiency owing to its fewer parameters, which results in reduced time and equipment costs. These findings underscore the superior efficacy and efficiency of TSCA-ViT, offering a promising approach for addressing the ongoing challenges in osteosarcoma diagnosis and treatment, particularly in settings with limited resources.

Residential greenness and cardiac conduction abnormalities: epidemiological evidence and an explainable machine learning modeling study

BACKGROUND

Previous studies indicated the beneficial influence of residential greenness on cardiovascular disease (CVD), however, the association of residential greenness with cardiac conduction performance remains unclear. This study aims to examine the epidemiological associations between residential greenness and cardiac conduction abnormalities in rural residents, simultaneously exploring the role of residential greenness for cardiac health in an explainable machine learning modeling study.

METHODS

A total of 27,294 participants were derived from the Henan Rural Cohort. Two satellite-based indices, the normalized difference vegetation index (NDVI) and the enhanced vegetation index (EVI), were used to estimate residential greenness. Independent and combined associations of residential greenness indices and physical activities with electrocardiogram (ECG) parameter abnormalities were evaluated using the logistic regression model and generalized linear model. The Gradient Boosting Machine (GBM) and the SHapely Additive exPlanations (SHAP) were employed in the modeling study.

RESULTS

The odds ratios (OR) and 95% confidence interval (CI) for QRS interval, heart rate (HR), QTc interval, and PR interval abnormalities with per interquartile range in NDVI were 0.896 (0.873-0.920), 0.955 (0.926-0.986), 1.015 (0.984-1.047), and 0.986 (0.929-1.045), respectively. Furthermore, the participants with higher physical activities plus residential greenness (assessed by EVI) were related to a 1.049-fold (1.017-1.081) and 1.298-fold (1.245-1.354) decreased risk for abnormal QRS interval and HR. Similar results were also observed in the sensitivity analysis. The NDVI ranked fifth (SHAP mean value 0.116) in the analysis for QRS interval abnormality risk in the modeling study.

CONCLUSION

A higher level of residential greenness was significantly associated with cardiac conduction abnormalities. This effect might be strengthened in residents with more physical activities. This study indicated the cruciality of environmental greenness to cardiac functions and also contributed to refining preventive medicine and greenness design strategies.

Leveraging the Academic Artificial Intelligence Silecosystem to Advance the Community Oncology Enterprise

Over the last 75 years, artificial intelligence has evolved from a theoretical concept and novel paradigm describing the role that computers might play in our society to a tool with which we daily engage. In this review, we describe AI in terms of its constituent elements, the synthesis of which we refer to as the AI Silecosystem. Herein, we provide an historical perspective of the evolution of the AI Silecosystem, conceptualized and summarized as a Kuhnian paradigm. This manuscript focuses on the role that the AI Silecosystem plays in oncology and its emerging importance in the care of the community oncology patient. We observe that this important role arises out of a unique alliance between the academic oncology enterprise and community oncology practices. We provide evidence of this alliance by illustrating the practical establishment of the AI Silecosystem at the City of Hope Comprehensive Cancer Center and its team utilization by community oncology providers.

Combining conventional ultrasound and ultrasound elastography to predict HER2 status in patients with breast cancer

Introduction: Identifying the HER2 status of breast cancer patients is important for treatment options. Previous studies have shown that ultrasound features are closely related to the subtype of breast cancer. Methods: In this study, we used features of conventional ultrasound and ultrasound elastography to predict HER2 status. Results and Discussion: The performance of model (AUROC) with features of conventional ultrasound and ultrasound elastography is higher than that of the model with features of conventional ultrasound (0.82 vs. 0.53). The SHAP method was used to explore the interpretability of the models. Compared with HER2- tumors, HER2+ tumors usually have greater elastic modulus parameters and microcalcifications. Therefore, we concluded that the features of conventional ultrasound combined with ultrasound elastography could improve the accuracy for predicting HER2 status.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions

OBJECTIVE

To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting.

BACKGROUND

To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown.

METHODS

Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

RESULTS

Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy.

CONCLUSIONS

Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

Artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer

Background

Gastric cancer with synchronous distant metastases indicates a dismal prognosis. The success in survival improvement mainly relies on our ability to predict the potential benefit of a therapy. Our objective is to develop an artificial intelligence annotated clinical-pathologic risk model to predict its outcomes.

Methods

In participants (n=47553) with gastric cancer of the surveillance, epidemiology, and end results program, we selected patients with distant metastases at first diagnosis, complete clinical-pathologic data and follow-up information. Patients were randomly divided into the training and test cohort at 7:3 ratio. 93 patients with advanced gastric cancer from six other cancer centers were collected as the external validation cohort. Multivariable analysis was used to identify the prognosis-related clinical-pathologic features. Then a survival prediction model was established and validated. Importantly, we provided explanations to the prediction with artificial intelligence SHAP (Shapley additive explanations) method. We also provide novel insights into treatment options.

Results

Data from a total 2549 patients were included in model development and internal test (median age, 61 years [range, 53-69 years]; 1725 [67.7%] male). Data from an additional 93 patients were collected as the external validation cohort (median age, 59 years [range, 48-66 years]; 51 [54.8%] male). The clinical-pathologic model achieved a consistently high accuracy for predicting prognosis in the training (C-index: 0.705 [range, 0.690-0.720]), test (C-index: 0.737 [range, 0.717-0.757]), and external validation (C-index: 0.694 [range, 0.562-0.826]) cohorts. Shapley values indicated that undergoing surgery, chemotherapy, young, absence of lung metastases and well differentiated were the top 5 contributors to the high likelihood of survival. A combination of surgery and chemotherapy had the greatest benefit. However, aggressive treatment did not equate to a survival benefit. SHAP dependence plots demonstrated insightful nonlinear interactive associations among predictors in survival benefit prediction. For example, patients who were elderly, or poor differentiated, or presence of lung or bone metastases had a worse prognosis if they undergo surgery or chemotherapy, while patients with metastases to liver alone seemed to gain benefit from surgery and chemotherapy.

Conclusion

In this large multicenter cohort study, we developed an artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer. It could be used to discuss treatment options.

An interpretable machine learning approach for predicting 30-day readmission after stroke

BACKGROUND

Stroke is the second leading cause of death worldwide and has a significantly high recurrence rate. We aimed to identify risk factors for stroke recurrence and develop an interpretable machine learning model to predict 30-day readmissions after stroke.

METHODS

Stroke patients deposited in electronic health records (EHRs) in Xuzhou Medical University Hospital between February 1, 2021, and November 30, 2021, were included in the study, and deceased patients were excluded. We extracted 74 features from EHRs, and the top 20 features (chi-2 value) were used to build machine learning models. 80% of the patients were used for pre-training. Subsequently, a 20% holdout dataset was used for verification. The Shapley Additive exPlanations (SHAP) method was used to explore the interpretability of the model.

RESULTS

The cohort included 6,558 patients, of whom the mean (SD) age was 65 (11) years, 3,926 were males (59.86 %), and 132 (2.01 %) were readmitted within 30 days. The area under the receiver operating characteristic curve (AUROC) for the optimized model was 0.80 (95 % CI 0.68-0.80). We used the SHAP method to identify the top 10 risk factors (i.e., severe carotid artery stenosis, weak, homocysteine, glycosylated hemoglobin, sex, lymphocyte percentage, neutrophilic granulocyte percentage, urine glucose, fresh cerebral infarction, and red blood cell count). The AUROC of a model with the 10 features was 0.80 (95 % CI 0.69-0.80) and was not significantly different from that of the model with 20 risk factors.

CONCLUSIONS

Our methods not only showed good performance in predicting 30-day readmissions after stroke but also revealed risk factors that provided valuable insights for treatments.