Leveraging electronic health records for predictive modeling of post-surgical complications

Eight quick tips for biologically and medically informed machine learning

Machine learning has become a powerful tool for computational analysis in the biomedical sciences, with its effectiveness significantly enhanced by integrating domain-specific knowledge. This integration has give rise to informed machine learning, in contrast to studies that lack domain knowledge and treat all variables equally (uninformed machine learning). While the application of informed machine learning to bioinformatics and health informatics datasets has become more seamless, the likelihood of errors has also increased. To address this drawback, we present eight guidelines outlining best practices for employing informed machine learning methods in biomedical sciences. These quick tips offer recommendations on various aspects of informed machine learning analysis, aiming to assist researchers in generating more robust, explainable, and dependable results. Even if we originally crafted these eight simple suggestions for novices, we believe they are deemed relevant for expert computational researchers as well.

Systematic evaluation of machine learning models for postoperative surgical site infection prediction

BACKGROUND

Surgical site infections (SSIs) lead to increased mortality and morbidity, as well as increased healthcare costs. Multiple models for the prediction of this serious surgical complication have been developed, with an increasing use of machine learning (ML) tools.

OBJECTIVE

The aim of this systematic review was to assess the performance as well as the methodological quality of validated ML models for the prediction of SSIs.

METHODS

A systematic search in PubMed, Embase and the Cochrane library was performed from inception until July 2023. Exclusion criteria were the absence of reported model validation, SSIs as part of a composite adverse outcome, and pediatric populations. ML performance measures were evaluated, and ML performances were compared to regression-based methods for studies that reported both methods. Risk of bias (ROB) of the studies was assessed using the Prediction model Risk of Bias Assessment Tool.

RESULTS

Of the 4,377 studies screened, 24 were included in this review, describing 85 ML models. Most models were only internally validated (81%). The C-statistic was the most used performance measure (reported in 96% of the studies) and only two studies reported calibration metrics. A total of 116 different predictors were described, of which age, steroid use, sex, diabetes, and smoking were most frequently (100% to 75%) incorporated. Thirteen studies compared ML models to regression-based models and showed a similar performance of both modelling methods. For all included studies, the overall ROB was high or unclear.

CONCLUSIONS

A multitude of ML models for the prediction of SSIs are available, with large variability in performance. However, most models lacked external validation, performance was reported limitedly, and the risk of bias was high. In studies describing both ML models and regression-based models, one modelling method did not outperform the other.

Can machine learning models improve the prediction of surgical site infection in abdominal surgery than traditional statistical models?

OBJECTIVE

To externally validate by revision and update the study on the efficacy of nosocomial infection control (SENIC) model of surgical site infection (SSI) using logistic regression (LR) and machine learning (ML) approaches.

METHODS

A retrospective analysis of hospital database-derived data from patients that had undergone gastrointestinal, colorectal and hernia surgeries (identified by ICD-9-CM). The SENIC index was calculated and fitted in an LR. MLs were developed using decision-tree (DT), random forest (RF), extreme-gradient-boosting (XGBoost) and Naïve Bayes (NB).

RESULTS

The prevalence of an SSI was 3.21% (404 of 12 596 surgeries; 95% confidence interval [CI] 2.91%, 3.53%). The C-statistic for the original SENIC model was 0.668 (95% CI 0.648, 0.688) with an observed/expected (O/E) ratio of 0.998 (interquartile range [IQR] 0.750, 1.047). An updated-SENIC-LR model with six predictors had a C-statistic of 0.768 (95% CI 0.745, 0.790) and O/E ratio of 0.999 (IQR 0.976, 1.004). The performance of MLs considering 14 predictors was poorer than the updated-SENIC-LR with C-statistics of 0.679, 0.675, 0.656 and 0.651 for NB, XGBoost, RF and DT, respectively. Overfitting was detected for ML approaches, particularly for DT, RF and XGBoost.

CONCLUSION

The updated-SENIC-LR model and NB may be useful for monitoring SSI risk following abdominal surgery.

Can serious postoperative complications in patients with Crohn's disease be predicted using machine learning?

The routine introduction of novel anti-inflammatory therapies into the management algorithms of patients with Crohn's disease over the last 2 decades has not substantially changed the likelihood of ultimate surgery. Rather it has delayed the operative need and altered the presentation phenotype. The prospect of complications continues to remain high in this modern era but depending upon the cohort assessed, it remains difficult to make strict comparisons between individual specialist centres. Those patients who present rather late after their diagnosis with a septic complication like an intra-abdominal abscess and a penetrating/fistulizing pattern of disease are more likely to have a complicated course particularly if they have clinical features such as difficult percutaneous access to the collection or multilocularity both of which can make preoperative drainage unsuccessful. Equally, those cases with extensive adhesions where an initial laparoscopic approach needs open conversion and where there is an extended operative time, unsurprisingly will suffer more significant complications that impact their length of hospital stay. The need for a protective stoma also introduces its own derivative costs, utilizing a range of health resources as well as resulting in important alterations in quality of life outcomes. Having established the parameters of the problem can the statistical analysis of the available data identify high-risk cases, promote the notion of centralization of specialist services or improve the allocation of disease-specific health expenditure?

Automated Identification of Postoperative Infections to Allow Prediction and Surveillance Based on Electronic Health Record Data: Scoping Review

BACKGROUND

Postoperative infections remain a crucial challenge in health care, resulting in high morbidity, mortality, and costs. Accurate identification and labeling of patients with postoperative bacterial infections is crucial for developing prediction models, validating biomarkers, and implementing surveillance systems in clinical practice.

OBJECTIVE

This scoping review aimed to explore methods for identifying patients with postoperative infections using electronic health record (EHR) data to go beyond the reference standard of manual chart review.

METHODS

We performed a systematic search strategy across PubMed, Embase, Web of Science (Core Collection), the Cochrane Library, and Emcare (Ovid), targeting studies addressing the prediction and fully automated surveillance (ie, without manual check) of diverse bacterial infections in the postoperative setting. For prediction modeling studies, we assessed the labeling methods used, categorizing them as either manual or automated. We evaluated the different types of EHR data needed for the surveillance and labeling of postoperative infections, as well as the performance of fully automated surveillance systems compared with manual chart review.

RESULTS

We identified 75 different methods and definitions used to identify patients with postoperative infections in studies published between 2003 and 2023. Manual labeling was the predominant method in prediction modeling research, 65% (49/75) of the identified methods use structured data, and 45% (34/75) use free text and clinical notes as one of their data sources. Fully automated surveillance systems should be used with caution because the reported positive predictive values are between 0.31 and 0.76.

CONCLUSIONS

There is currently no evidence to support fully automated labeling and identification of patients with infections based solely on structured EHR data. Future research should focus on defining uniform definitions, as well as prioritizing the development of more scalable, automated methods for infection detection using structured EHR data.

Machine learning prediction model for postoperative ileus following colorectal surgery

BACKGROUND

Postoperative ileus (POI) continues to be a major cause of morbidity following colorectal surgery. Despite best efforts, the incidence of POI in colorectal surgery remains high (~30%). This study aimed to investigate machine learning techniques to identify risk factors for POI in colorectal surgery patients, to help guide further preventative strategies.

METHODS

A TRIPOD-guideline-compliant retrospective study was conducted for major colorectal surgery patients at a single tertial care centre (2018-2022). The primary outcome was the occurrence of POI, defined as not achieving GI-2 (outcome measure of time to first stool and tolerance of oral diet) by day four. Multivariate logistic regression, decision trees, radial basis function and multilayer perceptron (MLP) models were trained using a random allocation of patients to training/testing data sets (80/20). The area under the receiver operating characteristic (AUROC) curves were used to evaluate model performance.

RESULTS

Of 504 colorectal surgery patients, 183 (36%) experienced POI. Multivariate logistic regression, decision trees, radial basis function and MLP models returned an AUROC of 0.722, 0.706, 0.712 and 0.800, respectively. The MLP model had the highest sensitivity and specificity values. In addition to well-known risk factors for POI, such as postoperative hypokalaemia, surgical approach, and opioid use, the MLP model identified sarcopenia (ranked 4/30) as a potentially modifiable risk factor for POI.

CONCLUSION

MLP outperformed other models in predicting POI. Machine learning can provide valuable insights into the importance and ranking of specific predictive variables for POI. Further research into the predictive value of preoperative sarcopenia for POI is required.

Testing Machine Learning Models to Predict Postoperative Ileus after Colorectal Surgery

Background: Postoperative ileus (POI) is a common complication after colorectal surgery, leading to increased hospital stay and costs. This study aimed to explore patient comorbidities that contribute to the development of POI in the colorectal surgical population and compare machine learning (ML) model accuracy to existing risk instruments. Study Design: In a retrospective study, data were collected on 316 adult patients who underwent colorectal surgery from January 2020 to December 2021. The study excluded patients undergoing multi-visceral resections, re-operations, or combined primary and metastatic resections. Patients lacking follow-up within 90 days after surgery were also excluded. Eight different ML models were trained and cross-validated using 29 patient comorbidities and four comorbidity risk indices (ASA Status, NSQIP, CCI, and ECI). Results: The study found that 6.33% of patients experienced POI. Age, BMI, gender, kidney disease, anemia, arrhythmia, rheumatoid arthritis, and NSQIP score were identified as significant predictors of POI. The ML models with the greatest accuracy were AdaBoost tuned with grid search (94.2%) and XG Boost tuned with grid search (85.2%). Conclusions: This study suggests that ML models can predict the risk of POI with high accuracy and may offer a new frontier in early detection and intervention for postoperative outcome optimization. ML models can greatly improve the prediction and prevention of POI in colorectal surgery patients, which can lead to improved patient outcomes and reduced healthcare costs. Further research is required to validate and assess the replicability of these results.

Machine learning improves prediction of postoperative outcomes after gastrointestinal surgery: a systematic review and meta-analysis

BACKGROUND

Machine learning (ML) approaches have become increasingly popular in predicting surgical outcomes. However, it is unknown whether they are superior to traditional statistical methods such as logistic regression (LR). This study aimed to perform a systematic review and meta-analysis to compare the performance of ML vs LR models in predicting postoperative outcomes for patients undergoing gastrointestinal (GI) surgery.

METHODS

A systematic search of Embase, MEDLINE, Cochrane, Web of Science, and Google Scholar was performed through December 2022. The primary outcome was the discriminatory performance of ML vs LR models as measured by the area under the receiver operating characteristic curve (AUC). A meta-analysis was then performed using a random effects model.

RESULTS

A total of 62 LR models and 143 ML models were included across 38 studies. On average, the best-performing ML models had a significantly higher AUC than the LR models (ΔAUC, 0.07; 95% CI, 0.04-0.09; P < .001). Similarly, on average, the best-performing ML models had a significantly higher logit (AUC) than the LR models (Δlogit [AUC], 0.41; 95% CI, 0.23-0.58; P < .001). Approximately half of studies (44%) were found to have a low risk of bias. Upon a subset analysis of only low-risk studies, the difference in logit (AUC) remained significant (ML vs LR, Δlogit [AUC], 0.40; 95% CI, 0.14-0.66; P = .009).

CONCLUSION

We found a significant improvement in discriminatory ability when using ML over LR algorithms in predicting postoperative outcomes for patients undergoing GI surgery. Subsequent efforts should establish standardized protocols for both developing and reporting studies using ML models and explore the practical implementation of these models.

Preoperative Prediction of Postoperative Infections Using Machine Learning and Electronic Health Record Data

OBJECTIVE

To estimate preoperative risk of postoperative infections using structured electronic health record (EHR) data.

BACKGROUND

Surveillance and reporting of postoperative infections is primarily done through costly, labor-intensive manual chart reviews on a small sample of patients. Automated methods using statistical models applied to postoperative EHR data have shown promise to augment manual review as they can cover all operations in a timely manner. However, there are no specific models for risk-adjusting infectious complication rates using EHR data.

METHODS

Preoperative EHR data from 30,639 patients (2013-2019) were linked to the American College of Surgeons National Surgical Quality Improvement Program preoperative data and postoperative infection outcomes data from 5 hospitals in the University of Colorado Health System. EHR data included diagnoses, procedures, operative variables, patient characteristics, and medications. Lasso and the knockoff filter were used to perform controlled variable selection. Outcomes included surgical site infection, urinary tract infection, sepsis/septic shock, and pneumonia up to 30 days postoperatively.

RESULTS

Among >15,000 candidate predictors, 7 were chosen for the surgical site infection model and 6 for each of the urinary tract infection, sepsis, and pneumonia models. Important variables included preoperative presence of the specific outcome, wound classification, comorbidities, and American Society of Anesthesiologists physical status classification. The area under the receiver operating characteristic curve for each model ranged from 0.73 to 0.89.

CONCLUSIONS

Parsimonious preoperative models for predicting postoperative infection risk using EHR data were developed and showed comparable performance to existing American College of Surgeons National Surgical Quality Improvement Program risk models that use manual chart review. These models can be used to estimate risk-adjusted postoperative infection rates applied to large volumes of EHR data in a timely manner.

Machine learning prediction model of major adverse outcomes after pediatric congenital heart surgery-a retrospective cohort study

BACKGROUND

Major adverse postoperative outcomes (APOs) can greatly affect mortality, hospital stay, care management and planning, and quality of life. This study aimed to evaluate the performance of five machine learning (ML) algorithms for predicting four major APOs after pediatric congenital heart surgery and their clinically meaningful model interpretations.

METHODS

Between August 2014 and December 2021, 23,000 consecutive pediatric patients receiving congenital heart surgery were enrolled. Based on the split date of 1 January 2019, we selected 13,927 participants for the training cohort, and 9,073 participants for the testing cohort. Four predefined major APOs including low cardiac output syndrome (LCOS), pneumonia, renal failure, and deep venous thrombosis (DVT) were investigated. 39 clinical and laboratory features were inputted in five ML models: light gradient boosting machine (LightGBM), logistic regression (LR), support vector machine, random forest, and CatBoost. The performance and interpretations of ML models were evaluated using the area under the receiver operating characteristic curve (AUC) and Shapley Additive Explanations (SHAP).

RESULTS

In the training cohort, CatBoost algorithms outperformed others with the mean AUCs of 0.908 for LCOS and 0.957 for renal failure, while LightGBM and LR achieved the best mean AUCs of 0.886 for pneumonia and 0.942 for DVT, respectively. In the testing cohort, the best-performing ML model for each major APOs with the following mean AUCs: LCOS (LightGBM), 0.893 (95% confidence interval (CI), 0.884-0.895); pneumonia (LR), 0.929 (95% CI, 0.926-0.931); renal failure (LightGBM), 0.963 (95% CI, 0.947-0.979), and DVT (LightGBM), 0.970 (95% CI, 0.953-0.982). The performance of ML models using only clinical variables was slightly lower than those using combined data, with the mean AUCs of 0.873 for LCOS, 0.894 for pneumonia, 0.953 for renal failure, and 0.933 for DVT. The SHAP showed that mechanical ventilation time was the most important contributor of four major APOs.

CONCLUSIONS

In pediatric congenital heart surgery, the established ML model can accurately predict the risk of four major APOs, providing reliable interpretations for high-risk contributor identification and informed clinical decisions making.

Machine learning models in clinical practice for the prediction of postoperative complications after major abdominal surgery

Complications after surgery have a major impact on short- and long-term outcomes, and decades of technological advancement have not yet led to the eradication of their risk. The accurate prediction of complications, recently enhanced by the development of machine learning algorithms, has the potential to completely reshape surgical patient management. In this paper, we reflect on multiple issues facing the implementation of machine learning, from the development to the actual implementation of machine learning models in daily clinical practice, providing suggestions on the use of machine learning models for predicting postoperative complications after major abdominal surgery.

Artificial intelligence in colorectal surgery: an AI-powered systematic review

Artificial intelligence (AI) has the potential to revolutionize surgery in the coming years. Still, it is essential to clarify what the meaningful current applications are and what can be reasonably expected. This AI-powered review assessed the role of AI in colorectal surgery. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-compliant systematic search of PubMed, Embase, Scopus, Cochrane Library databases, and gray literature was conducted on all available articles on AI in colorectal surgery (from January 1 1997 to March 1 2021), aiming to define the perioperative applications of AI. Potentially eligible studies were identified using novel software powered by natural language processing (NLP) and machine learning (ML) technologies dedicated to systematic reviews. Out of 1238 articles identified, 115 were included in the final analysis. Available articles addressed the role of AI in several areas of interest. In the preoperative phase, AI can be used to define tailored treatment algorithms, support clinical decision-making, assess the risk of complications, and predict surgical outcomes and survival. Intraoperatively, AI-enhanced surgery and integration of AI in robotic platforms have been suggested. After surgery, AI can be implemented in the Enhanced Recovery after Surgery (ERAS) pathway. Additional areas of applications included the assessment of patient-reported outcomes, automated pathology assessment, and research. Available data on these aspects are limited, and AI in colorectal surgery is still in its infancy. However, the rapid evolution of technologies makes it likely that it will increasingly be incorporated into everyday practice.

Digital Health Data Quality Issues: Systematic Review

BACKGROUND

The promise of digital health is principally dependent on the ability to electronically capture data that can be analyzed to improve decision-making. However, the ability to effectively harness data has proven elusive, largely because of the quality of the data captured. Despite the importance of data quality (DQ), an agreed-upon DQ taxonomy evades literature. When consolidated frameworks are developed, the dimensions are often fragmented, without consideration of the interrelationships among the dimensions or their resultant impact.

OBJECTIVE

The aim of this study was to develop a consolidated digital health DQ dimension and outcome (DQ-DO) framework to provide insights into 3 research questions: What are the dimensions of digital health DQ? How are the dimensions of digital health DQ related? and What are the impacts of digital health DQ?

METHODS

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, a developmental systematic literature review was conducted of peer-reviewed literature focusing on digital health DQ in predominately hospital settings. A total of 227 relevant articles were retrieved and inductively analyzed to identify digital health DQ dimensions and outcomes. The inductive analysis was performed through open coding, constant comparison, and card sorting with subject matter experts to identify digital health DQ dimensions and digital health DQ outcomes. Subsequently, a computer-assisted analysis was performed and verified by DQ experts to identify the interrelationships among the DQ dimensions and relationships between DQ dimensions and outcomes. The analysis resulted in the development of the DQ-DO framework.

RESULTS

The digital health DQ-DO framework consists of 6 dimensions of DQ, namely accessibility, accuracy, completeness, consistency, contextual validity, and currency; interrelationships among the dimensions of digital health DQ, with consistency being the most influential dimension impacting all other digital health DQ dimensions; 5 digital health DQ outcomes, namely clinical, clinician, research-related, business process, and organizational outcomes; and relationships between the digital health DQ dimensions and DQ outcomes, with the consistency and accessibility dimensions impacting all DQ outcomes.

CONCLUSIONS

The DQ-DO framework developed in this study demonstrates the complexity of digital health DQ and the necessity for reducing digital health DQ issues. The framework further provides health care executives with holistic insights into DQ issues and resultant outcomes, which can help them prioritize which DQ-related problems to tackle first.

Data-Driven Technologies as Enablers for Value Creation in the Prevention of Surgical Site Infections: a Systematic Review

Despite the advances in modern medicine, the use of data-driven technologies (DDTs) to prevent surgical site infections (SSIs) remains a major challenge. Scholars recognise that data management is the next frontier in infection prevention, but many aspects related to the benefits and advantages of using DDTs to mitigate SSI risk factors remain unclear and underexplored in the literature. This study explores how DDTs enable value creation in the prevention of SSIs. This study follows a systematic literature review approach and the PRISMA statement to analyse peer-reviewed articles from seven databases. Fifty-nine articles were included in the review and were analysed through a descriptive and a thematic analysis. The findings suggest a growing interest in DDTs in SSI prevention in the last 5 years, and that machine learning and smartphone applications are widely used in SSI prevention. DDTs are mainly applied to prevent SSIs in clean and clean-contaminated surgeries and often used to manage patient-related data in the postoperative stage. DDTs enable the creation of nine categories of value that are classified in four dimensions: cost/sacrifice, functional/instrumental, experiential/hedonic, and symbolic/expressive. This study offers a unique and systematic overview of the value creation aspects enabled by DDT applications in SSI prevention and suggests that additional research is needed in four areas: value co-creation and product-service systems, DDTs in contaminated and dirty surgeries, data legitimation and explainability, and data-driven interventions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41666-023-00129-2.

Performance of machine learning algorithms for surgical site infection case detection and prediction: A systematic review and meta-analysis

Background

Medical researchers and clinicians have shown much interest in developing machine learning (ML) algorithms to detect/predict surgical site infections (SSIs). However, little is known about the overall performance of ML algorithms in predicting SSIs and how to improve the algorithm's robustness. We conducted a systematic review and meta-analysis to summarize the performance of ML algorithms in SSIs case detection and prediction and to describe the impact of using unstructured and textual data in the development of ML algorithms.

Methods

MEDLINE, EMBASE, CINAHL, CENTRAL and Web of Science were searched from inception to March 25, 2021. Study characteristics and algorithm development information were extracted. Performance statistics (e.g., sensitivity, area under the receiver operating characteristic curve [AUC]) were pooled using a random effect model. Stratified analysis was applied to different study characteristic levels. Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Diagnostic Test Accuracy Studies (PRISMA-DTA) was followed.

Results

Of 945 articles identified, 108 algorithms from 32 articles were included in this review. The overall pooled estimate of the SSI incidence rate was 3.67%, 95% CI: 3.58-3.76. Mixed-use of structured and textual data-based algorithms (pooled estimates of sensitivity 0.83, 95% CI: 0.78-0.87, specificity 0.92, 95% CI: 0.86-0.95, AUC 0.92, 95% CI: 0.89-0.94) outperformed algorithms solely based on structured data (sensitivity 0.56, 95% CI:0.43-0.69, specificity 0.95, 95% CI:0.91-0.97, AUC = 0.90, 95% CI: 0.87-0.92).

Conclusions

ML algorithms developed with structured and textual data provided optimal performance. External validation of ML algorithms is needed to translate current knowledge into clinical practice.

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery

Importance

Cytoreductive surgery (CRS) is one of the most complex operations in surgical oncology with significant morbidity, and improved risk prediction tools are critically needed. Machine learning models can potentially overcome the limitations of traditional multiple logistic regression (MLR) models and provide accurate risk estimates.

Objective

To develop and validate an explainable machine learning model for predicting major postoperative complications in patients undergoing CRS.

Design, Setting, and Participants

This prognostic study used patient data from tertiary care hospitals with expertise in CRS included in the US Hyperthermic Intraperitoneal Chemotherapy Collaborative Database between 1998 and 2018. Information from 147 variables was extracted to predict the risk of a major complication. An ensemble-based machine learning (gradient-boosting) model was optimized on 80% of the sample with subsequent validation on a 20% holdout data set. The machine learning model was compared with traditional MLR models. The artificial intelligence SHAP (Shapley additive explanations) method was used for interpretation of patient- and cohort-level risk estimates and interactions to define novel surgical risk phenotypes. Data were analyzed between November 2019 and August 2021.

Exposures

Cytoreductive surgery.

Main Outcomes and Measures

Area under the receiver operating characteristics (AUROC); area under the precision recall curve (AUPRC).

Results

Data from a total 2372 patients were included in model development (mean age, 55 years [range, 11-95 years]; 1366 [57.6%] women). The optimized machine learning model achieved high discrimination (AUROC: mean cross-validation, 0.75 [range, 0.73-0.81]; test, 0.74) and precision (AUPRC: mean cross-validation, 0.50 [range, 0.46-0.58]; test, 0.42). Compared with the optimized machine learning model, the published MLR model performed worse (test AUROC and AUPRC: 0.54 and 0.18, respectively). Higher volume of estimated blood loss, having pelvic peritonectomy, and longer operative time were the top 3 contributors to the high likelihood of major complications. SHAP dependence plots demonstrated insightful nonlinear interactive associations between predictors and major complications. For instance, high estimated blood loss (ie, above 500 mL) was only detrimental when operative time exceeded 9 hours. Unsupervised clustering of patients based on similarity of sources of risk allowed identification of 6 distinct surgical risk phenotypes.

Conclusions and Relevance

In this prognostic study using data from patients undergoing CRS, an optimized machine learning model demonstrated a superior ability to predict individual- and cohort-level risk of major complications vs traditional methods. Using the SHAP method, 6 distinct surgical phenotypes were identified based on sources of risk of major complications.

Machine learning to guide clinical decision-making in abdominal surgery-a systematic literature review

PURPOSE

An indication for surgical therapy includes balancing benefits against risk, which remains a key task in all surgical disciplines. Decisions are oftentimes based on clinical experience while guidelines lack evidence-based background. Various medical fields capitalized the application of machine learning (ML), and preliminary research suggests promising implications in surgeons' workflow. Hence, we evaluated ML's contemporary and possible future role in clinical decision-making (CDM) focusing on abdominal surgery.

METHODS

Using the PICO framework, relevant keywords and research questions were identified. Following the PRISMA guidelines, a systemic search strategy in the PubMed database was conducted. Results were filtered by distinct criteria and selected articles were manually full text reviewed.

RESULTS

Literature review revealed 4,396 articles, of which 47 matched the search criteria. The mean number of patients included was 55,843. A total of eight distinct ML techniques were evaluated whereas AUROC was applied by most authors for comparing ML predictions vs. conventional CDM routines. Most authors (N = 30/47, 63.8%) stated ML's superiority in the prediction of benefits and risks of surgery. The identification of highly relevant parameters to be integrated into algorithms allowing a more precise prognosis was emphasized as the main advantage of ML in CDM.

CONCLUSIONS

A potential value of ML for surgical decision-making was demonstrated in several scientific articles. However, the low number of publications with only few collaborative studies between surgeons and computer scientists underpins the early phase of this highly promising field. Interdisciplinary research initiatives combining existing clinical datasets and emerging techniques of data processing may likely improve CDM in abdominal surgery in the future.

Machine Learning Algorithms for Predicting Surgical Outcomes after Colorectal Surgery: A Systematic Review

BACKGROUND

Machine learning (ML) has been introduced in various fields of healthcare. In colorectal surgery, the role of ML has yet to be reported. In this systematic review, an overview of machine learning models predicting surgical outcomes after colorectal surgery is provided.

METHODS

Databases PubMed, EMBASE, Cochrane, and Web of Science were searched for studies using machine learning models for patients undergoing colorectal surgery. To be eligible for inclusion, studies needed to apply machine learning models for patients undergoing colorectal surgery. Absence of machine learning or colorectal surgery or studies reporting on reviews, children, study abstracts were excluded. The Probast risk of bias tool was used to evaluate the methodological quality of machine learning models.

RESULTS

A total of 1821 studies were analysed, resulting in the inclusion of 31 articles. A vast proportion of ML algorithms have been used to predict the course of disease and response to neoadjuvant chemoradiotherapy. Radiomics have been applied most frequently, along with predictive accuracies up to 91%. However, most studies included a retrospective study design without external validation or calibration.

CONCLUSIONS

Machine learning models have shown promising potential in predicting surgical outcomes after colorectal surgery. However, large-scale data is warranted to bridge the gap between calibration and external validation. Clinical implementation is needed to demonstrate the contribution of ML within daily practice.

Characterization of Symptoms and Symptom Clusters for Type 2 Diabetes Using a Large Nationwide Electronic Health Record Database

OBJECTIVE

A variety of symptoms may be associated with type 2 diabetes and its complications. Symptoms in chronic diseases may be described in terms of prevalence, severity, and trajectory and often co-occur in groups, known as symptom clusters, which may be representative of a common etiology. The purpose of this study was to characterize type 2 diabetes-related symptoms using a large nationwide electronic health record (EHR) database.

METHODS

We acquired the Cerner Health Facts, a nationwide EHR database. The type 2 diabetes cohort (n = 1,136,301 patients) was identified using a rule-based phenotype method. A multistep procedure was then used to identify type 2 diabetes-related symptoms based on International Classification of Diseases, 9th and 10th revisions, diagnosis codes. Type 2 diabetes-related symptoms and co-occurring symptom clusters, including their temporal patterns, were characterized based the longitudinal EHR data.

RESULTS

Patients had a mean age of 61.4 years, 51.2% were female, and 70.0% were White. Among 1,136,301 patients, there were 8,008,276 occurrences of 59 symptoms. The most frequently reported symptoms included pain, heartburn, shortness of breath, fatigue, and swelling, which occurred in 21-60% of the patients. We also observed over-represented type 2 diabetes symptoms, including difficulty speaking, feeling confused, trouble remembering, weakness, and drowsiness/sleepiness. Some of these are rare and difficult to detect by traditional patient-reported outcomes studies.

CONCLUSION

To the best of our knowledge, this is the first study to use a nationwide EHR database to characterize type 2 diabetes-related symptoms and their temporal patterns. Fifty-nine symptoms, including both over-represented and rare diabetes-related symptoms, were identified.

The prediction of surgical complications using artificial intelligence in patients undergoing major abdominal surgery: A systematic review

BACKGROUND

Conventional statistics are based on a simple cause-and-effect principle. Postoperative complications, however, have a multifactorial and interrelated etiology. The application of artificial intelligence might be more accurate to predict postoperative outcomes. The objective of this study was to determine the current quality of studies describing the use of artificial intelligence in predicting complications in patients undergoing major abdominal surgery.

METHODS

A literature search was performed in PubMed, Embase, and Web of Science. Inclusion criteria were (1) empirical studies including patients undergoing (2) any type of gastrointestinal surgery, including hepatopancreaticobiliary surgery, whose (3) complications or mortality were predicted with the use of (4) any artificial intelligence system. Studies were screened for description of method of validation and testing in methodology. Outcome measurements were sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve.

RESULTS

From a total of 1,537 identified articles, 15 were included for the review. Among a large variety of algorithms used by the included studies, sensitivity was between 0.06 and 0.96, specificity was between 0.61 and 0.98, accuracy was between 0.78 and 0.95, and area under the receiver operating characteristic curve varied between 0.50 and 0.96.

CONCLUSION

Artificial intelligence algorithms have the ability to accurately predict postoperative complications. Nevertheless, algorithms should be properly tested and validated, both internally and externally. Furthermore, a complete database and the absence of unsampled imbalanced data are absolute prerequisites for algorithms to predict accurately.

Developing an Analytical Pipeline to Classify Patient Safety Event Reports Using Optimized Predictive Algorithms

BACKGROUND

 Patient safety event reports provide valuable insight into systemic safety issues but deriving insights from these reports requires computational tools to efficiently parse through large volumes of qualitative data. Natural language processing (NLP) combined with predictive learning provides an automated approach to evaluating these data and supporting the work of patient safety analysts.

OBJECTIVES

 The objective of this study was to use NLP and machine learning techniques to develop a generalizable, scalable, and reliable approach to classifying event reports for the purpose of driving improvements in the safety and quality of patient care.

METHODS

 Datasets for 14 different labels (themes) were vectorized using a bag-of-words, tf-idf, or document embeddings approach and then applied to a series of classification algorithms via a hyperparameter grid search to derive an optimized model. Reports were also analyzed for terms strongly associated with each theme using an adjusted F-score calculation.

RESULTS

 F₁ score for each optimized model ranged from 0.951 ("Fall") to 0.544 ("Environment"). The bag-of-words approach proved optimal for 12 of 14 labels, and the naïve Bayes algorithm performed best for nine labels. Linear support vector machine was demonstrated as optimal for three labels and XGBoost for four of the 14 labels. Labels with more distinctly associated terms performed better than less distinct themes, as shown by a Pearson's correlation coefficient of 0.634.

CONCLUSIONS

 We were able to demonstrate an analytical pipeline that broadly applies NLP and predictive modeling to categorize patient safety reports from multiple facilities. This pipeline allows analysts to more rapidly identify and structure information contained in patient safety data, which can enhance the evaluation and the use of this information over time.

Use of Machine Learning to Develop and Evaluate Models Using Preoperative and Intraoperative Data to Identify Risks of Postoperative Complications

IMPORTANCE

Postoperative complications can significantly impact perioperative care management and planning.

OBJECTIVES

To assess machine learning (ML) models for predicting postoperative complications using independent and combined preoperative and intraoperative data and their clinically meaningful model-agnostic interpretations.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study assessed 111 888 operations performed on adults at a single academic medical center from June 1, 2012, to August 31, 2016, with a mean duration of follow-up based on the length of postoperative hospital stay less than 7 days. Data analysis was performed from February 1 to September 31, 2020.

MAIN OUTCOMES AND MEASURES

Outcomes included 5 postoperative complications: acute kidney injury (AKI), delirium, deep vein thrombosis (DVT), pulmonary embolism (PE), and pneumonia. Patient and clinical characteristics available preoperatively, intraoperatively, and a combination of both were used as inputs for 5 candidate ML models: logistic regression, support vector machine, random forest, gradient boosting tree (GBT), and deep neural network (DNN). Model performance was compared using the area under the receiver operating characteristic curve (AUROC). Model interpretations were generated using Shapley Additive Explanations by transforming model features into clinical variables and representing them as patient-specific visualizations.

RESULTS

A total of 111 888 patients (mean [SD] age, 54.4 [16.8] years; 56 915 [50.9%] female; 82 533 [73.8%] White) were included in this study. The best-performing model for each complication combined the preoperative and intraoperative data with the following AUROCs: pneumonia (GBT), 0.905 (95% CI, 0.903-0.907); AKI (GBT), 0.848 (95% CI, 0.846-0.851); DVT (GBT), 0.881 (95% CI, 0.878-0.884); PE (DNN), 0.831 (95% CI, 0.824-0.839); and delirium (GBT), 0.762 (95% CI, 0.759-0.765). Performance of models that used only preoperative data or only intraoperative data was marginally lower than that of models that used combined data. When adding variables with missing data as input, AUROCs increased from 0.588 to 0.905 for pneumonia, 0.579 to 0.848 for AKI, 0.574 to 0.881 for DVT, 0.5 to 0.831 for PE, and 0.6 to 0.762 for delirium. The Shapley Additive Explanations analysis generated model-agnostic interpretation that illustrated significant clinical contributors associated with risks of postoperative complications.

CONCLUSIONS AND RELEVANCE

The ML models for predicting postoperative complications with model-agnostic interpretation offer opportunities for integrating risk predictions for clinical decision support. Such real-time clinical decision support can mitigate patient risks and help in anticipatory management for perioperative contingency planning.

Predicting outcomes of pelvic exenteration using machine learning

AIM

We aim to compare machine learning with neural network performance in predicting R0 resection (R0), length of stay > 14 days (LOS), major complication rates at 30 days postoperatively (COMP) and survival greater than 1 year (SURV) for patients having pelvic exenteration for locally advanced and recurrent rectal cancer.

METHOD

A deep learning computer was built and the programming environment was established. The PelvEx Collaborative database was used which contains anonymized data on patients who underwent pelvic exenteration for locally advanced or locally recurrent colorectal cancer between 2004 and 2014. Logistic regression, a support vector machine and an artificial neural network (ANN) were trained. Twenty per cent of the data were used as a test set for calculating prediction accuracy for R0, LOS, COMP and SURV. Model performance was measured by plotting receiver operating characteristic (ROC) curves and calculating the area under the ROC curve (AUROC).

RESULTS

Machine learning models and ANNs were trained on 1147 cases. The AUROC for all outcome predictions ranged from 0.608 to 0.793 indicating modest to moderate predictive ability. The models performed best at predicting LOS > 14 days with an AUROC of 0.793 using preoperative and operative data. Visualized logistic regression model weights indicate a varying impact of variables on the outcome in question.

CONCLUSION

This paper highlights the potential for predictive modelling of large international databases. Current data allow moderate predictive ability of both complex ANNs and more classic methods.

Prescriptive analytics for reducing 30-day hospital readmissions after general surgery

INTRODUCTION

New financial incentives, such as reduced Medicare reimbursements, have led hospitals to closely monitor their readmission rates and initiate efforts aimed at reducing them. In this context, many surgical departments participate in the American College of Surgeons National Surgical Quality Improvement Program (NSQIP), which collects detailed demographic, laboratory, clinical, procedure and perioperative occurrence data. The availability of such data enables the development of data science methods which predict readmissions and, as done in this paper, offer specific recommendations aimed at preventing readmissions.

MATERIALS AND METHODS

This study leverages NSQIP data for 722,101 surgeries to develop predictive and prescriptive models, predicting readmissions and offering real-time, personalized treatment recommendations for surgical patients during their hospital stay, aimed at reducing the risk of a 30-day readmission. We applied a variety of classification methods to predict 30-day readmissions and developed two prescriptive methods to recommend pre-operative blood transfusions to increase the patient's hematocrit with the objective of preventing readmissions. The effect of these interventions was evaluated using several predictive models.

RESULTS

Predictions of 30-day readmissions based on the entire collection of NSQIP variables achieve an out-of-sample accuracy of 87% (Area Under the Curve-AUC). Predictions based only on pre-operative variables have an accuracy of 74% AUC, out-of-sample. Personalized interventions, in the form of pre-operative blood transfusions identified by the prescriptive methods, reduce readmissions by 12%, on average, for patients considered as candidates for pre-operative transfusion (pre-operative hematoctic <30). The prediction accuracy of the proposed models exceeds results in the literature.

CONCLUSIONS

This study is among the first to develop a methodology for making specific, data-driven, personalized treatment recommendations to reduce the 30-day readmission rate. The reported predicted reduction in readmissions can lead to more than $20 million in savings in the U.S. annually.

Clinical Implementation of Predictive Models Embedded within Electronic Health Record Systems: A Systematic Review

Predictive analytics using electronic health record (EHR) data have rapidly advanced over the last decade. While model performance metrics have improved considerably, best practices for implementing predictive models into clinical settings for point-of-care risk stratification are still evolving. Here, we conducted a systematic review of articles describing predictive models integrated into EHR systems and implemented in clinical practice. We conducted an exhaustive database search and extracted data encompassing multiple facets of implementation. We assessed study quality and level of evidence. We obtained an initial 3393 articles for screening, from which a final set of 44 articles was included for data extraction and analysis. The most common clinical domains of implemented predictive models were related to thrombotic disorders/anticoagulation (25%) and sepsis (16%). The majority of studies were conducted in inpatient academic settings. Implementation challenges included alert fatigue, lack of training, and increased work burden on the care team. Of 32 studies that reported effects on clinical outcomes, 22 (69%) demonstrated improvement after model implementation. Overall, EHR-based predictive models offer promising results for improving clinical outcomes, although several gaps in the literature remain, and most study designs were observational. Future studies using randomized controlled trials may help improve the generalizability of findings.

Identification of postoperative complications using electronic health record data and machine learning

BACKGROUND

Using the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) complication status of patients who underwent an operation at the University of Colorado Hospital, we developed a machine learning algorithm for identifying patients with one or more complications using data from the electronic health record (EHR).

METHODS

We used an elastic-net model to estimate regression coefficients and carry out variable selection. International classification of disease codes (ICD-9), common procedural terminology (CPT) codes, medications, and CPT-specific complication event rate were included as predictors.

RESULTS

Of 6840 patients, 922 (13.5%) had at least one of the 18 complications tracked by NSQIP. The model achieved 88% specificity, 83% sensitivity, 97% negative predictive value, 52% positive predictive value, and an area under the curve of 0.93.

CONCLUSIONS

Using machine learning on EHR postoperative data linked to NSQIP outcomes data, a model with 163 predictors from the EHR identified complications well at our institution.

Prevalence of Machine Learning in Craniofacial Surgery

Machine learning (ML) revolves around the concept of using experience to teach computer-based programs to reliably perform specific tasks. Healthcare setting is an ideal environment for adaptation of ML applications given the multiple specific tasks that could be allocated to computer programs to perform. There have been several scoping reviews published in literature looking at the general acceptance and adaptability of surgical specialities to ML applications, but very few focusing on the application towards craniofacial surgery. This study aims to present a detailed scoping review regarding the use of ML applications in craniofacial surgery.

Machine learning in infection management using routine electronic health records: tools, techniques, and reporting of future technologies

BACKGROUND

Machine learning (ML) is increasingly being used in many areas of health care. Its use in infection management is catching up as identified in a recent review in this journal. We present here a complementary review to this work.

OBJECTIVES

To support clinicians and researchers in navigating through the methodological aspects of ML approaches in the field of infection management.

SOURCES

A Medline search was performed with the keywords artificial intelligence, machine learning, infection∗, and infectious disease∗ for the years 2014-2019. Studies using routinely available electronic hospital record data from an inpatient setting with a focus on bacterial and fungal infections were included.

CONTENT

Fifty-two studies were included and divided into six groups based on their focus. These studies covered detection/prediction of sepsis (n = 19), hospital-acquired infections (n = 11), surgical site infections and other postoperative infections (n = 11), microbiological test results (n = 4), infections in general (n = 2), musculoskeletal infections (n = 2), and other topics (urinary tract infections, deep fungal infections, antimicrobial prescriptions; n = 1 each). In total, 35 different ML techniques were used. Logistic regression was applied in 18 studies followed by random forest, support vector machines, and artificial neural networks in 18, 12, and seven studies, respectively. Overall, the studies were very heterogeneous in their approach and their reporting. Detailed information on data handling and software code was often missing. Validation on new datasets and/or in other institutions was rarely done. Clinical studies on the impact of ML in infection management were lacking.

IMPLICATIONS

Promising approaches for ML use in infectious diseases were identified. But building trust in these new technologies will require improved reporting. Explainability and interpretability of the models used were rarely addressed and should be further explored. Independent model validation and clinical studies evaluating the added value of ML approaches are needed.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Predicting diabetes-related hospitalizations based on electronic health records

Objective: To derive a predictive model to identify patients likely to be hospitalized during the following year due to complications attributed to Type II diabetes. Methods: A variety of supervised machine learning classification methods were tested and a new method that discovers hidden patient clusters in the positive class (hospitalized) was developed while, at the same time, sparse linear support vector machine classifiers were derived to separate positive samples from the negative ones (non-hospitalized). The convergence of the new method was established and theoretical guarantees were proved on how the classifiers it produces generalize to a test set not seen during training. Results: The methods were tested on a large set of patients from the Boston Medical Center - the largest safety net hospital in New England. It is found that our new joint clustering/classification method achieves an accuracy of 89% (measured in terms of area under the ROC Curve) and yields informative clusters which can help interpret the classification results, thus increasing the trust of physicians to the algorithmic output and providing some guidance towards preventive measures. While it is possible to increase accuracy to 92% with other methods, this comes with increased computational cost and lack of interpretability. The analysis shows that even a modest probability of preventive actions being effective (more than 19%) suffices to generate significant hospital care savings. Conclusions: Predictive models are proposed that can help avert hospitalizations, improve health outcomes and drastically reduce hospital expenditures. The scope for savings is significant as it has been estimated that in the USA alone, about $5.8 billion are spent each year on diabetes-related hospitalizations that could be prevented.

Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study

BACKGROUND

Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk.

METHODS AND FINDINGS

A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis.

CONCLUSIONS

Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.