Development and validation of models for detection of postoperative infections using structured electronic health records data and machine learning

Automated Identification of Postoperative Infections to Allow Prediction and Surveillance Based on Electronic Health Record Data: Scoping Review

BACKGROUND

Postoperative infections remain a crucial challenge in health care, resulting in high morbidity, mortality, and costs. Accurate identification and labeling of patients with postoperative bacterial infections is crucial for developing prediction models, validating biomarkers, and implementing surveillance systems in clinical practice.

OBJECTIVE

This scoping review aimed to explore methods for identifying patients with postoperative infections using electronic health record (EHR) data to go beyond the reference standard of manual chart review.

METHODS

We performed a systematic search strategy across PubMed, Embase, Web of Science (Core Collection), the Cochrane Library, and Emcare (Ovid), targeting studies addressing the prediction and fully automated surveillance (ie, without manual check) of diverse bacterial infections in the postoperative setting. For prediction modeling studies, we assessed the labeling methods used, categorizing them as either manual or automated. We evaluated the different types of EHR data needed for the surveillance and labeling of postoperative infections, as well as the performance of fully automated surveillance systems compared with manual chart review.

RESULTS

We identified 75 different methods and definitions used to identify patients with postoperative infections in studies published between 2003 and 2023. Manual labeling was the predominant method in prediction modeling research, 65% (49/75) of the identified methods use structured data, and 45% (34/75) use free text and clinical notes as one of their data sources. Fully automated surveillance systems should be used with caution because the reported positive predictive values are between 0.31 and 0.76.

CONCLUSIONS

There is currently no evidence to support fully automated labeling and identification of patients with infections based solely on structured EHR data. Future research should focus on defining uniform definitions, as well as prioritizing the development of more scalable, automated methods for infection detection using structured EHR data.

The association between participation in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) and postoperative outcomes: A comprehensive analysis of 7,474,298 patients

INTRODUCTION

Prior publications about the association between participation in the American College of Surgeons National Surgical Quality Improvement Program and improved postoperative outcomes have reported mixed results. We aimed to perform a comprehensive analysis of preoperative characteristics and unadjusted and risk-adjusted postoperative complication rates over time in the American College of Surgeons National Surgical Quality Improvement Program dataset.

METHODS

We used the American College of Surgeons National Surgical Quality Improvement Program database, 2005 to 2018, to analyze preoperative patient characteristics and unadjusted and risk-adjusted rates of adverse postoperative outcomes by year. Expected events were calculated using multiple logistic regression, with each complication as the dependent variable and the 28 non-laboratory preoperative American College of Surgeons National Surgical Quality Improvement Program variables as the independent variables. Annual observed-to-expected ratios for each outcome were used to risk-adjust outcomes over time.

RESULTS

The analytic cohort included 7,474,298 operations across 9 surgical specialties. Both the preoperative patient risk and the unadjusted rate of postoperative complications decreased over time. While the observed-to-expected ratio for mortality remained around 1, the observed-to-expected ratios for the other outcomes decreased over time from 2005 to 2018, except for the following cardiac complications: overall morbidity 1.11 (95% confidence interval: 1.10-1.13) to 0.97 (0.96-0.98); pulmonary 1.18 (1.15-1.21) to 0.91 (0.89-0.92); infection 1.19 (1.16-1.21) to 1.01 (1.00-1.01); urinary tract infection 1.29 (1.23-1.34) to 0.87 (0.86-0.89); venous thromboembolism 1.10 (1.03-1.16) to 0.92 (0.90-0.94) ; cardiac 0.76 (0.70-0.81) to 1.04 (1.01-1.07); renal 1.14 (1.08-1.21) to 0.96 (0.93-0.99); stroke 1.12 (1.00-1.25) to 0.98 (0.94-1.03); and bleeding 1.35 (1.33-1.36) to 0.80 (0.79-0.81).

CONCLUSION

Hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program have experienced a decrease in risk-adjusted postoperative surgical complications over time in all areas except for mortality and cardiac complications.

Artificial intelligence and machine learning applications in urinary tract infections identification and prediction: a systematic review and meta-analysis

BACKGROUND

Urinary tract infections (UTIs) have been one of the most common bacterial infections in clinical practice worldwide. Artificial intelligence (AI) and machine learning (ML) based algorithms have been increasingly applied in UTI case identification and prediction. However, the overall performance of AI/ML algorithms in identifying and predicting UTI has not been evaluated. The purpose of this paper is to quantitatively evaluate the application value of AI/ML in identifying and predicting UTI cases.

METHODS

MEDLINE, EMBASE, Web of Science, and PubMed databases were systematically searched for articles published up to December 31, 2023. Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2) and Prediction Model Risk of Bias Assessment Tool (PROBAST) were used to assess the risk of bias. Study characteristics and detailed algorithm information were extracted. Pooled sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were synthesized using a bivariate mix-effects model. Meta-regression and subgroup analysis were conducted to test the source of heterogeneity.

RESULTS

In total, 11 studies with 14 AI/ML models were included in the final meta-analysis. The overall pooled AUC was 0.89 (95%CI 0.86-0.92). Additionally, the pooled Sen, Spe, PLR, NLR, and DOR were 0.78 (95%CI 0.71-0.84), 0.89 (95%CI 0.83-0.93), 6.99 (95%CI 4.38-11.14), 0.25 (95%CI 0.18-0.34) and 28.07 (95%CI 14.27-55.20), respectively. The results of meta-regression suggested that reference standard definitions might be the source of heterogeneity.

CONCLUSION

AI/ML algorithms appear to be promising to help clinicians detect and identify patients at high risk of UTIs. However, further studies are demanded to evaluate the application value of AI/ML more thoroughly.

Predictive modeling for identifying infection risk following spinal surgery: Optimizing patient management

Infection is known to occur in a substantial proportion of patients following spinal surgery and predictive modeling may provide a useful means for identifying those at higher risk of complications and poor prognosis, which could help optimize pre- and postoperative management strategies. The outcome measure of the present study was to investigate the occurrence of all-cause infection during hospitalization following scoliosis surgery. To meet this aim, the present study retrospectively analyzed 370 patients who underwent surgery at the Second Affiliated Hospital, Zhejiang University School of Medicine (Hangzhou, China) between January 2016 and October 2022, and patients who either experienced or did not experience all-cause infection while in hospital were compared in terms of their clinicodemographic characteristics, surgical variables and laboratory test results. Logistic regression was subsequently applied to data from a subset of patients in order to build a model to predict infection, which was validated using another subset of patients. All-cause, in-hospital postoperative infections were found to have occurred in 66/370 patients (17.8%). The following variables were included in a predictive model: Sex, American Society of Anesthesiologists (ASA) classification, body mass index (BMI), diabetes mellitus, hypertension, preoperative levels of white blood cells and preoperative C-reactive protein (CRP) and duration of surgery. The model exhibited an area under the curve of 0.776 against the internal validation set. In conclusion, dynamic nomograms based on sex, ASA classification, BMI, diabetes mellitus, hypertension, preoperative levels of white blood cells and CRP and duration of surgery may have the potential to be a clinically useful predictor of all-cause infection following scoliosis. The predictive model constructed in the present study may potentially facilitate the real-time visualization of risk factors associated with all-cause infection following surgical procedures.

An explainable long short-term memory network for surgical site infection identification

BACKGROUND

Currently, surgical site infection surveillance relies on labor-intensive manual chart review. Recently suggested solutions involve machine learning to identify surgical site infections directly from the medical record. Deep learning is a form of machine learning that has historically performed better than traditional methods while being harder to interpret. We propose a deep learning model, a long short-term memory network, for the identification of surgical site infection from the medical record with an attention layer for explainability.

METHODS

We retrieved structured data and clinical notes from the University of Utah Health System's electronic health care record for operative events randomly selected for manual chart review from January 2016 to June 2021. Surgical site infection occurring within 30 days of surgery was determined according to the National Surgical Quality Improvement Program definition. We trained the long short-term memory model along with traditional machine learning models for comparison. We calculated several performance metrics from a holdout test set and performed additional analyses to understand the performance of the long short-term memory, including an explainability analysis.

RESULTS

Surgical site infection was present in 4.7% of the total 9,185 operative events. The area under the receiver operating characteristic curve and sensitivity of the long short-term memory was higher (area under the receiver operating characteristic curve: 0.954, sensitivity: 0.920) compared to the top traditional model (area under the receiver operating characteristic curve: 0.937, sensitivity: 0.736). The top 5 features of the long short-term memory included 2 procedure codes and 3 laboratory values.

CONCLUSION

Surgical site infection surveillance is vital for the reduction of surgical site infection rates. Our explainable long short-term memory achieved a comparable area under the receiver operating characteristic curve and greater sensitivity when compared to traditional machine learning methods. With explainable deep learning, automated surgical site infection surveillance could replace burdensome manual chart review processes.

Using risk-adjusted cumulative sum to evaluate surgeon, divisional, and institutional outcomes-a feasibility study

BACKGROUND

Few objective, real-time measurements of surgeon performance exist. The risk-adjusted cumulative sum is a novel method that can track surgeon-level outcomes on a continuous basis. The objective of this study was to demonstrate the feasibility of using risk-adjusted cumulative sum to monitor outcomes after colorectal operations and identify clinically relevant performance variations.

METHODS

The National Surgical Quality Improvement Program was queried to obtain patient-level data for 1,603 colorectal operations at a high-volume center from 2011 to 2020. For each case, expected risks of morbidity, mortality, reoperation, readmission, and prolonged length of stay were estimated using the National Surgical Quality Improvement Program risk calculator. Risk-adjusted cumulative sum curves were generated to signal observed-to-expected odds ratios of 1.5 (poor performance) and 0.5 (exceptional performance). Control limits were set based on a false positive rate of 5% (α = 0.05).

RESULTS

The cohort included data on 7 surgeons (those with more than 20 cases in the study period). Institutional observed versus expected outcomes were the following: morbidity 12.5% (vs 15.0%), mortality 2.5% (vs 2.0%), prolonged length of stay 19.7% (vs 19.1%), reoperation 11.1% (vs 11.3%), and 30-day readmission 6.1% (vs 4.8%). Risk-adjusted cumulative sum accurately demonstrated within- and between-surgeon performance variations across these metrics and proved effective when considering division-level data.

CONCLUSION

Risk-adjusted cumulative sum adjusts for patient-level risk factors to provide real-time data on surgeon-specific outcomes. This approach enables prompt identification of performance outliers and can contribute to quality assurance, root-cause analysis, and incentivization not only at the surgeon level but at divisional and institutional levels as well.

Preoperative Prediction of Postoperative Infections Using Machine Learning and Electronic Health Record Data

OBJECTIVE

To estimate preoperative risk of postoperative infections using structured electronic health record (EHR) data.

BACKGROUND

Surveillance and reporting of postoperative infections is primarily done through costly, labor-intensive manual chart reviews on a small sample of patients. Automated methods using statistical models applied to postoperative EHR data have shown promise to augment manual review as they can cover all operations in a timely manner. However, there are no specific models for risk-adjusting infectious complication rates using EHR data.

METHODS

Preoperative EHR data from 30,639 patients (2013-2019) were linked to the American College of Surgeons National Surgical Quality Improvement Program preoperative data and postoperative infection outcomes data from 5 hospitals in the University of Colorado Health System. EHR data included diagnoses, procedures, operative variables, patient characteristics, and medications. Lasso and the knockoff filter were used to perform controlled variable selection. Outcomes included surgical site infection, urinary tract infection, sepsis/septic shock, and pneumonia up to 30 days postoperatively.

RESULTS

Among >15,000 candidate predictors, 7 were chosen for the surgical site infection model and 6 for each of the urinary tract infection, sepsis, and pneumonia models. Important variables included preoperative presence of the specific outcome, wound classification, comorbidities, and American Society of Anesthesiologists physical status classification. The area under the receiver operating characteristic curve for each model ranged from 0.73 to 0.89.

CONCLUSIONS

Parsimonious preoperative models for predicting postoperative infection risk using EHR data were developed and showed comparable performance to existing American College of Surgeons National Surgical Quality Improvement Program risk models that use manual chart review. These models can be used to estimate risk-adjusted postoperative infection rates applied to large volumes of EHR data in a timely manner.

Innovative Techniques for Infection Control and Surveillance in Hospital Settings and Long-Term Care Facilities: A Scoping Review

Healthcare-associated infections (HAIs) pose significant challenges in healthcare systems, with preventable surveillance playing a crucial role. Traditional surveillance, although effective, is resource-intensive. The development of new technologies, such as artificial intelligence (AI), can support traditional surveillance in analysing an increasing amount of health data or meeting patient needs. We conducted a scoping review, following the PRISMA-ScR guideline, searching for studies of new digital technologies applied to the surveillance, control, and prevention of HAIs in hospitals and LTCFs published from 2018 to 4 November 2023. The literature search yielded 1292 articles. After title/abstract screening and full-text screening, 43 articles were included. The mean study duration was 43.7 months. Surgical site infections (SSIs) were the most-investigated HAI and machine learning was the most-applied technology. Three main themes emerged from the thematic analysis: patient empowerment, workload reduction and cost reduction, and improved sensitivity and personalization. Comparative analysis between new technologies and traditional methods showed different population types, with machine learning methods examining larger populations for AI algorithm training. While digital tools show promise in HAI surveillance, especially for SSIs, challenges persist in resource distribution and interdisciplinary integration in healthcare settings, highlighting the need for ongoing development and implementation strategies.

Formative evaluation of the development and implementation of the automated surveillance of postoperative infections tool

BACKGROUND

The gold standard for detecting postoperative complications uses databases like the American College of Surgeons National Surgical Quality Improvement Program, a multi-centered database based on manual chart review. However, their limitations and costs have led many centers to discontinue participation. Novel techniques to detect postoperative complications must be developed and implemented with surgeon involvement, which is paramount to their adoption. We sought to assess surgeons' opinions of a newly developed postoperative complication detection tool, the Automated Surveillance of Postoperative Infections, within the contextual clinical environment.

METHODS

This was a multi-site qualitative formative evaluation of surgeon perceptions of the Automated Surveillance of Postoperative Infections. We conducted semi-structured interviews and focus groups with surgeons and presented the Automated Surveillance of Postoperative Infections concept. Important domains and constructs, as categorized by Consolidated Framework for Implementation Research, were identified to support the successful adoption and implementation of the Automated Surveillance of Postoperative Infections.

RESULTS

Twenty-four surgeons with 10 surgical subspecialties were interviewed. The following 4 main themes were found: (1) perception of the Automated Surveillance of Postoperative Infections tool-to provide important data that can improve and support clinical outcomes; (2) environment for implementation-description of factors to support or impede implementation; (3) adaptability of the Automated Surveillance of Postoperative Infections-to work with the complexity of surgical cases; and (4) the Automated Surveillance of Postoperative Infections report format and details.

CONCLUSIONS

We successfully captured the perspectives and suggestions of surgeons to improve the Automated Surveillance of Postoperative Infections and potential barriers during the initial development phase. Barriers included fear of punitive action from reports and complex surgical cases. Facilitators identified were the need to improve clinical outcomes and organizational support. The results of this formative evaluation will be used to further develop Automated Surveillance of Postoperative Infections, starting with a prototype, the Automated Surveillance of Postoperative Infections 1.0.

Quantifying and Interpreting Inequality in Surgical Site Infections per Quarter Among Anesthetizing Locations and Specialties

Background Earlier studies have shown that prevention of surgical site infection can achieve net cost savings when targeted to operating rooms with the most surgical site infections. Methodology This retrospective cohort study included all 231,057 anesthetics between May 2017 and June 2022 at a large teaching hospital. The anesthetics were administered in operating rooms, procedure rooms, radiology, and other sites. The 8,941 postoperative infections were identified from International Classification of Diseases diagnosis codes relevant to surgical site infections documented during all follow-up encounters over 90 days postoperatively. To quantify the inequality in the counts of infections among anesthetizing locations, the Gini index was used, with the Gini index being proportional to the sum of the absolute pairwise differences among anesthetizing locations in the counts of infections. Results The Gini index for infections among the 112 anesthetizing locations at the hospital was 0.64 (99% confidence interval = 0.56 to 0.71). The value of 0.64 is so large that, for comparison, it exceeds nearly all countries' Gini index for income inequality. The 50% of locations with the fewest infections accounted for 5% of infections. The 10% of locations with the most infections accounted for 40% of infections and 15% of anesthetics. Among the 57 operating room locations, there was no association between counts of cases and infections (Spearman correlation coefficient r = 0.01). Among the non-operating room locations (e.g., interventional radiology), there was a significant association (Spearman r = 0.79). Conclusions Targeting specific anesthetizing locations is important for the multiple interventions to reduce surgical site infections that represent fixed costs irrespective of the number of patients (e.g., specialized ventilatory systems and nightly ultraviolet-C disinfection).