Accuracy of electronic surveillance of catheter-associated urinary tract infection at an academic medical center

Automated Identification of Postoperative Infections to Allow Prediction and Surveillance Based on Electronic Health Record Data: Scoping Review

BACKGROUND

Postoperative infections remain a crucial challenge in health care, resulting in high morbidity, mortality, and costs. Accurate identification and labeling of patients with postoperative bacterial infections is crucial for developing prediction models, validating biomarkers, and implementing surveillance systems in clinical practice.

OBJECTIVE

This scoping review aimed to explore methods for identifying patients with postoperative infections using electronic health record (EHR) data to go beyond the reference standard of manual chart review.

METHODS

We performed a systematic search strategy across PubMed, Embase, Web of Science (Core Collection), the Cochrane Library, and Emcare (Ovid), targeting studies addressing the prediction and fully automated surveillance (ie, without manual check) of diverse bacterial infections in the postoperative setting. For prediction modeling studies, we assessed the labeling methods used, categorizing them as either manual or automated. We evaluated the different types of EHR data needed for the surveillance and labeling of postoperative infections, as well as the performance of fully automated surveillance systems compared with manual chart review.

RESULTS

We identified 75 different methods and definitions used to identify patients with postoperative infections in studies published between 2003 and 2023. Manual labeling was the predominant method in prediction modeling research, 65% (49/75) of the identified methods use structured data, and 45% (34/75) use free text and clinical notes as one of their data sources. Fully automated surveillance systems should be used with caution because the reported positive predictive values are between 0.31 and 0.76.

CONCLUSIONS

There is currently no evidence to support fully automated labeling and identification of patients with infections based solely on structured EHR data. Future research should focus on defining uniform definitions, as well as prioritizing the development of more scalable, automated methods for infection detection using structured EHR data.

Development and validation of models for detection of postoperative infections using structured electronic health records data and machine learning

BACKGROUND

Postoperative infections constitute more than half of all postoperative complications. Surveillance of these complications is primarily done through manual chart review, which is time consuming, expensive, and typically only covers 10% to 15% of all operations. Automated surveillance would permit the timely evaluation of and reporting of all operations.

METHODS

The goal of this study was to develop and validate parsimonious, interpretable models for conducting surveillance of postoperative infections using structured electronic health records data. This was a retrospective study using 30,639 unique operations from 5 major hospitals between 2013 and 2019. Structured electronic health records data were linked to postoperative outcomes data from the American College of Surgeons National Surgical Quality Improvement Program. Predictors from the electronic health records included diagnoses, procedures, and medications. Infectious complications included surgical site infection, urinary tract infection, sepsis, and pneumonia within 30 days of surgery. The knockoff filter, a penalized regression technique that controls type I error, was applied for variable selection. Models were validated in a chronological held-out dataset.

RESULTS

Seven percent of patients experienced at least one type of postoperative infection. Models selected contained between 4 and 8 variables and achieved >0.91 area under the receiver operating characteristic curve, >81% specificity, >87% sensitivity, >99% negative predictive value, and 10% to 15% positive predictive value in a held-out test dataset.

CONCLUSION

Surveillance and reporting of postoperative infection rates can be implemented for all operations with high accuracy using electronic health records data and simple linear regression models.

Diagnostic accuracy of electronic surveillance tool for catheter-associated urinary tract infections in tertiary care hospitals: A meta-analysis

BACKGROUND

Automated systems have been developed to reduce labor-intensive manual recordings during nosocomial infection surveillance. The diagnostic accuracies of these systems have differed in various settings.

METHODS

We designed this meta-analysis to evaluate the diagnostic accuracy of an electronic surveillance tool for catheter-associated urinary tract infections (CAUTIs) in tertiary care hospitals. We systematically searched databases such as Medline, Scopus, Cochrane library and Embase (from inception until November 2019) for relevant studies. We assessed the quality of trials using the diagnostic accuracy studies-2 tool, and performed a meta-analysis to obtain a pooled sensitivity and specificity for electronic surveillance. We included 6 studies with 16,492 patients in the analysis.

RESULTS

We found a pooled sensitivity of electronic diagnostic surveillance for CAUTIs of 97.5% (95% confidence interval [CI], 67.6-99.9%) and a pooled specificity of 92.6% (95% CI, 55.2-99.2%). The diagnostic odds ratio was 494 (95% CI, 89-2747). The positive likelihood ratio was 13.1 (95% CI, 1.63-105.8) and the negative likelihood ratio 0.02 (95% CI, 0.001-0.40). A bivariate box plot indicated the possibility of heterogeneity between the included studies.

CONCLUSION

Our review suggests that electronic surveillance is useful for diagnosing CAUTIs among hospitalized patients in tertiary care hospitals due to its high sensitivity and specificity.

Electronically assisted surveillance systems of healthcare-associated infections: a systematic review

Background

Surveillance of healthcare-associated infections (HAI) is the basis of each infection control programme and, in case of acute care hospitals, should ideally include all hospital wards, medical specialties as well as all types of HAI. Traditional surveillance is labour intensive and electronically assisted surveillance systems (EASS) hold the promise to increase efficiency.

Objectives

To give insight in the performance characteristics of different approaches to EASS and the quality of the studies designed to evaluate them.

Methods

In this systematic review, online databases were searched and studies that compared an EASS with a traditional surveillance method were included. Two different indicators were extracted from each study, one regarding the quality of design (including reporting efficiency) and one based on the performance (e.g. specificity and sensitivity) of the EASS presented.

Results

A total of 78 studies were included. The majority of EASS (n = 72) consisted of an algorithm-based selection step followed by confirmatory assessment. The algorithms used different sets of variables. Only a minority (n = 7) of EASS were hospital-wide and designed to detect all types of HAI. Sensitivity of EASS was generally high (> 0.8), but specificity varied (0.37–1). Less than 20% (n = 14) of the studies presented data on the efficiency gains achieved.

Conclusions

Electronically assisted surveillance of HAI has yet to reach a mature stage and to be used routinely in healthcare settings. We recommend that future studies on the development and implementation of EASS of HAI focus on thorough validation, reproducibility, standardised datasets and detailed information on efficiency.

Identification of postoperative complications using electronic health record data and machine learning

BACKGROUND

Using the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) complication status of patients who underwent an operation at the University of Colorado Hospital, we developed a machine learning algorithm for identifying patients with one or more complications using data from the electronic health record (EHR).

METHODS

We used an elastic-net model to estimate regression coefficients and carry out variable selection. International classification of disease codes (ICD-9), common procedural terminology (CPT) codes, medications, and CPT-specific complication event rate were included as predictors.

RESULTS

Of 6840 patients, 922 (13.5%) had at least one of the 18 complications tracked by NSQIP. The model achieved 88% specificity, 83% sensitivity, 97% negative predictive value, 52% positive predictive value, and an area under the curve of 0.93.

CONCLUSIONS

Using machine learning on EHR postoperative data linked to NSQIP outcomes data, a model with 163 predictors from the EHR identified complications well at our institution.

Identification of urinary tract infections using electronic health record data

BACKGROUND

Population ascertainment of postoperative urinary tract infections (UTIs) is time-consuming and expensive, as it often requires manual chart review. Using the American College of Surgeons National Surgical Quality Improvement Program UTI status of patients who underwent an operation at the University of Colorado Hospital, we sought to develop an algorithm for identifying UTIs using data from the electronic health record.

METHODS

Data were split into training (operations occurring between 2013-2015) and test (operations in 2016) sets. A binomial generalized linear model with an elastic-net penalty was used to fit the model and carry out variables selection. International classification of disease codes, common procedural terminology codes, antibiotics, catheterization, and common procedural terminology-specific UTI event rates were included as predictors. The Youden's J statistic was used to determine the optimal classification threshold.

RESULTS

Of 6,840 patients, 134 (2.0%) had a UTI. The model achieved 92% specificity, 80% sensitivity, 100% negative predictive value, 16% positive predictive value, and an area under the curve of 0.94 using a decision threshold of 0.03.

CONCLUSIONS

A model with 14 predictors from the electronic health record identifies UTIs well, and it could be used to scale up UTI surveillance or to estimate the impact of large-scale interventions on UTI rates.

Electronic Surveillance For Catheter-Associated Urinary Tract Infection Using Natural Language Processing

Catheter-associated urinary tract infection (CAUTI) is a common and costly healthcare-associated infection, yet measuring it accurately is challenging and resource-intensive. Electronic surveillance promises to make this task more objective and efficient in an era of new financial and regulatory imperatives, but previous surveillance approaches have used a simplified version of the definition. We applied a complete definition, including subjective elements identified through natural language processing of clinical notes. Through examination of documentation practices, we defined a set of rules that identified positively and negatively asserted symptoms of CAUTI. Our algorithm was developed on a training set of 1421 catheterizedpatients and prospectively validated on 1567 catheterizedpatients. Compared to gold standard chart review, our tool had a sensitivity of 97.1%, specificity of 94.5% PPV of 66.7% and NPV of 99.6% for identifying CAUTI. We discuss sources of error and suggestions for more computable future definitions.

Community -and hospital laboratory-based surveillance for respiratory viruses

Traditional surveillance for respiratory viruses relies on symptom detection and laboratory detection during medically attended encounters for acute respiratory infection/influenza-like illness (ARI/ILI). Ecological momentary reporting using text messages is a novel method for surveillance. This study compares respiratory viral activity detected through longitudinal community-based surveillance using text message responses for sample acquisition and testing to respiratory viral activity obtained from hospital laboratory data from the same community. We demonstrate a significant correlation between community- and hospital laboratory-based surveillance for most respiratory viruses, although the relative proportions of viruses detected in the community and hospital differed significantly.

Data elements and validation methods used for electronic surveillance of health care-associated infections: a systematic review

BACKGROUND

We describe the primary data sources, data elements, and validation methods currently used in electronic surveillance systems (ESS) for identification and surveillance of health care-associated infections (HAIs), and compares these data elements and validation methods with recommended standards.

METHODS

Using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, a PubMed and manual search was conducted to identify research articles describing ESS for identification and surveillance of HAIs published January 1, 2009-August 31, 2014. Selected articles were evaluated to determine what data elements and validation methods were included.

RESULTS

Among the 509 articles identified in the original literature search, 30 met the inclusion criteria. Whereas the majority of studies (83%) used recommended data sources and validated the numerator (80%), only 10% of studies performed external and internal validation. In addition, there was variation in the ESS data formats used.

CONCLUSIONS

Our findings suggest that the majority of ESS for HAI surveillance use standard definitions, but the lack of widespread internal data, denominator, and external validation in these systems reduces the reliability of their findings. Additionally, advanced programming skills are required to create, implement, and maintain these systems and to reduce the variability in data formats.