Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool

Procalcitonin for safe reduction of unnecessary blood cultures in the emergency department: Development and validation of a prediction model

OBJECTIVES

Blood cultures (BCs) are commonly ordered in emergency departments (EDs), while a minority yields a relevant pathogen. Diagnostic stewardship is needed to safely reduce unnecessary BCs. We aimed to develop and validate a bacteremia prediction model for ED patients, with specific focus on the benefit of incorporating procalcitonin.

METHODS

We included adult patients with suspected bacteremia from a Dutch ED for a one-year period. We defined 23 candidate predictors for a "full model", of which nine were used for an automatable "basic model". Variations of both models with C-reactive protein and procalcitonin were constructed using LASSO regression, with bootstrapping for internal validation. External validation was done in an independent cohort of patients with confirmed infection from 71 Spanish EDs. We assessed discriminative performance using the C-statistic and calibration with calibration curves. Clinical usefulness was evaluated by sensitivity, specificity, saved BCs, and Net Benefit.

RESULTS

Among 2111 patients in the derivation cohort (mean age 63 years, 46% male), 273 (13%) had bacteremia, versus 896 (20%) in the external cohort (n = 4436). Adding procalcitonin substantially improved performance for all models. The basic model with procalcitonin showed most promise, with a C-statistic of 0.87 (0.86-0.88) upon external validation. At a 5% risk threshold, it showed a sensitivity of 99% and could have saved 29% of BCs while only missing 10 out of 896 (1.1%) bacteremia patients.

CONCLUSIONS

Procalcitonin-based bacteremia prediction models can safely reduce unnecessary BCs at the ED. Further validation is needed across a broader range of healthcare settings.

Blood culture algorithm implementation in emergency department patients as a diagnostic stewardship intervention

OBJECTIVE

Blood cultures (BCx) are important for selecting appropriate antibiotic treatment. Ordering BCx for conditions with a low probability of bacteremia has limited utility, thus improved guidance for ordering BCx is needed. Inpatient studies have implemented BCx algorithms, but no studies examine the intervention in an Emergency Department (ED) setting.

METHODS

We performed a quasi-experimental pre and postintervention study from January 12, 2020, to October 31, 2023, at a single academic adult ED and implemented a BCx algorithm. The primary outcome was the blood culture event rates (BCE per 100 ED admissions) pre and postintervention. Secondary outcomes included adverse event rates (30-day ED and hospital readmission and antibiotic days of therapy). Seven ED physicians and APP reviewed BCx for appropriateness, with monthly feedback provided to ED leadership and physicians.

RESULTS

After the BCx algorithm implementation, the BCE rate decreased from 12.17 BCE/100 ED admissions to 10.50 BCE/100 ED admissions. Of the 3,478 reviewed BCE, we adjudicated 2,153 BCE (62%) as appropriate, 653 (19%) as inappropriate, and 672 (19%) as uncertain. Adverse safety events were not statistically different pre and postintervention.

CONCLUSIONS

Implementation of an ED BCx algorithm demonstrated a reduction in BCE, without increased adverse safety events. Future studies should compare outcomes of BCx algorithm implementation in a community hospital ED without intensive chart review.

Artificial intelligence and prescription of antibiotic therapy: present and future

INTRODUCTION

In the past few years, the use of artificial intelligence in healthcare has grown exponentially. Prescription of antibiotics is not exempt from its rapid diffusion, and various machine learning (ML) techniques, from logistic regression to deep neural networks and large language models, have been explored in the literature to support decisions regarding antibiotic prescription.

AREAS COVERED

In this narrative review, we discuss promises and challenges of the application of ML-based clinical decision support systems (ML-CDSSs) for antibiotic prescription. A search was conducted in PubMed up to April 2024.

EXPERT OPINION

Prescribing antibiotics is a complex process involving various dynamic phases. In each of these phases, the support of ML-CDSSs has shown the potential, and also the actual ability in some studies, to favorably impacting relevant clinical outcomes. Nonetheless, before widely exploiting this massive potential, there are still crucial challenges ahead that are being intensively investigated, pertaining to the transparency of training data, the definition of the sufficient degree of prediction explanations when predictions are obtained through black box models, and the legal and ethical framework for decision responsibility whenever an antibiotic prescription is supported by ML-CDSSs.

A Comprehensive Evaluation of Emergency Department Utilization by Patients With Cirrhosis

INTRODUCTION

Emergency department (ED)-based care is required for cirrhosis management, yet the burden of cirrhosis-related ED healthcare utilization is understudied. We aimed to describe ED utilization within a statewide health system and compare the outcomes of high ED use (HEDU) vs non-HEDU in individuals with cirrhosis.

METHODS

We retrospectively reviewed charts of adults with cirrhosis who presented to any of 16 EDs within the Indiana University Health system in 2021. Patient characteristics, features of the initial ED visit, subsequent 90-day healthcare use, and 360-day outcomes were collected. Multivariable logistic regression models were used to identify predictors HEDU status which was defined as ≥2 ED visits within 90 days after the index ED visit.

RESULTS

There were 2,124 eligible patients (mean age 61.3 years, 53% male, and 91% White). Major etiologies of cirrhosis were alcohol (38%), metabolic dysfunction-associated steatohepatitis (27%), and viral hepatitis (21%). Cirrhosis was newly diagnosed in the ED visit for 18.4%. Most common reasons for ED visits were abdominal pain (21%), shortness of breath (19%), and ascites/volume overload (16%). Of the initial ED visits, 20% (n = 424) were potentially avoidable. The overall 90-day mortality was 16%. Within 90 days, there were 366 HEDU (20%). Notable variables independently associated with HEDU were model for end-stage liver disease-sodium (adjusted odds ratio [aOR] 1.044, 95% confidence interval [CI] 1.005-1.085), prior ED encounter (aOR 1.520, 95% CI 1.136-2.034), and avoidable initial ED visit (aOR 1.938, 95% CI 1.014-3.703).

DISCUSSION

Abdominal pain, shortness of breath, and ascites/fluid overload are the common presenting reasons for ED visits for patients with cirrhosis. Patients with cirrhosis presenting to the ED experience a 90-day mortality rate of 16%, and among those who initially visited the ED, 20% were HEDU. We identified several variables independently associated with HEDU. Our observations pave the way for developing interventions to optimize the care of patients with cirrhosis presenting to the ED and to lower repeated ED visits.

Appropriate use of blood cultures in the emergency department through machine learning (ABC): study protocol for a randomised controlled non-inferiority trial

INTRODUCTION

The liberal use of blood cultures in emergency departments (EDs) leads to low yields and high numbers of false-positive results. False-positive, contaminated cultures are associated with prolonged hospital stays, increased antibiotic usage and even higher hospital mortality rates. This trial aims to investigate whether a recently developed and validated machine learning model for predicting blood culture outcomes can safely and effectively guide clinicians in withholding unnecessary blood culture analysis.

METHODS AND ANALYSIS

A randomised controlled, non-inferiority trial comparing current practice with a machine learning-guided approach. The primary objective is to determine whether the machine learning based approach is non-inferior to standard practice based on 30-day mortality. Secondary outcomes include hospital length-of stay and hospital admission rates. Other outcomes include model performance and antibiotic usage. Participants will be recruited in the EDs of multiple hospitals in the Netherlands. A total of 7584 participants will be included.

ETHICS AND DISSEMINATION

Possible participants will receive verbal information and a paper information brochure regarding the trial. They will be given at least 1 hour consideration time before providing informed consent. Research results will be published in peer-reviewed journals. This study has been approved by the Amsterdam University Medical Centers' local medical ethics review committee (No 22.0567). The study will be conducted in concordance with the principles of the Declaration of Helsinki and in accordance with the Medical Research Involving Human Subjects Act, General Data Privacy Regulation and Medical Device Regulation.

TRIAL REGISTRATION NUMBER

NCT06163781.

Diagnostic stewardship in infectious diseases: a scoping review

Introduction. The term 'diagnostic stewardship' is relatively new, with a recent surge in its use within the literature. Despite its increasing popularity, a precise definition remains elusive. Various attempts have been made to define it, with some viewing it as an integral part of antimicrobial stewardship. The World Health Organization offers a broad definition, emphasizing the importance of timely, accurate diagnostics. However, inconsistencies in the use of this term still persist, necessitating further clarification.Gap Statement. There are currently inconsistencies in the definition of diagnostic stewardship used within the academic literature.Aim. This scoping review aims to categorize the use of diagnostic stewardship approaches and define this approach by identifying common characteristics and factors of its use within the literature.Methodology. This scoping review undertook a multi-database search from date of inception until October 2022. Any observational or experimental study where the authors define the intervention to be diagnostic stewardship from any clinical area was included. Screening of all papers was undertaken by a single reviewer with 10% verification by a second reviewer. Data extraction was undertaken by a single reviewer using a pre-piloted form. Given the wide variation in study design and intervention outcomes, a narrative synthesis approach was applied. Studies were clustered around common diagnostic stewardship interventions where appropriate.Results. After duplicate removal, a total of 1310 citations were identified, of which, after full-paper screening, 105 studies were included in this scoping review. The classification of an intervention as taking a diagnostic stewardship approach is a relatively recent development, with the first publication in this field dating back to 2017. The majority of research in this area has been conducted within the USA, with very few studies undertaken outside this region. Visual inspection of the citation map reveals that the current evidence base is interconnected, with frequent references to each other's work. The interventions commonly adopt a restrictive approach, utilizing hard and soft stops within the pre-analytical phase to restrict access to testing. Upon closer examination of the outcomes, it becomes evident that there is a predominant focus on reducing the number of tests rather than enhancing the current test protocol. This is further reflected in the limited number of studies that report on test performance (including protocol improvements, specificity and sensitivity).Conclusion. Diagnostic stewardship seems to have deviated from its intended course, morphing into a rather rudimentary instrument wielded not to enhance but to constrict the scope of testing. Despite the World Health Organization's advocacy for an ideology that promotes a more comprehensive approach to quality improvement, it may be more appropriate to consider alternative regional narratives when categorizing these types of quality improvement interventions.

A customised down-sampling machine learning approach for sepsis prediction

OBJECTIVE

Sepsis is a life-threatening condition in the ICU and requires treatment in time. Despite the accuracy of existing sepsis prediction models, insufficient focus on reducing alarms could worsen alarm fatigue and desensitisation in ICUs, potentially compromising patient safety. In this retrospective study, we aim to develop an accurate, robust, and readily deployable method in ICUs, only based on the vital signs and laboratory tests.

METHODS

Our method consists of a customised down-sampling process and a specific dynamic sliding window and XGBoost to offer sepsis prediction. The down-sampling process was applied to the retrospective data for training the XGBoost model. During the testing stage, the dynamic sliding window and the trained XGBoost were used to predict sepsis on the retrospective datasets, PhysioNet and FHC.

RESULTS

With the filtered data from PhysioNet, our method achieved 80.74% accuracy (77.90% sensitivity and 84.42% specificity) and 83.95% (84.82% sensitivity and 82.00% specificity) on the test set of PhysioNet-A and PhysioNet-B, respectively. The AUC score was 0.89 for both datasets. On the FHC dataset, our method achieved 92.38% accuracy (88.37% sensitivity and 95.16% specificity) and 0.98 AUC score on the test set of FHC.

CONCLUSION

Our results indicate that the down-sampling process and the dynamic sliding window with XGBoost brought robust and accurate performance to give sepsis prediction under various hospital settings. The localisation and robustness of our method can assist in sepsis diagnosis in different ICU settings.

Impact of Blood Culture Contamination on Antibiotic Use, Resource Utilization, and Clinical Outcomes: A Retrospective Cohort Study in Dutch and US Hospitals

Background

Blood culture contamination (BCC) has been associated with prolonged antibiotic use (AU) and increased health care utilization; however, this has not been widely reevaluated in the era of increased attention to antibiotic stewardship. We evaluated the impact of BCC on AU, resource utilization, and length of stay in Dutch and US patients.

Methods

This retrospective observational study examined adults admitted to 2 hospitals in the Netherlands and 5 hospitals in the United States undergoing ≥2 blood culture (BC) sets. Exclusion criteria included neutropenia, no hospital admission, or death within 48 hours of hospitalization. The impact of BCC on clinical outcomes-overall inpatient days of antibiotic therapy, test utilization, length of stay, and mortality-was determined via a multivariable regression model.

Results

An overall 22 927 patient admissions were evaluated: 650 (4.1%) and 339 (4.8%) with BCC and 11 437 (71.8%) and 4648 (66.3%) with negative BC results from the Netherlands and the United States, respectively. Dutch and US patients with BCC had a mean ± SE 1.74 ± 0.27 (P < .001) and 1.58 ± 0.45 (P < .001) more days of antibiotic therapy than patients with negative BC results. They also had 0.6 ± 0.1 (P < .001) more BCs drawn. Dutch but not US patients with BCC had longer hospital stays (3.36 days; P < .001). There was no difference in mortality between groups in either cohort. AU remained higher in US but not Dutch patients with BCC in a subanalysis limited to BC obtained within the first 24 hours of admission.

Conclusions

BCC remains associated with higher inpatient AU and health care utilization as compared with patients with negative BC results, although the impact on these outcomes differs by country.

Detecting changes in the performance of a clinical machine learning tool over time

BACKGROUND

Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models.

METHODS

A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time.

FINDINGS

Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%.

INTERPRETATION

Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool.

FUNDING

No funding to disclose.

Real-time artificial intelligence system for bacteremia prediction in adult febrile emergency department patients

BACKGROUND

Artificial intelligence (AI) holds significant potential to be a valuable tool in healthcare. However, its application for predicting bacteremia among adult febrile patients in the emergency department (ED) remains unclear. Therefore, we conducted a study to provide clarity on this issue.

METHODS

Adult febrile ED patients with blood cultures at Chi Mei Medical Center were divided into derivation (January 2017 to June 2019) and validation groups (July 2019 to December 2020). The derivation group was utilized to develop AI models using twenty-one feature variables and five algorithms to predict bacteremia. The performance of these models was compared with qSOFA score. The AI model with the highest area under the receiver operating characteristics curve (AUC) was chosen to implement the AI prediction system and tested on the validation group.

RESULTS

The study included 5,647 febrile patients. In the derivation group, there were 3,369 patients with a mean age of 61.4 years, and 50.7% were female, including 508 (13.8%) with bacteremia. The model with the best AUC was built using the random forest algorithm (0.761), followed by logistic regression (0.755). All five models demonstrated better AUC than the qSOFA score (0.560). The random forest model was adopted to build a real-time AI prediction system integrated into the hospital information system, and the AUC achieved 0.709 in the validation group.

CONCLUSION

The AI model shows promise to predict bacteremia in adult febrile ED patients; however, further external validation in different hospitals and populations is necessary to verify its effectiveness.

Comparison of Administrative versus Electronic Health Record-based Methods for Identifying Sepsis Hospitalizations

Rationale: Despite the importance of sepsis surveillance, no optimal approach for identifying sepsis hospitalizations exists. The Centers for Disease Control and Prevention Adult Sepsis Event Definition (CDC-ASE) is an electronic medical record-based algorithm that yields more stable estimates over time than diagnostic coding-based approaches but may still result in misclassification. Objectives: We sought to assess three approaches to identifying sepsis hospitalizations, including a modified CDC-ASE. Methods: This cross-sectional study included patients in the Veterans Affairs Ann Arbor Healthcare System admitted via the emergency department (February 2021 to February 2022) with at least one episode of acute organ dysfunction within 48 hours of emergency department presentation. Patients were assessed for community-onset sepsis using three methods: 1) explicit diagnosis codes, 2) the CDC-ASE, and 3) a modified CDC-ASE. The modified CDC-ASE required at least two systemic inflammatory response syndrome criteria instead of blood culture collection and had a more sensitive definition of respiratory dysfunction. Each method was compared with a reference standard of physician adjudication via medical record review. Patients were considered to have sepsis if they had at least one episode of acute organ dysfunction graded as "definitely" or "probably" infection related on physician review. Results: Of 821 eligible hospitalizations, 449 were selected for physician review. Of these, 98 (21.8%) were classified as sepsis by medical record review, 103 (22.9%) by the CDC-ASE, 132 (29.4%) by the modified CDC-ASE, and 37 (8.2%) by diagnostic codes. Accuracy was similar across the three methods of interest (80.6% for the CDC-ASE, 79.6% for the modified CDC-ADE, and 84.2% for diagnostic codes), but sensitivity and specificity varied. The CDC-ASE algorithm had sensitivity of 58.2% (95% confidence interval [CI], 47.2-68.1%) and specificity of 86.9% (95% CI, 82.9-90.2%). The modified CDC-ASE algorithm had greater sensitivity (69.4% [95% CI, 59.3-78.3%]) but lower specificity (81.8% [95% CI, 77.3-85.7%]). Diagnostic codes had lower sensitivity (32.7% [95% CI, 23.5-42.9%]) but greater specificity (98.6% [95% CI, 96.7-99.55%]). Conclusions: There are several approaches to identifying sepsis hospitalizations for surveillance that have acceptable accuracy. These approaches yield varying sensitivity and specificity, so investigators should carefully consider the test characteristics of each method before determining an appropriate method for their intended use.

Machine learning pipeline for blood culture outcome prediction using Sysmex XN-2000 blood sample results in Western Australia

BACKGROUND

Bloodstream infections (BSIs) are a significant burden on the global population and represent a key area of focus in the hospital environment. Blood culture (BC) testing is the standard diagnostic test utilised to confirm the presence of a BSI. However, current BC testing practices result in low positive yields and overuse of the diagnostic test. Diagnostic stewardship research regarding BC testing is increasing, and becoming more important to reduce unnecessary resource expenditure and antimicrobial use, especially as antimicrobial resistance continues to rise. This study aims to establish a machine learning (ML) pipeline for BC outcome prediction using data obtained from routinely analysed blood samples, including complete blood count (CBC), white blood cell differential (DIFF), and cell population data (CPD) produced by Sysmex XN-2000 analysers.

METHODS

ML models were trained using retrospective data produced between 2018 and 2019, from patients at Sir Charles Gairdner hospital, Nedlands, Western Australia, and processed at Pathwest Laboratory Medicine, Nedlands. Trained ML models were evaluated using stratified 10-fold cross validation.

RESULTS

Two ML models, an XGBoost model using CBC/DIFF/CPD features with boruta feature selection (BFS) , and a random forest model trained using CBC/DIFF features with BFS were selected for further validation after obtaining AUC scores of [Formula: see text] and [Formula: see text] respectively using stratified 10-fold cross validation. The XGBoost model obtained an AUC score of 0.76 on a internal validation set. The random forest model obtained AUC scores of 0.82 and 0.76 on internal and external validation datasets respectively.

CONCLUSIONS

We have demonstrated the utility of using an ML pipeline combined with CBC/DIFF, and CBC/DIFF/CPD feature spaces for BC outcome prediction. This builds on the growing body of research in the area of BC outcome prediction, and provides opportunity for further research.

Machine learning of cell population data, complete blood count, and differential count parameters for early prediction of bacteremia among adult patients with suspected bacterial infections and blood culture sampling in emergency departments

BACKGROUND

Bacteremia is a life-threatening complication of infectious diseases. Bacteremia can be predicted using machine learning (ML) models, but these models have not utilized cell population data (CPD).

METHODS

The derivation cohort from emergency department (ED) of China Medical University Hospital (CMUH) was used to develop the model and was prospectively validated in the same hospital. External validation was performed using cohorts from ED of Wei-Gong Memorial Hospital (WMH) and Tainan Municipal An-Nan Hospital (ANH). Adult patients who underwent complete blood count (CBC), differential count (DC), and blood culture tests were enrolled in the present study. The ML model was developed using CBC, DC, and CPD to predict bacteremia from positive blood cultures obtained within 4 h before or after the acquisition of CBC/DC blood samples.

RESULTS

This study included 20,636 patients from CMUH, 664 from WMH, and 1622 patients from ANH. Another 3143 patients were included in the prospective validation cohort of CMUH. The CatBoost model achieved an area under the receiver operating characteristic curve of 0.844 in the derivation cross-validation, 0.812 in the prospective validation, 0.844 in the WMH external validation, and 0.847 in the ANH external validation. The most valuable predictors of bacteremia in the CatBoost model were the mean conductivity of lymphocytes, nucleated red blood cell count, mean conductivity of monocytes, and neutrophil-to-lymphocyte ratio.

CONCLUSIONS

ML model that incorporated CBC, DC, and CPD showed excellent performance in predicting bacteremia among adult patients with suspected bacterial infections and blood culture sampling in emergency departments.

Embracing cohort heterogeneity in clinical machine learning development: a step toward generalizable models

This study is a simple illustration of the benefit of averaging over cohorts, rather than developing a prediction model from a single cohort. We show that models trained on data from multiple cohorts can perform significantly better in new settings than models based on the same amount of training data but from just a single cohort. Although this concept seems simple and obvious, no current prediction model development guidelines recommend such an approach.

Artificial Intelligence Can Guide Antibiotic Choice in Recurrent UTIs and Become an Important Aid to Improve Antimicrobial Stewardship

BACKGROUND

A correct approach to recurrent urinary tract infections (rUTIs) is an important pillar of antimicrobial stewardship. We aim to define an Artificial Neural Network (ANN) for predicting the clinical efficacy of the empiric antimicrobial treatment in women with rUTIs.

METHODS

We extracted clinical and microbiological data from 1043 women. We trained an ANN on 725 patients and validated it on 318.

RESULTS

The ANN showed a sensitivity of 87.8% and specificity of 97.3% in predicting the clinical efficacy of empirical therapy. The previous use of fluoroquinolones (HR = 4.23; p = 0.008) and cephalosporins (HR = 2.81; p = 0.003) as well as the presence of Escherichia coli with resistance against cotrimoxazole (HR = 3.54; p = 0.001) have been identified as the most important variables affecting the ANN output decision predicting the fluoroquinolones-based therapy failure. A previous isolation of Escherichia coli with resistance against fosfomycin (HR = 2.67; p = 0.001) and amoxicillin-clavulanic acid (HR = 1.94; p = 0.001) seems to be the most influential variable affecting the output decision predicting the cephalosporins- and cotrimoxazole-based therapy failure. The previously mentioned Escherichia coli with resistance against cotrimoxazole (HR = 2.35; p < 0.001) and amoxicillin-clavulanic acid (HR = 3.41; p = 0.007) seems to be the most influential variable affecting the output decision predicting the fosfomycin-based therapy failure.

CONCLUSIONS

ANNs seem to be an interesting tool to guide the antimicrobial choice in the management of rUTIs at the point of care.