Inconsistency of a model aimed at predicting bacteremia in hospitalized patients

Derivation and External Validation of the Ottawa Bloodstream Infection Model for Acutely Ill Adults

BACKGROUND

Knowing the probability that patients have a bloodstream infection (BSI) could influence the ordering of blood cultures and interpretation of their preliminary results. Many previous BSI probability models have limited applicability and accuracy. This study used currently recommended modeling techniques and a large sample to derive and validate the Ottawa BSI Model.

METHODS

At a tertiary care teaching hospital, we retrieved a random sample of 4180 adults having blood cultures in our emergency department or during the initial 48 h of the encounter. Variable selection was based on clinical experience and a systematic review of previous model performance. Model performance was measured in a temporal external validation group of 4680 patients.

RESULTS

A total of 327 derivation patients had a BSI (8.0%). BSI risk increased with increased number of culture sets (2 sets: adjusted odds ratio [aOR] 1.52 [1.10-2.11]; 3 sets: 1.99 [0.86-4.58]); with indwelling catheter (aOR 2.07 [1.34-3.20); with increasing temperature, heart rate, and neutrophil-lymphocyte ratio; and with decreasing systolic blood pressure, platelet count, urea-creatinine ratio, and estimated glomerular filtration rate. In the temporal external validation group, model discrimination was good (c-statistic 0.71 [0.69-0.74]) and calibration was very good (integrated calibration index .016 [.010-.024]). Exclusion of validation patients with acute SARS-CoV-2 infection improved discrimination slightly (c-statistic 0.73 [0.69-0.76]).

CONCLUSIONS

The Ottawa BSI Model uses commonly available data to return an expected BSI probability for acutely ill patients. However, it cannot exclude BSI and its complexity requires computational assistance to use.

Concurrent external validation of bloodstream infection probability models

OBJECTIVE

Accurately estimating the likelihood of bloodstream infection (BSI) can help clinicians make diagnostic and therapeutic decisions. Many multivariate models predicting BSI probability have been published. This study measured the performance of BSI probability models within the same patient sample.

METHODS

We retrieved validated BSI probability models included in a recently published systematic review that returned a patient-level BSI probability for adults. Model applicability, discrimination, and accuracy was measured in a simple random sample of 4485 admitted adults having blood cultures ordered in the emergency department or the initial 48 hours of hospitalization.

RESULTS

Ten models were included (publication years 1991-2015). Common methodological threats to model performance included overfitting and continuous variable categorization. Restrictive inclusion criteria caused seven models to apply to <15% of validation patients. Model discrimination was less than originally reported in derivation groups (median c-statistic 60%, range 48-69). The observed BSI risk frequently deviated from expected (median integrated calibration index 4.0%, range 0.8-12.4). Notable disagreement in expected BSI probabilities was seen between models (median (25th-75th percentile) relative difference between expected risks 68.0% (28.6-113.6%)).

DISCUSSION

In a large randomly selected external validation population, many published BSI probability models had restricted applicability, limited discrimination and calibration, and extensive inter-model disagreement. Direct comparison of model performance is hampered by dissimilarities between model-specific validation groups.

Prediction of Bacteremia Based on 12-Year Medical Data Using a Machine Learning Approach: Effect of Medical Data by Extraction Time

Early detection of bacteremia is important to prevent antibiotic abuse. Therefore, we aimed to develop a clinically applicable bacteremia prediction model using machine learning technology. Data from two tertiary medical centers’ electronic medical records during a 12-year-period were extracted. Multi-layer perceptron (MLP), random forest, and gradient boosting algorithms were applied for machine learning analysis. Clinical data within 12 and 24 hours of blood culture were analyzed and compared. Out of 622,771 blood cultures, 38,752 episodes of bacteremia were identified. In MLP with 128 hidden layer nodes, the area under the receiver operating characteristic curve (AUROC) of the prediction performance in 12- and 24-h data models was 0.762 (95% confidence interval (CI); 0.7617–0.7623) and 0.753 (95% CI; 0.7520–0.7529), respectively. AUROC of causative-pathogen subgroup analysis predictive value for Acinetobacter baumannii bacteremia was the highest at 0.839 (95% CI; 0.8388–0.8394). Compared to primary bacteremia, AUROC of sepsis caused by pneumonia was highest. Predictive performance of bacteremia was superior in younger age groups. Bacteremia prediction using machine learning technology appeared possible for acute infectious diseases. This model was more suitable especially to pneumonia caused by Acinetobacter baumannii. From the 24-h blood culture data, bacteremia was predictable by substituting only the continuously variable values.

Early Detection of Bacteraemia Using Ten Clinical Variables with an Artificial Neural Network Approach

An adequate model for predicting bacteraemia has not yet been developed. This study aimed to evaluate the performance of an artificial neural network (ANN)-based prediction model in comparison with previous statistical models. The performance of multi-layer perceptron (MLP), a representative ANN model, was verified via comparison with a non-neural network model. A total of 1260 bacteraemia episodes were identified in 13,402 patients. In MLP with 128 hidden layer nodes, the area under the receiver operating characteristic curve (AUC) of the prediction performance was 0.729 (95% confidence interval [CI]; 0.712–0.728), while in MLP with 256 hidden layer nodes, it was 0.727 (95% CI; 0.713–0.727). In a conventional Bayesian statistical method, the AUC was 0.7. The aforementioned two MLP models exhibited the highest sensitivity (0.810). The ranking of clinical variables was used to describe the influential power of the prediction. Serum alkaline phosphatase was one of the most influential clinical variables, and one-out search was the best ranking method for measuring the influence of the clinical variables. Furthermore, adding variables beyond the 10 top-ranking ones did not significantly affect the prediction of bacteraemia. The ANN model is not inferior to conventional statistical approaches. Bacteraemia can be predicted using only the top 10 clinical variables determined by a ranking method, and the model can be used in clinical practice by applying real-time monitoring.

Machine learning for fast identification of bacteraemia in SIRS patients treated on standard care wards: a cohort study

Bacteraemia is a life-threating condition requiring immediate diagnostic and therapeutic actions. Blood culture (BC) analyses often result in a low true positive result rate, indicating its improper usage. A predictive model might assist clinicians in deciding for whom to conduct or to avoid BC analysis in patients having a relevant bacteraemia risk. Predictive models were established by using linear and non-linear machine learning methods. To obtain proper data, a unique data set was collected prior to model estimation in a prospective cohort study, screening 3,370 standard care patients with suspected bacteraemia. Data from 466 patients fulfilling two or more systemic inflammatory response syndrome criteria (bacteraemia rate: 28.8%) were finally used. A 29 parameter panel of clinical data, cytokine expression levels and standard laboratory markers was used for model training. Model tuning was performed in a ten-fold cross validation and tuned models were validated in a test set (80:20 random split). The random forest strategy presented the best result in the test set validation (ROC-AUC: 0.729, 95%CI: 0.679–0.779). However, procalcitonin (PCT), as the best individual variable, yielded a similar ROC-AUC (0.729, 95%CI: 0.679–0.779). Thus, machine learning methods failed to improve the moderate diagnostic accuracy of PCT.

Systemic Inflammatory Response Syndrome Is Not an Indicator of Bacteremia in Hemodialysis Patients With Native Accesses: A Multicenter Study

Bloodstream infection (BSI) in hemodialysis (HD) patients is often difficult to diagnose. Systemic inflammatory response syndrome (SIRS) is a sensitive predictor of BSI in the general population. We aimed to assess the usefulness of SIRS in predicting BSI in HD patients. We designed a multicenter retrospective observational study of adult (age > 18 years) HD patients who underwent two sets of blood cultures for suspected BSI at first hospital visit from August 2011 to July 2012. Clinical, biological, and microbial data were evaluated to evaluate SIRS as a predictor of BSI upon initial presentation to the hospital. Data were obtained from 279 HD patients. Vascular access other than arteriovenous fistula and subcutaneously fixed superficial artery, and those administered antimicrobial drugs before visit were excluded; thus, a total of 202 patients were finally enrolled. Mean patient age was 71 years, 67.3% were male, 49.3% had diabetes, 28.2% had indwelling hardware, and 18.3% patients had BSI. Endocarditis and vertebral osteomyelitis were common infection sites, and Staphylococcus aureus was the most common pathogen. Of those with SIRS, 25.3% had BSI and 74.7% did not (odds ratio for SIRS, 2.10; 95% confidence interval, 0.90-4.91; p = 0.11). Thus, SIRS had a low sensitivity for predicting BSI in HD patients (sensitivity, 71.9%; specificity, 45.2%; positive likelihood ratio, 1.31; negative likelihood ratio, 0.62). Systemic inflammatory response syndrome has low sensitivity in identifying BSI in HD patients. A low threshold for drawing blood cultures and initiating antibiotic treatment should be considered for HD patients.

Predicting bacteraemia in validated models--a systematic review

Bacteraemia is associated with high mortality. Although many models for predicting bacteraemia have been developed, not all have been validated, and even when they were, the validation processes varied. We identified validated models that have been developed; asked whether they were successful in defining groups with a very low or high prevalence of bacteraemia; and whether they were used in clinical practice. Electronic databases were searched to identify studies that underwent validation on prediction of bacteraemia in adults. We included only studies that were able to define groups with low or high probabilities for bacteraemia (arbitrarily defined as below 3% or above 30%). Fifteen publications fulfilled inclusion criteria, including 59 276 patients. Eleven were prospective and four retrospective. Study populations and the parameters included in the different models were heterogeneous. Ten studies underwent internal validation; the model performed well in all of them. Twelve performed external validation. Of the latter, seven models were validated in a different hospital, using a new independent database. In five of these, the model performed well. After contacting authors, we found that none of the models was implemented in clinical practice. We conclude that heterogeneous studies have been conducted in different defined groups of patients with limited external validation. Significant savings to the system and the individual patient can be gained by refraining from performing blood cultures in groups of patients in which the probability of true bacteraemia is very low, while the probability of contamination is constant. Clinical trials of existing or new models should be done to examine whether models are helpful and safe in clinical use, preferably multicentre in order to secure utility and safety in diverse clinical settings.

A new statistical approach to predict bacteremia using electronic medical records

BACKGROUND

Previous attempts to predict bacteremia have focused on selecting significant variables. However, these approaches have had limitations such as poor reproducibility in prediction accuracy and inconsistency in predictor selection. Here we propose a Bayesian approach to predict bacteremia based on the statistical distributions of clinical variables of previous patients, which has recently become possible through the adoption of electronic medical records.

METHODS

In a derivation cohort, Bayesian prediction models were derived and their discriminative performance was compared with previous models under varying combinations of predictors. Then the Bayesian models were prospectively tested in a validation cohort. According to Bayesian probabilities of bacteremia, patients in both cohorts were grouped into bacteremia risk groups.

RESULTS

Using the same prediction variables, the Bayesian predictions were more accurate than conventional rule-based predictions. Moreover, their better discriminative performance remained consistent despite variations in clinical variables. The receiver operating characteristic (ROC) area of the Bayesian model with 20 predictors was 0.70 ± 0.007 in the derivation cohort and 0.70 ± 0.018 in the validation cohort. The prevalence of bacteremia in groups I, II, and VI (grouped according to probability ratio) were 1.9%, 3.4%, and 20.0% in the derivation cohort, and 0.4%, 3.2%, and 18.4% in the validation cohort, respectively. The overall prevalence of bacteremia was 6.9% in both cohorts.

CONCLUSIONS

In the present study, the Bayesian prediction model showed stable performance in predicting bacteremia and identifying risk groups, as the previous models did. The clinical significance of the Bayesian approach is expected to be demonstrated through a multicenter trial.

Factors associated with positive blood cultures in outpatients with suspected bacteremia

Blood cultures are routinely taken in outpatients with fever and suspected bacterial infections. However, in the majority of cases, they are not informative and of limited value for clinical decision making. The aim of this study was therefore to investigate factors associated with positive blood cultures in outpatients presenting to an outpatient clinic and emergency room. This was a case-control study of all outpatients with positive blood cultures from January 1, 2006 to October 31, 2007 and matched control patients with negative blood cultures in the same time period. Microbiology results and medical charts were reviewed to determine factors associated with positive blood cultures. The presence of a systemic inflammation response syndrome (SIRS) (OR 2.7, 95% Cl 1.0-7.2) and increased C-reactive protein (CRP) (OR 1.1 per 10 mg/l, 95% Cl 1.0-1.2) were the most powerful predictive values for the development of positive blood cultures. In positive cases serum albumin was lower (35 mg/l versus 39 mg/l) than in controls. SIRS, increasing CRP and low albumin were associated with positive blood cultures in outpatients. With simple clinical assessment and few laboratory tests indicative of infection, it is possible to define a group at higher risk for bacteremia in outpatients.

Additional value of procalcitonin for diagnosis of infection in patients with fever at the emergency department

OBJECTIVE

First, to determine whether procalcitonin (PCT) significantly adds diagnostic value in terms of sensitivity and specificity to a common set of markers of infection, including C-reactive protein (CRP), at the Emergency Department. Second, to create a simple scoring rule implementing PCT values. Third, to determine and compare associations of CRP and PCT with clinical outcomes.

DESIGN

The additional diagnostic value of PCT was determined using multiple logistic regression analysis. A score was developed to help distinguish patients with a culture-proven bacterial infection from patients not needing antibiotic treatment using 16 potential clinical and laboratory variables. The prognostic value of CRP and PCT was determined using Spearman's correlation and logistic regression.

SETTING

Emergency Department of a 310-bed teaching hospital.

PATIENTS

Patients between 18 and 85 years old presenting with fever to the Emergency Department.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

A total of 211 patients were studied (infection confirmed, n = 73; infection likely, n = 58; infection not excluded, n = 46; no infection, n = 34). CRP and chills were the strongest predictors for the diagnosis of bacterial infection. After addition of PCT to these parameters, model fit significantly improved (p = .003). The resulting scoring rule (score = 0.01 * CRP + 2 * chills + 1 * PCT) was characterized by an AUC value of 0.83 (sensitivity 79%; specificity of 71%), which was more accurate than physician judgment or SIRS (systemic inflammatory response syndrome). PCT levels were significantly associated with admission to a special care unit, duration of intravenous antibiotic use, total duration of antibiotic treatment, and length of hospital stay, whereas CRP was related only to the latter two variables.

CONCLUSIONS

These data suggest that PCT may be a valuable addition to currently used markers of infection for diagnosis of infection and prognosis in patients with fever at the Emergency Department.

Updated review of blood culture contamination

Blood culture contamination represents an ongoing source of frustration for clinicians and microbiologists alike. Ambiguous culture results often lead to diagnostic uncertainty in clinical management and are associated with increased health care costs due to unnecessary treatment and testing. A variety of strategies have been investigated and employed to decrease contamination rates. In addition, numerous approaches to increase our ability to distinguish between clinically significant bacteremia and contamination have been explored. In recent years, there has been an increase in the application of computer-based tools to support infection control activities as well as provide clinical decision support related to the management of infectious diseases. Finally, new approaches for estimating bacteremia risk which have the potential to decrease unnecessary blood culture utilization have been developed and evaluated. In this review, we provide an overview of blood culture contamination and describe the potential utility of a variety of approaches to improve both detection and prevention. While it is clear that progress is being made, fundamental challenges remain.

Prediction of bacteremia using TREAT, a computerized decision-support system

BACKGROUND

Prediction of bloodstream infection at the time of sepsis onset allows one to make appropriate and economical management decisions.

METHODS

The TREAT computerized decision-support system uses a causal probabilistic network, which is locally calibrated, to predict cases of bacteremia. We assessed the system's performance in 2 independent cohorts that included patients with suspected sepsis. Both studies were conducted in Israel, Italy, and Germany. Data were collected prospectively and were entered into the TREAT system at the time that blood samples were obtained for culture. Discriminative power was assessed using a receiver-operating characteristics curve.

RESULTS

In the first cohort, 790 patients were included. The area under the receiver-operating characteristics curve for prediction of bacteremia using the TREAT system was 0.68 (95% confidence interval [CI], 0.63-0.73). We used TREAT's prediction values to draw thresholds defining a low-, intermediate-, and high-risk groups for bacteremia, in which 3 (2.4%) of 123, 62 (12.8%) of 483, and 55 (29.9%) of 184 patients were bacteremic, respectively. In the second cohort, 1724 patients were included. The area under the receiver-operating characteristics curve was 0.70 (95% CI, 0.67-0.73). The prevalence of bacteremia observed in the low-, intermediate-, and high-risk groups defined by the first cohort were 1.3% (4 of 300 patients), 13.2% (150 of 1139 patients), and 28.1% (80 of 285 patients), respectively. The low-risk groups in the 2 cohorts comprised 15%-17% of all patients. Performance was stable in the 3 sites.

CONCLUSIONS

Using variables available at the time that blood cultures were performed, the TREAT system successfully stratified patients on the basis of the risk for bacteremia. The system's predictions were stable in 3 locations. The TREAT system can define a low-risk group of inpatients with suspected sepsis for whom blood cultures may not be needed.

The degree of chills for risk of bacteremia in acute febrile illness

PURPOSE

Patients with acute febrile illness may experience different degrees of chills. To evaluate the different degrees of chills in predicting risk of bacteremia in patients with acute febrile illness, we performed a single-center prospective observational study.

METHODS

We enrolled consecutive adult patients with acute febrile illness presenting to our emergency department. We defined mild chills as cold-feeling equivalent such as the need of an outer jacket; moderate chills as the need for a thick blanket; and shaking chills as whole-body shaking even under a thick blanket. We estimated risk ratios of the different degrees of chills for bacteremia using multivariable adjusted Poisson regression.

RESULTS

Of a total 526 patients, 40 patients (7.6%) had bacteremia. There were 65 patients (12.4%) with shaking chills, 100 (19%) with moderate chills, and 105 (20%) with mild chills. By comparing patients with no chills, the risk ratios of bacteremia were 12.1 (95% confidence interval [CI] 4.1-36.2) for shaking chills, 4.1 (95% CI 1.6-10.7) for moderate chills, and 1.8 (95% CI 0.9-3.3) for mild chills. Shaking chills showed a specificity of 90.3% (95% CI 89.2-91.5) and positive likelihood ratio of 4.65 (95% CI 2.95-6.86). The absence of chills showed a sensitivity of 87.5% (95% CI 74.4-94.5) and negative likelihood ratio of 0.24 (95% CI 0.11-0.51).

CONCLUSION

Evaluation of the degree of chills is important for estimating risk of bacteremia in patients with acute febrile illness. The more severe degree of chills suggests the higher risk of bacteremia.

A simple prediction algorithm for bacteraemia in patients with acute febrile illness

BACKGROUND

Existing prediction models for the risk of bacteraemia are complex and difficult to use. Physicians are likely to use a model only if it is simple and sensitive.

AIM

To develop a simple classification algorithm predicting the risk of bacteraemia.

DESIGN

Hospital-based study.

METHODS

We enrolled 526 adult consecutive patients with acute febrile illness (40 with bacteraemia) presenting to the emergency department at a community hospital in Okinawa, Japan. Recursive partitioning analysis was used to build the classification algorithm with V-fold cross-validation. We used two clinical scenarios: in the first, laboratory tests were not available; in the second, they were.

RESULTS

The two prediction algorithms generated three different risk groups for bacteraemia. In the first scenario, the important variables were chills, pulse, and physician diagnosis of a low-risk site. The low-risk group from this first algorithm included 68% of the total patients; sensitivity was 87.5% and the misclassification rate was 1.4% (5/358). In the second scenario, the important variables were chills, C-reactive protein, and physician diagnosis of a low-risk site. The low-risk group for the second algorithm included 62% of the total patients; sensitivity was 92.5% and misclassification rate 0.9% (3/328). The algorithms had negative predictive values of 98.6% (first scenario) and 99.1% (second).

DISCUSSION

This simple and sensitive prediction algorithm may be useful for identifying patients at low risk of bacteraemia. Prospective validation is needed in other settings.

Clinicians required very high sensitivity of a bacteremia prediction rule

BACKGROUND AND OBJECTIVE

Efforts to improve blood culture practice have focused on developing clinical prediction rules to identify patients at risk for bacteremia. However, no such models have been accepted into general clinical use. The goal of this study was to determine physicians' criteria for acceptability of a bacteremia clinical prediction rule.

METHOD

We conducted a survey of all medical and surgical house officers as well as all infectious diseases physicians at the University of Pennsylvania to identify physician requirements for the sensitivity of a bacteremia clinical prediction rule.

RESULTS

Of 225 eligible physicians, 149 (66.2%) completed the survey, including 110 house officers and 39 infectious disease physicians. The median (95% confidence interval) sensitivity of a bacteremia prediction rule required by respondents was 95% (95% confidence interval, 95.9%). Furthermore, 29 (19.5%) respondents required the sensitivity of a prediction rule to be at least 99%. The median required sensitivity was significantly higher for infectious diseases physicians than for house officers (98% and 95%, respectively) (P=.04).

CONCLUSION

Our survey of house staff and infectious diseases physicians demonstrates that the sensitivity of any bacteremia prediction rule must be extremely high (i.e., 99% to 100%) to be widely accepted by practicing clinicians. Elucidation of physician criteria for acceptability of clinical predictive models will be invaluable in future efforts to develop such prediction rules.

Predicting Bacteremia at the Bedside

Our aim was to develop a clinical prediction rule for detection of bacteremia in a cohort of patients observed prospectively at a reference center in Medellín, Colombia. The significant predictors of bacteremia were an age of >or=30 years (odds ratio [OR], 2.07; 95% confidence interval [CI], 1.19-3.60), a heart rate of >or=90 beats/min (OR, 1.90; 95% CI, 1.13-3.17), a temperature of >or=37.8 degrees C (OR, 2.42; 95% CI, 1.41-4.14), a leukocyte count of >or=12,000 cells/microL (OR, 2.40; 95% CI, 1.41-4.10), use of a central venous catheter (OR, 1.89; 95% CI, 1.02-3.50), and a length of hospitalization of >or=10 days (OR, 2.02; 95% CI, 1.25-3.24). The Hosmer-Lemeshow test revealed a goodness-of-fit of 2.99 (P=.981), and the area under the receiver operating characteristics curve was 0.7186. Simple variables obtained from the clinical history of patients are associated with bloodstream infection in a reproducible fashion and should be instrumental for prioritizing the requests for blood cultures by clinicians.

Blood cultures in the critical care unit: improving utilization and yield

Sepsis is a common cause of morbidity and death in critically ill patients, and blood culture samples are often drawn in an effort to identify a responsible pathogen. Blood culture results are usually negative, however, and even when positive are sometimes difficult to interpret. Distinguishing between true bacteremia and a false-positive blood culture result is important, but complicated by a variety of factors in the ICU. False-positive culture results are costly because they often prompt more diagnostic testing and more antibiotic prescriptions, and increase hospital length of stay. A number of factors influence the yield of blood cultures in critically ill patients, including the use of antibiotics, the volume of blood drawn, the frequency with which culture samples are drawn, and the site from which the culture samples are taken. Skin preparation techniques, handling of the cultures in the microbiology laboratory, and the type of blood culture system employed also influence blood culture yield. Attempts to identify predictors of true bacteremia in critically ill patients have been disappointing. In this review, we discuss factors that influence blood culture yield in critically ill patients, suggest ways to improve yield, and discuss true bacteremia vs false-positive blood culture results. We also discuss the costs and consequences of false-positive blood culture results, and list noninfectious causes of fever in the ICU.

Continuous quality improvement for introduction of automated blood culture instrument

Despite the critical nature and high cost of blood cultures, hospitals rely on manufacturers' test site data. As a result, in-hospital testing and compliance evaluation of newly acquired instruments are seldom done. The goal of this study was to apply a continuous quality improvement approach and to develop assessment criteria for all stages from the purchase order, through the on-site instrument evaluation, to the compliance evaluation. Despite the introduction of an automated high-blood-volume instrument (BacT/Alert) in our hospital, 56% of adult patients had only one venipuncture and 89.5% had < or = 20 ml of total blood volume sampled. False positives were associated with overfilling of bottles. These problems occurred because the phlebotomists did not like to perform multiple venipunctures on ill patients; therefore, they were drawing 20 ml of blood from one venipuncture and splitting it between two bottles. Unknown to the staff, the vacuum in the bottles draws significantly more than 10 ml of blood; therefore, the first bottle in the set was frequently overfilled and the second bottle was frequently underfilled. A diagrammatic guideline for a new blood culture protocol based on two venipunctures, taken one immediately after the other, to inoculate three bottles was developed. Compliance evaluation demonstrated that within 1 month of starting the new protocol, 74% of patients had at least two or more venipunctures and 60% had > or = 30 ml of blood drawn per patient episode. This study demonstrates the need for continuous quality improvement, including compliance evaluation, to ensure that the potential benefits of newer blood culture technology are actually realized.