A mathematical algorithm for detection of late-onset sepsis in very-low birth weight infants: a preliminary diagnostic test evaluation

Development and clinical impact assessment of a machine-learning model for early prediction of late-onset sepsis

BACKGROUND AND AIM

Preterm infants are prone to neonatal infections such as late-onset sepsis (LOS). The consequences of LOS can be severe and potentially life-threatening. Unfortunately, LOS often presents with unspecific symptoms, and early screening laboratory tests have limited diagnostic value and are often late. This study aimed to build a predictive algorithm to aid doctors in the early detection of LOS in very preterm infants.

METHODS

In a retrospective cohort study, all consecutively admitted preterm infants (GA ≤ 32 weeks) from 2008 until 2019 were included. They were classified as LOS or control according to blood culture results, currently the gold standard. To generate features, routine and continuously measured oxygen saturation and heart rate data with a minute-by-minute sampling rate were extracted from electronic medical records. Care was taken not to include variables indicative of existing LOS suspicion. The timing of a positive blood culture served as a proxy for LOS-onset. An equivalent timestamp was generated in gestational-age-matched control patients without a positive blood culture. Three machine learning (ML) techniques (generalized additive models, logistic regression, and XGBoost) were used to build a classification algorithm. To simulate the performance of the algorithm in clinical practice, a simulation using multiple alarm thresholds was performed on hourly predictions for the total hospitalization period.

RESULTS

292 infants with LOS were matched to 1497 controls. The median gestational age before matching was 28.1 and 30.3 weeks, respectively. Evaluation of the overall discriminative power of the LR algorithm yielded an AUC of 0.73 (p < 0.05) at the moment of clinical suspicion (t = 0). In the longitudinal simulation, our algorithm detects LOS in at least 47% of the patients before clinical suspicion without exceeding the alarm fatigue threshold of 3 alarms per day. Furthermore, medical experts evaluated the algorithm as clinically relevant regarding the feature contributions in the model explanations.

CONCLUSIONS

An ML algorithm was trained for the early detection of LOS. Performance was evaluated on both prediction horizons and in a clinical impact simulation. To the best of our knowledge, our assessment of clinical impact with a retrospective simulation on longitudinal data is the most extensive in the literature on LOS prediction to date. The clinically relevant algorithm, based on routinely collected data, can potentially accelerate clinical decisions in the early detection of LOS, even with limited inputs.

Vital sign-based detection of sepsis in neonates using machine learning

AIM

Sepsis is a leading cause of morbidity and mortality in neonates. Early diagnosis is key but difficult due to non-specific signs. We investigate the predictive value of machine learning-assisted analysis of non-invasive, high frequency monitoring data and demographic factors to detect neonatal sepsis.

METHODS

Single centre study, including a representative cohort of 325 infants (2866 hospitalisation days). Personalised event timelines including interventions and clinical findings were generated. Time-domain features from heart rate, respiratory rate and oxygen saturation values were calculated and demographic factors included. Sepsis prediction was performed using Naïve Bayes algorithm in a maximum a posteriori framework up to 24 h before clinical sepsis suspicion.

RESULTS

Twenty sepsis cases were identified. Combining multiple vital signs improved algorithm performance compared to heart rate characteristics alone. This enabled a prediction of sepsis with an area under the receiver operating characteristics curve of 0.82, up to 24 h before clinical sepsis suspicion. Moreover, 10 h prior to clinical suspicion, the risk of sepsis increased 150-fold.

CONCLUSION

The present algorithm using non-invasive patient data provides useful predictive value for neonatal sepsis detection. Machine learning-assisted algorithms are promising novel methods that could help individualise patient care and reduce morbidity and mortality.

Artificial and human intelligence for early identification of neonatal sepsis

Artificial intelligence may have a role in the early detection of sepsis in neonates. Machine learning can identify patterns that predict high or increasing risk for clinical deterioration from a sepsis-like illness. In developing this potential addition to NICU care, careful consideration should be given to the data and methods used to develop, validate, and evaluate prediction models. When an AI system alerts clinicians to a change in a patient's condition that warrants a bedside evaluation, human intelligence and experience come into play to determine an appropriate course of action: evaluate and treat or wait and watch closely. With intelligently developed, validated, and implemented AI sepsis systems, both clinicians and patients stand to benefit. IMPACT: This narrative review highlights the application of AI in neonatal sepsis prediction. It describes issues in clinical prediction model development specific to this population. This article reviews the methods, considerations, and literature on neonatal sepsis model development and validation. Challenges of AI technology and potential barriers to using sepsis AI systems in the NICU are discussed.

Neonatal sepsis prediction through clinical decision support algorithms: A systematic review

AIM

To systematically summarise the current evidence of employing clinical decision support algorithms (CDSAs) using non-invasive parameters for sepsis prediction in neonates.

METHODS

A comprehensive search in PubMed, CENTRAL and EMBASE was conducted. Screening, data extraction and risk of bias were performed by two authors. The certainty of the evidence was assessed using GRADE.

PROSPERO ID

CRD42020205143.

RESULTS

After abstract and full-text screening, 36 studies comprising 18,096 infants were included. Most CDSAs evaluated heart rate (HR)-based parameters. Two publications derived from one randomised-controlled trial assessing HR characteristics reported significant reduction in 30-day septicaemia-related mortality. Thirty-four non-randomised studies found promising yet inconclusive results.

CONCLUSION

Heart rate-based parameters are reliable components of CDSAs for sepsis prediction, particularly in combination with additional vital signs and demographics. However, inconclusive evidence and limited standardisation restricts clinical implementation of CDSAs outside of a controlled research environment. Further experimentation and comparison of parameter combinations and testing of new CDSAs are warranted.

Diagnosis of Neonatal Late-Onset Infection in Very Preterm Infant: Inter-Observer Agreement and International Classifications

Background: The definition of late-onset bacterial sepsis (LOS) in very preterm infants is not unified. The objective was to assess the concordance of LOS diagnosis between experts in neonatal infection and international classifications and to evaluate the potential impact on heart rate variability and rate of “bronchopulmonary dysplasia or death”. Methods: A retrospective (2017–2020) multicenter study including hospitalized infants born before 31 weeks of gestation with intention to treat at least 5-days with antibiotics was performed. LOS was classified as “certain or probable” or “doubtful” independently by five experts and according to four international classifications with concordance assessed by Fleiss’s kappa test. Results: LOS was suspected at seven days (IQR: 5–11) of life in 48 infants. Following expert classification, 36 of them (75%) were considered as “certain or probable” (kappa = 0.41). Following international classification, this number varied from 13 to 46 (kappa = −0.08). Using the expert classification, “bronchopulmonary dysplasia or death” occurred less frequently in the doubtful group (25% vs. 78%, p < 0.001). Differences existed in HRV changes between the two groups. Conclusion: The definition of LOS is not consensual with a low international and moderate inter-observer agreement. This affects the evaluation of associated organ dysfunction and prognosis.

Blood culture indications in critically ill neonates: a multicenter prospective cohort study

Due to potential lethality of healthcare-associated sepsis (HAS), a low threshold for blood culturing and antimicrobial therapy (ABT) initiation is accepted. We assessed variability in the trigger for blood culturing between three neonatal intensive care units. A multicenter prospective cohort study was conducted. In newborns with suspicion of HAS, 10 predefined clinical signs, nosocomial sepsis (NOSEP) score, C-reactive protein, ABT initiation, and risk factors were registered at time of culturing. Outcome was lab-confirmed HAS, defined according to the NeoKISS-criteria. Two hundred ninety-nine suspected HAS episodes were considered in 212 infants, of which 118 had birth-weight ≤ 1500 g; proportion of lab-confirmed HAS per suspected episode was 30/192 (center 1), 28/60 (center 2), and 8/47 (center 3) (p < 0.001). Median C-reactive protein and number of clinical signs at time of culturing differed between centers 1, 2, and 3 (respectively 11 vs. 5 vs. 3 mg/L, p = 0.001; 1 sign [IQR 0-2, center 1] vs. 3 signs [IQR 2-4, centers 2 and 3], p < 0.001). Median NOSEP score at time of culturing was 5 (IQR 3-8, center 1), 5 (IQR 3-9, center 2), and 8 (IQR 5-11, center 3) (p = 0.016). Difference in ABT initiation was noticed (82 vs. 93 vs. 74%, p = 0.05).

CONCLUSION

Center heterogeneity in sampling practice is substantial. Optimizing sampling practice can be recommended. What is Known: • Blood culture test is a common diagnostic procedure in critically-ill newborns. • A low threshold for sampling and antimicrobial therapy initiation is accepted. What is New: • Variability in blood culture practice was assessed between 3 neonatal intensive care units by the registration of sampling frequencies, clinical indications, and antimicrobial therapy initiation.

Vital signs analysis algorithm detects inflammatory response in premature infants with late onset sepsis and necrotizing enterocolitis

BACKGROUND

Nonspecific clinical signs and suboptimal diagnostic tests limit accurate identification of late onset sepsis (LOS) and necrotizing enterocolitis (NEC) in premature infants, resulting in significant morbidity and antibiotic overuse. An infant's systemic inflammatory response may be identified earlier than clinical suspicion through analysis of multiple vital signs by a computerized algorithm (RALIS).

AIM

To evaluate the revised RALIS algorithm for detection of LOS and NEC in preterm infants.

METHODS

In this nested case-control study, VS data (heart rate, respiratory rate, temperature, desaturations, bradycardias) were extracted from medical records of infants 23-32 weeks gestation. RALIS generated an output, with score ≥ 5 triggering an alert. Patient episodes were classified based on culture, radiograph, and antibiotic data into categories: LOS, expanded LOS, NEC, and controls. Paired t-tests, linear regression and cross-validation analyses were used to evaluate the relationship between RALIS alert and LOS/NEC.

RESULTS

Among 155 infants with 161 episodes, there were 41 expanded LOS (+blood, CSF, urine, respiratory culture), 31 LOS (+blood, CSF, urine), 9 NEC, and 93 controls. RALIS alert was 43.1 ± 79 h before culture in LOS (p = .012). There was a significant association between RALIS alert and LOS/NEC (β = 0.72, p < .0001). Sensitivity and specificity for LOS/NEC were 84% and 80%, (PPV = 63%; NPV = 93%). The regression model demonstrated an AUC of 89.9%.

CONCLUSIONS

For infants ≤32 weeks, RALIS detects systemic inflammatory responses in LOS and NEC in the first month of life. The algorithm can identify infection earlier than clinical suspicion, even for NEC with negative cultures. RALIS has high NPV to rule-out LOS and NEC, and may, after prospective validation, aid in antibiotic treatment decisions.

Computerized vital signs analysis and late onset infections in extremely low gestational age infants

AIM

Current clinical and laboratory diagnostics for neonatal infection are inadequate. An infant's systemic inflammatory response may be identified earlier than clinical suspicion by a computerized algorithm (RALIS) incorporating multiple vital signs (VS). We tested the ability of RALIS to detect late onset infection (LOI) earlier than clinically suspected.

METHODS

We conducted a retrospective review of infants enrolled in a birth cohort study at Prentice Women's Hospital. VS data (heart rate, respirations, temperature, desaturation, bradycardia) were extracted from electronic records of 73 premature infants (born ≤28 weeks' gestation; survived first month). RALIS generated a continuous output for the first 28 days of life. A score ≥5 for 6 h triggered an alert. The time of RALIS alert to time of clinical suspicion of infection (time culture sent) was measured for each episode of suspected and/or confirmed LOI.

RESULTS

Among the 73 infants followed with RALIS, there were 34 episodes of culture-positive LOI, seven culture-negative but treated episodes, and 13 false-positive culture (untreated) episodes. Twenty-five infants had no culture-positive or treated sepsis events during the observation period. There was a positive linear association between alert and culture (β=0.88, P<0.001). Mean absolute time difference between alert and culture was 59.4 h before culture. Sensitivity and specificity of RALIS for LOI were 0.82 and 0.44.

CONCLUSION

The RALIS algorithm is a sensitive indicator for early detection of infection in preterm infants. Further modifications to improve the specificity of the algorithm are needed prior to application of VS modeling to patient antibiotic treatment decisions.