1
|
van den Berg MAM, Medina OOAG, Loohuis IIP, van der Flier MM, Dudink JJ, Benders MMJNL, Bartels RRT, Vijlbrief DDC. Development and clinical impact assessment of a machine-learning model for early prediction of late-onset sepsis. Comput Biol Med 2023; 163:107156. [PMID: 37369173 DOI: 10.1016/j.compbiomed.2023.107156] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 05/24/2023] [Accepted: 06/07/2023] [Indexed: 06/29/2023]
Abstract
BACKGROUND AND AIM Preterm infants are prone to neonatal infections such as late-onset sepsis (LOS). The consequences of LOS can be severe and potentially life-threatening. Unfortunately, LOS often presents with unspecific symptoms, and early screening laboratory tests have limited diagnostic value and are often late. This study aimed to build a predictive algorithm to aid doctors in the early detection of LOS in very preterm infants. METHODS In a retrospective cohort study, all consecutively admitted preterm infants (GA ≤ 32 weeks) from 2008 until 2019 were included. They were classified as LOS or control according to blood culture results, currently the gold standard. To generate features, routine and continuously measured oxygen saturation and heart rate data with a minute-by-minute sampling rate were extracted from electronic medical records. Care was taken not to include variables indicative of existing LOS suspicion. The timing of a positive blood culture served as a proxy for LOS-onset. An equivalent timestamp was generated in gestational-age-matched control patients without a positive blood culture. Three machine learning (ML) techniques (generalized additive models, logistic regression, and XGBoost) were used to build a classification algorithm. To simulate the performance of the algorithm in clinical practice, a simulation using multiple alarm thresholds was performed on hourly predictions for the total hospitalization period. RESULTS 292 infants with LOS were matched to 1497 controls. The median gestational age before matching was 28.1 and 30.3 weeks, respectively. Evaluation of the overall discriminative power of the LR algorithm yielded an AUC of 0.73 (p < 0.05) at the moment of clinical suspicion (t = 0). In the longitudinal simulation, our algorithm detects LOS in at least 47% of the patients before clinical suspicion without exceeding the alarm fatigue threshold of 3 alarms per day. Furthermore, medical experts evaluated the algorithm as clinically relevant regarding the feature contributions in the model explanations. CONCLUSIONS An ML algorithm was trained for the early detection of LOS. Performance was evaluated on both prediction horizons and in a clinical impact simulation. To the best of our knowledge, our assessment of clinical impact with a retrospective simulation on longitudinal data is the most extensive in the literature on LOS prediction to date. The clinically relevant algorithm, based on routinely collected data, can potentially accelerate clinical decisions in the early detection of LOS, even with limited inputs.
Collapse
Affiliation(s)
- Merel A M van den Berg
- Department of Neonatology, Wilhelmina Children's Hospital, UMC Utrecht, Utrecht, the Netherlands
| | | | | | - Michiel M van der Flier
- Department of Pediatric Infectious Disease, Wilhelmina Children's Hospital, UMC Utrecht, Utrecht, the Netherlands
| | - Jeroen J Dudink
- Department of Neonatology, Wilhelmina Children's Hospital, UMC Utrecht, Utrecht, the Netherlands
| | - Manon M J N L Benders
- Department of Neonatology, Wilhelmina Children's Hospital, UMC Utrecht, Utrecht, the Netherlands
| | | | - Daniel D C Vijlbrief
- Department of Neonatology, Wilhelmina Children's Hospital, UMC Utrecht, Utrecht, the Netherlands.
| |
Collapse
|
2
|
Honoré A, Forsberg D, Adolphson K, Chatterjee S, Jost K, Herlenius E. Vital sign-based detection of sepsis in neonates using machine learning. Acta Paediatr 2023; 112:686-696. [PMID: 36607251 DOI: 10.1111/apa.16660] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 12/29/2022] [Accepted: 01/04/2023] [Indexed: 01/07/2023]
Abstract
AIM Sepsis is a leading cause of morbidity and mortality in neonates. Early diagnosis is key but difficult due to non-specific signs. We investigate the predictive value of machine learning-assisted analysis of non-invasive, high frequency monitoring data and demographic factors to detect neonatal sepsis. METHODS Single centre study, including a representative cohort of 325 infants (2866 hospitalisation days). Personalised event timelines including interventions and clinical findings were generated. Time-domain features from heart rate, respiratory rate and oxygen saturation values were calculated and demographic factors included. Sepsis prediction was performed using Naïve Bayes algorithm in a maximum a posteriori framework up to 24 h before clinical sepsis suspicion. RESULTS Twenty sepsis cases were identified. Combining multiple vital signs improved algorithm performance compared to heart rate characteristics alone. This enabled a prediction of sepsis with an area under the receiver operating characteristics curve of 0.82, up to 24 h before clinical sepsis suspicion. Moreover, 10 h prior to clinical suspicion, the risk of sepsis increased 150-fold. CONCLUSION The present algorithm using non-invasive patient data provides useful predictive value for neonatal sepsis detection. Machine learning-assisted algorithms are promising novel methods that could help individualise patient care and reduce morbidity and mortality.
Collapse
Affiliation(s)
- Antoine Honoré
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.,Astrid Lindgren Children's Hospital, Karolinska University Hospital, Stockholm, Sweden.,Division of Information Science and Engineering, Royal Institute of Technology - KTH, Stockholm, Sweden
| | - David Forsberg
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.,Astrid Lindgren Children's Hospital, Karolinska University Hospital, Stockholm, Sweden
| | - Katja Adolphson
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.,Astrid Lindgren Children's Hospital, Karolinska University Hospital, Stockholm, Sweden
| | - Saikat Chatterjee
- Division of Information Science and Engineering, Royal Institute of Technology - KTH, Stockholm, Sweden
| | - Kerstin Jost
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.,Astrid Lindgren Children's Hospital, Karolinska University Hospital, Stockholm, Sweden
| | - Eric Herlenius
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.,Astrid Lindgren Children's Hospital, Karolinska University Hospital, Stockholm, Sweden
| |
Collapse
|
3
|
Sullivan BA, Kausch SL, Fairchild KD. Artificial and human intelligence for early identification of neonatal sepsis. Pediatr Res 2023; 93:350-356. [PMID: 36127407 DOI: 10.1038/s41390-022-02274-7] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Revised: 07/29/2022] [Accepted: 08/05/2022] [Indexed: 11/09/2022]
Abstract
Artificial intelligence may have a role in the early detection of sepsis in neonates. Machine learning can identify patterns that predict high or increasing risk for clinical deterioration from a sepsis-like illness. In developing this potential addition to NICU care, careful consideration should be given to the data and methods used to develop, validate, and evaluate prediction models. When an AI system alerts clinicians to a change in a patient's condition that warrants a bedside evaluation, human intelligence and experience come into play to determine an appropriate course of action: evaluate and treat or wait and watch closely. With intelligently developed, validated, and implemented AI sepsis systems, both clinicians and patients stand to benefit. IMPACT: This narrative review highlights the application of AI in neonatal sepsis prediction. It describes issues in clinical prediction model development specific to this population. This article reviews the methods, considerations, and literature on neonatal sepsis model development and validation. Challenges of AI technology and potential barriers to using sepsis AI systems in the NICU are discussed.
Collapse
Affiliation(s)
- Brynne A Sullivan
- Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA, USA.
| | - Sherry L Kausch
- Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA, USA
| | - Karen D Fairchild
- Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA, USA
| |
Collapse
|
4
|
Persad E, Jost K, Honoré A, Forsberg D, Coste K, Olsson H, Rautiainen S, Herlenius E. Neonatal sepsis prediction through clinical decision support algorithms: A systematic review. Acta Paediatr 2021; 110:3201-3226. [PMID: 34432903 DOI: 10.1111/apa.16083] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Revised: 08/14/2021] [Accepted: 08/24/2021] [Indexed: 12/12/2022]
Abstract
AIM To systematically summarise the current evidence of employing clinical decision support algorithms (CDSAs) using non-invasive parameters for sepsis prediction in neonates. METHODS A comprehensive search in PubMed, CENTRAL and EMBASE was conducted. Screening, data extraction and risk of bias were performed by two authors. The certainty of the evidence was assessed using GRADE. PROSPERO ID CRD42020205143. RESULTS After abstract and full-text screening, 36 studies comprising 18,096 infants were included. Most CDSAs evaluated heart rate (HR)-based parameters. Two publications derived from one randomised-controlled trial assessing HR characteristics reported significant reduction in 30-day septicaemia-related mortality. Thirty-four non-randomised studies found promising yet inconclusive results. CONCLUSION Heart rate-based parameters are reliable components of CDSAs for sepsis prediction, particularly in combination with additional vital signs and demographics. However, inconclusive evidence and limited standardisation restricts clinical implementation of CDSAs outside of a controlled research environment. Further experimentation and comparison of parameter combinations and testing of new CDSAs are warranted.
Collapse
Affiliation(s)
- Emma Persad
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
- Karl Landsteiner University of Health Sciences Krems Austria
- Department of Evidence‐based Medicine and Evaluation Danube University Krems Krems Austria
| | - Kerstin Jost
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
| | - Antoine Honoré
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
- Division of Information Science and Engineering KTH Royal Institute of Technology Stockholm Sweden
| | - David Forsberg
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
| | - Karen Coste
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- CNRS INSERM GReD Université Clermont Auvergne Clermont‐Ferrand France
| | - Hanna Olsson
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
| | - Susanne Rautiainen
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
- Department of Global Public Health Karolinska Institutet Stockholm Sweden
| | - Eric Herlenius
- Department of Women's & Children’s Health Karolinska Institutet Stockholm Sweden
- Astrid Lindgren Children’s HospitalKarolinska University Hospital Stockholm Sweden
| |
Collapse
|
5
|
Diagnosis of Neonatal Late-Onset Infection in Very Preterm Infant: Inter-Observer Agreement and International Classifications. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18030882. [PMID: 33498557 PMCID: PMC7908350 DOI: 10.3390/ijerph18030882] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/26/2020] [Revised: 01/14/2021] [Accepted: 01/14/2021] [Indexed: 02/05/2023]
Abstract
Background: The definition of late-onset bacterial sepsis (LOS) in very preterm infants is not unified. The objective was to assess the concordance of LOS diagnosis between experts in neonatal infection and international classifications and to evaluate the potential impact on heart rate variability and rate of “bronchopulmonary dysplasia or death”. Methods: A retrospective (2017–2020) multicenter study including hospitalized infants born before 31 weeks of gestation with intention to treat at least 5-days with antibiotics was performed. LOS was classified as “certain or probable” or “doubtful” independently by five experts and according to four international classifications with concordance assessed by Fleiss’s kappa test. Results: LOS was suspected at seven days (IQR: 5–11) of life in 48 infants. Following expert classification, 36 of them (75%) were considered as “certain or probable” (kappa = 0.41). Following international classification, this number varied from 13 to 46 (kappa = −0.08). Using the expert classification, “bronchopulmonary dysplasia or death” occurred less frequently in the doubtful group (25% vs. 78%, p < 0.001). Differences existed in HRV changes between the two groups. Conclusion: The definition of LOS is not consensual with a low international and moderate inter-observer agreement. This affects the evaluation of associated organ dysfunction and prognosis.
Collapse
|
6
|
Blood culture indications in critically ill neonates: a multicenter prospective cohort study. Eur J Pediatr 2018; 177:1565-1572. [PMID: 30051146 DOI: 10.1007/s00431-018-3203-1] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/15/2018] [Revised: 06/29/2018] [Accepted: 07/01/2018] [Indexed: 12/29/2022]
Abstract
UNLABELLED Due to potential lethality of healthcare-associated sepsis (HAS), a low threshold for blood culturing and antimicrobial therapy (ABT) initiation is accepted. We assessed variability in the trigger for blood culturing between three neonatal intensive care units. A multicenter prospective cohort study was conducted. In newborns with suspicion of HAS, 10 predefined clinical signs, nosocomial sepsis (NOSEP) score, C-reactive protein, ABT initiation, and risk factors were registered at time of culturing. Outcome was lab-confirmed HAS, defined according to the NeoKISS-criteria. Two hundred ninety-nine suspected HAS episodes were considered in 212 infants, of which 118 had birth-weight ≤ 1500 g; proportion of lab-confirmed HAS per suspected episode was 30/192 (center 1), 28/60 (center 2), and 8/47 (center 3) (p < 0.001). Median C-reactive protein and number of clinical signs at time of culturing differed between centers 1, 2, and 3 (respectively 11 vs. 5 vs. 3 mg/L, p = 0.001; 1 sign [IQR 0-2, center 1] vs. 3 signs [IQR 2-4, centers 2 and 3], p < 0.001). Median NOSEP score at time of culturing was 5 (IQR 3-8, center 1), 5 (IQR 3-9, center 2), and 8 (IQR 5-11, center 3) (p = 0.016). Difference in ABT initiation was noticed (82 vs. 93 vs. 74%, p = 0.05). CONCLUSION Center heterogeneity in sampling practice is substantial. Optimizing sampling practice can be recommended. What is Known: • Blood culture test is a common diagnostic procedure in critically-ill newborns. • A low threshold for sampling and antimicrobial therapy initiation is accepted. What is New: • Variability in blood culture practice was assessed between 3 neonatal intensive care units by the registration of sampling frequencies, clinical indications, and antimicrobial therapy initiation.
Collapse
|
7
|
Mithal LB, Yogev R, Palac HL, Kaminsky D, Gur I, Mestan KK. Vital signs analysis algorithm detects inflammatory response in premature infants with late onset sepsis and necrotizing enterocolitis. Early Hum Dev 2018; 117:83-89. [PMID: 29351876 PMCID: PMC5983899 DOI: 10.1016/j.earlhumdev.2018.01.008] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/17/2017] [Revised: 12/13/2017] [Accepted: 01/10/2018] [Indexed: 12/21/2022]
Abstract
BACKGROUND Nonspecific clinical signs and suboptimal diagnostic tests limit accurate identification of late onset sepsis (LOS) and necrotizing enterocolitis (NEC) in premature infants, resulting in significant morbidity and antibiotic overuse. An infant's systemic inflammatory response may be identified earlier than clinical suspicion through analysis of multiple vital signs by a computerized algorithm (RALIS). AIM To evaluate the revised RALIS algorithm for detection of LOS and NEC in preterm infants. METHODS In this nested case-control study, VS data (heart rate, respiratory rate, temperature, desaturations, bradycardias) were extracted from medical records of infants 23-32 weeks gestation. RALIS generated an output, with score ≥ 5 triggering an alert. Patient episodes were classified based on culture, radiograph, and antibiotic data into categories: LOS, expanded LOS, NEC, and controls. Paired t-tests, linear regression and cross-validation analyses were used to evaluate the relationship between RALIS alert and LOS/NEC. RESULTS Among 155 infants with 161 episodes, there were 41 expanded LOS (+blood, CSF, urine, respiratory culture), 31 LOS (+blood, CSF, urine), 9 NEC, and 93 controls. RALIS alert was 43.1 ± 79 h before culture in LOS (p = .012). There was a significant association between RALIS alert and LOS/NEC (β = 0.72, p < .0001). Sensitivity and specificity for LOS/NEC were 84% and 80%, (PPV = 63%; NPV = 93%). The regression model demonstrated an AUC of 89.9%. CONCLUSIONS For infants ≤32 weeks, RALIS detects systemic inflammatory responses in LOS and NEC in the first month of life. The algorithm can identify infection earlier than clinical suspicion, even for NEC with negative cultures. RALIS has high NPV to rule-out LOS and NEC, and may, after prospective validation, aid in antibiotic treatment decisions.
Collapse
Affiliation(s)
- Leena B. Mithal
- Department of Pediatrics, Division of Infectious Diseases, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Ram Yogev
- Department of Pediatrics, Division of Infectious Diseases, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Hannah L. Palac
- Department of Preventive Medicine, Biostatistics Collaboration Center, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | | | - Ilan Gur
- Bikurofe Institute, Tel Aviv, Israel,St. Joseph Hospital, East Jerusalem, Israel
| | - Karen K. Mestan
- Department of Pediatrics, Division of Neonatology, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| |
Collapse
|
8
|
Mithal LB, Yogev R, Palac H, Gur I, Mestan KK. Computerized vital signs analysis and late onset infections in extremely low gestational age infants. J Perinat Med 2016; 44:491-7. [PMID: 26845716 DOI: 10.1515/jpm-2015-0362] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/29/2015] [Accepted: 12/22/2015] [Indexed: 11/15/2022]
Abstract
AIM Current clinical and laboratory diagnostics for neonatal infection are inadequate. An infant's systemic inflammatory response may be identified earlier than clinical suspicion by a computerized algorithm (RALIS) incorporating multiple vital signs (VS). We tested the ability of RALIS to detect late onset infection (LOI) earlier than clinically suspected. METHODS We conducted a retrospective review of infants enrolled in a birth cohort study at Prentice Women's Hospital. VS data (heart rate, respirations, temperature, desaturation, bradycardia) were extracted from electronic records of 73 premature infants (born ≤28 weeks' gestation; survived first month). RALIS generated a continuous output for the first 28 days of life. A score ≥5 for 6 h triggered an alert. The time of RALIS alert to time of clinical suspicion of infection (time culture sent) was measured for each episode of suspected and/or confirmed LOI. RESULTS Among the 73 infants followed with RALIS, there were 34 episodes of culture-positive LOI, seven culture-negative but treated episodes, and 13 false-positive culture (untreated) episodes. Twenty-five infants had no culture-positive or treated sepsis events during the observation period. There was a positive linear association between alert and culture (β=0.88, P<0.001). Mean absolute time difference between alert and culture was 59.4 h before culture. Sensitivity and specificity of RALIS for LOI were 0.82 and 0.44. CONCLUSION The RALIS algorithm is a sensitive indicator for early detection of infection in preterm infants. Further modifications to improve the specificity of the algorithm are needed prior to application of VS modeling to patient antibiotic treatment decisions.
Collapse
|