Clinical Decision Support in the PICU: Implications for Design and Evaluation

The Pediatric Data Science and Analytics Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators Network: Use of Supervised Machine Learning Applications in Pediatric Critical Care Medicine Research

OBJECTIVE

Perform a scoping review of supervised machine learning in pediatric critical care to identify published applications, methodologies, and implementation frequency to inform best practices for the development, validation, and reporting of predictive models in pediatric critical care.

DESIGN

Scoping review and expert opinion.

SETTING

We queried CINAHL Plus with Full Text (EBSCO), Cochrane Library (Wiley), Embase (Elsevier), Ovid Medline, and PubMed for articles published between 2000 and 2022 related to machine learning concepts and pediatric critical illness. Articles were excluded if the majority of patients were adults or neonates, if unsupervised machine learning was the primary methodology, or if information related to the development, validation, and/or implementation of the model was not reported. Article selection and data extraction were performed using dual review in the Covidence tool, with discrepancies resolved by consensus.

SUBJECTS

Articles reporting on the development, validation, or implementation of supervised machine learning models in the field of pediatric critical care medicine.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Of 5075 identified studies, 141 articles were included. Studies were primarily (57%) performed at a single site. The majority took place in the United States (70%). Most were retrospective observational cohort studies. More than three-quarters of the articles were published between 2018 and 2022. The most common algorithms included logistic regression and random forest. Predicted events were most commonly death, transfer to ICU, and sepsis. Only 14% of articles reported external validation, and only a single model was implemented at publication. Reporting of validation methods, performance assessments, and implementation varied widely. Follow-up with authors suggests that implementation remains uncommon after model publication.

CONCLUSIONS

Publication of supervised machine learning models to address clinical challenges in pediatric critical care medicine has increased dramatically in the last 5 years. While these approaches have the potential to benefit children with critical illness, the literature demonstrates incomplete reporting, absence of external validation, and infrequent clinical implementation.

Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit

OBJECTIVE

 Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals.

METHODS

 We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized.

RESULTS

 Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80-95), with over 96% rating the tool's usability as "good" or "excellent." Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows.

CONCLUSION

 The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.

Leveraging Clinical Informatics and Data Science to Improve Care and Facilitate Research in Pediatric Acute Respiratory Distress Syndrome: From the Second Pediatric Acute Lung Injury Consensus Conference

OBJECTIVES

The use of electronic algorithms, clinical decision support systems, and other clinical informatics interventions is increasing in critical care. Pediatric acute respiratory distress syndrome (PARDS) is a complex, dynamic condition associated with large amounts of clinical data and frequent decisions at the bedside. Novel data-driven technologies that can help screen, prompt, and support clinician decision-making could have a significant impact on patient outcomes. We sought to identify and summarize relevant evidence related to clinical informatics interventions in both PARDS and adult respiratory distress syndrome (ARDS), for the second Pediatric Acute Lung Injury Consensus Conference.

DATA SOURCES

MEDLINE (Ovid), Embase (Elsevier), and CINAHL Complete (EBSCOhost).

STUDY SELECTION

We included studies of pediatric or adult critically ill patients with or at risk of ARDS that examined automated screening tools, electronic algorithms, or clinical decision support systems.

DATA EXTRACTION

Title/abstract review, full text review, and data extraction using a standardized data extraction form.

DATA SYNTHESIS

The Grading of Recommendations Assessment, Development and Evaluation approach was used to identify and summarize evidence and develop recommendations. Twenty-six studies were identified for full text extraction to address the Patient/Intervention/Comparator/Outcome questions, and 14 were used for the recommendations/statements. Two clinical recommendations were generated, related to the use of electronic screening tools and automated monitoring of compliance with best practice guidelines. Two research statements were generated, related to the development of multicenter data collaborations and the design of generalizable algorithms and electronic tools. One policy statement was generated, related to the provision of material and human resources by healthcare organizations to empower clinicians to develop clinical informatics interventions to improve the care of patients with PARDS.

CONCLUSIONS

We present two clinical recommendations and three statements (two research one policy) for the use of electronic algorithms and clinical informatics tools for patients with PARDS based on a systematic review of the literature and expert consensus.

The use of machine learning and artificial intelligence within pediatric critical care

The field of pediatric critical care has been hampered in the era of precision medicine by our inability to accurately define and subclassify disease phenotypes. This has been caused by heterogeneity across age groups that further challenges the ability to perform randomized controlled trials in pediatrics. One approach to overcome these inherent challenges include the use of machine learning algorithms that can assist in generating more meaningful interpretations from clinical data. This review summarizes machine learning and artificial intelligence techniques that are currently in use for clinical data modeling with relevance to pediatric critical care. Focus has been placed on the differences between techniques and the role of each in the clinical arena. The various forms of clinical decision support that utilize machine learning are also described. We review the applications and limitations of machine learning techniques to empower clinicians to make informed decisions at the bedside. IMPACT: Critical care units generate large amounts of under-utilized data that can be processed through artificial intelligence. This review summarizes the machine learning and artificial intelligence techniques currently being used to process clinical data. The review highlights the applications and limitations of these techniques within a clinical context to aid providers in making more informed decisions at the bedside.

Pediatric Acute Lung Injury and Sepsis Investigators (PALISI): Evolution of an Investigator-Initiated Research Network

The Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network originated over 20 years ago to foster research to optimize the care of critically ill infants and children. Over this period, PALISI has seen two major evolutions: formalization of our network infrastructure and a broadening of our clinical research focus. First, the network is unique in that its activities and meetings are funded by subscriptions from members who now comprise a multidisciplinary group of investigators from over 90 PICUs all over the United States (US) and Canada, with collaborations across the globe. In 2020, the network converted into a standalone, nonprofit organizational structure (501c3), making the PALISI Network formally independent of academic and clinical institutions or professional societies. Such an approach allows us to invest in infrastructure and future initiatives with broader opportunities for fund raising. Second, our research investigations have expanded beyond the original focus on sepsis and acute lung injury, to incorporate the whole field of pediatric critical care, for example, efficient liberation from mechanical ventilator support, prudent use of blood products, improved safety of intubation practices, optimal sedation practices and glucose control, and pandemic research on influenza and COVID-19. Our network approach in each field follows, where necessary, the full spectrum of clinical and translational research, including: immunobiology studies for understanding basic pathologic mechanisms; surveys to explore contemporary clinical practice; consensus conferences to establish agreement about literature evidence; observational prevalence and incidence studies to measure scale of a clinical issue or question; case control studies as preliminary best evidence for design of definitive prospective studies; and, randomized controlled trials for informing clinical care. As a research network, PALISI and its related subgroups have published over 350 peer-reviewed publications from 2002 through September 2022.