Continuous Prediction of Mortality in the PICU: A Recurrent Neural Network Model in a Single-Center Dataset

Physical, Cognitive, Emotional, and Social Health Outcomes of Children in the First 6 Months After Childhood Critical Illness: A Prospective Single-Center Study

OBJECTIVES

To describe physical, cognitive, emotional, and social health outcomes of children and their trajectory in the first 6 months after PICU discharge.

DESIGN

Prospective, longitudinal observational cohort study.

SETTING

PICU in a tertiary pediatric hospital in Singapore from January 2021 to June 2022.

PATIENTS

One hundred thirty-five children (1 mo to 18 yr), admitted for greater than or equal to 48 hours with at least one organ dysfunction and received PICU therapy.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Serial self/parent proxy-reported assessments were obtained at: PICU admission, PICU discharge, and 1, 3, and 6 months after PICU discharge. The Pediatric Quality of Life Inventory (PedsQL) scale, Functional Status Scale (FSS), and measures of post-traumatic stress disorder (PTSD) using the Young Child PTSD Screen and the Child and Adolescent PTSD Screen-Parent Version were used. Trajectory groups were identified using group-based trajectory model. One hundred thirty-five children (mean [sd] age, 5.6 yr [5.5 yr]) were recruited. Seventy-eight (52%) were male. The mean (sd) Pediatric Index of Mortality III score was 3.2 (4.1) and PICU length of stay was 10.0 days (21.0 d). The mean (sd) PedsQL total scores were 66.5 (21.1) at baseline, 69.7 (21.4), 75.6 (19.7), and 78.4 (19.8) at 1, 3, and 6 months after PICU discharge, respectively. Overall, the PedsQL and FSS plateaued at 3 months. Our model revealed three distinct trajectory groups. Seventeen and 103 children in the mild and moderate trajectory groups, respectively, demonstrated recovery to baseline. Fifteen children in the severe trajectory group were older in age (mean [sd] 8.3 yr [6.4 yr]), with higher proportion (11/15) of preexisting illness. Five of 15 children in the severe group experienced posttraumatic stress syndrome (PTSS) at 6 months post-discharge.

CONCLUSIONS

In our cohort of PICU patients, we found three unique trajectory groups. Children in the severe group were older, more likely to have preexisting conditions and at increased risk for PTSS. Early identification and intervention may improve recovery in patients with severe PICU trajectories.

Identification of mycoplasma pneumonia in children based on fusion of multi-modal clinical free-text description and structured test data

Mycoplasma pneumonia may lead to hospitalizations and pose life-threatening risks in children. The automated identification of mycoplasma pneumonia from electronic medical records holds significant potential for improving the efficiency of hospital resource allocation. In this study, we proposed a novel method for identifying mycoplasma pneumonia by integrating multi-modal features derived from both free-text descriptions and structured test data in electronic medical records. Our approach begins with the extraction of free-text and structured data from clinical records through a systematic preprocessing pipeline. Subsequently, we employ a pre-trained transformer language model to extract features from the free-text, while multiple additive regression trees are used to transform features from the structured data. An attention-based fusion mechanism is then applied to integrate these multi-modal features for effective classification. We validated our method using clinic records of 7157 patients, retrospectively collected for training and testing purposes. The experimental results demonstrate that our proposed multi-modal fusion approach achieves significant improvements over other methods across four key performance metrics.

Development of a deep learning model for predicting critical events in a pediatric intensive care unit

BACKGROUND

Identifying critically ill patients at risk of cardiac arrest is important because it offers the opportunity for early intervention and increased survival. The aim of this study was to develop a deep learning model to predict critical events, such as cardiopulmonary resuscitation or mortality.

METHODS

This retrospective observational study was conducted at a tertiary university hospital. All patients younger than 18 years who were admitted to the pediatric intensive care unit from January 2010 to May 2023 were included. The main outcome was prediction performance of the deep learning model at forecasting critical events. Long short-term memory was used as a deep learning algorithm. The five-fold cross validation method was employed for model learning and testing.

RESULTS

Among the vital sign measurements collected during the study period, 11,660 measurements were used to develop the model after preprocessing; 1,060 of these data points were measurements that corresponded to critical events. The prediction performance of the model was the area under the receiver operating characteristic curve (95% confidence interval) of 0.988 (0.9751.000), and the area under the precision-recall curve was 0.862 (0.700-1.000).

CONCLUSIONS

The performance of the developed model at predicting critical events was excellent. However, follow-up research is needed for external validation.

Delay Between Actual Occurrence of Patient Vital Sign and the Nominal Appearance in the Electronic Health Record: Single-Center, Retrospective Study of PICU Data, 2014-2018

OBJECTIVES

Patient vital sign data charted in the electronic health record (EHR) are used for time-sensitive decisions, yet little is known about when these data become nominally available compared with when the vital sign was actually measured. The objective of this study was to determine the magnitude of any delay between when a vital sign was actually measured in a patient and when it nominally appears in the EHR.

DESIGN

We performed a single-center retrospective cohort study.

SETTING

Tertiary academic children's hospital.

PATIENTS

A total of 5,458 patients were admitted to a PICU from January 2014 to December 2018.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We analyzed entry and display times of all vital signs entered in the EHR. The primary outcome measurement was time between vital sign occurrence and nominal timing of the vital sign in the EHR. An additional outcome measurement was the frequency of batch charting. A total of 9,818,901 vital sign recordings occurred during the study period. Across the entire cohort the median (interquartile range [IQR]) difference between time of occurrence and nominal time in the EHR was in hours:minutes:seconds, 00:41:58 (IQR 00:13:42-01:44:10). Lag in the first 24 hours of PICU admission was 00:47:34 (IQR 00:15:23-02:19:00), lag in the last 24 hours was 00:38:49 (IQR 00:13:09-01:29:22; p < 0.001). There were 1,892,143 occurrences of batch charting.

CONCLUSIONS

This retrospective study shows a lag between vital sign occurrence and its appearance in the EHR, as well as a frequent practice of batch charting. The magnitude of the delay-median ~40 minutes-suggests that vital signs available in the EHR for clinical review and incorporation into clinical alerts may be outdated by the time they are available.

Machine Learning for Benchmarking Critical Care Outcomes

OBJECTIVES

Enhancing critical care efficacy involves evaluating and improving system functioning. Benchmarking, a retrospective comparison of results against standards, aids risk-adjusted assessment and helps healthcare providers identify areas for improvement based on observed and predicted outcomes. The last two decades have seen the development of several models using machine learning (ML) for clinical outcome prediction. ML is a field of artificial intelligence focused on creating algorithms that enable computers to learn from and make predictions or decisions based on data. This narrative review centers on key discoveries and outcomes to aid clinicians and researchers in selecting the optimal methodology for critical care benchmarking using ML.

METHODS

We used PubMed to search the literature from 2003 to 2023 regarding predictive models utilizing ML for mortality (592 articles), length of stay (143 articles), or mechanical ventilation (195 articles). We supplemented the PubMed search with Google Scholar, making sure relevant articles were included. Given the narrative style, papers in the cohort were manually curated for a comprehensive reader perspective.

RESULTS

Our report presents comparative results for benchmarked outcomes and emphasizes advancements in feature types, preprocessing, model selection, and validation. It showcases instances where ML effectively tackled critical care outcome-prediction challenges, including nonlinear relationships, class imbalances, missing data, and documentation variability, leading to enhanced results.

CONCLUSIONS

Although ML has provided novel tools to improve the benchmarking of critical care outcomes, areas that require further research include class imbalance, fairness, improved calibration, generalizability, and long-term validation of published models.

Prognostic models of in-hospital mortality of intensive care patients using neural representation of unstructured text: A systematic review and critical appraisal

OBJECTIVE

To review and critically appraise published and preprint reports of prognostic models of in-hospital mortality of patients in the intensive-care unit (ICU) based on neural representations (embeddings) of clinical notes.

METHODS

PubMed and arXiv were searched up to August 1, 2022. At least two reviewers independently selected the studies that developed a prognostic model of in-hospital mortality of intensive-care patients using free-text represented as embeddings and extracted data using the CHARMS checklist. Risk of bias was assessed using PROBAST. Reporting on the model was assessed with the TRIPOD guideline. To assess the machine learning components that were used in the models, we present a new descriptive framework based on different techniques to represent text and provide predictions from text. The study protocol was registered in the PROSPERO database (CRD42022354602).

RESULTS

Eighteen studies out of 2,825 were included. All studies used the publicly-available MIMIC dataset. Context-independent word embeddings are widely used. Model discrimination was provided by all studies (AUROC 0.75-0.96), but measures of calibration were scarce. Seven studies used both structural clinical variables and notes. Model discrimination improved when adding clinical notes to variables. None of the models was externally validated and often a simple train/test split was used for internal validation. Our critical appraisal demonstrated a high risk of bias in all studies and concerns regarding their applicability in clinical practice.

CONCLUSION

All studies used a neural architecture for prediction and were based on one publicly available dataset. Clinical notes were reported to improve predictive performance when used in addition to only clinical variables. Most studies had methodological, reporting, and applicability issues. We recommend reporting both model discrimination and calibration, using additional data sources, and using more robust evaluation strategies, including prospective and external validation. Finally, sharing data and code is encouraged to improve study reproducibility.

Editor's Choice Articles for September

The September 2023 issue and this year has already proven to be important for improving our understanding of pediatric acute respiratory distress syndrome (PARDS); Pediatric Critical Care Medicine (PCCM) has published 16 articles so far. Therefore, my three Editor's Choice articles this month highlight yet more PCCM material about PARDS by covering the use of noninvasive ventilation (NIV), the trajectory in cytokine profile during illness, and a new look at lung mechanics. The PCCM Connections for Readers give us the opportunity to focus on some clinical biomarkers of severity and mortality risk during critical illness.

Assessment of Prediction Tasks and Time Window Selection in Temporal Modeling of Electronic Health Record Data: a Systematic Review

Temporal electronic health record (EHR) data are often preferred for clinical prediction tasks because they offer more complete representations of a patient's pathophysiology than static data. A challenge when working with temporal EHR data is problem formulation, which includes defining the time windows of interest and the prediction task. Our objective was to conduct a systematic review that assessed the definition and reporting of concepts relevant to temporal clinical prediction tasks. We searched PubMed® and IEEE Xplore® databases for studies from January 1, 2010 applying machine learning models to EHR data for patient outcome prediction. Publications applying time-series methods were selected for further review. We identified 92 studies and summarized them by clinical context and definition and reporting of the prediction problem. For the time windows of interest, 12 studies did not discuss window lengths, 57 used a single set of window lengths, and 23 evaluated the relationship between window length and model performance. We also found that 72 studies had appropriate reporting of the prediction task. However, evaluation of prediction problem formulation for temporal EHR data was complicated by heterogeneity in assessing and reporting of these concepts. Even among studies modeling similar clinical outcomes, there were variations in terminology used to describe the prediction problem, rationale for window lengths, and determination of the outcome of interest. As temporal modeling using EHR data expands, minimal reporting standards should include time-series specific concerns to promote rigor and reproducibility in future studies and facilitate model implementation in clinical settings.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41666-023-00143-4.

Narrowing the gap: expected versus deployment performance

OBJECTIVES

Successful model development requires both an accurate a priori understanding of future performance and high performance on deployment. Optimistic estimations of model performance that are unrealized in real-world clinical settings can contribute to nonuse of predictive models. This study used 2 tasks, predicting ICU mortality and Bi-Level Positive Airway Pressure failure, to quantify: (1) how well internal test performances derived from different methods of partitioning data into development and test sets estimate future deployment performance of Recurrent Neural Network models and (2) the effects of including older data in the training set on models' performance.

MATERIALS AND METHODS

The cohort consisted of patients admitted between 2010 and 2020 to the Pediatric Intensive Care Unit of a large quaternary children's hospital. 2010-2018 data were partitioned into different development and test sets to measure internal test performance. Deployable models were trained on 2010-2018 data and assessed on 2019-2020 data, which was conceptualized to represent a real-world deployment scenario. Optimism, defined as the overestimation of the deployed performance by internal test performance, was measured. Performances of deployable models were also compared with each other to quantify the effect of including older data during training.

RESULTS, DISCUSSION, AND CONCLUSION

Longitudinal partitioning methods, where models are tested on newer data than the development set, yielded the least optimism. Including older years in the training dataset did not degrade deployable model performance. Using all available data for model development fully leveraged longitudinal partitioning by measuring year-to-year performance.

Wearable Biosensors in Congenital Heart Disease: Needs to Advance the Field

Traditional measures of clinical status and physiology have generally been based in health care settings, episodic, short in duration, and performed at rest. Wearable biosensors provide an opportunity to obtain continuous non-invasive physiologic data from patients with congenital heart disease (CHD) in the real-world setting, over longer durations, and across varying levels of activity. However, there are significant technical limitations to the use of wearable biosensors in CHD. Here, we review current applications of wearable biosensors in CHD; how clinical and research uses of wearable biosensors must consider various CHD physiologies; the technical challenges in developing wearable biosensors for CHD; and special considerations for digital biomarkers in CHD.

Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa

Objectives

Delays in identification, resuscitation and referral have been identified as a preventable cause of avoidable severity of illness and mortality in South African children. To address this problem, a machine learning model to predict a compound outcome of death prior to discharge from hospital and/or admission to the PICU was developed. A key aspect of developing machine learning models is the integration of human knowledge in their development. The objective of this study is to describe how this domain knowledge was elicited, including the use of a documented literature search and Delphi procedure.

Design

A prospective mixed methodology development study was conducted that included qualitative aspects in the elicitation of domain knowledge, together with descriptive and analytical quantitative and machine learning methodologies.

Setting

A single centre tertiary hospital providing acute paediatric services.

Participants

Three paediatric intensivists, six specialist paediatricians and three specialist anaesthesiologists.

Interventions

None.

Measurements and main results

The literature search identified 154 full-text articles reporting risk factors for mortality in hospitalised children. These factors were most commonly features of specific organ dysfunction. 89 of these publications studied children in lower- and middle-income countries. The Delphi procedure included 12 expert participants and was conducted over 3 rounds. Respondents identified a need to achieve a compromise between model performance, comprehensiveness and veracity and practicality of use. Participants achieved consensus on a range of clinical features associated with severe illness in children. No special investigations were considered for inclusion in the model except point-of-care capillary blood glucose testing. The results were integrated by the researcher and a final list of features was compiled.

Conclusion

The elicitation of domain knowledge is important in effective machine learning applications. The documentation of this process enhances rigour in such models and should be reported in publications. A documented literature search, Delphi procedure and the integration of the domain knowledge of the researchers contributed to problem specification and selection of features prior to feature engineering, pre-processing and model development.

Data harnessing to nurture the human mind for a tailored approach to the child

Big data in pediatrics is an ocean of structured and unstructured data. Big data analysis helps to dive into the ocean of data to filter out information that can guide pediatricians in their decision making, precision diagnosis, and targeted therapy. In addition, big data and its analysis have helped in the surveillance, prevention, and performance of the health system. There has been a considerable amount of work in pediatrics that we have tried to highlight in this review and some of it has been already incorporated into the health system. Work in specialties of pediatrics is still forthcoming with the creation of a common data model and amalgamation of the huge "omics" database. The physicians entrusted with the care of children must be aware of the outcome so that they can play a role to ensure that big data algorithms have a clinically relevant effect in improving the health of their patients. They will apply the outcome of big data and its analysis in patient care through clinical algorithms or with the help of embedded clinical support alerts from the electronic medical records. IMPACT: Big data in pediatrics include structured, unstructured data, waveform data, biological, and social data. Big data analytics has unraveled significant information from these databases. This is changing how pediatricians will look at the body of available evidence and translate it into their clinical practice. Data harnessed so far is implemented in certain fields while in others it is in the process of development to become a clinical adjunct to the physician. Common databases are being prepared for future work. Diagnostic and prediction models when incorporated into the health system will guide the pediatrician to a targeted approach to diagnosis and therapy.

The use of machine learning and artificial intelligence within pediatric critical care

The field of pediatric critical care has been hampered in the era of precision medicine by our inability to accurately define and subclassify disease phenotypes. This has been caused by heterogeneity across age groups that further challenges the ability to perform randomized controlled trials in pediatrics. One approach to overcome these inherent challenges include the use of machine learning algorithms that can assist in generating more meaningful interpretations from clinical data. This review summarizes machine learning and artificial intelligence techniques that are currently in use for clinical data modeling with relevance to pediatric critical care. Focus has been placed on the differences between techniques and the role of each in the clinical arena. The various forms of clinical decision support that utilize machine learning are also described. We review the applications and limitations of machine learning techniques to empower clinicians to make informed decisions at the bedside. IMPACT: Critical care units generate large amounts of under-utilized data that can be processed through artificial intelligence. This review summarizes the machine learning and artificial intelligence techniques currently being used to process clinical data. The review highlights the applications and limitations of these techniques within a clinical context to aid providers in making more informed decisions at the bedside.

Machine Learning-Based Systems for the Anticipation of Adverse Events After Pediatric Cardiac Surgery

Pediatric congenital heart disease (CHD) patients are at higher risk of postoperative complications and clinical deterioration either due to their underlying pathology or due to the cardiac surgery, contributing significantly to mortality, morbidity, hospital and family costs, and poor quality of life. In current clinical practice, clinical deterioration is detected, in most of the cases, when it has already occurred. Several early warning scores (EWS) have been proposed to assess children at risk of clinical deterioration using vital signs and risk indicators, in order to intervene in a timely manner to reduce the impact of deterioration and risk of death among children. However, EWS are based on measurements performed at a single time point without incorporating trends nor providing information about patient's risk trajectory. Moreover, some of these measurements rely on subjective assessment making them susceptible to different interpretations. All these limitations could explain why the implementation of EWS in high-resource settings failed to show a significant decrease in hospital mortality. By means of machine learning (ML) based algorithms we could integrate heterogeneous and complex data to predict patient's risk of deterioration. In this perspective article, we provide a brief overview of the potential of ML technologies to improve the identification of pediatric CHD patients at high-risk for clinical deterioration after cardiac surgery, and present the CORTEX traffic light, a ML-based predictive system that Sant Joan de Déu Barcelona Children's Hospital is implementing, as an illustration of the application of an ML-based risk stratification system in a relevant hospital setting.

Clinical Informatics and Quality Improvement in the Pediatric Intensive Care Unit

Clinical informatics can support quality improvement and patient safety in the pediatric intensive care unit (PICU) in several ways including data extraction, analysis, and decision support enabled by electronic health records (EHRs), and databases and registries. Clinical decision support (CDS), embedded in EHRs, now an integral part of the workflow in the PICU, includes several tools and is increasingly leveraging artificial intelligence (AI). Understanding the opportunities and challenges can improve the engagement of clinicians with the design, validation, and implementation of CDS, improve satisfaction with CDS, and improve patient safety, care quality, and value.

Development of a deep learning model that predicts Bi-level positive airway pressure failure

Delaying intubation for patients failing Bi-Level Positive Airway Pressure (BIPAP) may be associated with harm. The objective of this study was to develop a deep learning model capable of aiding clinical decision making by predicting Bi-Level Positive Airway Pressure (BIPAP) failure. This was a retrospective cohort study in a tertiary pediatric intensive care unit (PICU) between 2010 and 2020. Three machine learning models were developed to predict BIPAP failure: two logistic regression models and one deep learning model, a recurrent neural network with a Long Short-Term Memory (LSTM-RNN) architecture. Model performance was evaluated in a holdout test set. 175 (27.7%) of 630 total BIPAP sessions were BIPAP failures. Patients in the BIPAP failure group were on BIPAP for a median of 32.8 (9.2-91.3) hours prior to intubation. Late BIPAP failure (intubation after using BIPAP > 24 h) patients had fewer 28-day Ventilator Free Days (13.40 [0.68-20.96]), longer ICU length of stay and more post-extubation BIPAP days compared to those who were intubated ≤ 24 h from BIPAP initiation. An AUROC above 0.5 indicates that a model has extracted new information, potentially valuable to the clinical team, about BIPAP failure. Within 6 h of BIPAP initiation, the LSTM-RNN model predicted which patients were likely to fail BIPAP with an AUROC of 0.81 (0.80, 0.82), superior to all other models. Within 6 h of BIPAP initiation, the LSTM-RNN model would identify nearly 80% of BIPAP failures with a 50% false alarm rate, equal to an NNA of 2. In conclusion, a deep learning method using readily available data from the electronic health record can identify which patients on BIPAP are likely to fail with good discrimination, oftentimes days before they are intubated in usual practice.

Dynamic Mortality Risk Predictions for Children in ICUs: Development and Validation of Machine Learning Models

OBJECTIVES

Assess a machine learning method of serially updated mortality risk.

DESIGN

Retrospective analysis of a national database (Health Facts; Cerner Corporation, Kansas City, MO).

SETTING

Hospitals caring for children in ICUs.

PATIENTS

A total of 27,354 admissions cared for in ICUs from 2009 to 2018.

INTERVENTIONS

None.

MAIN OUTCOME

Hospital mortality risk estimates determined at 6-hour time periods during care in the ICU. Models were truncated at 180 hours due to decreased sample size secondary to discharges and deaths.

MEASUREMENTS AND MAIN RESULTS

The Criticality Index, based on physiology, therapy, and care intensity, was computed for each admission for each time period and calibrated to hospital mortality risk (Criticality Index-Mortality [CI-M]) at each of 29 time periods (initial assessment: 6 hr; last assessment: 180 hr). Performance metrics and clinical validity were determined from the held-out test sample (n = 3,453, 13%). Discrimination assessed with the area under the receiver operating characteristic curve was 0.852 (95% CI, 0.843-0.861) overall and greater than or equal to 0.80 for all individual time periods. Calibration assessed by the Hosmer-Lemeshow goodness-of-fit test showed good fit overall (p = 0.196) and was statistically not significant for 28 of the 29 time periods. Calibration plots for all models revealed the intercept ranged from--0.002 to 0.009, the slope ranged from 0.867 to 1.415, and the R2 ranged from 0.862 to 0.989. Clinical validity assessed using population trajectories and changes in the risk status of admissions (clinical volatility) revealed clinical trajectories consistent with clinical expectations and greater clinical volatility in deaths than survivors (p < 0.001).

CONCLUSIONS

Machine learning models incorporating physiology, therapy, and care intensity can track changes in hospital mortality risk during intensive care. The CI-M's framework and modeling method are potentially applicable to monitoring clinical improvement and deterioration in real time.

Development of artificial neural network models for paediatric critical illness in South Africa

OBJECTIVES

Failures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation.

DESIGN

Prospective, analytical cohort study.

SETTING

A single centre tertiary hospital in South Africa providing acute paediatric services.

PATIENTS

Children, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations.

OUTCOMES

Predictive models for a composite outcome of death before discharge from hospital or admission to the PICU.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit.

CONCLUSIONS

All models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.

Severity of illness and organ dysfunction scoring systems in pediatric critical care: The impacts on clinician's practices and the future

Severity and organ dysfunction (OD) scores are increasingly used in pediatric intensive care units (PICU). Therefore, this review aims to provide 1/ an updated state-of-the-art of severity scoring systems and OD scores in pediatric critical care, which explains 2/ the performance measurement tools and the significance of each tool in clinical practice and provides 3/ the usefulness, limits, and impact on future scores in PICU. The following two pediatric systems have been proposed: the PRISMIV, is used to collect data between 2 h before PICU admission and the first 4 h after PICU admission; the PIM3, is used to collect data during the first hour after PICU admission. The PELOD-2 and SOFApediatric scores were the most common OD scores available. Scores used in the PICU should help clinicians answer the following three questions: 1/ Are the most severely ill patients dying in my service: a good discrimination allow us to interpret that there are the most severe patients who died in my service. 2/ Does the overall number of deaths observed in my department consistent with the severity of patients? The standard mortality ratio allow us to determine whether the total number of deaths observed in our service over a given period is in adequacy with the number of deaths predicted, by considering the severity of patients on admission? 3/ Does the number of deaths observed by severity level in my department consistent with the severity of patients? The calibration enabled us to determine whether the number of deaths observed according to the severity of patients at PICU admission in a department over a given period is in adequacy with the number of deaths predicted, according to the severity of the patients at PICU admission. These scoring systems are not interpretable at the patient level. Scoring systems are used to describe patients with PICU in research and evaluate the service's case mix and performance. Therefore, the prospect of automated data collection, which permits their calculation, facilitated by the computerization of services, is a necessity that manufacturers should consider.

An Artificial Neural Network Model for Pediatric Mortality Prediction in Two Tertiary Pediatric Intensive Care Units in South Africa. A Development Study

OBJECTIVES

The performance of mortality prediction models remain a challenge in lower- and middle-income countries. We developed an artificial neural network (ANN) model for the prediction of mortality in two tertiary pediatric intensive care units (PICUs) in South Africa using free to download and use software and commercially available computers. These models were compared to a logistic regression model and a recalibrated version of the Pediatric Index of Mortality 3.

DESIGN

This study used data from a retrospective cohort study to develop an artificial neural model and logistic regression model for mortality prediction. The outcome evaluated was death in PICU.

SETTING

Two tertiary PICUs in South Africa.

PATIENTS

2,089 patients up to the age of 13 completed years were included in the study.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The AUROC was higher for the ANN (0.89) than for the logistic regression model (LR) (0.87) and the recalibrated PIM3 model (0.86). The precision recall curve however favors the ANN over logistic regression and recalibrated PIM3 (AUPRC = 0.6 vs. 0.53 and 0.58, respectively. The slope of the calibration curve was 1.12 for the ANN model (intercept 0.01), 1.09 for the logistic regression model (intercept 0.05) and 1.02 (intercept 0.01) for the recalibrated version of PIM3. The calibration curve was however closer to the diagonal for the ANN model.

CONCLUSIONS

Artificial neural network models are a feasible method for mortality prediction in lower- and middle-income countries but significant challenges exist. There is a need to conduct research directed toward the acquisition of large, complex data sets, the integration of documented clinical care into clinical research and the promotion of the development of electronic health record systems in lower and middle income settings.

The criticality Index-mortality: A dynamic machine learning prediction algorithm for mortality prediction in children cared for in an ICU

BACKGROUND

The Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change.

OBJECTIVES

To assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions.

POPULATION

There were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing.

MODEL

A neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression.

RESULTS

Discrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R² was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors.

CONCLUSIONS

The Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.

Ignorance Isn't Bliss: We Must Close the Machine Learning Knowledge Gap in Pediatric Critical Care

Pediatric intensivists are bombarded with more patient data than ever before. Integration and interpretation of data from patient monitors and the electronic health record (EHR) can be cognitively expensive in a manner that results in delayed or suboptimal medical decision making and patient harm. Machine learning (ML) can be used to facilitate insights from healthcare data and has been successfully applied to pediatric critical care data with that intent. However, many pediatric critical care medicine (PCCM) trainees and clinicians lack an understanding of foundational ML principles. This presents a major problem for the field. We outline the reasons why in this perspective and provide a roadmap for competency-based ML education for PCCM trainees and other stakeholders.

Predicting High-Flow Nasal Cannula Failure in an ICU Using a Recurrent Neural Network with Transfer Learning and Input Data Perseveration: A Retrospective Analysis (Preprint)

Background

High flow nasal cannula (HFNC) provides noninvasive respiratory support for children who are critically ill who may tolerate it more readily than other noninvasive ventilation (NIV) techniques such as bilevel positive airway pressure and continuous positive airway pressure. Moreover, HFNC may preclude the need for mechanical ventilation (intubation). Nevertheless, NIV or intubation may ultimately be necessary for certain patients. Timely prediction of HFNC failure can provide an indication for increasing respiratory support.

Objective

The aim of this study is to develop and compare machine learning (ML) models to predict HFNC failure.

Methods

A retrospective study was conducted using the Virtual Pediatric Intensive Care Unit database of electronic medical records of patients admitted to a tertiary pediatric intensive care unit between January 2010 and February 2020. Patients aged <19 years, without apnea, and receiving HFNC treatment were included. A long short-term memory (LSTM) model using 517 variables (vital signs, laboratory data, and other clinical parameters) was trained to generate a continuous prediction of HFNC failure, defined as escalation to NIV or intubation within 24 hours of HFNC initiation. For comparison, 7 other models were trained: a logistic regression (LR) using the same 517 variables, another LR using only 14 variables, and 5 additional LSTM-based models using the same 517 variables as the first LSTM model and incorporating additional ML techniques (transfer learning, input perseveration, and ensembling). Performance was assessed using the area under the receiver operating characteristic (AUROC) curve at various times following HFNC initiation. The sensitivity, specificity, and positive and negative predictive values of predictions at 2 hours after HFNC initiation were also evaluated. These metrics were also computed for a cohort with primarily respiratory diagnoses.

Results

A total of 834 HFNC trials (455 [54.6%] training, 173 [20.7%] validation, and 206 [24.7%] test) met the inclusion criteria, of which 175 (21%; training: 103/455, 22.6%; validation: 30/173, 17.3%; test: 42/206, 20.4%) escalated to NIV or intubation. The LSTM models trained with transfer learning generally performed better than the LR models, with the best LSTM model achieving an AUROC of 0.78 versus 0.66 for the 14-variable LR and 0.71 for the 517-variable LR 2 hours after initiation. All models except for the 14-variable LR achieved higher AUROCs in the respiratory cohort than in the general intensive care unit population.

Conclusions

ML models trained using electronic medical record data were able to identify children at risk of HFNC failure within 24 hours of initiation. LSTM models that incorporated transfer learning, input data perseveration, and ensembling showed improved performance compared with the LR and standard LSTM models.

Interpreting a recurrent neural network's predictions of ICU mortality risk

Deep learning has demonstrated success in many applications; however, their use in healthcare has been limited due to the lack of transparency into how they generate predictions. Algorithms such as Recurrent Neural Networks (RNNs) when applied to Electronic Medical Records (EMR) introduce additional barriers to transparency because of the sequential processing of the RNN and the multi-modal nature of EMR data. This work seeks to improve transparency by: 1) introducing Learned Binary Masks (LBM) as a method for identifying which EMR variables contributed to an RNN model's risk of mortality (ROM) predictions for critically ill children; and 2) applying KernelSHAP for the same purpose. Given an individual patient, LBM and KernelSHAP both generate an attribution matrix that shows the contribution of each input feature to the RNN's sequence of predictions for that patient. Attribution matrices can be aggregated in many ways to facilitate different levels of analysis of the RNN model and its predictions. Presented are three methods of aggregations and analyses: 1) over volatile time periods within individual patient predictions, 2) over populations of ICU patients sharing specific diagnoses, and 3) across the general population of critically ill children.