1
|
Heneghan JA, Walker SB, Fawcett A, Bennett TD, Dziorny AC, Sanchez-Pinto LN, Farris RW, Winter MC, Badke C, Martin B, Brown SR, McCrory MC, Ness-Cochinwala M, Rogerson C, Baloglu O, Harwayne-Gidansky I, Hudkins MR, Kamaleswaran R, Gangadharan S, Tripathi S, Mendonca EA, Markovitz BP, Mayampurath A, Spaeder MC. The Pediatric Data Science and Analytics Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators Network: Use of Supervised Machine Learning Applications in Pediatric Critical Care Medicine Research. Pediatr Crit Care Med 2024; 25:364-374. [PMID: 38059732 PMCID: PMC10994770 DOI: 10.1097/pcc.0000000000003425] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/08/2023]
Abstract
OBJECTIVE Perform a scoping review of supervised machine learning in pediatric critical care to identify published applications, methodologies, and implementation frequency to inform best practices for the development, validation, and reporting of predictive models in pediatric critical care. DESIGN Scoping review and expert opinion. SETTING We queried CINAHL Plus with Full Text (EBSCO), Cochrane Library (Wiley), Embase (Elsevier), Ovid Medline, and PubMed for articles published between 2000 and 2022 related to machine learning concepts and pediatric critical illness. Articles were excluded if the majority of patients were adults or neonates, if unsupervised machine learning was the primary methodology, or if information related to the development, validation, and/or implementation of the model was not reported. Article selection and data extraction were performed using dual review in the Covidence tool, with discrepancies resolved by consensus. SUBJECTS Articles reporting on the development, validation, or implementation of supervised machine learning models in the field of pediatric critical care medicine. INTERVENTIONS None. MEASUREMENTS AND MAIN RESULTS Of 5075 identified studies, 141 articles were included. Studies were primarily (57%) performed at a single site. The majority took place in the United States (70%). Most were retrospective observational cohort studies. More than three-quarters of the articles were published between 2018 and 2022. The most common algorithms included logistic regression and random forest. Predicted events were most commonly death, transfer to ICU, and sepsis. Only 14% of articles reported external validation, and only a single model was implemented at publication. Reporting of validation methods, performance assessments, and implementation varied widely. Follow-up with authors suggests that implementation remains uncommon after model publication. CONCLUSIONS Publication of supervised machine learning models to address clinical challenges in pediatric critical care medicine has increased dramatically in the last 5 years. While these approaches have the potential to benefit children with critical illness, the literature demonstrates incomplete reporting, absence of external validation, and infrequent clinical implementation.
Collapse
Affiliation(s)
- Julia A. Heneghan
- Division of Pediatric Critical Care, University of Minnesota Masonic Children’s Hospital; Minneapolis, MN
| | - Sarah B. Walker
- Department of Pediatrics (Critical Care), Northwestern University Feinberg School of Medicine and Ann & Robert H. Lurie Children’s Hospital of Chicago; Chicago, IL
| | - Andrea Fawcett
- Department of Clinical and Organizational Development; Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, IL
| | - Tellen D. Bennett
- Departments of Biomedical Informatics and Pediatrics (Critical Care Medicine), University of Colorado School of Medicine; Aurora, CO
| | - Adam C. Dziorny
- Department of Pediatrics, University of Rochester; Rochester, NY
| | - L. Nelson Sanchez-Pinto
- Department of Pediatrics (Critical Care) and Preventive Medicine (Health & Biomedical Informatics), Northwestern University Feinberg School of Medicine and Ann & Robert H. Lurie Children’s Hospital of Chicago; Chicago, IL
| | - Reid W.D. Farris
- Department of Pediatrics, University of Washington and Seattle Children’s Hospital; Seattle, WA
| | - Meredith C. Winter
- Department of Anesthesiology Critical Care Medicine, Children’s Hospital Los Angeles and Keck School of Medicine, University of Southern California; Los Angeles, CA
| | - Colleen Badke
- Department of Pediatrics (Critical Care), Northwestern University Feinberg School of Medicine and Ann & Robert H. Lurie Children’s Hospital of Chicago; Chicago, IL
| | - Blake Martin
- Departments of Biomedical Informatics and Pediatrics (Critical Care Medicine), University of Colorado School of Medicine; Aurora, CO
| | - Stephanie R. Brown
- Section of Pediatric Critical Care, Oklahoma Children’s Hospital and Department of Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, OK
| | - Michael C. McCrory
- Department of Anesthesiology, Wake Forest University School of Medicine; Winston Salem, NC
| | | | - Colin Rogerson
- Division of Critical Care, Department of Pediatrics, Indiana University; Indianapolis, IN
| | - Orkun Baloglu
- Pediatric Critical Care Medicine and Pediatric Cardiology, Cleveland Clinic Children’s Center for Artificial Intelligence (C4AI), Cleveland Clinic; Cleveland, OH
| | | | - Matthew R. Hudkins
- Division of Pediatric Critical Care, Department of Pediatrics, Oregon Health & Science University; Portland, OR
| | - Rishikesan Kamaleswaran
- Departments of Biomedical Informatics and Pediatrics, Emory University School of Medicine; Department of Biomedical Engineering, Georgia Institute of Technology; Atlanta, GA
| | - Sandeep Gangadharan
- Department of Pediatrics, Mount Sinai Icahn School of Medicine; New York, NY
| | - Sandeep Tripathi
- Department of Pediatrics. University of Illinois College of Medicine at Peoria/OSF HealthCare, Children’s Hospital of Illinois; Peoria, IL
| | - Eneida A. Mendonca
- Division of Biomedical Informatics, Department of Pediatrics, Cincinnati Children’s Hospital Medical Center and University of Cincinnati; Cincinnati, OH
| | - Barry P. Markovitz
- Division of Pediatric Critical Care, Department of Pediatrics, University of Utah Spencer F Eccles School of Medicine, Intermountain Primary Children’s Hospital; Salt Lake City, UT
| | - Anoop Mayampurath
- Department of Biostatistics & Medical Informatics, University of Wisconsin-Madison; Madison, WI
| | - Michael C. Spaeder
- Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA
| |
Collapse
|
2
|
DeWitt PE, Rebull MA, Bennett TD. Open source and reproducible and inexpensive infrastructure for data challenges and education. Sci Data 2024; 11:8. [PMID: 38167901 PMCID: PMC10762166 DOI: 10.1038/s41597-023-02854-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 12/12/2023] [Indexed: 01/05/2024] Open
Abstract
Data sharing is necessary to maximize the actionable knowledge generated from research data. Data challenges can encourage secondary analyses of datasets. Data challenges in biomedicine often rely on advanced cloud-based computing infrastructure and expensive industry partnerships. Examples include challenges that use Google Cloud virtual machines and the Sage Bionetworks Dream Challenges platform. Such robust infrastructures can be financially prohibitive for investigators without substantial resources. Given the potential to develop scientific and clinical knowledge and the NIH emphasis on data sharing and reuse, there is a need for inexpensive and computationally lightweight methods for data sharing and hosting data challenges. To fill that gap, we developed a workflow that allows for reproducible model training, testing, and evaluation. We leveraged public GitHub repositories, open-source computational languages, and Docker technology. In addition, we conducted a data challenge using the infrastructure we developed. In this manuscript, we report on the infrastructure, workflow, and data challenge results. The infrastructure and workflow are likely to be useful for data challenges and education.
Collapse
Affiliation(s)
- Peter E DeWitt
- Department of Biomedical Informatics, University of Colorado School of Medicine, University of Colorado, Aurora, CO, USA.
| | - Margaret A Rebull
- Department of Biomedical Informatics, University of Colorado School of Medicine, University of Colorado, Aurora, CO, USA
| | - Tellen D Bennett
- Department of Biomedical Informatics, University of Colorado School of Medicine, University of Colorado, Aurora, CO, USA.
- Section of Critical Care Medicine, Department of Pediatrics, University of Colorado School of Medicine, University of Colorado, Aurora, CO, USA.
- Children's Hospital Colorado, Aurora, CO, USA.
| |
Collapse
|
3
|
Fonseca J, Liu X, Oliveira HP, Pereira T. Mortality prediction using medical time series on TBI patients. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 242:107806. [PMID: 37832428 DOI: 10.1016/j.cmpb.2023.107806] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 07/29/2023] [Accepted: 09/08/2023] [Indexed: 10/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Traumatic Brain Injury (TBI) is one of the leading causes of injury-related mortality in the world, with severe cases reaching mortality rates of 30-40%. It is highly heterogeneous both in causes and consequences making more complex the medical interpretation and prognosis. Gathering clinical, demographic, and laboratory data to perform a prognosis requires time and skill in several clinical specialties. Artificial intelligence (AI) methods can take advantage of existing data by performing helpful predictions and guiding physicians toward a better prognosis and, consequently, better healthcare. The objective of this work was to develop learning models and evaluate their capability of predicting the mortality of TBI. The predictive model would allow the early assessment of the more serious cases and scarce medical resources can be pointed toward the patients who need them most. METHODS Long Short Term Memory (LSTM) and Transformer architectures were tested and compared in performance, coupled with data imbalance, missing data, and feature selection strategies. From the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, a cohort of TBI patients was selected and an analysis of the first 48 hours of multiple time series sequential variables was done to predict hospital mortality. RESULTS The best performance was obtained with the Transformer architecture, achieving an AUC of 0.907 with the larger group of features and trained with class proportion class weights and binary cross entropy loss. CONCLUSIONS Using the time series sequential data, LSTM and Transformers proved to be both viable options for predicting TBI hospital mortality in 48 hours after admission. Overall, using sequential deep learning models with time series data to predict TBI mortality is viable and can be used as a helpful indicator of the well-being of patients.
Collapse
Affiliation(s)
- João Fonseca
- INESC TEC - Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal; FEUP - Faculty of Engineering, University of Porto, Porto, Portugal
| | - Xiuyun Liu
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin, China
| | - Hélder P Oliveira
- INESC TEC - Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal; FCUP - Faculty of Science, University of Porto, Porto, Portugal
| | - Tania Pereira
- INESC TEC - Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal; FEUP - Faculty of Engineering, University of Porto, Porto, Portugal.
| |
Collapse
|
4
|
Appiah Balaji NN, Beaulieu CL, Bogner J, Ning X. Traumatic Brain Injury Rehabilitation Outcome Prediction Using Machine Learning Methods. Arch Rehabil Res Clin Transl 2023; 5:100295. [PMID: 38163039 PMCID: PMC10757159 DOI: 10.1016/j.arrct.2023.100295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2024] Open
Abstract
Objective To investigate the performance of machine learning (ML) methods for predicting outcomes from inpatient rehabilitation for subjects with TBI using a dataset with a large number of predictor variables. Our second objective was to identify top predictive features selected by the ML models for each outcome and to validate the interpretability of the models. Design Secondary analysis using computational modeling of relationships between patients, injury and treatment activities and 6 outcomes, applied to the large multi-site, prospective, longitudinal observational dataset collected during the traumatic brain injury inpatient rehabilitation study. Setting Acute inpatient rehabilitation. Participants 1946 patients aged 14 years or older, who sustained a severe, moderate, or complicated mild TBI, and were admitted to 1 of 9 US inpatient rehabilitation sites between 2008 and 2011 (N=1946). Main Outcome Measures Rehabilitation length of stay, discharge to home, FIM cognitive and FIM motor at discharge and at 9-months post discharge. Results Advanced ML models, specifically gradient boosting tree model, performed consistently better than all other models, including classical linear regression models. Top ranked predictive features were identified for each of the 6 outcome variables. Level of effort, days to rehabilitation admission, age at rehabilitation admission, and advanced mobility activities were the most frequently top ranked predictive features. The highest-ranking predictive feature differed across the specific outcome variable. Conclusions Identifying patient, injury, and rehabilitation treatment variables that are predictive of better outcomes will contribute to cost-effective care delivery and guide evidence-based clinical practice. ML methods can contribute to these efforts.
Collapse
Affiliation(s)
| | - Cynthia L. Beaulieu
- Department of Physical Medicine and Rehabilitation, The Ohio State University College of Medicine, Columbus, OH
| | - Jennifer Bogner
- Department of Physical Medicine and Rehabilitation, The Ohio State University College of Medicine, Columbus, OH
| | - Xia Ning
- Department of Computer Science and Engineering, The Ohio State University, Columbus, OH
- Department of Biomedical Informatics, The Ohio State University, Columbus, OH
- Translational Data Analytics Institute, The Ohio State University, Columbus, OH
| |
Collapse
|
5
|
Mekkodathil A, El-Menyar A, Naduvilekandy M, Rizoli S, Al-Thani H. Machine Learning Approach for the Prediction of In-Hospital Mortality in Traumatic Brain Injury Using Bio-Clinical Markers at Presentation to the Emergency Department. Diagnostics (Basel) 2023; 13:2605. [PMID: 37568968 PMCID: PMC10417008 DOI: 10.3390/diagnostics13152605] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Revised: 07/11/2023] [Accepted: 08/02/2023] [Indexed: 08/13/2023] Open
Abstract
BACKGROUND Accurate prediction of in-hospital mortality is essential for better management of patients with traumatic brain injury (TBI). Machine learning (ML) algorithms have been shown to be effective in predicting clinical outcomes. This study aimed to identify predictors of in-hospital mortality in TBI patients using ML algorithms. MATERIALS AND METHOD A retrospective study was performed using data from both the trauma registry and electronic medical records among TBI patients admitted to the Hamad Trauma Center in Qatar between June 2016 and May 2021. Thirteen features were selected for four ML models including a Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XgBoost), to predict the in-hospital mortality. RESULTS A dataset of 922 patients was analyzed, of which 78% survived and 22% died. The AUC scores for SVM, LR, XgBoost, and RF models were 0.86, 0.84, 0.85, and 0.86, respectively. XgBoost and RF had good AUC scores but exhibited significant differences in log loss between the training and testing sets (% difference in logloss of 79.5 and 41.8, respectively), indicating overfitting compared to the other models. The feature importance trend across all models indicates that aPTT, INR, ISS, prothrombin time, and lactic acid are the most important features in prediction. Magnesium also displayed significant importance in the prediction of mortality among serum electrolytes. CONCLUSIONS SVM was found to be the best-performing ML model in predicting the mortality of TBI patients. It had the highest AUC score and did not show overfitting, making it a more reliable model compared to LR, XgBoost, and RF.
Collapse
Affiliation(s)
- Ahammed Mekkodathil
- Clinical Research, Trauma and Vascular Surgery, Hamad Medical Corporation, Doha P.O. Box 3050, Qatar;
| | - Ayman El-Menyar
- Clinical Research, Trauma and Vascular Surgery, Hamad Medical Corporation, Doha P.O. Box 3050, Qatar;
- Clinical Medicine, Weill Cornell Medical College, Doha P.O. Box 24144, Qatar
| | | | - Sandro Rizoli
- Trauma Surgery Section, Hamad General Hospital (HGH), Doha P.O. Box 3050, Qatar; (S.R.)
| | - Hassan Al-Thani
- Trauma Surgery Section, Hamad General Hospital (HGH), Doha P.O. Box 3050, Qatar; (S.R.)
| |
Collapse
|