1
|
Nikravangolsefid N, Reddy S, Truong HH, Charkviani M, Ninan J, Prokop LJ, Suppadungsuk S, Singh W, Kashani KB, Garces JPD. Machine learning for predicting mortality in adult critically ill patients with Sepsis: A systematic review. J Crit Care 2024; 84:154889. [PMID: 39059094 DOI: 10.1016/j.jcrc.2024.154889] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2024] [Revised: 07/10/2024] [Accepted: 07/18/2024] [Indexed: 07/28/2024]
Abstract
INTRODUCTION Various Machine Learning (ML) models have been used to predict sepsis-associated mortality. We conducted a systematic review to evaluate the methodologies employed in studies to predict mortality among patients with sepsis. METHODS Following a pre-established protocol registered at the International Prospective Register of Systematic Reviews, we performed a comprehensive search of databases from inception to February 2024. We included peer-reviewed articles reporting predicting mortality in critically ill adult patients with sepsis. RESULTS Among the 1822 articles, 31 were included, involving 1,477,200 adult patients with sepsis. Nineteen studies had a high risk of bias. Among the diverse ML models, Logistic regression and eXtreme Gradient Boosting were the most frequently used, in 22 and 16 studies, respectively. Nine studies performed internal and external validation. Compared with conventional scoring systems such as SOFA, the ML models showed slightly higher performance in predicting mortality (AUROC ranges: 0.62-0.90 vs. 0.47-0.86). CONCLUSIONS ML models demonstrate a modest improvement in predicting sepsis-associated mortality. The certainty of these findings remains low due to the high risk of bias and significant heterogeneity. Studies should include comprehensive methodological details on calibration and hyperparameter selection, adopt a standardized definition of sepsis, and conduct multicenter prospective designs along with external validations.
Collapse
Affiliation(s)
- Nasrin Nikravangolsefid
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA; Division of Pulmonary and Critical Care Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Swetha Reddy
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Hong Hieu Truong
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA; Saint Francis Hospital, Department of Medicine, Evanston, IL, USA
| | - Mariam Charkviani
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Jacob Ninan
- Department of Nephrology and Critical Care, MultiCare Capital Medical Center, Olympia, WA, USA
| | | | - Supawadee Suppadungsuk
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Waryaam Singh
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Kianoush B Kashani
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA; Division of Pulmonary and Critical Care Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | - Juan Pablo Domecq Garces
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN, USA; Department of Critical Care Medicine, Mayo Clinic Health System, Mankato, MN, USA.
| |
Collapse
|
2
|
Matos J, Gallifant J, Chowdhury A, Economou-Zavlanos N, Charpignon ML, Gichoya J, Celi LA, Nazer L, King H, Wong AKI. A Clinician's Guide to Understanding Bias in Critical Clinical Prediction Models. Crit Care Clin 2024; 40:827-857. [PMID: 39218488 DOI: 10.1016/j.ccc.2024.05.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/04/2024]
Abstract
This narrative review focuses on the role of clinical prediction models in supporting informed decision-making in critical care, emphasizing their 2 forms: traditional scores and artificial intelligence (AI)-based models. Acknowledging the potential for both types to embed biases, the authors underscore the importance of critical appraisal to increase our trust in models. The authors outline recommendations and critical care examples to manage risk of bias in AI models. The authors advocate for enhanced interdisciplinary training for clinicians, who are encouraged to explore various resources (books, journals, news Web sites, and social media) and events (Datathons) to deepen their understanding of risk of bias.
Collapse
Affiliation(s)
- João Matos
- University of Porto (FEUP), Porto, Portugal; Institute for Systems and Computer Engineering, Technology and Science (INESC TEC), Porto, Portugal; Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Jack Gallifant
- Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Critical Care, Guy's and St Thomas' NHS Trust, London, UK
| | - Anand Chowdhury
- Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, Duke University, Durham, NC, USA
| | | | - Marie-Laure Charpignon
- Institute for Data Systems and Society, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Judy Gichoya
- Department of Radiology, Emory University, Atlanta, GA, USA
| | - Leo Anthony Celi
- Laboratory for Computational Physiology, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Lama Nazer
- Department of Pharmacy, King Hussein Cancer Center, Amman, Jordan
| | - Heather King
- Durham VA Health Care System, Health Services Research and Development, Center of Innovation to Accelerate Discovery and Practice Transformation (ADAPT), Durham, NC, USA; Department of Population Health Sciences, Duke University, Durham, NC, USA; Division of General Internal Medicine, Duke University, Duke University School of Medicine, Durham, NC, USA
| | - An-Kwok Ian Wong
- Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, Duke University, Durham, NC, USA; Department of Biostatistics and Bioinformatics, Duke University, Division of Translational Biomedical Informatics, Durham, NC, USA.
| |
Collapse
|
3
|
G A, K L N, M S AS. Improving sepsis classification performance with artificial intelligence algorithms: A comprehensive overview of healthcare applications. J Crit Care 2024; 83:154815. [PMID: 38723336 DOI: 10.1016/j.jcrc.2024.154815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2023] [Revised: 04/10/2024] [Accepted: 04/13/2024] [Indexed: 08/04/2024]
Abstract
PURPOSE This study investigates the potential of machine learning (ML) algorithms in improving sepsis diagnosis and prediction, focusing on their relevance in healthcare decision-making. The primary objective is to contribute to healthcare decision-making by evaluating the performance of various supervised and unsupervised models. MATERIALS AND METHODS Through an extensive literature review, optimal ML models used in sepsis research were identified. Diverse datasets from relevant sources were employed, and rigorous evaluation metrics, including accuracy, specificity, and sensitivity, were applied. Innovative techniques were introduced, such as a Stacked Blended Ensemble Model and Skopt Optimization with Blended Ensemble, incorporating Bayesian optimization for hyperparameter tuning. RESULTS ML algorithms demonstrate efficacy in sepsis diagnosis, presenting an improved balance between specificity and sensitivity, critical for effective clinical decision-making. Classifier ensemble models show enhanced accuracy and efficiency, with novel optimization techniques contributing to improved adaptability. CONCLUSION The study emphasizes the potential benefits of ML algorithms in sepsis management, advocating for ongoing research to optimize performance and ensure ethical utilization in healthcare decision-making. Ethical considerations, interpretability, and transparency are crucial factors in implementing these algorithms in clinical practice.
Collapse
Affiliation(s)
- Anjana G
- Department of Electronics and Communication Engineering, Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India
| | - Nisha K L
- Department of Electronics and Communication Engineering, Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India.
| | - Arun Sankar M S
- School of CS&IT, University College Cork, Cork, Ireland; Electronic and Communications Department, South East Technological University, Carlow, Ireland
| |
Collapse
|
4
|
Davies SJ, Sessler DI, Jian Z, Fleming NW, Mythen M, Maheshwari K, Veelo DP, Vlaar APJ, Settels J, Scheeren T, van der Ster BJP, Sander M, Cannesson M, Hatib F. Comparison of Differences in Cohort (Forward) and Case Control (Backward) Methodologic Approaches for Validation of the Hypotension Prediction Index. Anesthesiology 2024; 141:443-452. [PMID: 38557791 PMCID: PMC11323758 DOI: 10.1097/aln.0000000000004989] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Accepted: 03/18/2024] [Indexed: 04/04/2024]
Abstract
BACKGROUND The Hypotension Prediction Index (the index) software is a machine learning algorithm that detects physiologic changes that may lead to hypotension. The original validation used a case control (backward) analysis that has been suggested to be biased. This study therefore conducted a cohort (forward) analysis and compared this to the original validation technique. METHODS A retrospective analysis of data from previously reported studies was conducted. All data were analyzed identically with two different methodologies, and receiver operating characteristic curves were constructed. Both backward and forward analyses were performed to examine differences in area under the receiver operating characteristic curves for the Hypotension Prediction Index and other hemodynamic variables to predict a mean arterial pressure (MAP) less than 65 mmHg for at least 1 min 5, 10, and 15 min in advance. RESULTS The analysis included 2,022 patients, yielding 4,152,124 measurements taken at 20-s intervals. The area under the curve for the index predicting hypotension analyzed by backward and forward methodologies respectively was 0.957 (95% CI, 0.947 to 0.964) versus 0.923 (95% CI, 0.912 to 0.933) 5 min in advance, 0.933 (95% CI, 0.924 to 0.942) versus 0.923 (95% CI, 0.911 to 0.933) 10 min in advance, and 0.929 (95% CI, 0.918 to 0.938) versus 0.926 (95% CI, 0.914 to 0.937) 15 min in advance. No variable other than MAP had an area under the curve greater than 0.7. The areas under the curve using forward analysis for MAP predicting hypotension 5, 10, and 15 min in advance were 0.932 (95% CI, 0.920 to 0.940), 0.929 (95% CI, 0.918 to 0.938), and 0.932 (95% CI, 0.921 to 0.940), respectively. The R2 for the variation in the index due to MAP was 0.77. CONCLUSIONS Using an updated methodology, the study found that the utility of the Hypotension Prediction Index to predict future hypotensive events is high, with an area under the receiver operating characteristics curve similar to that of the original validation method. EDITOR’S PERSPECTIVE
Collapse
Affiliation(s)
- Simon J. Davies
- Department of Anaesthesia, Critical Care and Perioperative Medicine, York and Scarborough Teaching Hospitals National Health Service Foundation Trust, York, United Kingdom; and Centre for Health and Population Science, Hull York Medical School, York, United Kingdom
| | | | | | - Neal W. Fleming
- University of California–Davis School of Medicine, Sacramento, California
| | - Monty Mythen
- Edwards Lifesciences, Irvine, California; and University College London/University College London Hospital, National Institute of Health Research Biomedical Research Centre, London, United Kingdom
| | - Kamal Maheshwari
- Department of Outcomes Research, Cleveland Clinic, Cleveland, Ohio
| | - Denise P. Veelo
- Departments of Anaesthesia and Intensive Care, Amsterdam University Medical Center, Amsterdam, The Netherlands
| | - Alexander P. J. Vlaar
- Departments of Anaesthesia and Intensive Care, Amsterdam University Medical Center, Amsterdam, The Netherlands
| | | | - Thomas Scheeren
- Edwards Lifesciences, Irvine, California; and Department of Anesthesiology, University Medical Centre Groningen, Groningen, The Netherlands
| | - B. J. P. van der Ster
- Departments of Anaesthesia and Intensive Care, Amsterdam University Medical Center, Amsterdam, The Netherlands; and Erasmus Medical Center, Rotterdam, The Netherlands
| | - Michael Sander
- Department of Anaesthesiology, Intensive Care Medicine and Pain Medicine, University Hospital Giessen, Giessen, Germany
| | - Maxime Cannesson
- Department of Anesthesiology and Perioperative Medicine, UCLA, California
| | | |
Collapse
|
5
|
Wong DCW, Bonnici T, Gerry S, Birks J, Watkinson PJ. Effect of Digital Early Warning Scores on Hospital Vital Sign Observation Protocol Adherence: Stepped-Wedge Evaluation. J Med Internet Res 2024; 26:e46691. [PMID: 38900529 PMCID: PMC11224703 DOI: 10.2196/46691] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 11/17/2023] [Accepted: 04/08/2024] [Indexed: 06/21/2024] Open
Abstract
BACKGROUND Early warning scores (EWS) are routinely used in hospitals to assess a patient's risk of deterioration. EWS are traditionally recorded on paper observation charts but are increasingly recorded digitally. In either case, evidence for the clinical effectiveness of such scores is mixed, and previous studies have not considered whether EWS leads to changes in how deteriorating patients are managed. OBJECTIVE This study aims to examine whether the introduction of a digital EWS system was associated with more frequent observation of patients with abnormal vital signs, a precursor to earlier clinical intervention. METHODS We conducted a 2-armed stepped-wedge study from February 2015 to December 2016, over 4 hospitals in 1 UK hospital trust. In the control arm, vital signs were recorded using paper observation charts. In the intervention arm, a digital EWS system was used. The primary outcome measure was time to next observation (TTNO), defined as the time between a patient's first elevated EWS (EWS ≥3) and subsequent observations set. Secondary outcomes were time to death in the hospital, length of stay, and time to unplanned intensive care unit admission. Differences between the 2 arms were analyzed using a mixed-effects Cox model. The usability of the system was assessed using the system usability score survey. RESULTS We included 12,802 admissions, 1084 in the paper (control) arm and 11,718 in the digital EWS (intervention) arm. The system usability score was 77.6, indicating good usability. The median TTNO in the control and intervention arms were 128 (IQR 73-218) minutes and 131 (IQR 73-223) minutes, respectively. The corresponding hazard ratio for TTNO was 0.99 (95% CI 0.91-1.07; P=.73). CONCLUSIONS We demonstrated strong clinical engagement with the system. We found no difference in any of the predefined patient outcomes, suggesting that the introduction of a highly usable electronic system can be achieved without impacting clinical care. Our findings contrast with previous claims that digital EWS systems are associated with improvement in clinical outcomes. Future research should investigate how digital EWS systems can be integrated with new clinical pathways adjusting staff behaviors to improve patient outcomes.
Collapse
Affiliation(s)
- David Chi-Wai Wong
- Leeds Institute of Health Sciences, School of Medicine, University of Leeds, Leeds, United Kingdom
| | - Timothy Bonnici
- Critical Care Division, University College Hospital London NHS Foundation Trust, London, United Kingdom
| | - Stephen Gerry
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Jacqueline Birks
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
| | - Peter J Watkinson
- Oxford University Hospitals NHS Trust, Oxford, United Kingdom
- NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
- Nuffield Department of Clinical Neurosciences, Kadoorie Centre for Critical Care Research and Education, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
6
|
Choudhery MS, Arif T, Mahmood R, Harris DT. CAR-T-Cell-Based Cancer Immunotherapies: Potentials, Limitations, and Future Prospects. J Clin Med 2024; 13:3202. [PMID: 38892913 PMCID: PMC11172642 DOI: 10.3390/jcm13113202] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2024] [Revised: 05/17/2024] [Accepted: 05/23/2024] [Indexed: 06/21/2024] Open
Abstract
Cancer encompasses various elements occurring at the cellular and genetic levels, necessitating an immunotherapy capable of efficiently addressing both aspects. T cells can combat cancer cells by specifically recognizing antigens on them. This innate capability of T cells has been used to develop cellular immunotherapies, but most of them can only target antigens through major histocompatibility complexes (MHCs). New gene-editing techniques such as clustered regularly interspaced short palindromic repeat (CRISPR)-associated protein 9 (CRISPR-cas9) can precisely edit the DNA sequences. CRISPR-cas9 has made it possible to generate genetically engineered chimeric antigen receptors (CARs) that can overcome the problems associated with old immunotherapies. In chimeric antigen receptor T (CAR-T) cell therapy, the patient's T cells are isolated and genetically modified to exhibit synthetic CAR(s). CAR-T cell treatment has shown remarkably positive clinical outcomes in cancers of various types. Nevertheless, there are various challenges that reduce CAR-T effectiveness in solid tumors. It is required to address these challenges in order to make CAR-T cell therapy a better and safer option. Combining CAR-T treatment with other immunotherapies that target multiple antigens has shown positive outcomes. Moreover, recently generated Boolean logic-gated advanced CARs along with artificial intelligence has expanded its potential to treat solid tumors in addition to blood cancers. This review aims to describe the structure, types, and various methods used to develop CAR-T cells. The clinical applications of CAR-T cells in hematological malignancies and solid tumours have been described in detail. In addition, this discussion has addressed the limitations associated with CAR-T cells, explored potential strategies to mitigate CAR-T-related toxicities, and delved into future perspectives.
Collapse
Affiliation(s)
- Mahmood S. Choudhery
- Department of Human Genetics & Molecular Biology, University of Health Sciences, Lahore 54600, Pakistan;
| | - Taqdees Arif
- Department of Human Genetics & Molecular Biology, University of Health Sciences, Lahore 54600, Pakistan;
| | - Ruhma Mahmood
- Jinnah Hospital, Allama Iqbal Medical College, Lahore 54700, Pakistan;
| | - David T. Harris
- Department of Immunobiology, College of Medicine, University of Arizona Health Sciences Biorepository, The University of Arizona, Tucson, AZ 85724-5221, USA;
| |
Collapse
|
7
|
Yang HS. Machine Learning for Sepsis Prediction: Prospects and Challenges. Clin Chem 2024; 70:465-467. [PMID: 38431277 DOI: 10.1093/clinchem/hvae006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Accepted: 01/11/2024] [Indexed: 03/05/2024]
Affiliation(s)
- He S Yang
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, NY, 10065, United States
| |
Collapse
|
8
|
Baddal B, Taner F, Uzun Ozsahin D. Harnessing of Artificial Intelligence for the Diagnosis and Prevention of Hospital-Acquired Infections: A Systematic Review. Diagnostics (Basel) 2024; 14:484. [PMID: 38472956 DOI: 10.3390/diagnostics14050484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2023] [Revised: 01/23/2024] [Accepted: 02/19/2024] [Indexed: 03/14/2024] Open
Abstract
Healthcare-associated infections (HAIs) are the most common adverse events in healthcare and constitute a major global public health concern. Surveillance represents the foundation for the effective prevention and control of HAIs, yet conventional surveillance is costly and labor intensive. Artificial intelligence (AI) and machine learning (ML) have the potential to support the development of HAI surveillance algorithms for the understanding of HAI risk factors, the improvement of patient risk stratification as well as the prediction and timely detection and prevention of infections. AI-supported systems have so far been explored for clinical laboratory testing and imaging diagnosis, antimicrobial resistance profiling, antibiotic discovery and prediction-based clinical decision support tools in terms of HAIs. This review aims to provide a comprehensive summary of the current literature on AI applications in the field of HAIs and discuss the future potentials of this emerging technology in infection practice. Following the PRISMA guidelines, this study examined the articles in databases including PubMed and Scopus until November 2023, which were screened based on the inclusion and exclusion criteria, resulting in 162 included articles. By elucidating the advancements in the field, we aim to highlight the potential applications of AI in the field, report related issues and shortcomings and discuss the future directions.
Collapse
Affiliation(s)
- Buket Baddal
- Department of Medical Microbiology and Clinical Microbiology, Faculty of Medicine, Near East University, North Cyprus, Mersin 10, 99138 Nicosia, Turkey
- DESAM Research Institute, Near East University, North Cyprus, Mersin 10, 99138 Nicosia, Turkey
| | - Ferdiye Taner
- Department of Medical Microbiology and Clinical Microbiology, Faculty of Medicine, Near East University, North Cyprus, Mersin 10, 99138 Nicosia, Turkey
- DESAM Research Institute, Near East University, North Cyprus, Mersin 10, 99138 Nicosia, Turkey
| | - Dilber Uzun Ozsahin
- Department of Medical Diagnostic Imaging, College of Health Science, University of Sharjah, Sharjah 27272, United Arab Emirates
- Research Institute for Medical and Health Sciences, University of Sharjah, Sharjah 27272, United Arab Emirates
- Operational Research Centre in Healthcare, Near East University, North Cyprus, Mersin 10, 99138 Nicosia, Turkey
| |
Collapse
|
9
|
De Backer D, Deutschman CS, Hellman J, Myatra SN, Ostermann M, Prescott HC, Talmor D, Antonelli M, Pontes Azevedo LC, Bauer SR, Kissoon N, Loeches IM, Nunnally M, Tissieres P, Vieillard-Baron A, Coopersmith CM. Surviving Sepsis Campaign Research Priorities 2023. Crit Care Med 2024; 52:268-296. [PMID: 38240508 DOI: 10.1097/ccm.0000000000006135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/23/2024]
Abstract
OBJECTIVES To identify research priorities in the management, epidemiology, outcome, and pathophysiology of sepsis and septic shock. DESIGN Shortly after publication of the most recent Surviving Sepsis Campaign Guidelines, the Surviving Sepsis Research Committee, a multiprofessional group of 16 international experts representing the European Society of Intensive Care Medicine and the Society of Critical Care Medicine, convened virtually and iteratively developed the article and recommendations, which represents an update from the 2018 Surviving Sepsis Campaign Research Priorities. METHODS Each task force member submitted five research questions on any sepsis-related subject. Committee members then independently ranked their top three priorities from the list generated. The highest rated clinical and basic science questions were developed into the current article. RESULTS A total of 81 questions were submitted. After merging similar questions, there were 34 clinical and ten basic science research questions submitted for voting. The five top clinical priorities were as follows: 1) what is the best strategy for screening and identification of patients with sepsis, and can predictive modeling assist in real-time recognition of sepsis? 2) what causes organ injury and dysfunction in sepsis, how should it be defined, and how can it be detected? 3) how should fluid resuscitation be individualized initially and beyond? 4) what is the best vasopressor approach for treating the different phases of septic shock? and 5) can a personalized/precision medicine approach identify optimal therapies to improve patient outcomes? The five top basic science priorities were as follows: 1) How can we improve animal models so that they more closely resemble sepsis in humans? 2) What outcome variables maximize correlations between human sepsis and animal models and are therefore most appropriate to use in both? 3) How does sepsis affect the brain, and how do sepsis-induced brain alterations contribute to organ dysfunction? How does sepsis affect interactions between neural, endocrine, and immune systems? 4) How does the microbiome affect sepsis pathobiology? 5) How do genetics and epigenetics influence the development of sepsis, the course of sepsis and the response to treatments for sepsis? CONCLUSIONS Knowledge advances in multiple clinical domains have been incorporated in progressive iterations of the Surviving Sepsis Campaign guidelines, allowing for evidence-based recommendations for short- and long-term management of sepsis. However, the strength of existing evidence is modest with significant knowledge gaps and mortality from sepsis remains high. The priorities identified represent a roadmap for research in sepsis and septic shock.
Collapse
Affiliation(s)
- Daniel De Backer
- Department of Intensive Care, CHIREC Hospitals, Université Libre de Bruxelles, Brussels, Belgium
| | - Clifford S Deutschman
- Department of Pediatrics, Cohen Children's Medical Center, Northwell Health, New Hyde Park, NY
- Sepsis Research Lab, the Feinstein Institutes for Medical Research, Manhasset, NY
| | - Judith Hellman
- Department of Anesthesia and Perioperative Care, University of California, San Francisco, CA
| | - Sheila Nainan Myatra
- Department of Anaesthesiology, Critical Care and Pain, Tata Memorial Hospital, Homi Bhabha National Institute, Mumbai, India
| | - Marlies Ostermann
- Department of Critical Care, King's College London, Guy's & St Thomas' Hospital, London, United Kingdom
| | - Hallie C Prescott
- Department of Internal Medicine, University of Michigan, Ann Arbor, MI
| | - Daniel Talmor
- Department of Anesthesia, Critical Care and Pain Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Massimo Antonelli
- Department of Intensive Care, Emergency Medicine and Anesthesiology, Fondazione Policlinico Universitario A.Gemelli IRCCS, Rome, Italy
- Istituto di Anestesiologia e Rianimazione, Università Cattolica del Sacro Cuore, Rome, Italy
| | | | - Seth R Bauer
- Department of Pharmacy, Cleveland Clinic, Cleveland, OH
| | - Niranjan Kissoon
- Department of Pediatrics, University of British Columbia, Vancouver, BC, Canada
| | - Ignacio-Martin Loeches
- Department of Intensive Care Medicine, Multidisciplinary Intensive Care Research Organization (MICRO), St James's Hospital, Leinster, Dublin, Ireland
| | | | - Pierre Tissieres
- Pediatric Intensive Care, Neonatal Medicine and Pediatric Emergency, AP-HP Paris Saclay University, Bicêtre Hospital, Le Kremlin-Bicêtre, France
| | - Antoine Vieillard-Baron
- Service de Medecine Intensive Reanimation, Hopital Ambroise Pare, Universite Paris-Saclay, Le Kremlin-Bicêtre, France
| | | |
Collapse
|
10
|
Foote HP, Shaikh Z, Witt D, Shen T, Ratliff W, Shi H, Gao M, Nichols M, Sendak M, Balu S, Osborne K, Kumar KR, Jackson K, McCrary AW, Li JS. Development and Temporal Validation of a Machine Learning Model to Predict Clinical Deterioration. Hosp Pediatr 2024; 14:11-20. [PMID: 38053467 PMCID: PMC11293885 DOI: 10.1542/hpeds.2023-007308] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
OBJECTIVES Early warning scores detecting clinical deterioration in pediatric inpatients have wide-ranging performance and use a limited number of clinical features. This study developed a machine learning model leveraging multiple static and dynamic clinical features from the electronic health record to predict the composite outcome of unplanned transfer to the ICU within 24 hours and inpatient mortality within 48 hours in hospitalized children. METHODS Using a retrospective development cohort of 17 630 encounters across 10 388 patients, 2 machine learning models (light gradient boosting machine [LGBM] and random forest) were trained on 542 features and compared with our institutional Pediatric Early Warning Score (I-PEWS). RESULTS The LGBM model significantly outperformed I-PEWS based on receiver operating characteristic curve (AUROC) for the composite outcome of ICU transfer or mortality for both internal validation and temporal validation cohorts (AUROC 0.785 95% confidence interval [0.780-0.791] vs 0.708 [0.701-0.715] for temporal validation) as well as lead-time before deterioration events (median 11 hours vs 3 hours; P = .004). However, LGBM performance as evaluated by precision recall curve was lesser in the temporal validation cohort with associated decreased positive predictive value (6% vs 29%) and increased number needed to evaluate (17 vs 3) compared with I-PEWS. CONCLUSIONS Our electronic health record based machine learning model demonstrated improved AUROC and lead-time in predicting clinical deterioration in pediatric inpatients 24 to 48 hours in advance compared with I-PEWS. Further work is needed to optimize model positive predictive value to allow for integration into clinical practice.
Collapse
Affiliation(s)
| | - Zohaib Shaikh
- Duke Institute for Health Innovation
- Department of Medicine, Weill Cornell Medical Center
| | - Daniel Witt
- Duke Institute for Health Innovation
- Mayo Clinic Alix School of Medicine
| | - Tong Shen
- Department of Biomedical Engineering, Duke University
| | | | | | | | | | | | | | | | - Karan R. Kumar
- Division of Pediatric Critical Care Medicine, Duke University
| | | | | | | |
Collapse
|
11
|
Feng X, Zhu S, Shen Y, Zhu H, Yan M, Cai G, Ning G. Multi-organ spatiotemporal information aware model for sepsis mortality prediction. Artif Intell Med 2024; 147:102746. [PMID: 38184353 DOI: 10.1016/j.artmed.2023.102746] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 12/05/2023] [Accepted: 12/05/2023] [Indexed: 01/08/2024]
Abstract
BACKGROUND Sepsis is a syndrome involving multi-organ dysfunction, and the mortality in sepsis patients correlates with the number of lesioned organs. Precise prognosis models play a pivotal role in enabling healthcare practitioners to administer timely and accurate interventions for sepsis, thereby augmenting patient outcomes. Nevertheless, the majority of available models consider the overall physiological attributes of patients, overlooking the asynchronous spatiotemporal interactions among multiple organ systems. These constraints hinder a full application of such models, particularly when dealing with limited clinical data. To surmount these challenges, a comprehensive model, denoted as recurrent Graph Attention Network-multi Gated Recurrent Unit (rGAT-mGRU), was proposed. Taking into account the intricate spatiotemporal interactions among multiple organ systems, the model predicted in-hospital mortality of sepsis using data collected within the 48-hour period post-diagnosis. MATERIAL AND METHODS Multiple parallel GRU sub-models were formulated to investigate the temporal physiological variations of single organ systems. Meanwhile, a GAT structure featuring a memory unit was constructed to capture spatiotemporal connections among multi-organ systems. Additionally, an attention-injection mechanism was employed to govern the data flowing within the network pertaining to multi-organ systems. The proposed model underwent training and testing using a dataset of 10,181 sepsis cases extracted from the Medical Information Mart for Intensive Care III (MIMIC-III) database. To evaluate the model's superiority, it was compared with the existing common baseline models. Furthermore, ablation experiments were designed to elucidate the rationale and robustness of the proposed model. RESULTS Compared with the baseline models for predicting mortality of sepsis, the rGAT-mGRU model demonstrated the largest area under the receiver operating characteristic curve (AUROC) of 0.8777 ± 0.0039 and the maximum area under the precision-recall curve (AUPRC) of 0.5818 ± 0.0071, with sensitivity of 0.8358 ± 0.0302 and specificity of 0.7727 ± 0.0229, respectively. The proposed model was capable of delineating the varying contribution of the involved organ systems at distinct moments, as specifically illustrated by the attention weights. Furthermore, it exhibited consistent performance even in the face of limited clinical data. CONCLUSION The rGAT-mGRU model has the potential to indicate sepsis prognosis by extracting the dynamic spatiotemporal interplay information inherent in multi-organ systems during critical diseases, thereby providing clinicians with auxiliary decision-making support.
Collapse
Affiliation(s)
- Xue Feng
- Department of Biomedical Engineering, Zhejiang University, Hangzhou 310027, China
| | - Siyi Zhu
- Department of Biomedical Engineering, Zhejiang University, Hangzhou 310027, China
| | - Yanfei Shen
- Intensive Care Unit, Zhejiang Hospital, Hangzhou 310013, China
| | - Huaiping Zhu
- Department of Mathematics and Statistics, York University, Toronto M3J1P3, Canada
| | - Molei Yan
- Intensive Care Unit, Zhejiang Hospital, Hangzhou 310013, China
| | - Guolong Cai
- Intensive Care Unit, Zhejiang Hospital, Hangzhou 310013, China.
| | - Gangmin Ning
- Department of Biomedical Engineering, Zhejiang University, Hangzhou 310027, China; Research Center for Healthcare Data Science, Zhejiang Lab, Hangzhou 311121, China.
| |
Collapse
|
12
|
Rashid A, Al-Obeida F, Hafez W, Benakatti G, Malik RA, Koutentis C, Sharief J, Brierley J, Quraishi N, Malik ZA, Anwary A, Alkhzaimi H, Zaki SA, Khilnani P, Kadwa R, Phatak R, Schumacher M, Shaikh G, Al-Dubai A, Hussain A. ADVANCING THE UNDERSTANDING OF CLINICAL SEPSIS USING GENE EXPRESSION-DRIVEN MACHINE LEARNING TO IMPROVE PATIENT OUTCOMES. Shock 2024; 61:4-18. [PMID: 37752080 DOI: 10.1097/shk.0000000000002227] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/28/2023]
Abstract
ABSTRACT Sepsis remains a major challenge that necessitates improved approaches to enhance patient outcomes. This study explored the potential of machine learning (ML) techniques to bridge the gap between clinical data and gene expression information to better predict and understand sepsis. We discuss the application of ML algorithms, including neural networks, deep learning, and ensemble methods, to address key evidence gaps and overcome the challenges in sepsis research. The lack of a clear definition of sepsis is highlighted as a major hurdle, but ML models offer a workaround by focusing on endpoint prediction. We emphasize the significance of gene transcript information and its use in ML models to provide insights into sepsis pathophysiology and biomarker identification. Temporal analysis and integration of gene expression data further enhance the accuracy and predictive capabilities of ML models for sepsis. Although challenges such as interpretability and bias exist, ML research offers exciting prospects for addressing critical clinical problems, improving sepsis management, and advancing precision medicine approaches. Collaborative efforts between clinicians and data scientists are essential for the successful implementation and translation of ML models into clinical practice. Machine learning has the potential to revolutionize our understanding of sepsis and significantly improve patient outcomes. Further research and collaboration between clinicians and data scientists are needed to fully understand the potential of ML in sepsis management.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Joe Brierley
- Great Ormond Street Children's Hospital, London, UK
| | - Nasir Quraishi
- Centre for Spinal Studies & Surgery, Queen's Medical Centre. The University of Nottingham. Nottingham, UK
| | - Zainab A Malik
- College of Medicine, Mohammed Bin Rashid University of Medicine and Health Sciences. Dubai, U.A.E
| | - Arif Anwary
- School of Computing, Edinburgh Napier University. Edinburgh, UK
| | | | | | | | | | - Rajesh Phatak
- Pediatric Intensive Care, Burjeel Hospital, Najda, Abu Dhabi
| | | | - Guftar Shaikh
- Endocrinology, Royal Hospital for Children. Glasgow, UK
| | - Ahmed Al-Dubai
- School of Computing, Edinburgh Napier University. Edinburgh, UK
| | - Amir Hussain
- School of Computing, Edinburgh Napier University. Edinburgh, UK
| |
Collapse
|
13
|
Davis SE, Matheny ME, Balu S, Sendak MP. A framework for understanding label leakage in machine learning for health care. J Am Med Inform Assoc 2023; 31:274-280. [PMID: 37669138 PMCID: PMC10746313 DOI: 10.1093/jamia/ocad178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Revised: 07/24/2023] [Accepted: 08/19/2023] [Indexed: 09/07/2023] Open
Abstract
INTRODUCTION The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule." FRAMEWORK We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice. RECOMMENDATIONS Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.
Collapse
Affiliation(s)
- Sharon E Davis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Tennessee Valley Healthcare System VA Medical Center, Veterans Health Administration, Nashville, TN 37232, United States
| | - Suresh Balu
- Duke Institute for Health Innovation, Duke University School of Medicine, Durham, NC 27701, United States
| | - Mark P Sendak
- Duke Institute for Health Innovation, Duke University School of Medicine, Durham, NC 27701, United States
| |
Collapse
|
14
|
Patton MJ, Liu VX. Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges. Crit Care Clin 2023; 39:647-673. [PMID: 37704332 DOI: 10.1016/j.ccc.2023.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2023]
Abstract
The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.
Collapse
Affiliation(s)
- Michael J Patton
- Medical Scientist Training Program, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA; Hugh Kaul Precision Medicine Institute at the University of Alabama at Birmingham, 720 20th Street South, Suite 202, Birmingham, Alabama, 35233, USA.
| | - Vincent X Liu
- Kaiser Permanente Division of Research, Oakland, CA, USA.
| |
Collapse
|
15
|
Bäckel N, Hort S, Kis T, Nettleton DF, Egan JR, Jacobs JJL, Grunert D, Schmitt RH. Elaborating the potential of Artificial Intelligence in automated CAR-T cell manufacturing. FRONTIERS IN MOLECULAR MEDICINE 2023; 3:1250508. [PMID: 39086671 PMCID: PMC11285580 DOI: 10.3389/fmmed.2023.1250508] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Accepted: 08/28/2023] [Indexed: 08/02/2024]
Abstract
This paper discusses the challenges of producing CAR-T cells for cancer treatment and the potential for Artificial Intelligence (AI) for its improvement. CAR-T cell therapy was approved in 2018 as the first Advanced Therapy Medicinal Product (ATMP) for treating acute leukemia and lymphoma. ATMPs are cell- and gene-based therapies that show great promise for treating various cancers and hereditary diseases. While some new ATMPs have been approved, ongoing clinical trials are expected to lead to the approval of many more. However, the production of CAR-T cells presents a significant challenge due to the high costs associated with the manufacturing process, making the therapy very expensive (approx. $400,000). Furthermore, autologous CAR-T therapy is limited to a make-to-order approach, which makes scaling economical production difficult. First attempts are being made to automate this multi-step manufacturing process, which will not only directly reduce the high manufacturing costs but will also enable comprehensive data collection. AI technologies have the ability to analyze this data and convert it into knowledge and insights. In order to exploit these opportunities, this paper analyses the data potential in the automated CAR-T production process and creates a mapping to the capabilities of AI applications. The paper explores the possible use of AI in analyzing the data generated during the automated process and its capabilities to further improve the efficiency and cost-effectiveness of CAR-T cell production.
Collapse
Affiliation(s)
- Niklas Bäckel
- Fraunhofer Institute for Production Technology IPT, Aachen, Germany
| | - Simon Hort
- Fraunhofer Institute for Production Technology IPT, Aachen, Germany
| | - Tamás Kis
- Institute for Computer Science and Control, Hungarian Research Network, Budapest, Hungary
| | | | - Joseph R. Egan
- Department of Biochemical Engineering, Mathematical Modelling of Cell and Gene Therapies, University College London, London, United Kingdom
| | | | - Dennis Grunert
- Fraunhofer Institute for Production Technology IPT, Aachen, Germany
| | - Robert H. Schmitt
- Fraunhofer Institute for Production Technology IPT, Aachen, Germany
- Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University, Aachen, Germany
| |
Collapse
|
16
|
Ferreira LD, McCants D, Velamuri S. Using machine learning for process improvement in sepsis management. J Healthc Qual Res 2023; 38:304-311. [PMID: 36319584 DOI: 10.1016/j.jhqr.2022.09.006] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Revised: 08/18/2022] [Accepted: 09/26/2022] [Indexed: 06/16/2023]
Abstract
INTRODUCTION In the U.S., sepsis afflicts 1.7 million adults, causing 270,000 deaths each year. Early detection of sepsis could decrease the number of deaths by 92,000 annually and decrease hospital expenditures by 1.5 billion USD. Few prior studies and reviews have presented a holistic understanding of the relationship between machine learning and existing process improvement measures. This study, in addition to discussing machine learning and existing process improvements measures, elaborates on the disadvantages and the barriers to integrating machine learning into the clinic. This article synthesizes previous studies to educate healthcare professionals on effectively managing sepsis by leveraging the benefits of machine learning. METHODS This study used the PubMed database. Search terms include sepsis antibiotics, sepsis process improvement, sepsis machine learning. Our search criteria included previous studies published between January 1, 2017, and February 1, 2022. RESULTS/DISCUSSION Although machine learning algorithms have better predictive capabilities, their effectiveness in the clinical setting is limited as studies show mixed results because the medical staff often fails to intervene. To overcome poor interventional response, clinicians need to work with the facility's IT department to ensure integration into clinical workflow and minimize alert-fatigue. Algorithms should enhance the productivity of clinical teams, not attempt to replace them entirely. CONCLUSION Hospitals can employ process improvement measures that effectively utilize machine learning algorithms to ensure integration into clinical workflows. Healthcare professionals can utilize workflow tools in addition to the predictive capabilities of machine learning to enhance clinical decisions in sepsis.
Collapse
Affiliation(s)
- L D Ferreira
- Department of Student Affairs, Baylor College of Medicine, United States.
| | - D McCants
- Department of Internal Medicine, Baylor College of Medicine, United States
| | - S Velamuri
- Department of Internal Medicine, Baylor College of Medicine, United States; Luminare, Inc. United States
| |
Collapse
|
17
|
Jazayeri A, Yang CC, Capan M. Frequent temporal patterns of physiological and biological biomarkers and their evolution in sepsis. Artif Intell Med 2023; 143:102576. [PMID: 37673556 PMCID: PMC10483113 DOI: 10.1016/j.artmed.2023.102576] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2021] [Revised: 02/01/2023] [Accepted: 05/09/2023] [Indexed: 09/08/2023]
Abstract
Sepsis is one of the most challenging health conditions worldwide, with relatively high incidence and mortality rates. It is shown that preventing sepsis is the key to avoid potentially irreversible organ dysfunction. However, data-driven early identification of sepsis is challenging as sepsis shares signs and symptoms with other health conditions. This paper adopts a temporal pattern mining approach to identify frequent temporal and evolving patterns of physiological and biological biomarkers in sepsis patients. We show that using these frequent patterns as features for classifying sepsis and non-sepsis patients can improve the prediction accuracy and performance up to 7%. Most of the temporal modeling approaches adopted in the sepsis literature are based on deep learning methods. Although these approaches produce high accuracy, they generally have limited model explainability and interpretability. Using the adopted methods in this study, we could identify the most important features contributing to the patients' sepsis incidence, such as fluctuations in platelet, lactate, and creatinine, or evolution of patterns including renal and metabolic organ systems, and consequently, enhance the findings' clinical interpretability.
Collapse
Affiliation(s)
- Ali Jazayeri
- College of Computing & Informatics, Drexel University, Philadelphia, PA, United States of America.
| | - Christopher C Yang
- College of Computing & Informatics, Drexel University, Philadelphia, PA, United States of America
| | - Muge Capan
- Mechanical and Industrial Engineering, College of Engineering, University of Massachusetts Amherst, Amherst, MA, United States of America
| |
Collapse
|
18
|
Pungitore S, Subbian V. Assessment of Prediction Tasks and Time Window Selection in Temporal Modeling of Electronic Health Record Data: a Systematic Review. JOURNAL OF HEALTHCARE INFORMATICS RESEARCH 2023; 7:313-331. [PMID: 37637723 PMCID: PMC10449760 DOI: 10.1007/s41666-023-00143-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2022] [Revised: 04/12/2023] [Accepted: 07/28/2023] [Indexed: 08/29/2023]
Abstract
Temporal electronic health record (EHR) data are often preferred for clinical prediction tasks because they offer more complete representations of a patient's pathophysiology than static data. A challenge when working with temporal EHR data is problem formulation, which includes defining the time windows of interest and the prediction task. Our objective was to conduct a systematic review that assessed the definition and reporting of concepts relevant to temporal clinical prediction tasks. We searched PubMed® and IEEE Xplore® databases for studies from January 1, 2010 applying machine learning models to EHR data for patient outcome prediction. Publications applying time-series methods were selected for further review. We identified 92 studies and summarized them by clinical context and definition and reporting of the prediction problem. For the time windows of interest, 12 studies did not discuss window lengths, 57 used a single set of window lengths, and 23 evaluated the relationship between window length and model performance. We also found that 72 studies had appropriate reporting of the prediction task. However, evaluation of prediction problem formulation for temporal EHR data was complicated by heterogeneity in assessing and reporting of these concepts. Even among studies modeling similar clinical outcomes, there were variations in terminology used to describe the prediction problem, rationale for window lengths, and determination of the outcome of interest. As temporal modeling using EHR data expands, minimal reporting standards should include time-series specific concerns to promote rigor and reproducibility in future studies and facilitate model implementation in clinical settings. Supplementary Information The online version contains supplementary material available at 10.1007/s41666-023-00143-4.
Collapse
Affiliation(s)
- Sarah Pungitore
- Program in Applied Mathematics, Department of Mathematics, 617 N Santa Rita Ave, Tucson, AZ 85721 USA
| | - Vignesh Subbian
- Department of Biomedical Engineering, The University of Arizona, Tucson, AZ 85721-0020 USA
- Department of Systems and Industrial Engineering, The University of Arizona, Tucson, AZ 85721-0020 USA
| |
Collapse
|
19
|
Islam KR, Prithula J, Kumar J, Tan TL, Reaz MBI, Sumon MSI, Chowdhury MEH. Machine Learning-Based Early Prediction of Sepsis Using Electronic Health Records: A Systematic Review. J Clin Med 2023; 12:5658. [PMID: 37685724 PMCID: PMC10488449 DOI: 10.3390/jcm12175658] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 08/13/2023] [Accepted: 08/23/2023] [Indexed: 09/10/2023] Open
Abstract
BACKGROUND Sepsis, a life-threatening infection-induced inflammatory condition, has significant global health impacts. Timely detection is crucial for improving patient outcomes as sepsis can rapidly progress to severe forms. The application of machine learning (ML) and deep learning (DL) to predict sepsis using electronic health records (EHRs) has gained considerable attention for timely intervention. METHODS PubMed, IEEE Xplore, Google Scholar, and Scopus were searched for relevant studies. All studies that used ML/DL to detect or early-predict the onset of sepsis in the adult population using EHRs were considered. Data were extracted and analyzed from all studies that met the criteria and were also evaluated for their quality. RESULTS This systematic review examined 1942 articles, selecting 42 studies while adhering to strict criteria. The chosen studies were predominantly retrospective (n = 38) and spanned diverse geographic settings, with a focus on the United States. Different datasets, sepsis definitions, and prevalence rates were employed, necessitating data augmentation. Heterogeneous parameter utilization, diverse model distribution, and varying quality assessments were observed. Longitudinal data enabled early sepsis prediction, and quality criteria fulfillment varied, with inconsistent funding-article quality correlation. CONCLUSIONS This systematic review underscores the significance of ML/DL methods for sepsis detection and early prediction through EHR data.
Collapse
Affiliation(s)
- Khandaker Reajul Islam
- Department of Physiology, Faculty of Medicine, University Kebangsaan Malaysia, Kuala Lumpur 56000, Malaysia
| | - Johayra Prithula
- Department of Electrical and Electronics Engineering, University of Dhaka, Dhaka 1000, Bangladesh
| | - Jaya Kumar
- Department of Physiology, Faculty of Medicine, University Kebangsaan Malaysia, Kuala Lumpur 56000, Malaysia
| | - Toh Leong Tan
- Department of Emergency Medicine, Faculty of Medicine, Universiti Kebangsaan Malaysia, Kuala Lumpur 56000, Malaysia
| | - Mamun Bin Ibne Reaz
- Department of Electrical and Electronic Engineering, Independent University, Bangladesh Bashundhara, Dhaka 1229, Bangladesh
| | - Md. Shaheenur Islam Sumon
- Department of Biomedical Engineering, Military Institute of Science and Technology (MIST), Dhaka 1216, Bangladesh
| | | |
Collapse
|
20
|
Valik JK, Ward L, Tanushi H, Johansson AF, Färnert A, Mogensen ML, Pickering BW, Herasevich V, Dalianis H, Henriksson A, Nauclér P. Predicting sepsis onset using a machine learned causal probabilistic network algorithm based on electronic health records data. Sci Rep 2023; 13:11760. [PMID: 37474597 PMCID: PMC10359402 DOI: 10.1038/s41598-023-38858-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Accepted: 07/16/2023] [Indexed: 07/22/2023] Open
Abstract
Sepsis is a leading cause of mortality and early identification improves survival. With increasing digitalization of health care data automated sepsis prediction models hold promise to aid in prompt recognition. Most previous studies have focused on the intensive care unit (ICU) setting. Yet only a small proportion of sepsis develops in the ICU and there is an apparent clinical benefit to identify patients earlier in the disease trajectory. In this cohort of 82,852 hospital admissions and 8038 sepsis episodes classified according to the Sepsis-3 criteria, we demonstrate that a machine learned score can predict sepsis onset within 48 h using sparse routine electronic health record data outside the ICU. Our score was based on a causal probabilistic network model-SepsisFinder-which has similarities with clinical reasoning. A prediction was generated hourly on all admissions, providing a new variable was registered. Compared to the National Early Warning Score (NEWS2), which is an established method to identify sepsis, the SepsisFinder triggered earlier and had a higher area under receiver operating characteristic curve (AUROC) (0.950 vs. 0.872), as well as area under precision-recall curve (APR) (0.189 vs. 0.149). A machine learning comparator based on a gradient-boosting decision tree model had similar AUROC (0.949) and higher APR (0.239) than SepsisFinder but triggered later than both NEWS2 and SepsisFinder. The precision of SepsisFinder increased if screening was restricted to the earlier admission period and in episodes with bloodstream infection. Furthermore, the SepsisFinder signaled median 5.5 h prior to antibiotic administration. Identifying a high-risk population with this method could be used to tailor clinical interventions and improve patient care.
Collapse
Affiliation(s)
- John Karlsson Valik
- Division of Infectious Diseases, Department of Medicine, Karolinska Institutet, Solna, Stockholm, Sweden.
- Department of Infectious Diseases, Karolinska University Hospital, Stockholm, Sweden.
| | - Logan Ward
- Treat Systems ApS, Aalborg, Denmark
- Department of Health Science and Technology, Center for Model-Based Medical Decision Support, Aalborg University, Aalborg, Denmark
| | - Hideyuki Tanushi
- Division of Infectious Diseases, Department of Medicine, Karolinska Institutet, Solna, Stockholm, Sweden
| | - Anders F Johansson
- Department of Clinical Microbiology and the Laboratory for Molecular Infection Medicine (MIMS), Umeå University, Umeå, Sweden
| | - Anna Färnert
- Division of Infectious Diseases, Department of Medicine, Karolinska Institutet, Solna, Stockholm, Sweden
- Department of Infectious Diseases, Karolinska University Hospital, Stockholm, Sweden
| | | | - Brian W Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Vitaly Herasevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Hercules Dalianis
- Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden
| | - Aron Henriksson
- Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden
| | - Pontus Nauclér
- Division of Infectious Diseases, Department of Medicine, Karolinska Institutet, Solna, Stockholm, Sweden
- Department of Infectious Diseases, Karolinska University Hospital, Stockholm, Sweden
| |
Collapse
|
21
|
Okeibunor JC, Jaca A, Iwu-Jaja CJ, Idemili-Aronu N, Ba H, Zantsi ZP, Ndlambe AM, Mavundza E, Muneene D, Wiysonge CS, Makubalo L. The use of artificial intelligence for delivery of essential health services across WHO regions: a scoping review. Front Public Health 2023; 11:1102185. [PMID: 37469694 PMCID: PMC10352788 DOI: 10.3389/fpubh.2023.1102185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 06/19/2023] [Indexed: 07/21/2023] Open
Abstract
Background Artificial intelligence (AI) is a broad outlet of computer science aimed at constructing machines capable of simulating and performing tasks usually done by human beings. The aim of this scoping review is to map existing evidence on the use of AI in the delivery of medical care. Methods We searched PubMed and Scopus in March 2022, screened identified records for eligibility, assessed full texts of potentially eligible publications, and extracted data from included studies in duplicate, resolving differences through discussion, arbitration, and consensus. We then conducted a narrative synthesis of extracted data. Results Several AI methods have been used to detect, diagnose, classify, manage, treat, and monitor the prognosis of various health issues. These AI models have been used in various health conditions, including communicable diseases, non-communicable diseases, and mental health. Conclusions Presently available evidence shows that AI models, predominantly deep learning, and machine learning, can significantly advance medical care delivery regarding the detection, diagnosis, management, and monitoring the prognosis of different illnesses.
Collapse
Affiliation(s)
| | - Anelisa Jaca
- Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
| | | | - Ngozi Idemili-Aronu
- Department of Sociology/Anthropology, University of Nigeria, Nsukka, Nigeria
| | - Housseynou Ba
- World Health Organization Regional Office for Africa, Brazzaville, Republic of Congo
| | - Zukiswa Pamela Zantsi
- Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
| | - Asiphe Mavis Ndlambe
- Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
| | - Edison Mavundza
- World Health Organization Regional Office for Africa, Brazzaville, Republic of Congo
| | | | - Charles Shey Wiysonge
- Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa
- HIV and Other Infectious Diseases Research Unit, South African Medical Research Council, Durban, South Africa
| | - Lindiwe Makubalo
- World Health Organization Regional Office for Africa, Brazzaville, Republic of Congo
| |
Collapse
|
22
|
Kaya U, Yılmaz A, Aşar S. Sepsis Prediction by Using a Hybrid Metaheuristic Algorithm: A Novel Approach for Optimizing Deep Neural Networks. Diagnostics (Basel) 2023; 13:2023. [PMID: 37370918 DOI: 10.3390/diagnostics13122023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 05/11/2023] [Accepted: 05/21/2023] [Indexed: 06/29/2023] Open
Abstract
The early diagnosis of sepsis reduces the risk of the patient's death. Gradient-based algorithms are applied to the neural network models used in the estimation of sepsis in the literature. However, these algorithms become stuck at the local minimum in solution space. In recent years, swarm intelligence and an evolutionary approach have shown proper results. In this study, a novel hybrid metaheuristic algorithm was proposed for optimization with regard to the weights of the deep neural network and applied for the early diagnosis of sepsis. The proposed algorithm aims to reach the global minimum with a local search strategy capable of exploring and exploiting particles in Particle Swarm Optimization (PSO) and using the mental search operator of the Human Mental Search algorithm (HMS). The benchmark functions utilized to compare the performance of HMS, PSO, and HMS-PSO revealed that the proposed approach is more reliable, durable, and adjustable than other applied algorithms. HMS-PSO is integrated with a deep neural network (HMS-PSO-DNN). The study focused on predicting sepsis with HMS-PSO-DNN, utilizing a dataset of 640 patients aged 18 to 60. The HMS-PSO-DNN model gave a better mean squared error (MSE) result than other algorithms in terms of accuracy, robustness, and performance. We obtained the MSE value of 0.22 with 30 independent runs.
Collapse
Affiliation(s)
- Umut Kaya
- Faculty of Engineering and Architecture, Department of Software Engineering, İstanbul Beykent University, Istanbul 34398, Turkey
| | - Atınç Yılmaz
- Faculty of Engineering and Architecture, Department of Computer Engineering, İstanbul Beykent University, Istanbul 34398, Turkey
| | - Sinan Aşar
- Intensive Care Unit, Bakirkoy Dr. Sadi Konuk Training and Research Hospital, Istanbul 34147, Turkey
| |
Collapse
|
23
|
van der Vegt AH, Scott IA, Dermawan K, Schnetler RJ, Kalke VR, Lane PJ. Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework. J Am Med Inform Assoc 2023:7161075. [PMID: 37172264 DOI: 10.1093/jamia/ocad075] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 04/04/2023] [Accepted: 04/23/2023] [Indexed: 05/14/2023] Open
Abstract
OBJECTIVE To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework. MATERIALS AND METHODS Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework. RESULTS The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework. DISCUSSION Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process. CONCLUSIONS A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.
Collapse
Affiliation(s)
- Anton H van der Vegt
- Queensland Digital Health Centre, The University of Queensland, Brisbane, Queensland, Australia
| | - Ian A Scott
- Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, Australia
| | - Krishna Dermawan
- Centre for Information Resilience, The University of Queensland, St Lucia, Australia
| | - Rudolf J Schnetler
- School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, Australia
| | - Vikrant R Kalke
- Patient Safety and Quality, Clinical Excellence Queensland, Queensland Health, Brisbane, Australia
| | - Paul J Lane
- Safety Quality & Innovation, The Prince Charles Hospital, Queensland Health, Brisbane, Australia
| |
Collapse
|
24
|
Sandhu S, Sendak MP, Ratliff W, Knechtle W, Fulkerson WJ, Balu S. Accelerating health system innovation: principles and practices from the Duke Institute for Health Innovation. PATTERNS (NEW YORK, N.Y.) 2023; 4:100710. [PMID: 37123436 PMCID: PMC10140606 DOI: 10.1016/j.patter.2023.100710] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
The Duke Institute for Health Innovation (DIHI) was launched in 2013. Frontline staff members submit proposals for innovation projects that align with strategic priorities set by organizational leadership. Funded projects receive operational and technical support from institute staff members and a transdisciplinary network of collaborators to develop and implement solutions as part of routine clinical care, ranging from machine learning algorithms to mobile applications. DIHI's operations are shaped by four guiding principles: build to show value, build to integrate, build to scale, and build responsibly. Between 2013 and 2021, more than 600 project proposals have been submitted to DIHI. More than 85 innovation projects, both through the application process and other strategic partnerships, have been supported and implemented. DIHI's funding has incubated 12 companies, engaged more than 300 faculty members, staff members, and students, and contributed to more than 50 peer-reviewed publications. DIHI's practices can serve as a model for other health systems to systematically source, develop, implement, and scale innovations.
Collapse
Affiliation(s)
- Sahil Sandhu
- Duke Institute for Health Innovation, Durham, NC, USA
- Harvard Medical School, Boston, MA, USA
| | | | | | | | - William J. Fulkerson
- Duke University School of Medicine, Durham, NC, USA
- Duke University Health System, Durham, NC, USA
| | - Suresh Balu
- Duke Institute for Health Innovation, Durham, NC, USA
- Duke University School of Medicine, Durham, NC, USA
- Corresponding author
| |
Collapse
|
25
|
Gholamzadeh M, Abtahi H, Safdari R. Comparison of different machine learning algorithms to classify patients suspected of having sepsis infection in the intensive care unit. INFORMATICS IN MEDICINE UNLOCKED 2023. [DOI: 10.1016/j.imu.2023.101236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/31/2023] Open
|
26
|
Ossa LA, Rost M, Lorenzini G, Shaw DM, Elger BS. A smarter perspective: Learning with and from AI-cases. Artif Intell Med 2023; 135:102458. [PMID: 36628794 DOI: 10.1016/j.artmed.2022.102458] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Revised: 09/16/2022] [Accepted: 11/18/2022] [Indexed: 11/24/2022]
Abstract
Artificial intelligence (AI) has only partially (or not at all) been integrated into medical education, leading to growing concerns regarding how to train healthcare practitioners to handle the changes brought about by the introduction of AI. Programming lessons and other technical information into healthcare curricula has been proposed as a solution to support healthcare personnel in using AI or other future technology. However, integrating these core elements of computer science knowledge might not meet the observed need that students will benefit from gaining practical experience with AI in the direct application area. Therefore, this paper proposes a dynamic approach to case-based learning that utilizes the scenarios where AI is currently used in clinical practice as examples. This approach will support students' understanding of technical aspects. Case-based learning with AI as an example provides additional benefits: (1) it allows doctors to compare their thought processes to the AI suggestions and critically reflect on the assumptions and biases of AI and clinical practice; (2) it incentivizes doctors to discuss and address ethical issues inherent to technology and those already existing in current clinical practice; (3) it serves as a foundation for fostering interdisciplinary collaboration via discussion of different views between technologists, multidisciplinary experts, and healthcare professionals. The proposed knowledge shift from AI as a technical focus to AI as an example for case-based learning aims to encourage a different perspective on educational needs. Technical education does not need to compete with other essential clinical skills as it could serve as a basis for supporting them, which leads to better medical education and practice, ultimately benefiting patients.
Collapse
Affiliation(s)
| | - Michael Rost
- Institute for Biomedical Ethics, University of Basel, Basel, Switzerland
| | - Giorgia Lorenzini
- Institute for Biomedical Ethics, University of Basel, Basel, Switzerland
| | - David M Shaw
- Institute for Biomedical Ethics, University of Basel, Basel, Switzerland; Care and Public Health Research Institute, Maastricht University, Netherlands
| | - Bernice Simone Elger
- Institute for Biomedical Ethics, University of Basel, Basel, Switzerland; Center for Legal Medicine (CURML), University of Geneva, Switzerland
| |
Collapse
|
27
|
Rangan ES, Pathinarupothi RK, Anand KJS, Snyder MP. Performance effectiveness of vital parameter combinations for early warning of sepsis-an exhaustive study using machine learning. JAMIA Open 2022; 5:ooac080. [PMID: 36267121 PMCID: PMC9566305 DOI: 10.1093/jamiaopen/ooac080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2022] [Revised: 09/07/2022] [Accepted: 09/20/2022] [Indexed: 11/15/2022] Open
Abstract
Objective To carry out exhaustive data-driven computations for the performance of noninvasive vital signs heart rate (HR), respiratory rate (RR), peripheral oxygen saturation (SpO2), and temperature (Temp), considered both independently and in all possible combinations, for early detection of sepsis. Materials and methods By extracting features interpretable by clinicians, we applied Gradient Boosted Decision Tree machine learning on a dataset of 2630 patients to build 240 models. Validation was performed on a geographically distinct dataset. Relative to onset, predictions were clocked as per 16 pairs of monitoring intervals and prediction times, and the outcomes were ranked. Results The combination of HR and Temp was found to be a minimal feature set yielding maximal predictability with area under receiver operating curve 0.94, sensitivity of 0.85, and specificity of 0.90. Whereas HR and RR each directly enhance prediction, the effects of SpO2 and Temp are significant only when combined with HR or RR. In benchmarking relative to standard methods Systemic Inflammatory Response Syndrome (SIRS), National Early Warning Score (NEWS), and quick-Sequential Organ Failure Assessment (qSOFA), Vital-SEP outperformed all 3 of them. Conclusion It can be concluded that using intensive care unit data even 2 vital signs are adequate to predict sepsis upto 6 h in advance with promising accuracy comparable to standard scoring methods and other sepsis predictive tools reported in literature. Vital-SEP can be used for fast-track prediction especially in limited resource hospital settings where laboratory based hematologic or biochemical assays may be unavailable, inaccurate, or entail clinically inordinate delays. A prospective study is essential to determine the clinical impact of the proposed sepsis prediction model and evaluate other outcomes such as mortality and duration of hospital stay.
Collapse
Affiliation(s)
- Ekanath Srihari Rangan
- Department of Genetics, Stanford University School of Medicine, Stanford, California, USA
| | | | - Kanwaljeet J S Anand
- Division of Critical Care, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA
| | - Michael P Snyder
- Department of Genetics, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
28
|
Accuracy of International Classification of Diseases, 10th Revision Codes for Identifying Sepsis: A Systematic Review and Meta-Analysis. Crit Care Explor 2022; 4:e0788. [PMID: 36382338 PMCID: PMC9649267 DOI: 10.1097/cce.0000000000000788] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022] Open
Abstract
UNLABELLED Administrative databases are increasingly used in research studies to capture clinical outcomes such as sepsis. This systematic review and meta-analysis examines the accuracy of International Classification of Diseases, 10th revision (ICD-10), codes for identifying sepsis in adult and pediatric patients. DATA SOURCES We searched MEDLINE, EMBASE, Web of Science, CENTRAL, Epistemonikos, and McMaster Superfilters from inception to September 7, 2021. STUDY SELECTION We included studies that validated the accuracy of sepsis ICD-10 codes against any reference standard. DATA EXTRACTION Three authors, working in duplicate, independently extracted data. We conducted meta-analysis using a random effects model to pool sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). We evaluated individual study risk of bias using the Quality Assessment of Diagnostic Accuracy Studies tool and assessed certainty in pooled diagnostic effect measures using the Grading of Recommendations Assessment, Development, and Evaluation framework. DATA SYNTHESIS Thirteen eligible studies were included in the qualitative synthesis and the meta-analysis. Eleven studies used manual chart review as the reference standard, and four studies used registry databases. Only one study evaluated pediatric patients exclusively. Compared with the reference standard of detailed chart review and/or registry databases, the pooled sensitivity for sepsis ICD-10 codes was 35% (95% CI, 22-48, low certainty), whereas the pooled specificity was 98% (95% CI: 98-99, low certainty). The PPV for ICD-10 codes ranged from 9.8% to 100% (median, 72.0%; interquartile range [IQR], 50.0-84.7%). NPV ranged from 54.7% to 99.1% (median, 95.9%; interquartile range, 85.5-98.3%). CONCLUSIONS Sepsis is undercoded in administrative databases. Future research is needed to explore if greater consistency in ICD-10 code definitions and enhanced quality measures for ICD-10 coders can improve the coding accuracy of sepsis in large databases.
Collapse
|
29
|
|
30
|
Adams R, Henry KE, Sridharan A, Soleimani H, Zhan A, Rawat N, Johnson L, Hager DN, Cosgrove SE, Markowski A, Klein EY, Chen ES, Saheed MO, Henley M, Miranda S, Houston K, Linton RC, Ahluwalia AR, Wu AW, Saria S. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med 2022; 28:1455-1460. [PMID: 35864252 DOI: 10.1038/s41591-022-01894-0] [Citation(s) in RCA: 90] [Impact Index Per Article: 45.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2021] [Accepted: 06/08/2022] [Indexed: 12/20/2022]
Abstract
Early recognition and treatment of sepsis are linked to improved patient outcomes. Machine learning-based early warning systems may reduce the time to recognition, but few systems have undergone clinical evaluation. In this prospective, multi-site cohort study, we examined the association between patient outcomes and provider interaction with a deployed sepsis alert system called the Targeted Real-time Early Warning System (TREWS). During the study, 590,736 patients were monitored by TREWS across five hospitals. We focused our analysis on 6,877 patients with sepsis who were identified by the alert before initiation of antibiotic therapy. Adjusting for patient presentation and severity, patients in this group whose alert was confirmed by a provider within 3 h of the alert had a reduced in-hospital mortality rate (3.3%, confidence interval (CI) 1.7, 5.1%, adjusted absolute reduction, and 18.7%, CI 9.4, 27.0%, adjusted relative reduction), organ failure and length of stay compared with patients whose alert was not confirmed by a provider within 3 h. Improvements in mortality rate (4.5%, CI 0.8, 8.3%, adjusted absolute reduction) and organ failure were larger among those patients who were additionally flagged as high risk. Our findings indicate that early warning systems have the potential to identify sepsis patients early and improve patient outcomes and that sepsis patients who would benefit the most from early treatment can be identified and prioritized at the time of the alert.
Collapse
Affiliation(s)
- Roy Adams
- Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD, USA.,Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Katharine E Henry
- Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA.,Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | | | - Hossein Soleimani
- Health Informatics, University of California, San Francisco, CA, USA
| | - Andong Zhan
- Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA.,Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | - Nishi Rawat
- Armstrong Institute for Patient Safety and Quality, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Lauren Johnson
- Department of Quality Improvement, Johns Hopkins Hospital, Baltimore, MD, USA
| | - David N Hager
- Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Sara E Cosgrove
- Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | | | - Eili Y Klein
- Department of Emergency Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Edward S Chen
- Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Mustapha O Saheed
- Department of Emergency Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA
| | - Maureen Henley
- Department of Quality Improvement, Johns Hopkins Hospital, Baltimore, MD, USA
| | - Sheila Miranda
- Department of Medicine, Johns Hopkins Hospital, Baltimore, MD, USA
| | - Katrina Houston
- Department of Quality Improvement, Johns Hopkins Hospital, Baltimore, MD, USA
| | | | | | - Albert W Wu
- Armstrong Institute for Patient Safety and Quality, Johns Hopkins School of Medicine, Baltimore, MD, USA. .,Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA. .,Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. .,Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. .,Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
| | - Suchi Saria
- Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD, USA. .,Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. .,Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA. .,Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. .,Bayesian Health, New York, NY, USA.
| |
Collapse
|
31
|
Sadasivuni S, Saha M, Bhatia N, Banerjee I, Sanyal A. Fusion of fully integrated analog machine learning classifier with electronic medical records for real-time prediction of sepsis onset. Sci Rep 2022; 12:5711. [PMID: 35383233 PMCID: PMC8983688 DOI: 10.1038/s41598-022-09712-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2021] [Accepted: 03/28/2022] [Indexed: 12/20/2022] Open
Abstract
The objective of this work is to develop a fusion artificial intelligence (AI) model that combines patient electronic medical record (EMR) and physiological sensor data to accurately predict early risk of sepsis. The fusion AI model has two components—an on-chip AI model that continuously analyzes patient electrocardiogram (ECG) data and a cloud AI model that combines EMR and prediction scores from on-chip AI model to predict fusion sepsis onset score. The on-chip AI model is designed using analog circuits for sepsis prediction with high energy efficiency for integration with resource constrained wearable device. Combination of EMR and sensor physiological data improves prediction performance compared to EMR or physiological data alone, and the late fusion model has an accuracy of 93% in predicting sepsis 4 h before onset. The key differentiation of this work over existing sepsis prediction literature is the use of single modality patient vital (ECG) and simple demographic information, instead of comprehensive laboratory test results and multiple vital signs. Such simple configuration and high accuracy makes our solution favorable for real-time, at-home use for self-monitoring.
Collapse
Affiliation(s)
| | - Monjoy Saha
- Department of Biomedical Informatics, Emory University, Atlanta, GA, 30322, USA
| | - Neal Bhatia
- Division of Cardiology, Emory University School of Medicine, Atlanta, GA, 30322, USA
| | - Imon Banerjee
- Department of Biomedical Informatics, Emory University, Atlanta, GA, 30322, USA.,Department of Radiology, Emory University, Atlanta, GA, 30322, USA
| | - Arindam Sanyal
- Electrical Engineering, University at Buffalo, Buffalo, NY, 14260, USA.
| |
Collapse
|
32
|
Morse KE, Brown C, Fleming S, Todd I, Powell A, Russell A, Scheinker D, Sutherland SM, Lu J, Watkins B, Shah NH, Pageler NM, Palma JP. Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model. Appl Clin Inform 2022; 13:431-438. [PMID: 35508197 PMCID: PMC9068274 DOI: 10.1055/s-0042-1746168] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 03/01/2022] [Indexed: 12/29/2022] Open
Abstract
OBJECTIVE The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
Collapse
Affiliation(s)
- Keith E. Morse
- Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
| | - Conner Brown
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Scott Fleming
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Irene Todd
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Austin Powell
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Alton Russell
- Harvard Medical School, Boston, Massachusetts, United States
| | - David Scheinker
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Scott M. Sutherland
- Division of Nephrology, Department of Pediatrics, Stanford University, Stanford, California, United States
| | - Jonathan Lu
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Brendan Watkins
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Nigam H. Shah
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Natalie M. Pageler
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
- Division of Systems Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
| | - Jonathan P. Palma
- Division of Neonatology, Department of Pediatrics, Orlando Health, Orlando, Florida, United States
| |
Collapse
|
33
|
Kim JK, Ahn W, Park S, Lee SH, Kim L. Early Prediction of Sepsis Onset Using Neural Architecture Search Based on Genetic Algorithms. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:ijerph19042349. [PMID: 35206537 PMCID: PMC8872017 DOI: 10.3390/ijerph19042349] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 02/13/2022] [Accepted: 02/14/2022] [Indexed: 12/14/2022]
Abstract
Sepsis is a life-threatening condition with a high mortality rate. Early prediction and treatment are the most effective strategies for increasing survival rates. This paper proposes a neural architecture search (NAS) model to predict the onset of sepsis with a low computational cost and high search performance by applying a genetic algorithm (GA). The proposed model shares the weights of all possible connection nodes internally within the neural network. Externally, the search cost is reduced through the weight-sharing effect between the genotypes of the GA. A predictive analysis was performed using the Medical Information Mart for Intensive Care III (MIMIC-III), a medical time-series dataset, with the primary objective of predicting sepsis onset 3 h before occurrence. In addition, experiments were conducted under various prediction times (0-12 h) for comparison. The proposed model exhibited an area under the receiver operating characteristic curve (AUROC) score of 0.94 (95% CI: 0.92-0.96) for 3 h, which is 0.31-0.26 higher than the scores obtained using the Sequential Organ Failure Assessment (SOFA), quick SOFA (qSOFA), and Simplified Acute Physiology Score (SAPS) II scoring systems. Furthermore, the proposed model exhibited a 12% improvement in the AUROC value over a simple model based on the long short-term memory neural network. Additionally, it is not only optimally searchable for sepsis onset prediction, but also outperforms conventional models that use similar predictive purposes and datasets. Notably, it is sufficiently robust to shape changes in the input data and has less structural dependence.
Collapse
Affiliation(s)
- Jae Kwan Kim
- Center for Bionics, Korea Institute of Science and Technology, Seoul 02792, Korea
- School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea
| | - Wonbin Ahn
- Applied AI Research Lab, LG AI Research, Seoul 07796, Korea
| | - Sangin Park
- Center for Bionics, Korea Institute of Science and Technology, Seoul 02792, Korea
| | - Soo-Hong Lee
- School of Mechanical Engineering, Yonsei University, Seoul 03722, Korea
| | - Laehyun Kim
- Center for Bionics, Korea Institute of Science and Technology, Seoul 02792, Korea
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul 04763, Korea
| |
Collapse
|
34
|
Chalumuri YR, Kimball JP, Mousavi A, Zia JS, Rolfes C, Parreira JD, Inan OT, Hahn JO. Classification of Blood Volume Decompensation State via Machine Learning Analysis of Multi-Modal Wearable-Compatible Physiological Signals. SENSORS (BASEL, SWITZERLAND) 2022; 22:1336. [PMID: 35214238 PMCID: PMC8963055 DOI: 10.3390/s22041336] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/21/2021] [Revised: 01/14/2022] [Accepted: 01/25/2022] [Indexed: 12/15/2022]
Abstract
This paper presents a novel computational algorithm to estimate blood volume decompensation state based on machine learning (ML) analysis of multi-modal wearable-compatible physiological signals. To the best of our knowledge, our algorithm may be the first of its kind which can not only discriminate normovolemia from hypovolemia but also classify hypovolemia into absolute hypovolemia and relative hypovolemia. We realized our blood volume classification algorithm by (i) extracting a multitude of features from multi-modal physiological signals including the electrocardiogram (ECG), the seismocardiogram (SCG), the ballistocardiogram (BCG), and the photoplethysmogram (PPG), (ii) constructing two ML classifiers using the features, one to classify normovolemia vs. hypovolemia and the other to classify hypovolemia into absolute hypovolemia and relative hypovolemia, and (iii) sequentially integrating the two to enable multi-class classification (normovolemia, absolute hypovolemia, and relative hypovolemia). We developed the blood volume decompensation state classification algorithm using the experimental data collected from six animals undergoing normovolemia, relative hypovolemia, and absolute hypovolemia challenges. Leave-one-subject-out analysis showed that our classification algorithm achieved an F1 score and accuracy of (i) 0.93 and 0.89 in classifying normovolemia vs. hypovolemia, (ii) 0.88 and 0.89 in classifying hypovolemia into absolute hypovolemia and relative hypovolemia, and (iii) 0.77 and 0.81 in classifying the overall blood volume decompensation state. The analysis of the features embedded in the ML classifiers indicated that many features are physiologically plausible, and that multi-modal SCG-BCG fusion may play an important role in achieving good blood volume classification efficacy. Our work may complement existing computational algorithms to estimate blood volume compensatory reserve as a potential decision-support tool to provide guidance on context-sensitive hypovolemia therapeutic strategy.
Collapse
Affiliation(s)
- Yekanth Ram Chalumuri
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA; (A.M.); (J.D.P.)
| | - Jacob P. Kimball
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30308, USA; (J.P.K.); (J.S.Z.); (O.T.I.)
| | - Azin Mousavi
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA; (A.M.); (J.D.P.)
| | - Jonathan S. Zia
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30308, USA; (J.P.K.); (J.S.Z.); (O.T.I.)
| | - Christopher Rolfes
- Global Center for Medical Innovation, Translational Training and Testing Laboratories, Inc. (T3 Labs), Atlanta, GA 30313, USA;
| | - Jesse D. Parreira
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA; (A.M.); (J.D.P.)
| | - Omer T. Inan
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30308, USA; (J.P.K.); (J.S.Z.); (O.T.I.)
| | - Jin-Oh Hahn
- Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA; (A.M.); (J.D.P.)
| |
Collapse
|
35
|
Evaluating machine learning models for sepsis prediction: A systematic review of methodologies. iScience 2022; 25:103651. [PMID: 35028534 PMCID: PMC8741489 DOI: 10.1016/j.isci.2021.103651] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2021] [Revised: 11/16/2021] [Accepted: 12/15/2021] [Indexed: 12/29/2022] Open
Abstract
Studies for sepsis prediction using machine learning are developing rapidly in medical science recently. In this review, we propose a set of new evaluation criteria and reporting standards to assess 21 qualified machine learning models for quality analysis based on PRISMA. Our assessment shows that (1.) the definition of sepsis is not consistent among the studies; (2.) data sources and data preprocessing methods, machine learning models, feature engineering, and inclusion types vary widely among the studies; (3.) the closer to the onset of sepsis, the higher the value of AUROC is; (4.) the improvement in AUROC is primarily due to using machine learning as a feature engineering tool; (5.) deep neural networks coupled with Sepsis-3 diagnostic criteria tend to yield better results on the time series data collected from patients with sepsis. The new evaluation criteria and reporting standards will facilitate the development of improved machine learning models for clinical applications. New evaluation/reporting standard for sepsis prediction machine learning models Major limitations in the current models for sepsis prediction have been identified We strongly suggest using machine learning as a feature engineering tool Recommending multilayer neural networks and Sepsis 3.0 for yield better result
Collapse
|
36
|
Eisenberg MA, Balamuth F. Pediatric sepsis screening in US hospitals. Pediatr Res 2022; 91:351-358. [PMID: 34417563 PMCID: PMC8378117 DOI: 10.1038/s41390-021-01708-y] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Revised: 07/28/2021] [Accepted: 08/04/2021] [Indexed: 11/09/2022]
Abstract
Sepsis is a major cause of morbidity and mortality in children. While adverse outcomes can be reduced through prompt initiation of sepsis protocols including fluid resuscitation and antibiotics, provision of these therapies relies on clinician recognition of sepsis. Recognition is challenging in children because early signs of shock such as tachycardia and tachypnea have low specificity while hypotension often does not occur until late in the clinical course. This narrative review highlights the important context that has led to the rapid growth of pediatric sepsis screening in the United States. In this review, we (1) describe different screening tools used in US emergency department, inpatient, and intensive care unit settings; (2) highlight details of the design, implementation, and evaluation of specific tools; (3) review the available data on the process of integrating sepsis screening into an overall sepsis quality improvement program and on the effect of these screening tools on patient outcomes; (4) discuss potential harms of sepsis screening including alarm fatigue; and (5) highlight several future directions in sepsis screening, such as novel tools that incorporate artificial intelligence and machine learning methods to augment sepsis identification with the ultimate goal of precision-based approaches to sepsis recognition and treatment. IMPACT: This narrative review highlights the context that has led to the rapid growth of pediatric sepsis screening nationally. Screening tools used in US emergency department, inpatient, and intensive care unit settings are described in terms of their design, implementation, and clinical performance. Limitations and potential harms of these tools are highlighted, as well as future directions that may lead to a more precision-based approach to sepsis recognition and treatment.
Collapse
Affiliation(s)
- Matthew A. Eisenberg
- grid.38142.3c000000041936754XDepartments of Pediatrics and Emergency Medicine, Harvard Medical School, Boston, MA USA ,grid.2515.30000 0004 0378 8438Division of Emergency Medicine, Boston Children’s Hospital, Boston, MA USA
| | - Fran Balamuth
- grid.25879.310000 0004 1936 8972Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA USA ,grid.239552.a0000 0001 0680 8770Division of Emergency Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA USA ,grid.239552.a0000 0001 0680 8770Pediatric Sepsis Program, Children’s Hospital of Philadelphia, Philadelphia, PA USA ,grid.239552.a0000 0001 0680 8770Center for Pediatric Clinical Effectiveness, Children’s Hospital of Philadelphia, Philadelphia, PA USA
| |
Collapse
|
37
|
Yang H, Tourani R, Zhu Y, Kumar V, Melton GB, Steinbach M, Simon G. Strategies for building robust prediction models using data unavailable at prediction time. J Am Med Inform Assoc 2021; 29:72-79. [PMID: 34963141 PMCID: PMC8714287 DOI: 10.1093/jamia/ocab229] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2021] [Revised: 08/27/2021] [Accepted: 10/13/2021] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE Hospital-acquired infections (HAIs) are associated with significant morbidity, mortality, and prolonged hospital length of stay. Risk prediction models based on pre- and intraoperative data have been proposed to assess the risk of HAIs at the end of the surgery, but the performance of these models lag behind HAI detection models based on postoperative data. Postoperative data are more predictive than pre- or interoperative data since it is closer to the outcomes in time, but it is unavailable when the risk models are applied (end of surgery). The objective is to study whether such data, which is temporally unavailable at prediction time (TUP) (and thus cannot directly enter the model), can be used to improve the performance of the risk model. MATERIALS AND METHODS An extensive array of 12 methods based on logistic/linear regression and deep learning were used to incorporate the TUP data using a variety of intermediate representations of the data. Due to the hierarchical structure of different HAI outcomes, a comparison of single and multi-task learning frameworks is also presented. RESULTS AND DISCUSSION The use of TUP data was always advantageous as baseline methods, which cannot utilize TUP data, never achieved the top performance. The relative performances of the different models vary across the different outcomes. Regarding the intermediate representation, we found that its complexity was key and that incorporating label information was helpful. CONCLUSIONS Using TUP data significantly helped predictive performance irrespective of the model complexity.
Collapse
Affiliation(s)
- Haoyu Yang
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, Minnesota, USA
| | - Roshan Tourani
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Ying Zhu
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Vipin Kumar
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, Minnesota, USA
| | - Genevieve B Melton
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
- Department of Surgery, University of Minnesota, Minneapolis, Minnesota, USA
| | - Michael Steinbach
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, Minnesota, USA
| | - Gyorgy Simon
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
- Department of Internal Medicine, University of Minnesota, Minneapolis, Minnesota, USA
| |
Collapse
|
38
|
Lauritsen SM, Thiesson B, Jørgensen MJ, Riis AH, Espelund US, Weile JB, Lange J. The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards. NPJ Digit Med 2021; 4:158. [PMID: 34782696 PMCID: PMC8593052 DOI: 10.1038/s41746-021-00529-x] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2021] [Accepted: 10/19/2021] [Indexed: 11/09/2022] Open
Abstract
Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.
Collapse
Affiliation(s)
- Simon Meyer Lauritsen
- Enversion A/S, Fiskerivej 12, 1st floor, 8000, Aarhus C, Denmark. .,Department of Clinical Medicine, Aarhus University, Aarhus N, Denmark.
| | - Bo Thiesson
- Enversion A/S, Fiskerivej 12, 1st floor, 8000, Aarhus C, Denmark.,Department of Engineering, Aarhus University, Aarhus C, Denmark
| | | | | | - Ulrick Skipper Espelund
- Department of Research, Horsens Regional Hospital, Horsens, Denmark.,Department of Anesthesiology, Horsens Regional Hospital, Horsens, Denmark
| | - Jesper Bo Weile
- Emergency Department, Horsens Regional Hospital, Horsens, Denmark.,Research Center for Emergency Medicine, Aarhus University Hospital, Aarhus, Denmark
| | - Jeppe Lange
- Department of Clinical Medicine, Aarhus University, Aarhus N, Denmark.,Department of Research, Horsens Regional Hospital, Horsens, Denmark
| |
Collapse
|
39
|
Sendak MP, Gao M, Ratliff W, Nichols M, Bedoya A, O'Brien C, Balu S. Looking for clinician involvement under the wrong lamp post: The need for collaboration measures. J Am Med Inform Assoc 2021; 28:2541-2542. [PMID: 34498049 DOI: 10.1093/jamia/ocab129] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2021] [Accepted: 06/04/2021] [Indexed: 12/16/2022] Open
Affiliation(s)
- Mark P Sendak
- Duke Institute for Health Innovation, Durham, North Carolina, USA
| | - Michael Gao
- Duke Institute for Health Innovation, Durham, North Carolina, USA
| | - William Ratliff
- Duke Institute for Health Innovation, Durham, North Carolina, USA
| | - Marshall Nichols
- Duke Institute for Health Innovation, Durham, North Carolina, USA
| | - Armando Bedoya
- Department of Internal Medicine, Duke University School of Medicine, Durham, North Carolina, USA
| | - Cara O'Brien
- Department of Internal Medicine, Duke University School of Medicine, Durham, North Carolina, USA
| | - Suresh Balu
- Duke Institute for Health Innovation, Durham, North Carolina, USA.,Duke University School of Medicine, Durham, North Carolina, USA
| |
Collapse
|
40
|
Banerjee R, Shah N, Dicker AP. Next-Generation Implementation of Chimeric Antigen Receptor T-Cell Therapy Using Digital Health. JCO Clin Cancer Inform 2021; 5:668-678. [PMID: 34110929 DOI: 10.1200/cci.21.00023] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022] Open
Abstract
Chimeric antigen receptor T-cell (CAR-T) therapy is a paradigm-shifting immunotherapy modality in oncology; however, unique toxicities such as cytokine release syndrome (CRS) and immune effector cell-associated neurotoxicity syndrome limit its ability to be implemented more widely in the outpatient setting or at smaller-volume centers. Three operational challenges with CAR-T therapy include the following: (1) the logistics of toxicity monitoring, ie, with frequent vital sign checks and neurologic assessments; (2) the specialized knowledge required for toxicity management, particularly with regard to CRS and immune effector cell-associated neurotoxicity syndrome; and (3) the need for high-quality symptomatic and supportive care during this intensive period. In this review, we explore potential niches for digital innovations that can improve the implementation of CAR-T therapy in each of these domains. These tools include patient-facing technologies and provider-facing platforms: for example, wearable devices and mobile health apps to screen for fevers and encephalopathy, electronic patient-reported outcome assessments-based workflows to assist with symptom management, machine learning algorithms to predict emerging CRS in real time, clinical decision support systems to assist with toxicity management, and digital coaching to help maintain wellness. Televisits, which have grown in prominence since the novel coronavirus pandemic, will continue to play a key role in the monitoring and management of CAR-T-related toxicities as well. Limitations of these strategies include the need to ensure care equity and stakeholder buy-in, both operationally and financially. Nevertheless, once developed and validated, the next-generation implementation of CAR-T therapy using these digital tools may improve both its safety and accessibility.
Collapse
Affiliation(s)
- Rahul Banerjee
- Division of Hematology/Oncology, Department of Medicine, University of California San Francisco, San Francisco, CA
| | - Nina Shah
- Division of Hematology/Oncology, Department of Medicine, University of California San Francisco, San Francisco, CA
| | - Adam P Dicker
- Department of Radiation Oncology, Jefferson University, Philadelphia, PA.,Jefferson Center for Digital Health, Jefferson University, Philadelphia, PA
| |
Collapse
|
41
|
Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records. Nat Protoc 2021; 16:2765-2787. [PMID: 33953393 DOI: 10.1038/s41596-021-00513-5] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2020] [Accepted: 01/25/2021] [Indexed: 02/03/2023]
Abstract
Early prediction of patient outcomes is important for targeting preventive care. This protocol describes a practical workflow for developing deep-learning risk models that can predict various clinical and operational outcomes from structured electronic health record (EHR) data. The protocol comprises five main stages: formal problem definition, data pre-processing, architecture selection, calibration and uncertainty, and generalizability evaluation. We have applied the workflow to four endpoints (acute kidney injury, mortality, length of stay and 30-day hospital readmission). The workflow can enable continuous (e.g., triggered every 6 h) and static (e.g., triggered at 24 h after admission) predictions. We also provide an open-source codebase that illustrates some key principles in EHR modeling. This protocol can be used by interdisciplinary teams with programming and clinical expertise to build deep-learning prediction models with alternate data sources and prediction tasks.
Collapse
|
42
|
Pinevich Y, Amos-Binks A, Burris CS, Rule G, Bogojevic M, Flint I, Pickering BW, Nemeth CP, Herasevich V. Validation of a Machine Learning Model for Early Shock Detection. Mil Med 2021; 187:82-88. [PMID: 34056656 DOI: 10.1093/milmed/usab220] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 03/27/2021] [Accepted: 05/18/2021] [Indexed: 12/17/2022] Open
Abstract
OBJECTIVES The objectives of this study were to test in real time a Trauma Triage, Treatment, and Training Decision Support (4TDS) machine learning (ML) model of shock detection in a prospective silent trial, and to evaluate specificity, sensitivity, and other estimates of diagnostic performance compared to the gold standard of electronic medical records (EMRs) review. DESIGN We performed a single-center diagnostic performance study. PATIENTS AND SETTING A prospective cohort consisted of consecutive patients aged 18 years and older who were admitted from May 1 through September 30, 2020 to six Mayo Clinic intensive care units (ICUs) and five progressive care units. MEASUREMENTS AND MAIN RESULTS During the study time, 5,384 out of 6,630 hospital admissions were eligible. During the same period, the 4TDS shock model sent 825 alerts and 632 were eligible. Among 632 hospital admissions with alerts, 287 were screened positive and 345 were negative. Among 4,752 hospital admissions without alerts, 78 were screened positive and 4,674 were negative. The area under the receiver operating characteristics curve for the 4TDS shock model was 0.86 (95% CI 0.85-0.87%). The 4TDS shock model demonstrated a sensitivity of 78.6% (95% CI 74.1-82.7%) and a specificity of 93.1% (95% CI 92.4-93.8%). The model showed a positive predictive value of 45.4% (95% CI 42.6-48.3%) and a negative predictive value of 98.4% (95% CI 98-98.6%). CONCLUSIONS We successfully validated an ML model to detect circulatory shock in a prospective observational study. The model used only vital signs and showed moderate performance compared to the gold standard of clinician EMR review when applied to an ICU patient cohort.
Collapse
Affiliation(s)
- Yuliya Pinevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | | | | | - Gregory Rule
- Applied Research Associates, Albuquerque, NM 87110, USA
| | - Marija Bogojevic
- Division of Nephrology and Hypertension, Mayo Clinic, Rochester, MN 55905, USA
| | - Isaac Flint
- Applied Research Associates, Albuquerque, NM 87110, USA
| | - Brian W Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | | | - Vitaly Herasevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA
| |
Collapse
|
43
|
Wu M, Du X, Gu R, Wei J. Artificial Intelligence for Clinical Decision Support in Sepsis. Front Med (Lausanne) 2021; 8:665464. [PMID: 34055839 PMCID: PMC8155362 DOI: 10.3389/fmed.2021.665464] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2021] [Accepted: 04/06/2021] [Indexed: 11/13/2022] Open
Abstract
Sepsis is one of the main causes of death in critically ill patients. Despite the continuous development of medical technology in recent years, its morbidity and mortality are still high. This is mainly related to the delay in starting treatment and non-adherence of clinical guidelines. Artificial intelligence (AI) is an evolving field in medicine, which has been used to develop a variety of innovative Clinical Decision Support Systems. It has shown great potential in predicting the clinical condition of patients and assisting in clinical decision-making. AI-derived algorithms can be applied to multiple stages of sepsis, such as early prediction, prognosis assessment, mortality prediction, and optimal management. This review describes the latest literature on AI for clinical decision support in sepsis, and outlines the application of AI in the prediction, diagnosis, subphenotyping, prognosis assessment, and clinical management of sepsis. In addition, we discussed the challenges of implementing and accepting this non-traditional methodology for clinical purposes.
Collapse
Affiliation(s)
- Miao Wu
- Department of Emergency, Renmin Hospital of Wuhan University, Wuhan, China
| | - Xianjin Du
- Department of Critical Care Medicine, Renmin Hospital of Wuhan University, Wuhan, China
| | - Raymond Gu
- Department of Surgery, State University of New York Upstate Medical University, Syracuse, NY, United States
| | - Jie Wei
- Department of Emergency, Renmin Hospital of Wuhan University, Wuhan, China
| |
Collapse
|
44
|
Giacobbe DR, Signori A, Del Puente F, Mora S, Carmisciano L, Briano F, Vena A, Ball L, Robba C, Pelosi P, Giacomini M, Bassetti M. Early Detection of Sepsis With Machine Learning Techniques: A Brief Clinical Perspective. Front Med (Lausanne) 2021; 8:617486. [PMID: 33644097 PMCID: PMC7906970 DOI: 10.3389/fmed.2021.617486] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2020] [Accepted: 01/19/2021] [Indexed: 12/15/2022] Open
Abstract
Sepsis is a major cause of death worldwide. Over the past years, prediction of clinically relevant events through machine learning models has gained particular attention. In the present perspective, we provide a brief, clinician-oriented vision on the following relevant aspects concerning the use of machine learning predictive models for the early detection of sepsis in the daily practice: (i) the controversy of sepsis definition and its influence on the development of prediction models; (ii) the choice and availability of input features; (iii) the measure of the model performance, the output, and their usefulness in the clinical practice. The increasing involvement of artificial intelligence and machine learning in health care cannot be disregarded, despite important pitfalls that should be always carefully taken into consideration. In the long run, a rigorous multidisciplinary approach to enrich our understanding in the application of machine learning techniques for the early recognition of sepsis may show potential to augment medical decision-making when facing this heterogeneous and complex syndrome.
Collapse
Affiliation(s)
- Daniele Roberto Giacobbe
- Infectious Diseases Unit, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - Alessio Signori
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - Filippo Del Puente
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - Sara Mora
- Department of Informatics Bioengineering, Robotics, and Systems Engineering (DIBRIS), University of Genoa, Genoa, Italy
| | - Luca Carmisciano
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - Federica Briano
- Infectious Diseases Unit, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - Antonio Vena
- Infectious Diseases Unit, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
| | - Lorenzo Ball
- Anaesthesia and Intensive Care, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Surgical Sciences and Integrated Diagnostics (DISC), University of Genoa, Genoa, Italy
| | - Chiara Robba
- Anaesthesia and Intensive Care, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Surgical Sciences and Integrated Diagnostics (DISC), University of Genoa, Genoa, Italy
| | - Paolo Pelosi
- Anaesthesia and Intensive Care, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Surgical Sciences and Integrated Diagnostics (DISC), University of Genoa, Genoa, Italy
| | - Mauro Giacomini
- Department of Informatics Bioengineering, Robotics, and Systems Engineering (DIBRIS), University of Genoa, Genoa, Italy
| | - Matteo Bassetti
- Infectious Diseases Unit, San Martino Policlinico Hospital – IRCCS for Oncology and Neurosciences, Genoa, Italy
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| |
Collapse
|
45
|
Sandhu S, Lin AL, Brajer N, Sperling J, Ratliff W, Bedoya AD, Balu S, O'Brien C, Sendak MP. Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study. J Med Internet Res 2020; 22:e22421. [PMID: 33211015 PMCID: PMC7714645 DOI: 10.2196/22421] [Citation(s) in RCA: 51] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2020] [Revised: 09/16/2020] [Accepted: 10/26/2020] [Indexed: 12/22/2022] Open
Abstract
Background Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. Objective This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. Methods We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. Results A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. Conclusions This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.
Collapse
Affiliation(s)
- Sahil Sandhu
- Trinity College of Arts & Sciences, Duke University, Durham, NC, United States
| | - Anthony L Lin
- Duke University School of Medicine, Durham, NC, United States
| | - Nathan Brajer
- Duke University School of Medicine, Durham, NC, United States
| | - Jessica Sperling
- Social Science Research Institute, Duke University, Durham, NC, United States
| | - William Ratliff
- Duke Institute for Health Innovation, Durham, NC, United States
| | - Armando D Bedoya
- Division of Pulmonary, Allergy, and Critical Care Medicine, Duke University School of Medicine, Durham, NC, United States
| | - Suresh Balu
- Duke Institute for Health Innovation, Durham, NC, United States
| | - Cara O'Brien
- Department of Medicine, Duke University School of Medicine, Durham, NC, United States
| | - Mark P Sendak
- Duke Institute for Health Innovation, Durham, NC, United States
| |
Collapse
|