1
|
Byrd TF, Phelan TA, Ingraham NE, Langworthy BW, Bhasin A, Kc A, Melton-Meaux GB, Tignanelli CJ. Beyond Unplanned ICU Transfers: Linking a Revised Definition of Deterioration to Patient Outcomes. Crit Care Med 2024; 52:e439-e449. [PMID: 38832836 DOI: 10.1097/ccm.0000000000006333] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/06/2024]
Abstract
OBJECTIVES To develop an electronic descriptor of clinical deterioration for hospitalized patients that predicts short-term mortality and identifies patient deterioration earlier than current standard definitions. DESIGN A retrospective study using exploratory record review, quantitative analysis, and regression analyses. SETTING Twelve-hospital community-academic health system. PATIENTS All adult patients with an acute hospital encounter between January 1, 2018, and December 31, 2022. INTERVENTIONS Not applicable. MEASUREMENTS AND MAIN RESULTS Clinical trigger events were selected and used to create a revised electronic definition of deterioration, encompassing signals of respiratory failure, bleeding, and hypotension occurring in proximity to ICU transfer. Patients meeting the revised definition were 12.5 times more likely to die within 7 days (adjusted odds ratio 12.5; 95% CI, 8.9-17.4) and had a 95.3% longer length of stay (95% CI, 88.6-102.3%) compared with those who were transferred to the ICU or died regardless of meeting the revised definition. Among the 1812 patients who met the revised definition of deterioration before ICU transfer (52.4%), the median detection time was 157.0 min earlier (interquartile range 64.0-363.5 min). CONCLUSIONS The revised definition of deterioration establishes an electronic descriptor of clinical deterioration that is strongly associated with short-term mortality and length of stay and identifies deterioration over 2.5 hours earlier than ICU transfer. Incorporating the revised definition of deterioration into the training and validation of early warning system algorithms may enhance their timeliness and clinical accuracy.
Collapse
Affiliation(s)
- Thomas F Byrd
- Division of Hospital Medicine, Department of Medicine, University of Minnesota, Minneapolis, MN
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN
| | | | - Nicholas E Ingraham
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN
- Division of Pulmonary and Critical Care, Department of Medicine, University of Minnesota, Minneapolis, MN
| | - Benjamin W Langworthy
- Division of Biostatistics, University of Minnesota School of Public Health, Minneapolis, MN
| | - Ajay Bhasin
- Division of Hospital Medicine, Department of Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - Abhinab Kc
- University of Minnesota Medical School, Minneapolis, MN
| | - Genevieve B Melton-Meaux
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN
- Division of Colon and Rectal Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN
| | - Christopher J Tignanelli
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN
- Division of Acute Care Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN
| |
Collapse
|
2
|
Pinsky MR, Bedoya A, Bihorac A, Celi L, Churpek M, Economou-Zavlanos NJ, Elbers P, Saria S, Liu V, Lyons PG, Shickel B, Toral P, Tscholl D, Clermont G. Use of artificial intelligence in critical care: opportunities and obstacles. Crit Care 2024; 28:113. [PMID: 38589940 PMCID: PMC11000355 DOI: 10.1186/s13054-024-04860-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2024] [Accepted: 03/05/2024] [Indexed: 04/10/2024] Open
Abstract
BACKGROUND Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed. MAIN BODY Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools. CONCLUSIONS AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.
Collapse
Affiliation(s)
- Michael R Pinsky
- Department of Critical Care Medicine, School of Medicine, University of Pittsburgh, 638 Scaife Hall, 3550 Terrace Street, Pittsburgh, PA, 15261, USA.
| | - Armando Bedoya
- Algorithm-Based Clinical Decision Support (ABCDS) Oversight, Office of Vice Dean of Data Science, School of Medicine, Duke University, Durham, NC, 27705, USA
- Division of Pulmonary Critical Care Medicine, Duke University School of Medicine, Durham, NC, 27713, USA
| | - Azra Bihorac
- Department of Medicine, University of Florida College of Medicine Gainesville, Malachowsky Hall, 1889 Museum Road, Suite 2410, Gainesville, FL, 32611, USA
| | - Leo Celi
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Matthew Churpek
- Department of Medicine, University of Wisconsin, 600 Highland Ave, Madison, WI, 53792, USA
| | - Nicoleta J Economou-Zavlanos
- Algorithm-Based Clinical Decision Support (ABCDS) Oversight, Office of Vice Dean of Data Science, School of Medicine, Duke University, Durham, NC, 27705, USA
| | - Paul Elbers
- Department of Intensive Care, Amsterdam UMC, Amsterdam, NL, USA
- Amsterdam UMC, ZH.7D.167, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands
| | - Suchi Saria
- Department of Computer Science, Whiting School of Engineering, Johns Hopkins Medical Institutions, Johns Hopkins University, 333 Malone Hall, 300 Wolfe Street, Baltimore, MD, USA
- Department of Medicine, Johns Hopkins School of Medicine, AI and Health Lab, Johns Hopkins University, Baltimore, MD, USA
- Bayesian Health, New york, NY, 10282, USA
| | - Vincent Liu
- Department of Medicine, Oregon Health & Science University, 3181 S.W. Sam Jackson Park Road, Mail Code UHN67, Portland, OR, 97239-3098, USA
- , 2000 Broadway, Oakland, CA, 94612, USA
| | - Patrick G Lyons
- Department of Medicine, Oregon Health & Science University, 3181 S.W. Sam Jackson Park Road, Mail Code UHN67, Portland, OR, 97239-3098, USA
| | - Benjamin Shickel
- Department of Medicine, University of Florida College of Medicine Gainesville, Malachowsky Hall, 1889 Museum Road, Suite 2410, Gainesville, FL, 32611, USA
- Amsterdam UMC, ZH.7D.167, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands
| | - Patrick Toral
- Department of Intensive Care, Amsterdam UMC, Amsterdam, NL, USA
- Amsterdam UMC, ZH.7D.165, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands
| | - David Tscholl
- Institute of Anesthesiology, University Hospital Zurich, University of Zurich, Frauenklinikstrasse 10, 8091, Zurich, Switzerland
| | - Gilles Clermont
- Department of Critical Care Medicine, School of Medicine, University of Pittsburgh, 638 Scaife Hall, 3550 Terrace Street, Pittsburgh, PA, 15261, USA
- VA Pittsburgh Health System, 131A Building 30, 4100 Allequippa St, Pittsburgh, PA, 15240, USA
| |
Collapse
|
3
|
Salehinejad H, Meehan AM, Rahman PA, Core MA, Borah BJ, Caraballo PJ. Novel machine learning model to improve performance of an early warning system in hospitalized patients: a retrospective multisite cross-validation study. EClinicalMedicine 2023; 66:102312. [PMID: 38192596 PMCID: PMC10772226 DOI: 10.1016/j.eclinm.2023.102312] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 10/26/2023] [Accepted: 10/26/2023] [Indexed: 01/10/2024] Open
Abstract
Background Threshold-based early warning systems (EWS) are used to predict adverse events (Aes). Machine learning (ML) algorithms that incorporate all EWS scores prior to an event may perform better in hospitalized patients. Methods The deterioration index (DI) is a proprietary EWS. A threshold of DI >60 is used to predict a composite AE: all-cause mortality, cardiac arrest, transfer to intensive care, and evaluation by the rapid response team in practice. The DI scores were collected for adult patients (≥18 y-o) hospitalized on medical or surgical services during 8-23-2021 to 3-31-2022 from four different Mayo Clinic sites in the United States. A novel ML model was developed and trained on a retrospective cohort of hospital encounters. DI scores were represented in a high-dimensional space using random convolution kernels to facilitate training of a classifier and the area under the receiver operator characteristics curve (AUC) was calculated. Multiple time intervals prior to an AE were analyzed. A leave-one-out cross-validation protocol was used to evaluate performance across separate clinic sites. Findings Three different classifiers were trained on 59,617 encounter-derived DI scores in high-dimensional feature space and the AUCs were compared to two threshold models. All three tested classifiers improved the AUC over the threshold approaches from 0.56 and 0.57 to 0.76, 0.85 and 0.94. Time interval analysis of the top performing classifier showed best accuracy in the hour before an event occurred (AUC 0.91), but prediction held up even in the 12 h before an AE (AUC 0.80 at minus 12 h, 0.81 at minus 9 h, 0.85 at minus 6 h, and 0.88 at minus 3 h before an AE). Multisite cross-validation using leave-one-out approach on data from four different clinical sites showed broad generalization performance of the top performing ML model with AUC of 0.91, 0.91, 0.95, and 0.91. Interpretation A novel ML model that incorporates all the longitudinal DI scores prior to an AE in a hospitalized patient performs better at outcome prediction than the currently used threshold model. The use of clinical data, a generalized ML technique, and successful multisite cross-validation demonstrate the feasibility of our model in clinical implementation. Funding No funding to report.
Collapse
Affiliation(s)
- Hojjat Salehinejad
- Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | | | - Parvez A. Rahman
- Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA
| | - Marcia A. Core
- Department of Information Technology, Mayo Clinic, Rochester, MN, USA
| | - Bijan J. Borah
- Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA
| | - Pedro J. Caraballo
- Department of Medicine, Mayo Clinic, Rochester, MN, USA
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
4
|
Byrd TF, Southwell B, Ravishankar A, Tran T, Kc A, Phelan T, Melton-Meaux GB, Usher MG, Scheppmann D, Switzer S, Simon G, Tignanelli CJ. Validation of a Proprietary Deterioration Index Model and Performance in Hospitalized Adults. JAMA Netw Open 2023; 6:e2324176. [PMID: 37486632 PMCID: PMC10366696 DOI: 10.1001/jamanetworkopen.2023.24176] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/25/2023] Open
Abstract
Importance The Deterioration Index (DTI), used by hospitals for predicting patient deterioration, has not been extensively validated externally, raising concerns about performance and equitable predictions. Objective To locally validate DTI performance and assess its potential for bias in predicting patient clinical deterioration. Design, Setting, and Participants This retrospective prognostic study included 13 737 patients admitted to 8 heterogenous Midwestern US hospitals varying in size and type, including academic, community, urban, and rural hospitals. Patients were 18 years or older and admitted between January 1 and May 31, 2021. Exposure DTI predictions made every 15 minutes. Main Outcomes and Measures Deterioration, defined as the occurrence of any of the following while hospitalized: mechanical ventilation, intensive care unit transfer, or death. Performance of the DTI was evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Bias measures were calculated across demographic subgroups. Results A total of 5 143 513 DTI predictions were made for 13 737 patients across 14 834 hospitalizations. Among 13 918 encounters, the mean (SD) age of patients was 60.3 (19.2) years; 7636 (54.9%) were female, 11 345 (81.5%) were White, and 12 392 (89.0%) were of other ethnicity than Hispanic or Latino. The prevalence of deterioration was 10.3% (n = 1436). The DTI produced AUROCs of 0.759 (95% CI, 0.756-0.762) at the observation level and 0.685 (95% CI, 0.671-0.700) at the encounter level. Corresponding AUPRCs were 0.039 (95% CI, 0.037-0.040) at the observation level and 0.248 (95% CI, 0.227-0.273) at the encounter level. Bias measures varied across demographic subgroups and were 14.0% worse for patients identifying as American Indian or Alaska Native and 19.0% worse for those who chose not to disclose their ethnicity. Conclusions and Relevance In this prognostic study, the DTI had modest ability to predict patient deterioration, with varying degrees of performance at the observation and encounter levels and across different demographic groups. Disparate performance across subgroups suggests the need for more transparency in model training data and reinforces the need to locally validate externally developed prediction models.
Collapse
Affiliation(s)
- Thomas F Byrd
- Department of Medicine, University of Minnesota, Minneapolis
- Institute for Health Informatics, University of Minnesota, Minneapolis
| | | | | | - Travis Tran
- Department of Medicine, University of Minnesota, Minneapolis
| | - Abhinab Kc
- University of Minnesota Medical School, University of Minnesota, Minneapolis
| | - Tom Phelan
- Fairview Health Services, Minneapolis, Minnesota
| | - Genevieve B Melton-Meaux
- Institute for Health Informatics, University of Minnesota, Minneapolis
- Department of Surgery, University of Minnesota, Minneapolis
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis
| | - Michael G Usher
- Department of Medicine, University of Minnesota, Minneapolis
- Institute for Health Informatics, University of Minnesota, Minneapolis
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis
| | - Daren Scheppmann
- Institute for Health Informatics, University of Minnesota, Minneapolis
| | - Sean Switzer
- Department of Medicine, University of Minnesota, Minneapolis
| | - Gyorgy Simon
- Department of Medicine, University of Minnesota, Minneapolis
- Institute for Health Informatics, University of Minnesota, Minneapolis
| | - Christopher J Tignanelli
- Institute for Health Informatics, University of Minnesota, Minneapolis
- Department of Surgery, University of Minnesota, Minneapolis
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis
| |
Collapse
|