1
|
Amin KD, Weissler EH, Ratliff W, Sullivan AE, Holder TA, Bury C, Francis S, Theiling BJ, Hintze B, Gao M, Nichols M, Balu S, Jones WS, Sendak M. Development and Validation of a Natural Language Processing Model to Identify Low-Risk Pulmonary Embolism in Real Time to Facilitate Safe Outpatient Management. Ann Emerg Med 2024; 84:118-127. [PMID: 38441514 DOI: 10.1016/j.annemergmed.2024.01.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 01/24/2024] [Accepted: 01/25/2024] [Indexed: 07/22/2024]
Abstract
STUDY OBJECTIVE This study aimed to (1) develop and validate a natural language processing model to identify the presence of pulmonary embolism (PE) based on real-time radiology reports and (2) identify low-risk PE patients based on previously validated risk stratification scores using variables extracted from the electronic health record at the time of diagnosis. The combination of these approaches yielded an natural language processing-based clinical decision support tool that can identify patients presenting to the emergency department (ED) with low-risk PE as candidates for outpatient management. METHODS Data were curated from all patients who received a PE-protocol computed tomography pulmonary angiogram (PE-CTPA) imaging study in the ED of a 3-hospital academic health system between June 1, 2018 and December 31, 2020 (n=12,183). The "preliminary" radiology reports from these imaging studies made available to ED clinicians at the time of diagnosis were adjudicated as positive or negative for PE by the clinical team. The reports were then divided into development, internal validation, and temporal validation cohorts in order to train, test, and validate an natural language processing model that could identify the presence of PE based on unstructured text. For risk stratification, patient- and encounter-level data elements were curated from the electronic health record and used to compute a real-time simplified pulmonary embolism severity (sPESI) score at the time of diagnosis. Chart abstraction was performed on all low-risk PE patients admitted for inpatient management. RESULTS When applied to the internal validation and temporal validation cohorts, the natural language processing model identified the presence of PE from radiology reports with an area under the receiver operating characteristic curve of 0.99, sensitivity of 0.86 to 0.87, and specificity of 0.99. Across cohorts, 10.5% of PE-CTPA studies were positive for PE, of which 22.2% were classified as low-risk by the sPESI score. Of all low-risk PE patients, 74.3% were admitted for inpatient management. CONCLUSION This study demonstrates that a natural language processing-based model utilizing real-time radiology reports can accurately identify patients with PE. Further, this model, used in combination with a validated risk stratification score (sPESI), provides a clinical decision support tool that accurately identifies patients in the ED with low-risk PE as candidates for outpatient management.
Collapse
Affiliation(s)
- Krunal D Amin
- Department of Medicine, Duke University School of Medicine, Durham, NC.
| | | | | | | | - Tara A Holder
- Division of Cardiology, Vanderbilt University Medical Center, Nashville, TN
| | - Cathleen Bury
- Department of Surgery, Duke University School of Medicine, Durham, NC
| | - Samuel Francis
- Department of Surgery, Duke University School of Medicine, Durham, NC
| | | | | | - Michael Gao
- Duke Institute for Health Innovation, Durham, NC
| | | | - Suresh Balu
- Duke Institute for Health Innovation, Durham, NC
| | - William Schuyler Jones
- Division of Cardiology, Department of Medicine, Duke University School of Medicine, Durham, NC
| | - Mark Sendak
- Duke Institute for Health Innovation, Durham, NC
| |
Collapse
|
2
|
Fan G, Liu H, Yang S, Luo L, Pang M, Liu B, Zhang L, Han L, Rong L, Liao X. Early Prognostication of Critical Patients With Spinal Cord Injury: A Machine Learning Study With 1485 Cases. Spine (Phila Pa 1976) 2024; 49:754-762. [PMID: 37921018 DOI: 10.1097/brs.0000000000004861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Accepted: 10/14/2023] [Indexed: 11/04/2023]
Abstract
STUDY DESIGN A retrospective case-series. OBJECTIVE The study aims to use machine learning to predict the discharge destination of spinal cord injury (SCI) patients in the intensive care unit. SUMMARY OF BACKGROUND DATA Prognostication following SCI is vital, especially for critical patients who need intensive care. PATIENTS AND METHODS Clinical data of patients diagnosed with SCI were extracted from a publicly available intensive care unit database. The first recorded data of the included patients were used to develop a total of 98 machine learning classifiers, seeking to predict discharge destination (eg, death, further medical care, home, etc.). The microaverage area under the curve (AUC) was the main indicator to assess discrimination. The best average-AUC classifier and the best death-sensitivity classifier were integrated into an ensemble classifier. The discrimination of the ensemble classifier was compared with top death-sensitivity classifiers and top average-AUC classifiers. In addition, prediction consistency and clinical utility were also assessed. RESULTS A total of 1485 SCI patients were included. The ensemble classifier had a microaverage AUC of 0.851, which was only slightly inferior to the best average-AUC classifier ( P =0.10). The best average-AUC classifier death sensitivity was much lower than that of the ensemble classifier. The ensemble classifier had a death sensitivity of 0.452, which was inferior to the top 8 death-sensitivity classifiers, whose microaverage AUC were inferior to the ensemble classifier ( P <0.05). In addition, the ensemble classifier demonstrated a comparable Brier score and superior net benefit in the DCA when compared with the performance of the origin classifiers. CONCLUSIONS The ensemble classifier shows an overall superior performance in predicting discharge destination, considering discrimination ability, prediction consistency, and clinical utility. This classifier system may aid in the clinical management of critical SCI patients in the early phase following injury. LEVEL OF EVIDENCE Level 3.
Collapse
Affiliation(s)
- Guoxin Fan
- Department of Pain Medicine, Huazhong University of Science and Technology Union Shenzhen Hospital, Shenzhen, China
- Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen, China
- Department of Spine Surgery, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Huaqing Liu
- Artificial Intelligence Innovation Center, Research Institute of Tsinghua, Pearl River Delta, Guangzhou, China
| | - Sheng Yang
- Department of Orthopedic, Spinal Pain Research Institute, Shanghai Tenth People's Hospital, Tongji University School of Medicine, Shanghai, China
| | - Libo Luo
- Department of Medical Oncology, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China
| | - Mao Pang
- Department of Spine Surgery, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Bin Liu
- Department of Spine Surgery, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Liangming Zhang
- Department of Spine Surgery, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Lanqing Han
- Artificial Intelligence Innovation Center, Research Institute of Tsinghua, Pearl River Delta, Guangzhou, China
| | - Limin Rong
- Department of Spine Surgery, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Xiang Liao
- Department of Pain Medicine, Huazhong University of Science and Technology Union Shenzhen Hospital, Shenzhen, China
- Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen, China
| |
Collapse
|
3
|
Duong A, El Gamal A, Bilodeau V, Huot J, Delorme C, Poudrette J, Crevier B, Marsot A. Vancomycin: An analysis and evaluation of eight population pharmacokinetic models for clinical application in general adult population. Pharmacotherapy 2024; 44:425-434. [PMID: 38803279 DOI: 10.1002/phar.2941] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Revised: 03/30/2024] [Accepted: 04/21/2024] [Indexed: 05/29/2024]
Abstract
INTRODUCTION Based on the recent guidelines for vancomycin therapeutic drug monitoring (TDM), the area under the curve to minimum inhibitory concentration ratio was to be employed combined with the usage of population pharmacokinetic (popPK) model for dosing adaptation. Yet, deploying these models in a clinical setting requires an external evaluation of their performance. OBJECTIVES This study aimed to evaluate existing vancomycin popPK models from the literature for the use in TDM within the general patient population in a clinical setting. METHODS The models under external evaluation were chosen based on a review of literature covering vancomycin popPK models developed in general adult populations. Patients' data were collected from Charles-Le Moyne Hospital (CLMH). The external evaluation was performed with NONMEM® (v7.5). Additional analyses such as evaluating the impact of number of samples on external evaluation, Bayesian forecasting, and a priori dosing regimen simulations were performed on the best performing model. RESULTS Eight popPK models were evaluated with an independent dataset that included 40 patients and 252 samples. The model developed by Goti and colleagues demonstrated superior performance in diagnostic plots and population predictive performance, with bias and inaccuracy values of 0.251% and 22.7%, respectively, and for individual predictive performance, bias and inaccuracy were -4.90% and 12.1%, respectively. When limiting the independent dataset to one or two samples per patient, the Goti model exhibited inadequate predictive performance for inaccuracy, with values exceeding 30%. Moreover, the Goti model is suitable for Bayesian forecasting with at least two samples as prior for the prediction of the next trough concentration. Furthermore, the vancomycin dosing regimen that would maximize therapeutic targets of area under the curve to minimum inhibitory concentration ratio (AUC24/MIC) and trough concentrations (Ctrough) for the Goti model was 20 mg/kg/dose twice daily. CONCLUSION Considering the superior predictive performance and potential for Bayesian forecasting in the Goti model, future research aims to test its applicability in clinical settings at CLMH, both in a priori and a posteriori scenario.
Collapse
Affiliation(s)
- Alexandre Duong
- Faculté de Pharmacie, Université de Montréal, Montreal, Quebec, Canada
- Laboratoire STP2, Faculté de Pharmacie, Université de Montréal, Montreal, Quebec, Canada
| | - Ahmed El Gamal
- Faculté de Pharmacie, Université de Montréal, Montreal, Quebec, Canada
| | - Véronique Bilodeau
- Département de Pharmacie, Centre intégré de santé et de services sociaux Montérégie-Est, Longueuil, Quebec, Canada
| | - Justine Huot
- Département de Pharmacie, Centre intégré de santé et de services sociaux Montérégie-Centre, Longueuil, Quebec, Canada
| | - Carole Delorme
- Département de Pharmacie, Centre intégré de santé et de services sociaux Montérégie-Centre, Longueuil, Quebec, Canada
| | - Johanne Poudrette
- Département de Pharmacie, Centre intégré de santé et de services sociaux Montérégie-Centre, Longueuil, Quebec, Canada
| | - Benoît Crevier
- Département de Pharmacie, Centre intégré de santé et de services sociaux Montérégie-Centre, Longueuil, Quebec, Canada
| | - Amélie Marsot
- Faculté de Pharmacie, Université de Montréal, Montreal, Quebec, Canada
- Laboratoire STP2, Faculté de Pharmacie, Université de Montréal, Montreal, Quebec, Canada
- Centre de recherche, CHU Sainte-Justine, Montreal, Quebec, Canada
| |
Collapse
|
4
|
Nwaiwu CA, Rivera Perla KM, Abel LB, Sears IJ, Barton AT, Peterson RC, Liu YZ, Khatri IS, Sarkar IN, Shah N. Predicting Colonic Neoplasia Surgical Complications: A Machine Learning Approach. Dis Colon Rectum 2024; 67:700-713. [PMID: 38319746 DOI: 10.1097/dcr.0000000000003166] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/08/2024]
Abstract
BACKGROUND A range of statistical approaches have been used to help predict outcomes associated with colectomy. The multifactorial nature of complications suggests that machine learning algorithms may be more accurate in determining postoperative outcomes by detecting nonlinear associations, which are not readily measured by traditional statistics. OBJECTIVE The aim of this study was to investigate the utility of machine learning algorithms to predict complications in patients undergoing colectomy for colonic neoplasia. DESIGN Retrospective analysis using decision tree, random forest, and artificial neural network classifiers to predict postoperative outcomes. SETTINGS National Inpatient Sample database (2003-2017). PATIENTS Adult patients who underwent elective colectomy with anastomosis for neoplasia. MAIN OUTCOME MEASURES Performance was quantified using sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve to predict the incidence of anastomotic leak, prolonged length of stay, and inpatient mortality. RESULTS A total of 14,935 patients (4731 laparoscopic, 10,204 open) were included. They had an average age of 67 ± 12.2 years, and 53% of patients were women. The 3 machine learning models successfully identified patients who developed the measured complications. Although differences between model performances were largely insignificant, the neural network scored highest for most outcomes: predicting anastomotic leak, area under the receiver operating characteristic curve 0.88/0.93 (open/laparoscopic, 95% CI, 0.73-0.92/0.80-0.96); prolonged length of stay, area under the receiver operating characteristic curve 0.84/0.88 (open/laparoscopic, 95% CI, 0.82-0.85/0.85-0.91); and inpatient mortality, area under the receiver operating characteristic curve 0.90/0.92 (open/laparoscopic, 95% CI, 0.85-0.96/0.86-0.98). LIMITATIONS The patients from the National Inpatient Sample database may not be an accurate sample of the population of all patients undergoing colectomy for colonic neoplasia and does not account for specific institutional and patient factors. CONCLUSIONS Machine learning predicted postoperative complications in patients with colonic neoplasia undergoing colectomy with good performance. Although validation using external data and optimization of data quality will be required, these machine learning tools show great promise in assisting surgeons with risk-stratification of perioperative care to improve postoperative outcomes. See Video Abstract . PREDICCIN DE LAS COMPLICACIONES QUIRRGICAS DE LA NEOPLASIA DE COLON UN ENFOQUE DE MODELO DE APRENDIZAJE AUTOMTICO ANTECEDENTES:Se han utilizado una variedad de enfoques estadísticos para ayudar a predecir los resultados asociados con la colectomía. La naturaleza multifactorial de las complicaciones sugiere que los algoritmos de aprendizaje automático pueden ser más precisos en determinar los resultados posoperatorios al detectar asociaciones no lineales, que generalmente no se miden en las estadísticas tradicionales.OBJETIVO:El objetivo de este estudio fue investigar la utilidad de los algoritmos de aprendizaje automático para predecir complicaciones en pacientes sometidos a colectomía por neoplasia de colon.DISEÑO:Análisis retrospectivo utilizando clasificadores de árboles de decisión, bosques aleatorios y redes neuronales artificiales para predecir los resultados posoperatorios.AJUSTE:Base de datos de la Muestra Nacional de Pacientes Hospitalizados (2003-2017).PACIENTES:Pacientes adultos sometidos a colectomía electiva con anastomosis por neoplasia.INTERVENCIONES:N/A.PRINCIPALES MEDIDAS DE RESULTADO:El rendimiento se cuantificó utilizando la sensibilidad, especificidad, precisión y la característica operativa del receptor del área bajo la curva para predecir la incidencia de fuga anastomótica, duración prolongada de la estancia hospitalaria y mortalidad de los pacientes hospitalizados.RESULTADOS:Se incluyeron un total de 14.935 pacientes (4.731 laparoscópicos, 10.204 abiertos). Presentaron una edad promedio de 67 ± 12,2 años y el 53% eran mujeres. Los tres modelos de aprendizaje automático identificaron con éxito a los pacientes que desarrollaron las complicaciones medidas. Aunque las diferencias entre el rendimiento del modelo fueron en gran medida insignificantes, la red neuronal obtuvo la puntuación más alta para la mayoría de los resultados: predicción de fuga anastomótica, característica operativa del receptor del área bajo la curva 0,88/0,93 (abierta/laparoscópica, IC del 95%: 0,73-0,92/0,80-0,96); duración prolongada de la estancia hospitalaria, característica operativa del receptor del área bajo la curva 0,84/0,88 (abierta/laparoscópica, IC del 95%: 0,82-0,85/0,85-0,91); y mortalidad de pacientes hospitalizados, característica operativa del receptor del área bajo la curva 0,90/0,92 (abierto/laparoscópico, IC del 95%: 0,85-0,96/0,86-0,98).LIMITACIONES:Los pacientes de la base de datos de la Muestra Nacional de Pacientes Hospitalizados pueden no ser una muestra precisa de la población de todos los pacientes sometidos a colectomía por neoplasia de colon y no tienen en cuenta factores institucionales y específicos del paciente.CONCLUSIONES:El aprendizaje automático predijo con buen rendimiento las complicaciones postoperatorias en pacientes con neoplasia de colon sometidos a colectomía. Aunque será necesaria la validación mediante datos externos y la optimización de la calidad de los datos, estas herramientas de aprendizaje automático son muy prometedoras para ayudar a los cirujanos con la estratificación de riesgos de la atención perioperatoria para mejorar los resultados posoperatorios. (Traducción-Dr. Fidel Ruiz Healy ).
Collapse
Affiliation(s)
- Chibueze A Nwaiwu
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | - Krissia M Rivera Perla
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
- Harvard T.H. Chan School of Public Health, Boston, Massachusetts
| | - Logan B Abel
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | - Isaac J Sears
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | - Andrew T Barton
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | | | - Yao Z Liu
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | - Ishaani S Khatri
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| | - Indra N Sarkar
- Center for Biomedical Informatics, Brown University, Providence, Rhode Island
- Rhode Island Quality Institute, Providence, Rhode Island
| | - Nishit Shah
- Department of Surgery, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, Rhode Island
| |
Collapse
|
5
|
Göndöcs D, Dörfler V. AI in medical diagnosis: AI prediction & human judgment. Artif Intell Med 2024; 149:102769. [PMID: 38462271 DOI: 10.1016/j.artmed.2024.102769] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 12/02/2023] [Accepted: 01/14/2024] [Indexed: 03/12/2024]
Abstract
AI has long been regarded as a panacea for decision-making and many other aspects of knowledge work; as something that will help humans get rid of their shortcomings. We believe that AI can be a useful asset to support decision-makers, but not that it should replace decision-makers. Decision-making uses algorithmic analysis, but it is not solely algorithmic analysis; it also involves other factors, many of which are very human, such as creativity, intuition, emotions, feelings, and value judgments. We have conducted semi-structured open-ended research interviews with 17 dermatologists to understand what they expect from an AI application to deliver to medical diagnosis. We have found four aggregate dimensions along which the thinking of dermatologists can be described: the ways in which our participants chose to interact with AI, responsibility, 'explainability', and the new way of thinking (mindset) needed for working with AI. We believe that our findings will help physicians who might consider using AI in their diagnosis to understand how to use AI beneficially. It will also be useful for AI vendors in improving their understanding of how medics want to use AI in diagnosis. Further research will be needed to examine if our findings have relevance in the wider medical field and beyond.
Collapse
Affiliation(s)
| | - Viktor Dörfler
- University of Strathclyde Business School, United Kingdom.
| |
Collapse
|
6
|
Michelson AP, Oh I, Gupta A, Puri V, Kreisel D, Gelman AE, Nava R, Witt CA, Byers DE, Halverson L, Vazquez-Guillamet R, Payne PRO, Hachem RR. Developing machine learning models to predict primary graft dysfunction after lung transplantation. Am J Transplant 2024; 24:458-467. [PMID: 37468109 DOI: 10.1016/j.ajt.2023.07.008] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 06/21/2023] [Accepted: 07/04/2023] [Indexed: 07/21/2023]
Abstract
Primary graft dysfunction (PGD) is the leading cause of morbidity and mortality in the first 30 days after lung transplantation. Risk factors for the development of PGD include donor and recipient characteristics, but how multiple variables interact to impact the development of PGD and how clinicians should consider these in making decisions about donor acceptance remain unclear. This was a single-center retrospective cohort study to develop and evaluate machine learning pipelines to predict the development of PGD grade 3 within the first 72 hours of transplantation using donor and recipient variables that are known at the time of donor offer acceptance. Among 576 bilateral lung recipients, 173 (30%) developed PGD grade 3. The cohort underwent a 75% to 25% train-test split, and lasso regression was used to identify 11 variables for model development. A K-nearest neighbor's model showing the best calibration and performance with relatively small confidence intervals was selected as the final predictive model with an area under the receiver operating characteristics curve of 0.65. Machine learning models can predict the risk for development of PGD grade 3 based on data available at the time of donor offer acceptance. This may improve donor-recipient matching and donor utilization in the future.
Collapse
Affiliation(s)
- Andrew P Michelson
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA; Institute for Informatics, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Inez Oh
- Institute for Informatics, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Aditi Gupta
- Institute for Informatics, Washington University School of Medicine, Saint Louis, Missouri, USA; Division of Biostatistics, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Varun Puri
- Division of Cardiothoracic Surgery, Department of Surgery, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Daniel Kreisel
- Division of Cardiothoracic Surgery, Department of Surgery, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Andrew E Gelman
- Division of Cardiothoracic Surgery, Department of Surgery, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Ruben Nava
- Division of Cardiothoracic Surgery, Department of Surgery, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Chad A Witt
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Derek E Byers
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Laura Halverson
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Rodrigo Vazquez-Guillamet
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Philip R O Payne
- Institute for Informatics, Washington University School of Medicine, Saint Louis, Missouri, USA
| | - Ramsey R Hachem
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University School of Medicine, Saint Louis, Missouri, USA.
| |
Collapse
|
7
|
Hanneman K, Playford D, Dey D, van Assen M, Mastrodicasa D, Cook TS, Gichoya JW, Williamson EE, Rubin GD. Value Creation Through Artificial Intelligence and Cardiovascular Imaging: A Scientific Statement From the American Heart Association. Circulation 2024; 149:e296-e311. [PMID: 38193315 DOI: 10.1161/cir.0000000000001202] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/10/2024]
Abstract
Multiple applications for machine learning and artificial intelligence (AI) in cardiovascular imaging are being proposed and developed. However, the processes involved in implementing AI in cardiovascular imaging are highly diverse, varying by imaging modality, patient subtype, features to be extracted and analyzed, and clinical application. This article establishes a framework that defines value from an organizational perspective, followed by value chain analysis to identify the activities in which AI might produce the greatest incremental value creation. The various perspectives that should be considered are highlighted, including clinicians, imagers, hospitals, patients, and payers. Integrating the perspectives of all health care stakeholders is critical for creating value and ensuring the successful deployment of AI tools in a real-world setting. Different AI tools are summarized, along with the unique aspects of AI applications to various cardiac imaging modalities, including cardiac computed tomography, magnetic resonance imaging, and positron emission tomography. AI is applicable and has the potential to add value to cardiovascular imaging at every step along the patient journey, from selecting the more appropriate test to optimizing image acquisition and analysis, interpreting the results for classification and diagnosis, and predicting the risk for major adverse cardiac events.
Collapse
|
8
|
Gericke CA. Scope of practice regulation in medicine: balancing patient safety, access to care and professional autonomy. AUST HEALTH REV 2024; 48:1-3. [PMID: 38291673 DOI: 10.1071/ah24012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Accepted: 01/18/2024] [Indexed: 02/01/2024]
Abstract
Scope of practice regulation in medicine is crucial for ensuring patient safety, access to care and professional autonomy. This paper explores the impact of scope of practice regulation on healthcare delivery, professional responsibilities and patient outcomes. It discusses the variability in standards for safe practice, the challenges in defining boundaries between medical specialties and the recent controversies in cosmetic surgery practice. The paper also examines the potential benefits and drawbacks of rigorous scope of practice regulations, including their impact on clinical innovation, flexibility and access to care. Furthermore, it delves into the implications of defensive medicine and the consequences of restrictive regulations on patient care. The author proposes implementing a proactive, national, artificial intelligence-powered, real-time outcome monitoring system to address these challenges. This system aims to cover every patient undergoing a surgical procedure and could be gradually extended to non-surgical conditions, benefiting all key stakeholders in the health system. The paper emphasises the need for a balanced approach to scope of practice regulation to avoid stifling clinical innovation and professional autonomy, while ensuring patient safety and professional accountability.
Collapse
Affiliation(s)
- Christian A Gericke
- School of Medicine and Public Health, University of Newcastle, Callaghan, NSW, Australia
| |
Collapse
|
9
|
Patel AK, Trujillo-Rivera E, Chamberlain JM, Morizono H, Pollack MM. External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients. PLoS One 2024; 19:e0288233. [PMID: 38285704 PMCID: PMC10824440 DOI: 10.1371/journal.pone.0288233] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2022] [Accepted: 06/22/2023] [Indexed: 01/31/2024] Open
Abstract
OBJECTIVE To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future. DESIGN Prognostic study comparing multi-institutional CI-D models' performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset. PARTICIPANTS All pediatric inpatients admitted from January 1st 2018 -February 29th 2020 through the emergency department. MAIN OUTCOME(S) AND MEASURE(S) The main outcome was inpatient care in routine or ICU care locations. RESULTS A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15-157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493-0.545 and area under the precision-recall curve (AUPRC) was 0.262-0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906-0.944 and AUPRC range from 0.754-0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care. CONCLUSION AND RELEVANCE Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.
Collapse
Affiliation(s)
- Anita K. Patel
- Department of Pediatrics, Division of Critical Care Medicine, Children’s National Health System, Washington, DC, United States of America
- George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America
| | - Eduardo Trujillo-Rivera
- Department of Pediatrics, Division of Critical Care Medicine, Children’s National Health System, Washington, DC, United States of America
- George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America
- Children’s National Research Institute, Children’s National Hospital, Washington, DC, United States of America
| | - James M. Chamberlain
- George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America
- Department of Pediatrics, Division of Emergency Medicine, Children’s National Hospital, Washington, DC, United States of America
| | - Hiroki Morizono
- Children’s National Research Institute, Children’s National Hospital, Washington, DC, United States of America
- Department of Genomics and Precision Medicine, George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America
| | - Murray M. Pollack
- Department of Pediatrics, Division of Critical Care Medicine, Children’s National Health System, Washington, DC, United States of America
- George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America
| |
Collapse
|
10
|
van der Vegt AH, Campbell V, Mitchell I, Malycha J, Simpson J, Flenady T, Flabouris A, Lane PJ, Mehta N, Kalke VR, Decoyna JA, Es’haghi N, Liu CH, Scott IA. Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inform Assoc 2024; 31:509-524. [PMID: 37964688 PMCID: PMC10797271 DOI: 10.1093/jamia/ocad220] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 10/27/2023] [Accepted: 10/31/2023] [Indexed: 11/16/2023] Open
Abstract
OBJECTIVE To identify factors influencing implementation of machine learning algorithms (MLAs) that predict clinical deterioration in hospitalized adult patients and relate these to a validated implementation framework. MATERIALS AND METHODS A systematic review of studies of implemented or trialed real-time clinical deterioration prediction MLAs was undertaken, which identified: how MLA implementation was measured; impact of MLAs on clinical processes and patient outcomes; and barriers, enablers and uncertainties within the implementation process. Review findings were then mapped to the SALIENT end-to-end implementation framework to identify the implementation stages at which these factors applied. RESULTS Thirty-seven articles relating to 14 groups of MLAs were identified, each trialing or implementing a bespoke algorithm. One hundred and seven distinct implementation evaluation metrics were identified. Four groups reported decreased hospital mortality, 1 significantly. We identified 24 barriers, 40 enablers, and 14 uncertainties and mapped these to the 5 stages of the SALIENT implementation framework. DISCUSSION Algorithm performance across implementation stages decreased between in silico and trial stages. Silent plus pilot trial inclusion was associated with decreased mortality, as was the use of logistic regression algorithms that used less than 39 variables. Mitigation of alert fatigue via alert suppression and threshold configuration was commonly employed across groups. CONCLUSIONS : There is evidence that real-world implementation of clinical deterioration prediction MLAs may improve clinical outcomes. Various factors identified as influencing success or failure of implementation can be mapped to different stages of implementation, thereby providing useful and practical guidance for implementers.
Collapse
Affiliation(s)
- Anton H van der Vegt
- Centre for Health Services Research, The University of Queensland, Brisbane, QLD 4102, Australia
| | - Victoria Campbell
- Intensive Care Unit, Sunshine Coast Hospital and Health Service, Birtynia, QLD 4575, Australia
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Imogen Mitchell
- Office of Research and Education, Canberra Health Services, Canberra, ACT 2601, Australia
| | - James Malycha
- Department of Critical Care Medicine, The Queen Elizabeth Hospital, Woodville, SA 5011, Australia
| | - Joanna Simpson
- Eastern Health Intensive Care Services, Eastern Health, Box Hill, VIC 3128, Australia
| | - Tracy Flenady
- School of Nursing, Midwifery & Social Sciences, Central Queensland University, Rockhampton, QLD 4701, Australia
| | - Arthas Flabouris
- Intensive Care Department, Royal Adelaide Hospital, Adelaide, SA 5000, Australia
- Adelaide Medical School, University of Adelaide, Adelaide, SA 5005, Australia
| | - Paul J Lane
- Safety Quality & Innovation, The Prince Charles Hospital, Chermside, QLD 4032, Australia
| | - Naitik Mehta
- Patient Safety and Quality, Clinical Excellence Queensland, Brisbane, QLD 4001, Australia
| | - Vikrant R Kalke
- Patient Safety and Quality, Clinical Excellence Queensland, Brisbane, QLD 4001, Australia
| | - Jovie A Decoyna
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Nicholas Es’haghi
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Chun-Huei Liu
- School of Medicine and Dentistry, Griffith University, Gold Coast, QLD 4222, Australia
| | - Ian A Scott
- Centre for Health Services Research, The University of Queensland, Brisbane, QLD 4102, Australia
- Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, QLD 4102, Australia
| |
Collapse
|
11
|
Davis SE, Matheny ME, Balu S, Sendak MP. A framework for understanding label leakage in machine learning for health care. J Am Med Inform Assoc 2023; 31:274-280. [PMID: 37669138 PMCID: PMC10746313 DOI: 10.1093/jamia/ocad178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Revised: 07/24/2023] [Accepted: 08/19/2023] [Indexed: 09/07/2023] Open
Abstract
INTRODUCTION The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule." FRAMEWORK We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice. RECOMMENDATIONS Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.
Collapse
Affiliation(s)
- Sharon E Davis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States
- Tennessee Valley Healthcare System VA Medical Center, Veterans Health Administration, Nashville, TN 37232, United States
| | - Suresh Balu
- Duke Institute for Health Innovation, Duke University School of Medicine, Durham, NC 27701, United States
| | - Mark P Sendak
- Duke Institute for Health Innovation, Duke University School of Medicine, Durham, NC 27701, United States
| |
Collapse
|
12
|
Abujaber AA, Alkhawaldeh IM, Imam Y, Nashwan AJ, Akhtar N, Own A, Tarawneh AS, Hassanat AB. Predicting 90-day prognosis for patients with stroke: a machine learning approach. Front Neurol 2023; 14:1270767. [PMID: 38145122 PMCID: PMC10748594 DOI: 10.3389/fneur.2023.1270767] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 11/23/2023] [Indexed: 12/26/2023] Open
Abstract
Background Stroke is a significant global health burden and ranks as the second leading cause of death worldwide. Objective This study aims to develop and evaluate a machine learning-based predictive tool for forecasting the 90-day prognosis of stroke patients after discharge as measured by the modified Rankin Score. Methods The study utilized data from a large national multiethnic stroke registry comprising 15,859 adult patients diagnosed with ischemic or hemorrhagic stroke. Of these, 7,452 patients satisfied the study's inclusion criteria. Feature selection was performed using the correlation and permutation importance methods. Six classifiers, including Random Forest (RF), Classification and Regression Tree, Linear Discriminant Analysis, Support Vector Machine, and k-Nearest Neighbors, were employed for prediction. Results The RF model demonstrated superior performance, achieving the highest accuracy (0.823) and excellent discrimination power (AUC 0.893). Notably, stroke type, hospital acquired infections, admission location, and hospital length of stay emerged as the top-ranked predictors. Conclusion The RF model shows promise in predicting stroke prognosis, enabling personalized care plans and enhanced preventive measures for stroke patients. Prospective validation is essential to assess its real-world clinical performance and ensure successful implementation across diverse healthcare settings.
Collapse
Affiliation(s)
| | | | - Yahia Imam
- Neurology Section, Neuroscience Institute, Hamad Medical Corporation (HMC), Doha, Qatar
| | | | - Naveed Akhtar
- Neuroradiology Department, Neuroscience Institute, Hamad Medical Corporation (HMC), Doha, Qatar
| | - Ahmed Own
- Neuroradiology Department, Neuroscience Institute, Hamad Medical Corporation (HMC), Doha, Qatar
| | - Ahmad S. Tarawneh
- Faculty of Information Technology, Mutah University, Al-Karak, Jordan
| | - Ahmad B. Hassanat
- Faculty of Information Technology, Mutah University, Al-Karak, Jordan
| |
Collapse
|
13
|
Hydoub YM, Walker AP, Kirchoff RW, Alzu'bi HM, Chipi PY, Gerberi DJ, Burton MC, Murad MH, Dugani SB. Risk Prediction Models for Hospital Mortality in General Medical Patients: A Systematic Review. AMERICAN JOURNAL OF MEDICINE OPEN 2023; 10:100044. [PMID: 38090393 PMCID: PMC10715621 DOI: 10.1016/j.ajmo.2023.100044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/13/2022] [Revised: 03/20/2023] [Accepted: 05/27/2023] [Indexed: 07/20/2024]
Abstract
Objective To systematically review contemporary prediction models for hospital mortality developed or validated in general medical patients. Methods We screened articles in five databases, from January 1, 2010, through April 7, 2022, and the bibliography of articles selected for final inclusion. We assessed the quality for risk of bias and applicability using the Prediction Model Risk of Bias Assessment Tool (PROBAST) and extracted data using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist. Two investigators independently screened each article, assessed quality, and extracted data. Results From 20,424 unique articles, we identified 15 models in 8 studies across 10 countries. The studies included 280,793 general medical patients and 19,923 hospital deaths. Models included 7 early warning scores, 2 comorbidities indices, and 6 combination models. Ten models were studied in all general medical patients (general models) and 7 in general medical patients with infection (infection models). Of the 15 models, 13 were developed using logistic or Poisson regression and 2 using machine learning methods. Also, 4 of 15 models reported on handling of missing values. None of the infection models had high discrimination, whereas 4 of 10 general models had high discrimination (area under curve >0.8). Only 1 model appropriately assessed calibration. All models had high risk of bias; 4 of 10 general models and 5 of 7 infection models had low concern for applicability for general medical patients. Conclusion Mortality prediction models for general medical patients were sparse and differed in quality, applicability, and discrimination. These models require hospital-level validation and/or recalibration in general medical patients to guide mortality reduction interventions.
Collapse
Affiliation(s)
- Yousif M. Hydoub
- Division of Cardiology, Sheikh Shakhbout Medical City, Abu Dhabi, United Arab Emirates
| | - Andrew P. Walker
- Division of Hospital Internal Medicine, Mayo Clinic, Phoenix, Ariz
- Department of Critical Care Medicine, Mayo Clinic, Phoenix, Ariz
| | - Robert W. Kirchoff
- Division of Hospital Internal Medicine, Mayo Clinic, Phoenix, Ariz
- Division of Hospital Internal Medicine, Mayo Clinic, Rochester, Minn
| | | | - Patricia Y. Chipi
- Division of Hospital Internal Medicine, Mayo Clinic, Jacksonville, Fla
| | | | | | - M. Hassan Murad
- Evidence-Based Practice Center, Mayo Clinic, Rochester, Minn
| | - Sagar B. Dugani
- Division of Hospital Internal Medicine, Mayo Clinic, Rochester, Minn
- Division of Health Care Delivery Research, Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minn
| |
Collapse
|
14
|
Stoessel D, Fa R, Artemova S, von Schenck U, Nowparast Rostami H, Madiot PE, Landelle C, Olive F, Foote A, Moreau-Gaudry A, Bosson JL. Early prediction of in-hospital mortality utilizing multivariate predictive modelling of electronic medical records and socio-determinants of health of the first day of hospitalization. BMC Med Inform Decis Mak 2023; 23:259. [PMID: 37957690 PMCID: PMC10644472 DOI: 10.1186/s12911-023-02356-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2023] [Accepted: 10/27/2023] [Indexed: 11/15/2023] Open
Abstract
BACKGROUND In France an average of 4% of hospitalized patients die during their hospital stay. To aid medical decision making and the attribution of resources, within a few days of admission the identification of patients at high risk of dying in hospital is essential. METHODS We used de-identified routine patient data available in the first 2 days of hospitalization in a French University Hospital (between 2016 and 2018) to build models predicting in-hospital mortality (at ≥ 2 and ≤ 30 days after admission). We tested nine different machine learning algorithms with repeated 10-fold cross-validation. Models were trained with 283 variables including age, sex, socio-determinants of health, laboratory test results, procedures (Classification of Medical Acts), medications (Anatomical Therapeutic Chemical code), hospital department/unit and home address (urban, rural etc.). The models were evaluated using various performance metrics. The dataset contained 123,729 admissions, of which the outcome for 3542 was all-cause in-hospital mortality and 120,187 admissions (no death reported within 30 days) were controls. RESULTS The support vector machine, logistic regression and Xgboost algorithms demonstrated high discrimination with a balanced accuracy of 0.81 (95%CI 0.80-0.82), 0.82 (95%CI 0.80-0.83) and 0.83 (95%CI 0.80-0.83) and AUC of 0.90 (95%CI 0.88-0.91), 0.90 (95%CI 0.89-0.91) and 0.90 (95%CI 0.89-0.91) respectively. The most predictive variables for in-hospital mortality in all three models were older age (greater risk), and admission with a confirmed appointment (reduced risk). CONCLUSION We propose three highly discriminating machine-learning models that could improve clinical and organizational decision making for adult patients at hospital admission.
Collapse
Affiliation(s)
- Daniel Stoessel
- Life Science Analytics, Clinical Solutions, Elsevier, Berlin, Germany
| | - Rui Fa
- Elsevier Health Analytics, London, UK
| | - Svetlana Artemova
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France
| | | | | | | | - Caroline Landelle
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France
- TIMC CNRS UMR5525, Université Grenoble Alpes, Grenoble, F-38000, France
| | - Fréderic Olive
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France
| | - Alison Foote
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France
| | - Alexandre Moreau-Gaudry
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France
- TIMC CNRS UMR5525, Université Grenoble Alpes, Grenoble, F-38000, France
| | - Jean-Luc Bosson
- Public Health Department, CHU Grenoble Alpes, Grenoble, F-38000, France.
- TIMC CNRS UMR5525, Université Grenoble Alpes, Grenoble, F-38000, France.
| |
Collapse
|
15
|
Xie F, Beukelman T, Sun D, Yun H, Curtis JR. Identifying inpatient mortality in MarketScan claims data using machine learning. Pharmacoepidemiol Drug Saf 2023; 32:1299-1305. [PMID: 37344984 DOI: 10.1002/pds.5658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 05/24/2023] [Accepted: 06/19/2023] [Indexed: 06/23/2023]
Abstract
PURPOSE Inpatient mortality is an important variable in epidemiology studies using claims data. In 2016, MarketScan data began obscuring specific hospital discharge status types for patient privacy, including inpatient deaths, by setting the values to missing. We used a machine learning approach to correctly identify hospitalizations that resulted in inpatient death using data prior to 2016. METHODS All hospitalizations from 2011 to 2015 with discharge status of missing, died, or one of the other subsequently obscured values were identified and divided into a training set and two test sets. Predictor variables included age, sex, elapsed time from hospital discharge until last observed claim and until healthcare plan disenrollment, and absence of any discharge diagnoses. Four machine learning methods were used to train statistical models and assess sensitivity and positive predictive value (PPV) for inpatient mortality. RESULTS Overall 1 307 917 hospitalizations were included. All four machine learning approaches performed well in all datasets. Random forest performed best with 88% PPV and 93% sensitivity for the training set and both test sets. The two factors with the highest relative importance for identifying inpatient mortality were having no observed claims for the patient on days 2-91 following hospital discharge and patient disenrollment from the healthcare plan within 60 days following hospital discharge. CONCLUSION We successfully developed machine learning algorithms to identify inpatient mortality. This approach can be applied to obscured data to accurately identify inpatient mortality among hospitalizations with missing discharge status.
Collapse
Affiliation(s)
- Fenglong Xie
- Department of Medicine, Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, Alabama, USA
- Foundation for Science, Technology, Education, and Research (FASTER), Birmingham, Alabama, USA
| | - Timothy Beukelman
- Foundation for Science, Technology, Education, and Research (FASTER), Birmingham, Alabama, USA
| | - Dongmei Sun
- Department of Medicine, Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, Alabama, USA
| | - Huifeng Yun
- Department of Medicine, Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, Alabama, USA
| | - Jeffrey R Curtis
- Department of Medicine, Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, Alabama, USA
- Foundation for Science, Technology, Education, and Research (FASTER), Birmingham, Alabama, USA
| |
Collapse
|
16
|
Le JP, Shashikumar SP, Malhotra A, Nemati S, Wardi G. Making the Improbable Possible: Generalizing Models Designed for a Syndrome-Based, Heterogeneous Patient Landscape. Crit Care Clin 2023; 39:751-768. [PMID: 37704338 PMCID: PMC10758922 DOI: 10.1016/j.ccc.2023.02.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2023]
Abstract
Syndromic conditions, such as sepsis, are commonly encountered in the intensive care unit. Although these conditions are easy for clinicians to grasp, these conditions may limit the performance of machine-learning algorithms. Individual hospital practice patterns may limit external generalizability. Data missingness is another barrier to optimal algorithm performance and various strategies exist to mitigate this. Recent advances in data science, such as transfer learning, conformal prediction, and continual learning, may improve generalizability of machine-learning algorithms in critically ill patients. Randomized trials with these approaches are indicated to demonstrate improvements in patient-centered outcomes at this point.
Collapse
Affiliation(s)
- Joshua Pei Le
- School of Medicine, University of Limerick, Castletroy, Co, Limerick V94 T9PX, Ireland
| | | | - Atul Malhotra
- Division of Pulmonary, Critical Care and Sleep Medicine, University of California San Diego, San Diego, CA, USA
| | - Shamim Nemati
- Division of Biomedical Informatics, University of California San Diego, San Diego, CA, USA
| | - Gabriel Wardi
- Division of Pulmonary, Critical Care and Sleep Medicine, University of California San Diego, San Diego, CA, USA; Department of Emergency Medicine, University of California San Diego, 200 W Arbor Drive, San Diego, CA 92103, USA.
| |
Collapse
|
17
|
Patton MJ, Liu VX. Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges. Crit Care Clin 2023; 39:647-673. [PMID: 37704332 DOI: 10.1016/j.ccc.2023.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2023]
Abstract
The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.
Collapse
Affiliation(s)
- Michael J Patton
- Medical Scientist Training Program, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA; Hugh Kaul Precision Medicine Institute at the University of Alabama at Birmingham, 720 20th Street South, Suite 202, Birmingham, Alabama, 35233, USA.
| | - Vincent X Liu
- Kaiser Permanente Division of Research, Oakland, CA, USA.
| |
Collapse
|
18
|
Schlichte L, Setji N, Walter J, Acker Y, Casarett D, Pollak KI, Steinhauser K, Check DK, Lakis K, Schmid L, Ma JE. The Use of Templates for Documenting Advance Care Planning Conversations: A Descriptive Analysis. J Pain Symptom Manage 2023; 66:123-136. [PMID: 37080478 DOI: 10.1016/j.jpainsymman.2023.04.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 04/09/2023] [Accepted: 04/12/2023] [Indexed: 04/22/2023]
Abstract
CONTEXT While professional societies and expert panels have recommended quality indicators related to advance care planning (ACP) documentation, including using structured documentation templates, it is unclear how clinicians document these conversations. OBJECTIVE To explore how clinicians document ACP, specifically, which components of these conversations are documented. METHODS A codebook was developed based on existing frameworks for ACP conversations and documentation. ACP documentation from a hospital medicine quality improvement project conducted from November 2019 to April 2021 were included and assessed. Documentation was examined for the presence or absence of each component within the coding schema. Clinician documented ACP using three different note types: template (only template prompts were used), template plus (authors added additional text to the template), and free text only. ACP note components were analyzed by note type and author department. RESULTS A total of 182 ACP notes were identified and reviewed. The most common note type was template plus (58%), followed by free text (28%) and template (14%). The most frequent components across all note types were: important relationships to patient (92%), and discussion of life-sustaining treatment preferences (87%). There was considerable heterogeneity in the components across note types. The presence of components focused on treatment decisions and legal paperwork differed significantly between note types (P < 0.05). Components on preference for medical information, emotional state, or spiritual support were rarely included across all note types. CONCLUSION This study provides a preliminary exploration of ACP documentation and found that templates may influence what information is documented after an ACP conversation.
Collapse
Affiliation(s)
- Lindsay Schlichte
- Duke University School of Medicine (L.S.), Durham, North Carolina, USA
| | - Noppon Setji
- Division of General Internal Medicine, Department of Medicine (N.S., J.W., D.C., K.S., J.M.), Duke University School of Medicine, Durham, North Carolina, USA
| | - Jonathan Walter
- Division of General Internal Medicine, Department of Medicine (N.S., J.W., D.C., K.S., J.M.), Duke University School of Medicine, Durham, North Carolina, USA
| | - Yvonne Acker
- Patient Safety and Quality, Duke University Health System (Y.A.), Durham, North Carolina, USA
| | - David Casarett
- Division of General Internal Medicine, Department of Medicine (N.S., J.W., D.C., K.S., J.M.), Duke University School of Medicine, Durham, North Carolina, USA
| | - Kathryn I Pollak
- Department of Population Health Sciences (K.I.P., K.S., D.K.C.), Duke University, Durham, North Carolina, USA; Cancer Prevention and Control (K.I.P., K.S.), Duke Cancer Institute, Durham, North Carolina, USA; Department of Family Medicine and Community Health (K.I.P.), Duke University School of Medicine, Durham, North Carolina, USA
| | - Karen Steinhauser
- Division of General Internal Medicine, Department of Medicine (N.S., J.W., D.C., K.S., J.M.), Duke University School of Medicine, Durham, North Carolina, USA; Department of Population Health Sciences (K.I.P., K.S., D.K.C.), Duke University, Durham, North Carolina, USA; Cancer Prevention and Control (K.I.P., K.S.), Duke Cancer Institute, Durham, North Carolina, USA; Center of Innovation to Accelerate Discovery and Practice Transformation (ADAPT) (K.S.), Durham VA Health Care System, Durham, North Carolina, USA
| | - Devon K Check
- Department of Population Health Sciences (K.I.P., K.S., D.K.C.), Duke University, Durham, North Carolina, USA
| | - Kristen Lakis
- Office of Culture and Wellbeing Hub (K.L.), Duke University Health System, Durham North Carolina, USA
| | - Lorrie Schmid
- Social Sciences Research Institute (L.S.), Duke University, Durham, North Carolina, USA
| | - Jessica E Ma
- Division of General Internal Medicine, Department of Medicine (N.S., J.W., D.C., K.S., J.M.), Duke University School of Medicine, Durham, North Carolina, USA; Geriatric Research Education and Clinical Center (J.M.), Durham VA Health System, Duke University School of Medicine, Durham, North Carolina, USA.
| |
Collapse
|
19
|
Jang W, Choi YS, Kim JY, Yon DK, Lee YJ, Chung SH, Kim CY, Yeo SG, Lee J. Artificial Intelligence-Driven Respiratory Distress Syndrome Prediction for Very Low Birth Weight Infants: Korean Multicenter Prospective Cohort Study. J Med Internet Res 2023; 25:e47612. [PMID: 37428525 PMCID: PMC10366668 DOI: 10.2196/47612] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 05/04/2023] [Accepted: 06/14/2023] [Indexed: 07/11/2023] Open
Abstract
BACKGROUND Respiratory distress syndrome (RDS) is a disease that commonly affects premature infants whose lungs are not fully developed. RDS results from a lack of surfactant in the lungs. The more premature the infant is, the greater is the likelihood of having RDS. However, even though not all premature infants have RDS, preemptive treatment with artificial pulmonary surfactant is administered in most cases. OBJECTIVE We aimed to develop an artificial intelligence model to predict RDS in premature infants to avoid unnecessary treatment. METHODS In this study, 13,087 very low birth weight infants who were newborns weighing less than 1500 grams were assessed in 76 hospitals of the Korean Neonatal Network. To predict RDS in very low birth weight infants, we used basic infant information, maternity history, pregnancy/birth process, family history, resuscitation procedure, and test results at birth such as blood gas analysis and Apgar score. The prediction performances of 7 different machine learning models were compared, and a 5-layer deep neural network was proposed in order to enhance the prediction performance from the selected features. An ensemble approach combining multiple models from the 5-fold cross-validation was subsequently developed. RESULTS Our proposed ensemble 5-layer deep neural network consisting of the top 20 features provided high sensitivity (83.03%), specificity (87.50%), accuracy (84.07%), balanced accuracy (85.26%), and area under the curve (0.9187). Based on the model that we developed, a public web application that enables easy access for the prediction of RDS in premature infants was deployed. CONCLUSIONS Our artificial intelligence model may be useful for preparations for neonatal resuscitation, particularly in cases involving the delivery of very low birth weight infants, as it can aid in predicting the likelihood of RDS and inform decisions regarding the administration of surfactant.
Collapse
Affiliation(s)
- Woocheol Jang
- Biomedical Engineering, Kyung Hee University, Yongin-si, Republic of Korea
| | - Yong Sung Choi
- Department of Pediatrics, Kyung Hee University Medical Center, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Ji Yoo Kim
- Department of Pediatrics, Kyung Hee University Medical Center, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Dong Keon Yon
- Department of Pediatrics, Kyung Hee University Medical Center, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Young Joo Lee
- Department of Obstetrics and Gynecology, Kyung Hee University Medical Center, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Sung-Hoon Chung
- Department of Pediatrics, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Chae Young Kim
- Department of Pediatrics, Kyung Hee University Hospital at Gangdong, Kyung Hee University College of Medicine, Seoul, Republic of Korea
| | - Seung Geun Yeo
- Department of Otorhinolaryngology Head and Neck Surgery, Kyung Hee University Medical Center, Kyung Hee University School of Medicine, Seoul, Republic of Korea
| | - Jinseok Lee
- Biomedical Engineering, Kyung Hee University, Yongin-si, Republic of Korea
| |
Collapse
|
20
|
Kamineni M, Ötleş E, Oh J, Rao K, Young VB, Li BY, West LR, Hooper DC, Shenoy ES, Guttag JG, Wiens J, Makar M. Prospective evaluation of data-driven models to predict daily risk of Clostridioides difficile infection at 2 large academic health centers. Infect Control Hosp Epidemiol 2023; 44:1163-1166. [PMID: 36120815 PMCID: PMC10024639 DOI: 10.1017/ice.2022.218] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
Many data-driven patient risk stratification models have not been evaluated prospectively. We performed and compared the prospective and retrospective evaluations of 2 Clostridioides difficile infection (CDI) risk-prediction models at 2 large academic health centers, and we discuss the models' robustness to data-set shifts.
Collapse
Affiliation(s)
- Meghana Kamineni
- Electrical Engineering and Computer Science Department, Massachusetts Institute of Technology, Cambridge, Massachusetts
| | - Erkin Ötleş
- Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor, Michigan
- Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan
| | - Jeeheh Oh
- Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan
| | - Krishna Rao
- Department of Internal Medicine, Division of Infectious Diseases, University of Michigan Medical School, Ann Arbor, Michigan
| | - Vincent B Young
- Department of Internal Medicine, Division of Infectious Diseases, University of Michigan Medical School, Ann Arbor, Michigan
| | - Benjamin Y Li
- Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor, Michigan
- Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan
| | - Lauren R West
- Infection Control Unit, Massachusetts General Hospital, Boston, Massachusetts
| | - David C Hooper
- Infection Control Unit, Massachusetts General Hospital, Boston, Massachusetts
- Division of Infectious Diseases, Massachusetts General Hospital, Boston, Massachusetts
- Harvard Medical School, Boston, Massachusetts
| | - Erica S Shenoy
- Infection Control Unit, Massachusetts General Hospital, Boston, Massachusetts
- Division of Infectious Diseases, Massachusetts General Hospital, Boston, Massachusetts
- Harvard Medical School, Boston, Massachusetts
| | - John G Guttag
- Electrical Engineering and Computer Science Department, Massachusetts Institute of Technology, Cambridge, Massachusetts
| | - Jenna Wiens
- Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan
| | - Maggie Makar
- Electrical Engineering and Computer Science Department, Massachusetts Institute of Technology, Cambridge, Massachusetts
- Division of Computer Science and Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan
| |
Collapse
|
21
|
Cai W, Xu J, Chen Y, Wu X, Zeng Y, Yu F. Performance of Machine Learning Algorithms for Predicting Disease Activity in Inflammatory Bowel Disease. Inflammation 2023:10.1007/s10753-023-01827-0. [PMID: 37171693 DOI: 10.1007/s10753-023-01827-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Revised: 04/17/2023] [Accepted: 04/24/2023] [Indexed: 05/13/2023]
Abstract
This study aimed to explore the effectiveness of predicting disease activity in patients with inflammatory bowel disease (IBD), using machine learning (ML) models. A retrospective research was undertaken on IBD patients who were admitted into the First Affiliated Hospital of Wenzhou Medical University between September 2011 and September 2019. At first, data were randomly split into a 3:1 ratio of training to test set. The least absolute shrinkage and selection operator (LASSO) algorithm was applied to reduce the dimension of variables. These variables were used to generate seven ML algorithms, namely random forests (RFs), adaptive boosting (AdaBoost), K-nearest neighbors (KNNs), support vector machines (SVMs), naïve Bayes (NB), ridge regression, and eXtreme gradient boosting (XGBoost) to train to predict disease activity in IBD patients. SHapley Additive exPlanation (SHAP) analysis was performed to rank variable importance. A total of 876 participants with IBD, consisting of 275 ulcerative colitis (UC) and 601 Crohn's disease (CD), were retrospectively enrolled in the study. Thirty-three variables were obtained from the clinical characteristics and laboratory tests of the participants. Finally, after LASSO analysis, 11 and 5 variables were screened out to construct ML models for CD and UC, respectively. All seven ML models performed well in predicting disease activity in the CD and UC test sets. Among these ML models, SVM was more effective in predicting disease activity in the CD group, whose AUC reached 0.975, sensitivity 0.947, specificity 0.920, and accuracy 0.933. AdaBoost performed best for the UC group, with an AUC of 0.911, sensitivity 0.844, specificity 0.875, and accuracy 0.855. ML algorithms were available and capable of predicting disease activity in IBD patients. Based on clinical and laboratory variables, ML algorithms demonstrate great promise in guiding physicians' decision-making.
Collapse
Affiliation(s)
- Weimin Cai
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Jun Xu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Yihan Chen
- Department of Gastroenterology and Hepatology, Wenzhou Central Hospital, Wenzhou, 325000, China
| | - Xiao Wu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Yuan Zeng
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China
| | - Fujun Yu
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital of Wenzhou Medical University, No. 2, Fuxue Lane, Wenzhou, 325000, China.
| |
Collapse
|
22
|
Ledziński Ł, Grześk G. Artificial Intelligence Technologies in Cardiology. J Cardiovasc Dev Dis 2023; 10:jcdd10050202. [PMID: 37233169 DOI: 10.3390/jcdd10050202] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 05/03/2023] [Accepted: 05/04/2023] [Indexed: 05/27/2023] Open
Abstract
As the world produces exabytes of data, there is a growing need to find new methods that are more suitable for dealing with complex datasets. Artificial intelligence (AI) has significant potential to impact the healthcare industry, which is already on the road to change with the digital transformation of vast quantities of information. The implementation of AI has already achieved success in the domains of molecular chemistry and drug discoveries. The reduction in costs and in the time needed for experiments to predict the pharmacological activities of new molecules is a milestone in science. These successful applications of AI algorithms provide hope for a revolution in healthcare systems. A significant part of artificial intelligence is machine learning (ML), of which there are three main types-supervised learning, unsupervised learning, and reinforcement learning. In this review, the full scope of the AI workflow is presented, with explanations of the most-often-used ML algorithms and descriptions of performance metrics for both regression and classification. A brief introduction to explainable artificial intelligence (XAI) is provided, with examples of technologies that have developed for XAI. We review important AI implementations in cardiology for supervised, unsupervised, and reinforcement learning and natural language processing, emphasizing the used algorithm. Finally, we discuss the need to establish legal, ethical, and methodical requirements for the deployment of AI models in medicine.
Collapse
Affiliation(s)
- Łukasz Ledziński
- Department of Cardiology and Clinical Pharmacology, Faculty of Health Sciences, Collegium Medicum in Bydgoszcz, Nicolaus Copernicus University in Toruń, Ujejskiego 75, 85-168 Bydgoszcz, Poland
| | - Grzegorz Grześk
- Department of Cardiology and Clinical Pharmacology, Faculty of Health Sciences, Collegium Medicum in Bydgoszcz, Nicolaus Copernicus University in Toruń, Ujejskiego 75, 85-168 Bydgoszcz, Poland
| |
Collapse
|
23
|
Sandhu S, Sendak MP, Ratliff W, Knechtle W, Fulkerson WJ, Balu S. Accelerating health system innovation: principles and practices from the Duke Institute for Health Innovation. PATTERNS (NEW YORK, N.Y.) 2023; 4:100710. [PMID: 37123436 PMCID: PMC10140606 DOI: 10.1016/j.patter.2023.100710] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
The Duke Institute for Health Innovation (DIHI) was launched in 2013. Frontline staff members submit proposals for innovation projects that align with strategic priorities set by organizational leadership. Funded projects receive operational and technical support from institute staff members and a transdisciplinary network of collaborators to develop and implement solutions as part of routine clinical care, ranging from machine learning algorithms to mobile applications. DIHI's operations are shaped by four guiding principles: build to show value, build to integrate, build to scale, and build responsibly. Between 2013 and 2021, more than 600 project proposals have been submitted to DIHI. More than 85 innovation projects, both through the application process and other strategic partnerships, have been supported and implemented. DIHI's funding has incubated 12 companies, engaged more than 300 faculty members, staff members, and students, and contributed to more than 50 peer-reviewed publications. DIHI's practices can serve as a model for other health systems to systematically source, develop, implement, and scale innovations.
Collapse
Affiliation(s)
- Sahil Sandhu
- Duke Institute for Health Innovation, Durham, NC, USA
- Harvard Medical School, Boston, MA, USA
| | | | | | | | - William J. Fulkerson
- Duke University School of Medicine, Durham, NC, USA
- Duke University Health System, Durham, NC, USA
| | - Suresh Balu
- Duke Institute for Health Innovation, Durham, NC, USA
- Duke University School of Medicine, Durham, NC, USA
- Corresponding author
| |
Collapse
|
24
|
Chi S, Kim S, Reuter M, Ponzillo K, Oliver DP, Foraker R, Heard K, Liu J, Pitzer K, White P, Moore N. Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm. JAMA Netw Open 2023; 6:e238795. [PMID: 37071421 PMCID: PMC10114011 DOI: 10.1001/jamanetworkopen.2023.8795] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Accepted: 02/28/2023] [Indexed: 04/19/2023] Open
Abstract
Importance Goal-concordant care is an ongoing challenge in hospital settings. Identification of high mortality risk within 30 days may call attention to the need to have serious illness conversations, including the documentation of patient goals of care. Objective To examine goals of care discussions (GOCDs) in a community hospital setting with patients identified as having a high risk of mortality by a machine learning mortality prediction algorithm. Design, Setting, and Participants This cohort study took place at community hospitals within 1 health care system. Participants included adult patients with a high risk of 30-day mortality who were admitted to 1 of 4 hospitals between January 2 and July 15, 2021. Patient encounters of inpatients in the intervention hospital where physicians were notified of the computed high risk mortality score were compared with patient encounters of inpatients in 3 community hospitals without the intervention (ie, matched control). Intervention Physicians of patients with a high risk of mortality within 30 days received notification and were encouraged to arrange for GOCDs. Main Outcomes and Measures The primary outcome was the percentage change of documented GOCDs prior to discharge. Propensity-score matching was completed on a preintervention and postintervention period using age, sex, race, COVID-19 status, and machine learning-predicted mortality risk scores. A difference-in-difference analysis validated the results. Results Overall, 537 patients were included in this study with 201 in the preintervention period (94 in the intervention group; 104 in the control group) and 336 patients in the postintervention period. The intervention and control groups included 168 patients per group and were well-balanced in age (mean [SD], 79.3 [9.60] vs 79.6 [9.21] years; standardized mean difference [SMD], 0.03), sex (female, 85 [51%] vs 85 [51%]; SMD, 0), race (White patients, 145 [86%] vs 144 [86%]; SMD 0.006), and Charlson comorbidities (median [range], 8.00 [2.00-15.0] vs 9.00 [2.00 to 19.0]; SMD, 0.34). Patients in the intervention group from preintervention to postintervention period were associated with being 5 times more likely to have documented GOCDs (OR, 5.11 [95% CI, 1.93 to 13.42]; P = .001) by discharge compared with matched controls, and GOCD occurred significantly earlier in the hospitalization in the intervention patients as compared with matched controls (median, 4 [95% CI, 3 to 6] days vs 16 [95% CI, 15 to not applicable] days; P < .001). Similar findings were observed for Black patient and White patient subgroups. Conclusions and Relevance In this cohort study, patients whose physicians had knowledge of high-risk predictions from machine learning mortality algorithms were associated with being 5 times more likely to have documented GOCDs than matched controls. Additional external validation is needed to determine if similar interventions would be helpful at other institutions.
Collapse
Affiliation(s)
- Stephen Chi
- Division of Pulmonary and Critical Care Medicine, Washington University in St Louis, St Louis, Missouri
| | - Seunghwan Kim
- Institute for Informatics, Washington University in St Louis, St Louis, Missouri
| | | | | | - Debra Parker Oliver
- Division of Palliative Medicine, Department of Medicine, Washington University in St Louis, St Louis, Missouri
| | - Randi Foraker
- Institute for Informatics, Washington University in St Louis, St Louis, Missouri
| | | | - Jingxia Liu
- Division of Public Health Sciences, Department of Surgery, Washington University in St Louis, St Louis, Missouri
- Division of Biostatistics, Washington University in St Louis, St Louis, Missouri
| | - Kyle Pitzer
- Division of Palliative Medicine, Department of Medicine, Washington University in St Louis, St Louis, Missouri
- Division of Biostatistics, Washington University in St Louis, St Louis, Missouri
| | - Patrick White
- Division of Palliative Medicine, Department of Medicine, Washington University in St Louis, St Louis, Missouri
| | | |
Collapse
|
25
|
Wang X, Claggett BL, Tian L, Malachias MVB, Pfeffer MA, Wei LJ. Quantifying and Interpreting the Prediction Accuracy of Models for the Time of a Cardiovascular Event-Moving Beyond C Statistic: A Review. JAMA Cardiol 2023; 8:290-295. [PMID: 36723915 PMCID: PMC10660575 DOI: 10.1001/jamacardio.2022.5279] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
Importance For personalized or stratified medicine, it is critical to establish a reliable and efficient prediction model for a clinical outcome of interest. The goal is to develop a parsimonious model with fewer predictors for broad future application without compromising predictability. A general approach is to construct various empirical models via individual patients' specific baseline characteristics/biomarkers and then evaluate their relative merits. When the outcome of interest is the timing of a cardiovascular event, a commonly used metric to assess the adequacy of the fitted models is based on C statistics. These measures quantify a model's ability to separate those who develop events earlier from those who develop them later or not at all (discrimination), but they do not measure how closely model estimates match observed outcomes (prediction accuracy). Metrics that provide clinically interpretable measures to quantify prediction accuracy are needed. Observations C statistics measure the concordance between the risk scores derived from the model and the observed event time observations. However, C statistics do not quantify the model prediction accuracy. The integrated Brier Score, which calculates the mean squared distance between the empirical cumulative event-free curve and its individual patient's counterparts, estimates the prediction accuracy, but it is not clinically intuitive. A simple alternative measure is the average distance between the observed and predicted event times over the entire study population. This metric directly quantifies the model prediction accuracy and has often been used to evaluate the goodness of fit of the assumed models in settings other than survival data. This time-scale measure is easier to interpret than the C statistics or the Brier score. Conclusions and Relevance This article enhances our understanding of the model selection/evaluation process with respect to prediction accuracy. A simple, intuitive measure for quantifying such accuracy beyond C statistics can improve the reliability and efficiency of the selected model for personalized and stratified medicine.
Collapse
Affiliation(s)
- Xuan Wang
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Harvard University, Boston, Massachusetts
| | - Brian Lee Claggett
- Cardiovascular Division, Brigham and Women's Hospital, Harvard Medical School, Harvard University, Boston, Massachusetts
| | - Lu Tian
- Department of Biomedical Data Science, Stanford University, Stanford, California
| | | | - Marc A Pfeffer
- Cardiovascular Division, Brigham and Women's Hospital, Harvard Medical School, Harvard University, Boston, Massachusetts
| | - Lee-Jen Wei
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Harvard University, Boston, Massachusetts
| |
Collapse
|
26
|
Carra G, Güiza F, Piper I, Citerio G, Maas A, Depreitere B, Meyfroidt G. Development and External Validation of a Machine Learning Model for the Early Prediction of Doses of Harmful Intracranial Pressure in Patients with Severe Traumatic Brain Injury. J Neurotrauma 2023; 40:514-522. [PMID: 35950615 DOI: 10.1089/neu.2022.0251] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
Treatment and prevention of elevated intracranial pressure (ICP) is crucial in patients with severe traumatic brain injury (TBI). Elevated ICP is associated with secondary brain injury, and both intensity and duration of an episode of intracranial hypertension, often referred to as "ICP dose," are associated with worse outcomes. Prediction of such harmful episodes of ICP dose could allow for a more proactive and preventive management of TBI, with potential implications on patients' outcomes. The goal of this study was to develop and validate a machine-learning (ML) model to predict potentially harmful ICP doses in patients with severe TBI. The prediction target was defined based on previous studies and included a broad range of doses of elevated ICP that have been associated with poor long-term neurological outcomes. The ML models were used, with minute-by-minute ICP and mean arterial blood pressure signals as inputs. Harmful ICP episodes were predicted with a 30 min forewarning. Models were developed in a multi-center dataset of 290 adult patients with severe TBI and externally validated on 264 patients from the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) dataset. The external validation of the prediction model on the CENTER-TBI dataset demonstrated good discrimination and calibration (area under the curve: 0.94, accuracy: 0.89, precision: 0.87, sensitivity: 0.78, specificity: 0.94, calibration-in-the-large: 0.03, calibration slope: 0.93). The proposed prediction model provides accurate and timely predictions of harmful doses of ICP on the development and external validation dataset. A future interventional study is needed to assess whether early intervention on the basis of ICP dose predictions will result in improved outcomes.
Collapse
Affiliation(s)
- Giorgia Carra
- Clinical Division and Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine, and UZ Leuven and KU Leuven, Leuven, Belgium
| | - Fabian Güiza
- Clinical Division and Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine, and UZ Leuven and KU Leuven, Leuven, Belgium
| | - Ian Piper
- Intensive Care Monitoring Research and Paediatric Critical Care, Edinburgh Royal Hospital for Children and Young People, Edinburgh, United Kingdom
| | - Giuseppe Citerio
- University of Milan-Bicocca, School of Medicine and Surgery and Department of Emergency and Intensive Care, Monza, Italy
| | - Andrew Maas
- Department of Neurosurgery, Antwerp University Hospital and University of Antwerp, Edegem, Belgium
| | - Bart Depreitere
- Department of Neurosurgery, UZ Leuven and KU Leuven, Leuven, Belgium
| | - Geert Meyfroidt
- Clinical Division and Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine, and UZ Leuven and KU Leuven, Leuven, Belgium
| | | |
Collapse
|
27
|
Guan J, Leung E, Kwok KO, Chen FY. A hybrid machine learning framework to improve prediction of all-cause rehospitalization among elderly patients in Hong Kong. BMC Med Res Methodol 2023; 23:14. [PMID: 36639745 PMCID: PMC9837949 DOI: 10.1186/s12874-022-01824-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2022] [Accepted: 12/19/2022] [Indexed: 01/15/2023] Open
Abstract
BACKGROUND Accurately estimating elderly patients' rehospitalisation risk benefits clinical decisions and service planning. However, research in rehospitalisation and repeated hospitalisation yielded only models with modest performance, and the model performance deteriorates rapidly as the prediction timeframe expands beyond 28 days and for older participants. METHODS A temporal zero-inflated Poisson (tZIP) regression model was developed and validated retrospectively and prospectively. The data of the electronic health records (EHRs) contain cohorts (aged 60+) in a major public hospital in Hong Kong. Two temporal offset functions accounted for the associations between exposure time and parameters corresponding to the zero-inflated logistic component and the Poisson distribution's expected count. tZIP was externally validated with a retrospective cohort's rehospitalisation events up to 12 months after the discharge date. Subsequently, tZIP was validated prospectively after piloting its implementation at the study hospital. Patients discharged within the pilot period were tagged, and the proposed model's prediction of their rehospitalisation was verified monthly. Using a hybrid machine learning (ML) approach, the tZIP-based risk estimator's marginal effect on 28-day rehospitalisation was further validated, competing with other factors representing different post-acute and clinical statuses. RESULTS The tZIP prediction of rehospitalisation from 28 days to 365 days was achieved at above 80% discrimination accuracy retrospectively and prospectively in two out-of-sample cohorts. With a large margin, it outperformed the Cox proportional and linear models built with the same predictors. The hybrid ML revealed that the risk estimator's contribution to 28-day rehospitalisation outweighed other features relevant to service utilisation and clinical status. CONCLUSIONS A novel rehospitalisation risk model was introduced, and its risk estimators, whose importance outweighed all other factors of diverse post-acute care and clinical conditions, were derived. The proposed approach relies on four easily accessible variables easily extracted from EHR. Thus, clinicians could visualise patients' rehospitalisation risk from 28 days to 365 days after discharge and screen high-risk older patients for follow-up care at the proper time.
Collapse
Affiliation(s)
| | - Eman Leung
- grid.10784.3a0000 0004 1937 0482JC School of Public Health and Primary Care, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Kin-on Kwok
- grid.10784.3a0000 0004 1937 0482JC School of Public Health and Primary Care, The Chinese University of Hong Kong, Hong Kong SAR, China ,grid.10784.3a0000 0004 1937 0482Stanley Ho Centre for Emerging Infectious Diseases, The Chinese University of Hong Kong, Hong Kong SAR, China ,grid.10784.3a0000 0004 1937 0482Hong Kong Institute of Asia-Pacific Studies, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Frank Youhua Chen
- Department of Management Sciences, City University of Hong Kong, Hong Kong SAR, China.
| |
Collapse
|
28
|
Debnath S, Koppel R, Saadi N, Potak D, Weinberger B, Zanos TP. Prediction of intrapartum fever using continuously monitored vital signs and heart rate variability. Digit Health 2023; 9:20552076231187594. [PMID: 37448783 PMCID: PMC10336767 DOI: 10.1177/20552076231187594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 06/23/2023] [Indexed: 07/15/2023] Open
Abstract
Objectives Neonatal early onset sepsis (EOS), bacterial infection during the first seven days of life, is difficult to diagnose because presenting signs are non-specific, but early diagnosis before birth can direct life-saving treatment for mother and baby. Specifically, maternal fever during labor from placental infection is the strongest predictor of EOS. Alterations in maternal heart rate variability (HRV) may precede development of intrapartum fever, enabling incipient EOS detection. The objective of this work was to build a predictive model for intrapartum fever. Methods Continuously measured temperature, heart rate, and beat-to-beat RR intervals were obtained from wireless sensors on women (n = 141) in labor; traditional manual vital signs were taken every 3-6 hours. Validated measures of HRV were calculated in moving 5-minute windows of RR intervals: standard deviation of normal-to-normal intervals (SDNN) and root mean square of successive differences (RMSSD) between normal heartbeats. Results Fever (>38.0 °C) was detected by manual or continuous measurements in 48 women. Compared to afebrile mothers, average SDNN and RMSSD in febrile mothers decreased significantly (p < 0.001) at 2 and 3 hours before fever onset, respectively. This observed HRV divergence and raw recorded vitals were applied to a logistic regression model at various time horizons, up to 4-5 hours before fever onset. Model performance increased with decreasing time horizons, and a model built using continuous vital signs as input variables consistently outperformed a model built from episodic vital signs. Conclusions HRV-based predictive models could identify mothers at risk for fever and infants at risk for EOS, guiding maternal antibiotic prophylaxis and neonatal monitoring.
Collapse
Affiliation(s)
- Shubham Debnath
- Institute of Health System Science, Feinstein Institutes for Medical Research, Manhasset, NY, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Manhasset, NY, USA
| | - Robert Koppel
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
- Neonatal-Perinatal Medicine, Cohen Children's Medical Center, Queens, NY, USA
| | - Nafeesa Saadi
- Neonatal-Perinatal Medicine, Cohen Children's Medical Center, Queens, NY, USA
| | - Debra Potak
- Neonatal-Perinatal Medicine, Cohen Children's Medical Center, Queens, NY, USA
| | - Barry Weinberger
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
- Neonatal-Perinatal Medicine, Cohen Children's Medical Center, Queens, NY, USA
| | - Theodoros P Zanos
- Institute of Health System Science, Feinstein Institutes for Medical Research, Manhasset, NY, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Manhasset, NY, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| |
Collapse
|
29
|
Levy TJ, Coppa K, Cang J, Barnaby DP, Paradis MD, Cohen SL, Makhnevich A, van Klaveren D, Kent DM, Davidson KW, Hirsch JS, Zanos TP. Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients. Nat Commun 2022; 13:6812. [PMID: 36357420 PMCID: PMC9648888 DOI: 10.1038/s41467-022-34646-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 11/02/2022] [Indexed: 11/12/2022] Open
Abstract
Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.
Collapse
Affiliation(s)
- Todd J Levy
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
| | - Kevin Coppa
- Clinical Digital Solutions, Northwell Health, New Hyde Park, NY, 11042, USA
| | - Jinxuan Cang
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
| | - Douglas P Barnaby
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Marc D Paradis
- Northwell Holdings, Northwell Health, Manhasset, NY, 11030, USA
| | - Stuart L Cohen
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Alex Makhnevich
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - David van Klaveren
- Department of Public Health, Erasmus MC University Medical Center, Rotterdam, Netherlands
- Predictive Analytics and Comparative Effectiveness Center, Tufts Medical Center, Boston, MA, USA
| | - David M Kent
- Predictive Analytics and Comparative Effectiveness Center, Tufts Medical Center, Boston, MA, USA
| | - Karina W Davidson
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Jamie S Hirsch
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA
- Clinical Digital Solutions, Northwell Health, New Hyde Park, NY, 11042, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA
| | - Theodoros P Zanos
- Institute of Health System Science, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA.
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, 11030, USA.
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY, 11549, USA.
| |
Collapse
|
30
|
Machine learning methods to predict 30-day hospital readmission outcome among US adults with pneumonia: analysis of the national readmission database. BMC Med Inform Decis Mak 2022; 22:288. [DOI: 10.1186/s12911-022-01995-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Accepted: 09/14/2022] [Indexed: 11/10/2022] Open
Abstract
Abstract
Background
Hospital readmissions for pneumonia are a growing concern in the US, with significant consequences for costs and quality of care. This study developed the rule-based model and other machine learning (ML) models to predict 30-day readmission risk in patients with pneumonia and compared model performance.
Methods
This population-based study involved patients aged ≥ 18 years hospitalized with pneumonia from January 1, 2016, through November 30, 2016, using the Healthcare Cost and Utilization Project-National Readmission Database (HCUP-NRD). Rule-based algorithms and other ML algorithms, specifically decision trees, random forest, extreme gradient descent boosting (XGBoost), and Least Absolute Shrinkage and Selection Operator (LASSO), were used to model all-cause readmissions 30 days post-discharge from index pneumonia hospitalization. A total of 61 clinically relevant variables were included for ML model development. Models were trained on randomly partitioned 50% of the data and evaluated using the remaining dataset. Model hyperparameters were tuned using the ten-fold cross-validation on the resampled training dataset. The area under the receiver operating curves (AUROC) and area under precision-recall curves (AUPRC) were calculated for the testing set to evaluate the model performance.
Results
Of the 372,293 patients with an index hospital hospitalization for pneumonia, 48,280 (12.97%) were readmitted within 30 days. Judged by AUROC in the testing data, rule-based model (0.6591) significantly outperformed decision tree (0.5783, p value < 0.001), random forest (0.6509, p value < 0.01) and LASSO (0.6087, p value < 0.001), but was less superior than XGBoost (0.6606, p value = 0.015). The AUPRC of the rule-based model in the testing data (0.2146) was higher than the decision tree (0.1560), random forest (0.2052), and LASSO (0.2042), but was similar to XGBoost (0.2147). The top risk-predictive rules captured by the rule-based algorithm were comorbidities, illness severity, disposition locations, payer type, age, and length of stay. These predictive risk factors were also identified by other ML models with high variable importance.
Conclusion
The performance of machine learning models for predicting readmission in pneumonia patients varied. The XGboost was better than the rule-based model based on the AUROC. However, important risk factors for predicting readmission remained consistent across ML models.
Collapse
|
31
|
An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department. Sci Rep 2022; 12:17466. [PMID: 36261457 PMCID: PMC9580414 DOI: 10.1038/s41598-022-22233-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2022] [Accepted: 10/11/2022] [Indexed: 01/12/2023] Open
Abstract
Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients' ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.
Collapse
|
32
|
Chang HH, Chiang JH, Wang CS, Chiu PF, Abdel-Kader K, Chen H, Siew ED, Yabes J, Murugan R, Clermont G, Palevsky PM, Jhamb M. Predicting Mortality Using Machine Learning Algorithms in Patients Who Require Renal Replacement Therapy in the Critical Care Unit. J Clin Med 2022; 11:5289. [PMID: 36142936 PMCID: PMC9500742 DOI: 10.3390/jcm11185289] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 09/04/2022] [Accepted: 09/05/2022] [Indexed: 01/13/2023] Open
Abstract
Background: General severity of illness scores are not well calibrated to predict mortality among patients receiving renal replacement therapy (RRT) for acute kidney injury (AKI). We developed machine learning models to make mortality prediction and compared their performance to that of the Sequential Organ Failure Assessment (SOFA) and HEpatic failure, LactatE, NorepInephrine, medical Condition, and Creatinine (HELENICC) scores. Methods: We extracted routinely collected clinical data for AKI patients requiring RRT in the MIMIC and eICU databases. The development models were trained in 80% of the pooled dataset and tested in the rest of the pooled dataset. We compared the area under the receiver operating characteristic curves (AUCs) of four machine learning models (multilayer perceptron [MLP], logistic regression, XGBoost, and random forest [RF]) to that of the SOFA, nonrenal SOFA, and HELENICC scores and assessed calibration, sensitivity, specificity, positive (PPV) and negative (NPV) predicted values, and accuracy. Results: The mortality AUC of machine learning models was highest for XGBoost (0.823; 95% confidence interval [CI], 0.791−0.854) in the testing dataset, and it had the highest accuracy (0.758). The XGBoost model showed no evidence of lack of fit with the Hosmer−Lemeshow test (p > 0.05). Conclusion: XGBoost provided the highest performance of mortality prediction for patients with AKI requiring RRT compared with previous scoring systems.
Collapse
Affiliation(s)
- Hsin-Hsiung Chang
- Division of Nephrology, Department of Internal Medicine, Antai Medical Care Corporation Antai Tian-Sheng Memorial Hospital, Donggang 928, Taiwan
- Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan
- Renal-Electrolyte Division, Department of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Jung-Hsien Chiang
- Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan
| | - Chi-Shiang Wang
- Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan
| | - Ping-Fang Chiu
- Division of Nephrology, Department of Internal Medicine, Changhua Christian Hospital, Changhua 500, Taiwan
- Department of Hospitality Management, MingDao University, Changhua 500, Taiwan
| | - Khaled Abdel-Kader
- Division of Nephrology and Hypertension, Vanderbilt University Medical Center, Nashville, TN 37011, USA
- Vanderbilt Center for Kidney Disease (VCKD) and Integrated Program for AKI Research (VIP-AKI), Nashville, TN 37011, USA
| | - Huiwen Chen
- Renal-Electrolyte Division, Department of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Edward D. Siew
- Division of Nephrology and Hypertension, Vanderbilt University Medical Center, Nashville, TN 37011, USA
- Vanderbilt Center for Kidney Disease (VCKD) and Integrated Program for AKI Research (VIP-AKI), Nashville, TN 37011, USA
- Tennessee Valley Health Systems (TVHS), Veteran’s Health Administration, Nashville, TN 37212, USA
| | - Jonathan Yabes
- Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Raghavan Murugan
- Program for Critical Care Nephrology, CRISMA, Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA 15213, USA
| | - Gilles Clermont
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Paul M. Palevsky
- Renal-Electrolyte Division, Department of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
- Kidney Medicine Section, VA Pittsburgh Healthcare System, Pittsburgh, PA 15240, USA
| | - Manisha Jhamb
- Renal-Electrolyte Division, Department of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA
| |
Collapse
|
33
|
Alsoof D, McDonald CL, Kuris EO, Daniels AH. Machine Learning for the Orthopaedic Surgeon: Uses and Limitations. J Bone Joint Surg Am 2022; 104:1586-1594. [PMID: 35383655 DOI: 10.2106/jbjs.21.01305] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
➤ Machine learning is a subset of artificial intelligence in which computer algorithms are trained to make classifications and predictions based on patterns in data. The utilization of these techniques is rapidly expanding in the field of orthopaedic research. ➤ There are several domains in which machine learning has application to orthopaedics, including radiographic diagnosis, gait analysis, implant identification, and patient outcome prediction. ➤ Several limitations prevent the widespread use of machine learning in the daily clinical environment. However, future work can overcome these issues and enable machine learning tools to be a useful adjunct for orthopaedic surgeons in their clinical decision-making.
Collapse
Affiliation(s)
- Daniel Alsoof
- Department of Orthopedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | | | | | | |
Collapse
|
34
|
Xu Z, Guo K, Chu W, Lou J, Chen C. Performance of Machine Learning Algorithms for Predicting Adverse Outcomes in Community-Acquired Pneumonia. Front Bioeng Biotechnol 2022; 10:903426. [PMID: 35845426 PMCID: PMC9278327 DOI: 10.3389/fbioe.2022.903426] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Accepted: 05/16/2022] [Indexed: 12/31/2022] Open
Abstract
Background: The ability to assess adverse outcomes in patients with community-acquired pneumonia (CAP) could improve clinical decision-making to enhance clinical practice, but the studies remain insufficient, and similarly, few machine learning (ML) models have been developed. Objective: We aimed to explore the effectiveness of predicting adverse outcomes in CAP through ML models. Methods: A total of 2,302 adults with CAP who were prospectively recruited between January 2012 and March 2015 across three cities in South America were extracted from DryadData. After a 70:30 training set: test set split of the data, nine ML algorithms were executed and their diagnostic accuracy was measured mainly by the area under the curve (AUC). The nine ML algorithms included decision trees, random forests, extreme gradient boosting (XGBoost), support vector machines, Naïve Bayes, K-nearest neighbors, ridge regression, logistic regression without regularization, and neural networks. The adverse outcomes included hospital admission, mortality, ICU admission, and one-year post-enrollment status. Results: The XGBoost algorithm had the best performance in predicting hospital admission. Its AUC reached 0.921, and accuracy, precision, recall, and F1-score were better than those of other models. In the prediction of ICU admission, a model trained with the XGBoost algorithm showed the best performance with AUC 0.801. XGBoost algorithm also did a good job at predicting one-year post-enrollment status. The results of AUC, accuracy, precision, recall, and F1-score indicated the algorithm had high accuracy and precision. In addition, the best performance was seen by the neural network algorithm when predicting death (AUC 0.831). Conclusions: ML algorithms, particularly the XGBoost algorithm, were feasible and effective in predicting adverse outcomes of CAP patients. The ML models based on available common clinical features had great potential to guide individual treatment and subsequent clinical decisions.
Collapse
Affiliation(s)
- Zhixiao Xu
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Kun Guo
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Weiwei Chu
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Jingwen Lou
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China
| | - Chengshui Chen
- Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China.,The Interventional Pulmonary Key Laboratory of Zhejiang Province, Wenzhou, China
| |
Collapse
|
35
|
Machine learning-based suggestion for critical interventions in the management of potentially severe conditioned patients in emergency department triage. Sci Rep 2022; 12:10537. [PMID: 35732641 PMCID: PMC9218081 DOI: 10.1038/s41598-022-14422-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2022] [Accepted: 06/07/2022] [Indexed: 12/05/2022] Open
Abstract
Providing timely intervention to critically ill patients is a challenging task in emergency departments (ED). Our study aimed to predict early critical interventions (CrIs), which can be used as clinical recommendations. This retrospective observational study was conducted in the ED of a tertiary hospital located in a Korean metropolitan city. Patient who visited ED from January 1, 2016, to December 31, 2018, were included. Need of six CrIs were selected as prediction outcomes, namely, arterial line (A-line) insertion, oxygen therapy, high-flow nasal cannula (HFNC), intubation, Massive Transfusion Protocol (MTP), and inotropes and vasopressor. Extreme gradient boosting (XGBoost) prediction model was built by using only data available at the initial stage of ED. Overall, 137,883 patients were included in the study. The areas under the receiver operating characteristic curve for the prediction of A-line insertion was 0·913, oxygen therapy was 0.909, HFNC was 0.962, intubation was 0.945, MTP was 0.920, and inotropes or vasopressor administration was 0.899 in the XGBoost method. In addition, an increase in the need for CrIs was associated with worse ED outcomes. The CrIs model was integrated into the study site's electronic medical record and could be used to suggest early interventions for emergency physicians.
Collapse
|
36
|
Agarwal R, Domenico HJ, Balla SR, Byrne DW, Whisenant JG, Woods MC, Martin BJ, Karlekar MB, Bennett ML. Palliative Care Exposure Relative to Predicted Risk of Six-Month Mortality in Hospitalized Adults. J Pain Symptom Manage 2022; 63:645-653. [PMID: 35081441 PMCID: PMC9018538 DOI: 10.1016/j.jpainsymman.2022.01.013] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Revised: 01/14/2022] [Accepted: 01/18/2022] [Indexed: 12/25/2022]
Abstract
CONTEXT The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care, prognostic discussions, and palliative care interventions remains unknown. OBJECTIVES To develop and validate a real-time predictive model for 180 day mortality using routinely available clinical and laboratory admission data and determine if palliative care exposure varies with predicted mortality risk. METHODS Adult admissions between October 1, 2013 and October.1, 2017 were included for the model derivation. A separate cohort was collected between January 1, 2018 and July 31, 2020 for validation. Patients were followed for 180 days from discharge, and logistic regression with selected variables was used to estimate patients' risk for mortality. RESULTS In the model derivation cohort, 7963 events of 180 day mortality (4.5% event rate) were observed. Median age was 53.0 (IQR 24.0-66.0) with 92,734 females (52.5%). Variables with strongest association with 180 day mortality included: Braden Score (OR 0.83; 95% CI 0.82-0.84); admission Do Not Resuscitate orders (OR 2.61; 95% CI 2.43-2.79); admission service and admission status. The model yielded excellent discriminatory ability in both the derivation (c-statistic 0.873; 95% CI 0.870-0.877; Brier score 0.04) and validation cohorts (c-statistic 0.844; 95% CI 0.840-0.847; Brier score 0.072). Inpatient palliative care consultations increased from 3% of minimal-risk encounters to 41% of high-risk encounters (P < 0.01). CONCLUSION We developed and temporally validated a predictive mortality model for adults from a large retrospective cohort, which helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.
Collapse
Affiliation(s)
- Rajiv Agarwal
- Department of Medicine (R.A., J.G.W., M.B.K.), Vanderbilt University Medical Center, Nashville, Tennessee, USA; Vanderbilt-Ingram Cancer Center (R.A., J.G.W.), Nashville, Tennessee, USA.
| | - Henry J Domenico
- Department of Biostatistics (H.J.D., D.W.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA; Vanderbilt Office of Quality, Safety, and Risk Prevention (H.J.D., M.C.W., B.J.M., M.L.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Sreenivasa R Balla
- Health Information Technology (S.R.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Daniel W Byrne
- Department of Biostatistics (H.J.D., D.W.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Jennifer G Whisenant
- Department of Medicine (R.A., J.G.W., M.B.K.), Vanderbilt University Medical Center, Nashville, Tennessee, USA; Vanderbilt-Ingram Cancer Center (R.A., J.G.W.), Nashville, Tennessee, USA
| | - Marcella C Woods
- Vanderbilt Office of Quality, Safety, and Risk Prevention (H.J.D., M.C.W., B.J.M., M.L.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Barbara J Martin
- Vanderbilt Office of Quality, Safety, and Risk Prevention (H.J.D., M.C.W., B.J.M., M.L.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Mohana B Karlekar
- Department of Medicine (R.A., J.G.W., M.B.K.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Marc L Bennett
- Vanderbilt Office of Quality, Safety, and Risk Prevention (H.J.D., M.C.W., B.J.M., M.L.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA; Department of Otolaryngology Head and Neck Surgery (M.L.B.), Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
37
|
Chi S, Guo A, Heard K, Kim S, Foraker R, White P, Moore N. Development and Structure of an Accurate Machine Learning Algorithm to Predict Inpatient Mortality and Hospice Outcomes in the Coronavirus Disease 2019 Era. Med Care 2022; 60:381-386. [PMID: 35230273 PMCID: PMC8989608 DOI: 10.1097/mlr.0000000000001699] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
BACKGROUND The coronavirus disease 2019 (COVID-19) pandemic has challenged the accuracy and racial biases present in traditional mortality scores. An accurate prognostic model that can be applied to hospitalized patients irrespective of race or COVID-19 status may benefit patient care. RESEARCH DESIGN This cohort study utilized historical and ongoing electronic health record features to develop and validate a deep-learning model applied on the second day of admission predicting a composite outcome of in-hospital mortality, discharge to hospice, or death within 30 days of admission. Model features included patient demographics, diagnoses, procedures, inpatient medications, laboratory values, vital signs, and substance use history. Conventional performance metrics were assessed, and subgroup analysis was performed based on race, COVID-19 status, and intensive care unit admission. SUBJECTS A total of 35,521 patients hospitalized between April 2020 and October 2020 at a single health care system including a tertiary academic referral center and 9 community hospitals. RESULTS Of 35,521 patients, including 9831 non-White patients and 2020 COVID-19 patients, 2838 (8.0%) met the composite outcome. Patients who experienced the composite outcome were older (73 vs. 61 y old) with similar sex and race distributions between groups. The model achieved an area under the receiver operating characteristic curve of 0.89 (95% confidence interval: 0.88, 0.91) and an average positive predictive value of 0.46 (0.40, 0.52). Model performance did not differ significantly in White (0.89) and non-White (0.90) subgroups or when grouping by COVID-19 status and intensive care unit admission. CONCLUSION A deep-learning model using large-volume, structured electronic health record data can effectively predict short-term mortality or hospice outcomes on the second day of admission in the general inpatient population without significant racial bias.
Collapse
Affiliation(s)
- Stephen Chi
- Division of Pulmonary and Critical Care Medicine
| | - Aixia Guo
- Institute for Informatics, Washington University in St. Louis
| | | | - Seunghwan Kim
- Division of General Medical Sciences, School of Medicine, Washington University in St. Louis
| | - Randi Foraker
- Institute for Informatics, Washington University in St. Louis
| | - Patrick White
- Division of Palliative Medicine, Department of Medicine, Washington University in St. Louis
| | | |
Collapse
|
38
|
Trujillo Rivera EA, Chamberlain JM, Patel AK, Morizono H, Heneghan JA, Pollack MM. Dynamic Mortality Risk Predictions for Children in ICUs: Development and Validation of Machine Learning Models. Pediatr Crit Care Med 2022; 23:344-352. [PMID: 35190501 PMCID: PMC9117400 DOI: 10.1097/pcc.0000000000002910] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVES Assess a machine learning method of serially updated mortality risk. DESIGN Retrospective analysis of a national database (Health Facts; Cerner Corporation, Kansas City, MO). SETTING Hospitals caring for children in ICUs. PATIENTS A total of 27,354 admissions cared for in ICUs from 2009 to 2018. INTERVENTIONS None. MAIN OUTCOME Hospital mortality risk estimates determined at 6-hour time periods during care in the ICU. Models were truncated at 180 hours due to decreased sample size secondary to discharges and deaths. MEASUREMENTS AND MAIN RESULTS The Criticality Index, based on physiology, therapy, and care intensity, was computed for each admission for each time period and calibrated to hospital mortality risk (Criticality Index-Mortality [CI-M]) at each of 29 time periods (initial assessment: 6 hr; last assessment: 180 hr). Performance metrics and clinical validity were determined from the held-out test sample (n = 3,453, 13%). Discrimination assessed with the area under the receiver operating characteristic curve was 0.852 (95% CI, 0.843-0.861) overall and greater than or equal to 0.80 for all individual time periods. Calibration assessed by the Hosmer-Lemeshow goodness-of-fit test showed good fit overall (p = 0.196) and was statistically not significant for 28 of the 29 time periods. Calibration plots for all models revealed the intercept ranged from--0.002 to 0.009, the slope ranged from 0.867 to 1.415, and the R2 ranged from 0.862 to 0.989. Clinical validity assessed using population trajectories and changes in the risk status of admissions (clinical volatility) revealed clinical trajectories consistent with clinical expectations and greater clinical volatility in deaths than survivors (p < 0.001). CONCLUSIONS Machine learning models incorporating physiology, therapy, and care intensity can track changes in hospital mortality risk during intensive care. The CI-M's framework and modeling method are potentially applicable to monitoring clinical improvement and deterioration in real time.
Collapse
Affiliation(s)
| | - James M Chamberlain
- Department of Pediatrics, Division of Emergency Medicine, Children's National Hospital and George Washington University School of Medicine and Health Sciences, Washington, DC
| | - Anita K Patel
- Department of Pediatrics, Division of Critical Care Medicine, Children's National Hospital and George Washington University School of Medicine and Health Sciences, Washington, DC
| | - Hiroki Morizono
- Children's National Research Institute, Associate Research Professor of Genomics and Precision Medicine, George Washington University School of Medicine and Health Sciences, Washington, DC
| | - Julia A Heneghan
- Department of Pediatrics, Division of Critical Care Medicine, Children's National Hospital and George Washington University School of Medicine and Health Sciences, Washington, DC
| | - Murray M Pollack
- Department of Pediatrics, Division of Critical Care Medicine, Children's National Hospital and George Washington University School of Medicine and Health Sciences, Washington, DC
| |
Collapse
|
39
|
Gordon AJ, Govindarajan P, Bennett CL, Matheson L, Kohn MA, Camargo C, Kline J. External validation of the 4C Mortality Score for hospitalised patients with COVID-19 in the RECOVER network. BMJ Open 2022; 12:e054700. [PMID: 35450898 PMCID: PMC9023850 DOI: 10.1136/bmjopen-2021-054700] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/10/2021] [Accepted: 03/28/2022] [Indexed: 01/08/2023] Open
Abstract
OBJECTIVES Estimating mortality risk in hospitalised SARS-CoV-2+ patients may help with choosing level of care and discussions with patients. The Coronavirus Clinical Characterisation Consortium Mortality Score (4C Score) is a promising COVID-19 mortality risk model. We examined the association of risk factors with 30-day mortality in hospitalised, full-code SARS-CoV-2+ patients and investigated the discrimination and calibration of the 4C Score. This was a retrospective cohort study of SARS-CoV-2+ hospitalised patients within the RECOVER (REgistry of suspected COVID-19 in EmeRgency care) network. SETTING 99 emergency departments (EDs) across the USA. PARTICIPANTS Patients ≥18 years old, positive for SARS-CoV-2 in the ED, and hospitalised. PRIMARY OUTCOME Death within 30 days of the index visit. We performed logistic regression analysis, reporting multivariable risk ratios (MVRRs) and calculated the area under the ROC curve (AUROC) and mean prediction error for the original 4C Score and after dropping the C reactive protein (CRP) component. RESULTS Of 6802 hospitalised patients with COVID-19, 1149 (16.9%) died within 30 days. The 30-day mortality was increased with age 80+ years (MVRR=5.79, 95% CI 4.23 to 7.34); male sex (MVRR=1.17, 1.05 to 1.28); and nursing home/assisted living facility residence (MVRR=1.29, 1.1 to 1.48). The 4C Score had comparable discrimination in the RECOVER dataset compared with the original 4C validation dataset (AUROC: RECOVER 0.786 (95% CI 0.773 to 0.799), 4C validation 0.763 (95% CI 0.757 to 0.769). Score-specific mortalities in our sample were lower than in the 4C validation sample (mean prediction error 6.0%). Dropping the CRP component from the 4C Score did not substantially affect discrimination and 4C risk estimates were now close (mean prediction error 0.7%). CONCLUSIONS We independently validated 4C Score as predicting risk of 30-day mortality in hospitalised SARS-CoV-2+ patients. We recommend dropping the CRP component of the score and using our recalibrated mortality risk estimates.
Collapse
Affiliation(s)
- Alexandra June Gordon
- Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | | | - Christopher L Bennett
- Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
- Epidemiology, Stanford University School of Medicine, Stanford, California, USA
| | - Loretta Matheson
- Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Michael A Kohn
- Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
- Epidemiology and Biostatistics, University of California San Francisco, San Francisco, California, USA
| | - Carlos Camargo
- Emergency Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Jeffrey Kline
- Emergency Medicine, Wayne State University School of Medicine, Detroit, Michigan, USA
| |
Collapse
|
40
|
Singh H, Mhasawade V, Chunara R. Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database. PLOS DIGITAL HEALTH 2022; 1:e0000023. [PMID: 36812510 PMCID: PMC9931319 DOI: 10.1371/journal.pdig.0000023] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 02/17/2022] [Indexed: 12/23/2022]
Abstract
Modern predictive models require large amounts of data for training and evaluation, absence of which may result in models that are specific to certain locations, populations in them and clinical practices. Yet, best practices for clinical risk prediction models have not yet considered such challenges to generalizability. Here we ask whether population- and group-level performance of mortality prediction models vary significantly when applied to hospitals or geographies different from the ones in which they are developed. Further, what characteristics of the datasets explain the performance variation? In this multi-center cross-sectional study, we analyzed electronic health records from 179 hospitals across the US with 70,126 hospitalizations from 2014 to 2015. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by the race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" that infers paths of causal influence while identifying potential influences associated with unmeasured variables. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st-3rd quartile or IQR; median 0.801); calibration slope from 0.725 to 0.983 (IQR; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (IQR; median 0.092). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. The race variable also mediated differences in the relationship between clinical variables and mortality, by hospital/region. In conclusion, group-level performance should be assessed during generalizability checks to identify potential harms to the groups. Moreover, for developing methods to improve model performance in new environments, a better understanding and documentation of provenance of data and health processes are needed to identify and mitigate sources of variation.
Collapse
Affiliation(s)
| | | | - Rumi Chunara
- New York University, Tandon School of Engineering,New York University, School of Global Public Health
| |
Collapse
|
41
|
Variane GFT, Camargo JPV, Rodrigues DP, Magalhães M, Mimica MJ. Current Status and Future Directions of Neuromonitoring With Emerging Technologies in Neonatal Care. Front Pediatr 2022; 9:755144. [PMID: 35402367 PMCID: PMC8984110 DOI: 10.3389/fped.2021.755144] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/08/2021] [Accepted: 11/18/2021] [Indexed: 11/13/2022] Open
Abstract
Neonatology has experienced a significant reduction in mortality rates of the preterm population and critically ill infants over the last few decades. Now, the emphasis is directed toward improving long-term neurodevelopmental outcomes and quality of life. Brain-focused care has emerged as a necessity. The creation of neonatal neurocritical care units, or Neuro-NICUs, provides strategies to reduce brain injury using standardized clinical protocols, methodologies, and provider education and training. Bedside neuromonitoring has dramatically improved our ability to provide assessment of newborns at high risk. Non-invasive tools, such as continuous electroencephalography (cEEG), amplitude-integrated electroencephalography (aEEG), and near-infrared spectroscopy (NIRS), allow screening for seizures and continuous evaluation of brain function and cerebral oxygenation at the bedside. Extended and combined uses of these techniques, also described as multimodal monitoring, may allow practitioners to better understand the physiology of critically ill neonates. Furthermore, the rapid growth of technology in the Neuro-NICU, along with the increasing use of telemedicine and artificial intelligence with improved data mining techniques and machine learning (ML), has the potential to vastly improve decision-making processes and positively impact outcomes. This article will cover the current applications of neuromonitoring in the Neuro-NICU, recent advances, potential pitfalls, and future perspectives in this field.
Collapse
Affiliation(s)
- Gabriel Fernando Todeschi Variane
- Division of Neonatology, Department of Pediatrics, Irmandade de Misericordia da Santa Casa de São Paulo, São Paulo, Brazil
- Clinical Research Department, Protecting Brains and Saving Futures Organization, São Paulo, Brazil
- Division of Neonatology, Grupo Santa Joana, São Paulo, Brazil
| | - João Paulo Vasques Camargo
- Clinical Research Department, Protecting Brains and Saving Futures Organization, São Paulo, Brazil
- Data Science Department, OPD Team, São Paulo, Brazil
| | - Daniela Pereira Rodrigues
- Clinical Research Department, Protecting Brains and Saving Futures Organization, São Paulo, Brazil
- Pediatric Nursing Department, Escola Paulista de Enfermagem, Universidade Federal de São Paulo, São Paulo, Brazil
| | - Maurício Magalhães
- Division of Neonatology, Department of Pediatrics, Irmandade de Misericordia da Santa Casa de São Paulo, São Paulo, Brazil
- Clinical Research Department, Protecting Brains and Saving Futures Organization, São Paulo, Brazil
- Department of Pediatrics, Faculdade de Ciências Médicas da Santa Casa de São Paulo, São Paulo, Brazil
| | - Marcelo Jenné Mimica
- Department of Pathology, Faculdade de Ciências Médicas da Santa Casa de São Paulo, São Paulo, Brazil
- Department of Pediatrics, Irmandade da Santa Casa de Misericórdia de São Paulo, São Paulo, Brazil
| |
Collapse
|
42
|
Tran Z, Zhang W, Verma A, Cook A, Kim D, Burruss S, Ramezani R, Benharash P. The derivation of an International Classification of Diseases, Tenth Revision-based trauma-related mortality model using machine learning. J Trauma Acute Care Surg 2022; 92:561-566. [PMID: 34554135 DOI: 10.1097/ta.0000000000003416] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND Existing mortality prediction models have attempted to quantify injury burden following trauma-related admissions with the most notable being the Injury Severity Score (ISS). Although easy to calculate, it requires additional administrative coding. International Classification of Diseases (ICD)-based models such as the Trauma Mortality Prediction Model (TMPM-ICD10) circumvent these limitations, but they use linear modeling, which may not adequately capture the intricate relationships of injuries on mortality. Using ICD-10 coding and machine learning (ML) algorithms, the present study used the National Trauma Data Bank to develop mortality prediction models whose performance was compared with logistic regression, ISS, and TMPM-ICD10. METHODS The 2015 to 2017 National Trauma Data Bank was used to identify adults following trauma-related admissions. Of 8,021 ICD-10 codes, injuries were categorized into 1,495 unique variables. The primary outcome was in-hospital mortality. eXtreme Gradient Boosting (XGBoost), a ML technique that uses iterations of decision trees, was used to develop mortality models. Model discrimination was compared with logistic regression, ISS, and TMPM-ICD10 using receiver operating characteristic curve and probabilistic accuracy with calibration curves. RESULTS Of 1,611,063 patients, 54,870 (3.41%) experienced in-hospital mortality. Compared with those who survived, those who died more frequently suffered from penetrating trauma and had a greater number of injuries. The XGBoost model exhibited superior receiver operating characteristic curve (0.863 [95% confidence interval (CI), 0.862-0.864]) compared with logistic regression (0.845 [95% CI, 0.844-0.846]), ISS (0.828 [95% CI, 0.827-0.829]), and TMPM-ICD10 (0.861 [95% CI, 0.860-0.862]) (all p < 0.001). Importantly, the ML model also had significantly improved calibration compared with other methodologies (XGBoost, coefficient of determination (R2) = 0.993; logistic regression, R2 = 0.981; ISS, R2 = 0.649; TMPM-ICD10, R2 = 0.830). CONCLUSION Machine learning models using XGBoost demonstrated superior performance and calibration compared with logistic regression, ISS, and TMPM-ICD10. Such approaches in quantifying injury severity may improve its utility in mortality prognostication, quality improvement, and trauma research. LEVEL OF EVIDENCE Prognostic and Epidemiologic; level III.
Collapse
Affiliation(s)
- Zachary Tran
- From the Cardiovascular Outcomes Research Laboratories (Z.T., A.V., P.B.), David Geffen School of Medicine, University of California, Los Angeles, Los Angeles; Division of Acute Care Surgery, Department of Surgery (Z.T., S.B.), Loma Linda University Medical Center, Loma Linda; Department of Computer Science (W.Z., R.R.), University of California, Los Angeles, California; Department of Surgery (A.C.), University of Texas Health Science Center at Tyler, Tyler, Texas; and Department of Surgery (D.K.), Harbor-UCLA Medical Center, Torrance, California
| | | | | | | | | | | | | | | |
Collapse
|
43
|
Douthit BJ, Walden RL, Cato K, Coviak CP, Cruz C, D'Agostino F, Forbes T, Gao G, Kapetanovic TA, Lee MA, Pruinelli L, Schultz MA, Wieben A, Jeffery AD. Data Science Trends Relevant to Nursing Practice: A Rapid Review of the 2020 Literature. Appl Clin Inform 2022; 13:161-179. [PMID: 35139564 PMCID: PMC8828453 DOI: 10.1055/s-0041-1742218] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
Abstract
BACKGROUND The term "data science" encompasses several methods, many of which are considered cutting edge and are being used to influence care processes across the world. Nursing is an applied science and a key discipline in health care systems in both clinical and administrative areas, making the profession increasingly influenced by the latest advances in data science. The greater informatics community should be aware of current trends regarding the intersection of nursing and data science, as developments in nursing practice have cross-professional implications. OBJECTIVES This study aimed to summarize the latest (calendar year 2020) research and applications of nursing-relevant patient outcomes and clinical processes in the data science literature. METHODS We conducted a rapid review of the literature to identify relevant research published during the year 2020. We explored the following 16 topics: (1) artificial intelligence/machine learning credibility and acceptance, (2) burnout, (3) complex care (outpatient), (4) emergency department visits, (5) falls, (6) health care-acquired infections, (7) health care utilization and costs, (8) hospitalization, (9) in-hospital mortality, (10) length of stay, (11) pain, (12) patient safety, (13) pressure injuries, (14) readmissions, (15) staffing, and (16) unit culture. RESULTS Of 16,589 articles, 244 were included in the review. All topics were represented by literature published in 2020, ranging from 1 article to 59 articles. Numerous contemporary data science methods were represented in the literature including the use of machine learning, neural networks, and natural language processing. CONCLUSION This review provides an overview of the data science trends that were relevant to nursing practice in 2020. Examinations of such literature are important to monitor the status of data science's influence in nursing practice.
Collapse
Affiliation(s)
- Brian J. Douthit
- Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs; Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Rachel L. Walden
- Annette and Irwin Eskind Family Biomedical Library, Vanderbilt University, Nashville, Tennessee, United States
| | - Kenrick Cato
- Department of Emergency Medicine, Columbia University School of Nursing, New York, New York, United States
| | - Cynthia P. Coviak
- Professor Emerita of Nursing, Grand Valley State University, Allendale, Michigan, United States
| | - Christopher Cruz
- Global Health Technology and Informatics, Chevron, San Ramon, California, United States
| | - Fabio D'Agostino
- Department of Medicine and Surgery, Saint Camillus International University of Health Sciences, Rome, Italy
| | - Thompson Forbes
- College of Nursing, East Carolina University, Greenville, North California, United States
| | - Grace Gao
- Department of Nursing, St Catherine University, Saint Paul, Minnesota, United States
| | - Theresa A. Kapetanovic
- College of Nursing, East Carolina University, Greenville, North California, United States
| | - Mikyoung A. Lee
- College of Nursing, Texas Woman's University, Denton, Texas, United States
| | - Lisiane Pruinelli
- School of Nursing, University of Minnesota, Minneapolis, Minnesota, United States
| | - Mary A. Schultz
- Department of Nursing, California State University, San Bernardino, California, United States
| | - Ann Wieben
- School of Nursing, University of Wisconsin-Madison, Wisconsin, United States
| | - Alvin D. Jeffery
- School of Nursing, Vanderbilt University; Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, Tennessee, United States,Address for correspondence Alvin D. Jeffery, PhD, RN-BC, CCRN-K, FNP-BC 461 21st Avenue South, Nashville, TN 37240United States
| |
Collapse
|
44
|
Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study. Support Care Cancer 2022; 30:4363-4372. [DOI: 10.1007/s00520-021-06774-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Accepted: 12/18/2021] [Indexed: 10/19/2022]
|
45
|
Wang Z, Zhe S, Zimmerman J, Morrisey C, Tonna JE, Sharma V, Metcalf RA. Development and validation of a machine learning method to predict intraoperative red blood cell transfusions in cardiothoracic surgery. Sci Rep 2022; 12:1355. [PMID: 35079127 PMCID: PMC8789772 DOI: 10.1038/s41598-022-05445-y] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2021] [Accepted: 01/12/2022] [Indexed: 12/23/2022] Open
Abstract
Accurately predicting red blood cell (RBC) transfusion requirements in cardiothoracic (CT) surgery could improve blood inventory management and be used as a surrogate marker for assessing hemorrhage risk preoperatively. We developed a machine learning (ML) method to predict intraoperative RBC transfusions in CT surgery. A detailed database containing time-stamped clinical variables for all CT surgeries from 5/2014-6/2019 at a single center (n = 2410) was used for model development. After random forest feature selection, surviving features were inputs for ML algorithms using five-fold cross-validation. The dataset was updated with 437 additional cases from 8/2019-8/2020 for validation. We developed and validated a hybrid ML method given the skewed nature of the dataset. Our Gaussian Process (GP) regression ML algorithm accurately predicted RBC transfusion amounts of 0 and 1-3 units (root mean square error, RMSE 0.117 and 1.705, respectively) and our GP classification ML algorithm accurately predicted 4 + RBC units transfused (area under the curve, AUC = 0.826). The final prediction is the regression result if classification predicted < 4 units transfused, or the classification result if 4 + units were predicted. We developed and validated an ML method to accurately predict intraoperative RBC transfusions in CT surgery using local data.
Collapse
Affiliation(s)
- Zheng Wang
- School of Computing, University of Utah, Salt Lake City, UT, USA
| | - Shandian Zhe
- School of Computing, University of Utah, Salt Lake City, UT, USA
| | - Joshua Zimmerman
- Department of Anesthesiology, University of Utah, Salt Lake City, UT, USA
| | - Candice Morrisey
- Department of Anesthesiology, University of Utah, Salt Lake City, UT, USA
| | - Joseph E Tonna
- Division of Cardiothoracic Surgery, Department of Surgery, University of Utah, Salt Lake City, UT, USA
| | - Vikas Sharma
- Division of Cardiothoracic Surgery, Department of Surgery, University of Utah, Salt Lake City, UT, USA
| | - Ryan A Metcalf
- Department of Pathology, University of Utah, Salt Lake City, UT, USA.
- ARUP Laboratories, Salt Lake City, UT, USA.
| |
Collapse
|
46
|
Ibrahim ZM, Bean D, Searle T, Qian L, Wu H, Shek A, Kraljevic Z, Galloway J, Norton S, Teo JT, Dobson RJ. A Knowledge Distillation Ensemble Framework for Predicting Short- and Long-Term Hospitalization Outcomes From Electronic Health Records Data. IEEE J Biomed Health Inform 2022; 26:423-435. [PMID: 34129509 DOI: 10.1109/jbhi.2021.3089287] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
The ability to perform accurate prognosis is crucial for proactive clinical decision making, informed resource management and personalised care. Existing outcome prediction models suffer from a low recall of infrequent positive outcomes. We present a highly-scalable and robust machine learning framework to automatically predict adversity represented by mortality and ICU admission and readmission from time-series of vital signs and laboratory results obtained within the first 24 hours of hospital admission. The stacked ensemble platform comprises two components: a) an unsupervised LSTM Autoencoder that learns an optimal representation of the time-series, using it to differentiate the less frequent patterns which conclude with an adverse event from the majority patterns that do not, and b) a gradient boosting model, which relies on the constructed representation to refine prediction by incorporating static features. The model is used to assess a patient's risk of adversity and provides visual justifications of its prediction. Results of three case studies show that the model outperforms existing platforms in ICU and general ward settings, achieving average Precision-Recall Areas Under the Curve (PR-AUCs) of 0.891 (95% CI: 0.878-0.939) for mortality and 0.908 (95% CI: 0.870-0.935) in predicting ICU admission and readmission.
Collapse
|
47
|
Patel AK, Trujillo-Rivera E, Morizono H, Pollack MM. The criticality Index-mortality: A dynamic machine learning prediction algorithm for mortality prediction in children cared for in an ICU. Front Pediatr 2022; 10:1023539. [PMID: 36533242 PMCID: PMC9752098 DOI: 10.3389/fped.2022.1023539] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/23/2022] [Accepted: 10/26/2022] [Indexed: 12/03/2022] Open
Abstract
BACKGROUND The Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change. OBJECTIVES To assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions. POPULATION There were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing. MODEL A neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression. RESULTS Discrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R2 was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors. CONCLUSIONS The Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.
Collapse
Affiliation(s)
- Anita K Patel
- Department of Pediatrics, Division of Critical Care Medicine, Children's National Health System, George Washington University School of Medicine and Health Sciences, Washington, DC, United States
| | - Eduardo Trujillo-Rivera
- Department of Bio-Informatics, Children's National Health System, George Washington University School of Medicine and Health Sciences, Washington, DC, United States
| | - Hiroki Morizono
- Department of Pediatrics, Children's National Research Institute, George Washington University School of Medicine and Health Sciences, Washington, DC, United States
| | - Murray M Pollack
- Department of Pediatrics, Division of Critical Care Medicine, Children's National Health System, George Washington University School of Medicine and Health Sciences, Washington, DC, United States
| |
Collapse
|
48
|
Prediction across healthcare settings: a case study in predicting emergency department disposition. NPJ Digit Med 2021; 4:169. [PMID: 34912043 PMCID: PMC8674364 DOI: 10.1038/s41746-021-00537-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Accepted: 10/19/2021] [Indexed: 12/24/2022] Open
Abstract
Several approaches exist today for developing predictive models across multiple clinical sites, yet there is a lack of comparative data on their performance, especially within the context of EHR-based prediction models. We set out to provide a framework for prediction across healthcare settings. As a case study, we examined an ED disposition prediction model across three geographically and demographically diverse sites. We conducted a 1-year retrospective study, including all visits in which the outcome was either discharge-to-home or hospitalization. Four modeling approaches were compared: a ready-made model trained at one site and validated at other sites, a centralized uniform model incorporating data from all sites, multiple site-specific models, and a hybrid approach of a ready-made model re-calibrated using site-specific data. Predictions were performed using XGBoost. The study included 288,962 visits with an overall admission rate of 16.8% (7.9–26.9%). Some risk factors for admission were prominent across all sites (e.g., high-acuity triage emergency severity index score, high prior admissions rate), while others were prominent at only some sites (multiple lab tests ordered at the pediatric sites, early use of ECG at the adult site). The XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93), followed by the calibrated-model approach (AUC = 0.87–0.92), and the ready-made approach (AUC = 0.62–0.85). Our results show that site-specific customization is a key driver of predictive model performance.
Collapse
|
49
|
Naemi A, Schmidt T, Mansourvar M, Naghavi-Behzad M, Ebrahimi A, Wiil UK. Machine learning techniques for mortality prediction in emergency departments: a systematic review. BMJ Open 2021; 11:e052663. [PMID: 34728454 PMCID: PMC8565537 DOI: 10.1136/bmjopen-2021-052663] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/24/2021] [Accepted: 09/27/2021] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVES This systematic review aimed to assess the performance and clinical feasibility of machine learning (ML) algorithms in prediction of in-hospital mortality for medical patients using vital signs at emergency departments (EDs). DESIGN A systematic review was performed. SETTING The databases including Medline (PubMed), Scopus and Embase (Ovid) were searched between 2010 and 2021, to extract published articles in English, describing ML-based models utilising vital sign variables to predict in-hospital mortality for patients admitted at EDs. Critical appraisal and data extraction for systematic reviews of prediction modelling studies checklist was used for study planning and data extraction. The risk of bias for included papers was assessed using the prediction risk of bias assessment tool. PARTICIPANTS Admitted patients to the ED. MAIN OUTCOME MEASURE In-hospital mortality. RESULTS Fifteen articles were included in the final review. We found that eight models including logistic regression, decision tree, K-nearest neighbours, support vector machine, gradient boosting, random forest, artificial neural networks and deep neural networks have been applied in this domain. Most studies failed to report essential main analysis steps such as data preprocessing and handling missing values. Fourteen included studies had a high risk of bias in the statistical analysis part, which could lead to poor performance in practice. Although the main aim of all studies was developing a predictive model for mortality, nine articles did not provide a time horizon for the prediction. CONCLUSION This review provided an updated overview of the state-of-the-art and revealed research gaps; based on these, we provide eight recommendations for future studies to make the use of ML more feasible in practice. By following these recommendations, we expect to see more robust ML models applied in the future to help clinicians identify patient deterioration earlier.
Collapse
Affiliation(s)
- Amin Naemi
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Thomas Schmidt
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Marjan Mansourvar
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Mohammad Naghavi-Behzad
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Department of Nuclear Medicine, Odense University Hospital, Odense, Denmark
| | - Ali Ebrahimi
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Uffe Kock Wiil
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| |
Collapse
|
50
|
Pal S, Paul S. Linking hydrological security and landscape insecurity in the moribund deltaic wetland of India using tree-based hybrid ensemble method in python. ECOL INFORM 2021. [DOI: 10.1016/j.ecoinf.2021.101422] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
|