51
|
Ono S, Goto T. Introduction to supervised machine learning in clinical epidemiology. ANNALS OF CLINICAL EPIDEMIOLOGY 2022; 4:63-71. [PMID: 38504945 PMCID: PMC10760492 DOI: 10.37737/ace.22009] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 03/21/2024]
Abstract
Machine learning refers to a series of processes in which a computer finds rules from a vast amount of data. With recent advances in computer technology and the availability of a wide variety of health data, machine learning has rapidly developed and been applied in medical research. Currently, there are three types of machine learning: supervised, unsupervised, and reinforcement learning. In medical research, supervised learning is commonly used for diagnoses and prognoses, while unsupervised learning is used for phenotyping a disease, and reinforcement learning for maximizing favorable results, such as optimization of total patients' waiting time in the emergency department. The present article focuses on the concept and application of supervised learning in medicine, the most commonly used machine learning approach in medicine, and provides a brief explanation of four algorithms widely used for prediction (random forests, gradient-boosted decision tree, support vector machine, and neural network). Among these algorithms, the neural network has further developed into deep learning algorithms to solve more complex tasks. Along with simple classification problems, deep learning is commonly used to process medical imaging, such as retinal fundus photographs for diabetic retinopathy diagnosis. Although machine learning can bring new insights into medicine by processing a vast amount of data that are often beyond human capacity, algorithms can also fail when domain knowledge is neglected. The combination of algorithms and human cognitive ability is a key to the successful application of machine learning in medicine.
Collapse
Affiliation(s)
- Sachiko Ono
- Department of Eat-loss Medicine, Graduate School of Medicine, The University of Tokyo
| | - Tadahiro Goto
- Department of Clinical Epidemiology and Health Economics, The University of Tokyo
- TXP Medical Co. Ltd
| |
Collapse
|
52
|
Han JH, Yoon SJ, Lee HS, Park G, Lim J, Shin JE, Eun HS, Park MS, Lee SM. Application of Machine Learning Approaches to Predict Postnatal Growth Failure in Very Low Birth Weight Infants. Yonsei Med J 2022; 63:640-647. [PMID: 35748075 PMCID: PMC9226835 DOI: 10.3349/ymj.2022.63.7.640] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/24/2021] [Revised: 03/22/2022] [Accepted: 03/25/2022] [Indexed: 01/05/2023] Open
Abstract
PURPOSE The aims of the study were to develop and evaluate a machine learning model with which to predict postnatal growth failure (PGF) among very low birth weight (VLBW) infants. MATERIALS AND METHODS Of 10425 VLBW infants registered in the Korean Neonatal Network between 2013 and 2017, 7954 infants were included. PGF was defined as a decrease in Z score >1.28 at discharge, compared to that at birth. Six metrics [area under the receiver operating characteristic curve (AUROC), accuracy, precision, sensitivity, specificity, and F1 score] were obtained at five time points (at birth, 7 days, 14 days, 28 days after birth, and at discharge). Machine learning models were built using four different techniques [extreme gradient boosting (XGB), random forest, support vector machine, and convolutional neural network] to compare against the conventional multiple logistic regression (MLR) model. RESULTS The XGB algorithm showed the best performance with all six metrics across the board. When compared with MLR, XGB showed a significantly higher AUROC (p=0.03) for Day 7, which was the primary performance metric. Using optimal cut-off points, for Day 7, XGB still showed better performances in terms of AUROC (0.74), accuracy (0.68), and F1 score (0.67). AUROC values seemed to increase slightly from birth to 7 days after birth with significance, almost reaching a plateau after 7 days after birth. CONCLUSION We have shown the possibility of predicting PGF through machine learning algorithms, especially XGB. Such models may help neonatologists in the early diagnosis of high-risk infants for PGF for early intervention.
Collapse
Affiliation(s)
- Jung Ho Han
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - So Jin Yoon
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - Hye Sun Lee
- Biostatistics Collaboration Unit, Yonsei University College of Medicine, Seoul, Korea
| | - Goeun Park
- Biostatistics Collaboration Unit, Yonsei University College of Medicine, Seoul, Korea
| | - Joohee Lim
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - Jeong Eun Shin
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - Ho Seon Eun
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - Min Soo Park
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea
| | - Soon Min Lee
- Department of Pediatrics, Yonsei University College of Medicine, Seoul, Korea.
| |
Collapse
|
53
|
Machine learning-based suggestion for critical interventions in the management of potentially severe conditioned patients in emergency department triage. Sci Rep 2022; 12:10537. [PMID: 35732641 PMCID: PMC9218081 DOI: 10.1038/s41598-022-14422-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2022] [Accepted: 06/07/2022] [Indexed: 12/05/2022] Open
Abstract
Providing timely intervention to critically ill patients is a challenging task in emergency departments (ED). Our study aimed to predict early critical interventions (CrIs), which can be used as clinical recommendations. This retrospective observational study was conducted in the ED of a tertiary hospital located in a Korean metropolitan city. Patient who visited ED from January 1, 2016, to December 31, 2018, were included. Need of six CrIs were selected as prediction outcomes, namely, arterial line (A-line) insertion, oxygen therapy, high-flow nasal cannula (HFNC), intubation, Massive Transfusion Protocol (MTP), and inotropes and vasopressor. Extreme gradient boosting (XGBoost) prediction model was built by using only data available at the initial stage of ED. Overall, 137,883 patients were included in the study. The areas under the receiver operating characteristic curve for the prediction of A-line insertion was 0·913, oxygen therapy was 0.909, HFNC was 0.962, intubation was 0.945, MTP was 0.920, and inotropes or vasopressor administration was 0.899 in the XGBoost method. In addition, an increase in the need for CrIs was associated with worse ED outcomes. The CrIs model was integrated into the study site's electronic medical record and could be used to suggest early interventions for emergency physicians.
Collapse
|
54
|
Fan H, Cui Y, Xu X, Zhang D, Yang D, Huang L, Ding T, Lu G. Validation of a Classification Model Using Complete Blood Count to Predict Severe Human Adenovirus Lower Respiratory Tract Infections in Pediatric Cases. Front Pediatr 2022; 10:896606. [PMID: 35712623 PMCID: PMC9197341 DOI: 10.3389/fped.2022.896606] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Accepted: 04/25/2022] [Indexed: 11/13/2022] Open
Abstract
Background Human adenovirus (HAdV) lower respiratory tract infections (LRTIs) are prone to severe cases and even cause death in children. Here, we aimed to develop a classification model to predict severity in pediatric patients with HAdV LRTIs using complete blood count (CBC). Methods The CBC parameters from pediatric patients with a diagnosis of HAdV LRTIs from 2013 to 2019 were collected during the disease's course. The data were analyzed as potential predictors for severe cases and were selected using a random forest model. Results We enrolled 1,652 CBC specimens from 1,069 pediatric patients with HAdV LRTIs in the present study. Four hundred and seventy-four patients from 2017 to 2019 were used as the discovery cohort, and 470 patients from 2013 to 2016 were used as the validation cohort. The monocyte ratio (MONO%) was the most obvious difference between the mild and severe groups at onset, and could be used as a marker for the early accurate prediction of the severity [area under the subject operating characteristic curve (AUROC): 0.843]. Four risk factors [MONO%, hematocrit (HCT), red blood cell count (RBC), and platelet count (PLT)] were derived to construct a classification model of severe and mild cases using a random forest model (AUROC: 0.931 vs. 0.903). Conclusion Monocyte ratio can be used as an individual predictor of severe cases in the early stages of HAdV LRTIs. The four risk factors model is a simple and accurate risk assessment tool that can predict severe cases in the early stages of HAdV LRTIs.
Collapse
Affiliation(s)
- Huifeng Fan
- Department of Respiration, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Ying Cui
- Department of Immunology, Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou, China
| | - Xuehua Xu
- Pediatric Intensive Care Unit, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Dongwei Zhang
- Pediatric Intensive Care Unit, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Diyuan Yang
- Department of Respiration, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Li Huang
- Pediatric Intensive Care Unit, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Tao Ding
- Department of Immunology, Zhongshan School of Medicine, Sun Yat-sen University, Guangzhou, China
- Key Laboratory of Tropical Disease Control, Ministry of Education, Sun Yat-sen University, Guangzhou, China
- Provincial Engineering Technology Research Center for Biological Vector Control, Guangzhou, China
| | - Gen Lu
- Department of Respiration, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
| |
Collapse
|
55
|
Prehospital Factors Predict Outcomes in Pediatric Trauma: A Principal Component Analysis. J Trauma Acute Care Surg 2022; 93:291-298. [PMID: 35546247 DOI: 10.1097/ta.0000000000003680] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
BACKGROUND Trauma team activation leveling decisions are complex and based on many variables. Accurate triage decisions improve patient safety and resource utilization. Our purpose was to establish proof-of-concept for using principal component analysis (PCA) to identify multivariate predictors of injury severity and to assess their ability to predict outcomes in pediatric trauma patients. We hypothesized that we could identify significant principal components (PCs) among variables used for decisions regarding trauma team activation and that PC scores would be predictive of outcomes in pediatric trauma. METHODS We conducted a retrospective review of the trauma registry (1/2014-12/2020) at our pediatric trauma center, including all pediatric patients (age < 18 y) who triggered a trauma team activation. Data included patient demographics, prehospital report, Injury Severity Score, and outcomes. Four significant principal components were identified using PCA. Differences in outcome variables between the highest and lowest quartile for PC score were examined. RESULTS 1090 pediatric patients were included. The 4 significant PCs accounted for >96% of the overall date variance. The first PC was a composite of prehospital Glasgow Coma Scale and Revised Trauma Score (RTS) and was predictive of outcomes, including injury severity, length of stay, and mortality. The second PC was characterized primarily by prehospital systolic blood pressure (SBP) and high PC scores were associated with increased length of stay. The third and fourth PCs were characterized by patient age and by prehospital RTS and SBP, respectively. CONCLUSIONS We demonstrate that, using information available at the time of trauma team activation, PCA can be used to identify key predictors of patient outcome. While the ultimate goal is to create a machine learning-based predictive tool to support and improve clinical decision making, this study serves as a crucial step toward developing a deep understanding of the features of the model and their behavior with actual clinical data. LEVEL OF EVIDENCE III; Diagnostic test.
Collapse
|
56
|
Mo M, Guilak F, Elward A, Quayle K, Thompson D, Brouillet K, Luhmann SJ. The Use of Biomarkers in the Early Diagnosis of Septic Arthritis and Osteomyelitis-A Pilot Study. J Pediatr Orthop 2022; 42:e526-e532. [PMID: 35405729 DOI: 10.1097/bpo.0000000000002052] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
BACKGROUND The diagnosis of septic arthritis (SA) and osteomyelitis (OM) has remained challenging in the pediatric population, often accompanied by delays and requiring invasive interventions. The purpose of this pilot study is to identify a novel panel of biomarkers and cytokines that can accurately differentiate SA and OM at initial presentation using serum alone. METHODS Twenty patients below 18 years old whose working diagnosis included SA (n=10) and OM (n=10) were identified. Serum was collected at initial evaluation. Each sample underwent seven ELISA [C1-C2, COMP, CS-846, hyaluronan, procalcitonin, PIIANP, C-terminal telopeptide of type II collagen (CTX-II)] and 65-plex cytokine panels. Principal component and Lasso regression analysis were performed to identify a limited set of predictive biomarkers. RESULTS Mean age was 4.7 and 9.5 years in SA and OM patients, respectively (P=0.029). 50% of SA patients presented within 24 hours of symptom onset, compared with 0% of OM patients (P=0.033). 30% of SA patients were discharged home with an incorrect diagnosis and re-presented to the emergency department days later. At time of presentation: temperature ≥38.5°C was present in 10% of SA and 40% of OM patients (P=0.12), mean erythrocyte sedimentation rate (mm/h) was 51.6 in SA and 44.9 in OM patients (P=0.63), mean C-reactive protein (mg/dL) was 55.8 in SA and 71.8 in OM patients (P=0.53), and mean white blood cells (K/mm3) was 12.5 in SA and 10.4 in OM patients (P=0.34). 90% of SA patients presented with ≤2 of the Kocher criteria. 100% of SA and 40% of OM patients underwent surgery. 70% of SA cultures were culture negative, 10% MSSA, 10% Kingella, and 10% Strep pyogenes. 40% of OM cultures were culture negative, 50% MSSA, and 10% MRSA. Four biomarkers [CTx-II, transforming growth factor alpha (TGF-α), monocyte chemoattractant protein 1 (MCP-1), B cell-attracting chemokine 1] were identified that were able to classify and differentiate 18 of the 20 SA and OM cases correctly, with 90% sensitivity and 80% specificity. CONCLUSIONS This pilot study identifies a panel of biomarkers that can differentiate between SA and OM at initial presentation using serum alone. LEVEL OF EVIDENCE Level II-diagnostic study.
Collapse
Affiliation(s)
| | - Farshid Guilak
- Departments of Orthopedic Surgery
- Shriners Hospitals for Children, St. Louis, MO
| | | | - Kimberly Quayle
- Emergency Medicine, Washington University School of Medicine, Saint Louis Children's Hospital
| | - Dominic Thompson
- Departments of Orthopedic Surgery
- Shriners Hospitals for Children, St. Louis, MO
| | | | | |
Collapse
|
57
|
Filipow N, Main E, Sebire NJ, Booth J, Taylor AM, Davies G, Stanojevic S. Implementation of prognostic machine learning algorithms in paediatric chronic respiratory conditions: a scoping review. BMJ Open Respir Res 2022; 9:9/1/e001165. [PMID: 35297371 PMCID: PMC8928277 DOI: 10.1136/bmjresp-2021-001165] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Accepted: 03/06/2022] [Indexed: 11/23/2022] Open
Abstract
Machine learning (ML) holds great potential for predicting clinical outcomes in heterogeneous chronic respiratory diseases (CRD) affecting children, where timely individualised treatments offer opportunities for health optimisation. This paper identifies rate-limiting steps in ML prediction model development that impair clinical translation and discusses regulatory, clinical and ethical considerations for ML implementation. A scoping review of ML prediction models in paediatric CRDs was undertaken using the PRISMA extension scoping review guidelines. From 1209 results, 25 articles published between 2013 and 2021 were evaluated for features of a good clinical prediction model using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines. Most of the studies were in asthma (80%), with few in cystic fibrosis (12%), bronchiolitis (4%) and childhood wheeze (4%). There were inconsistencies in model reporting and studies were limited by a lack of validation, and absence of equations or code for replication. Clinician involvement during ML model development is essential and diversity, equity and inclusion should be assessed at each step of the ML pipeline to ensure algorithms do not promote or amplify health disparities among marginalised groups. As ML prediction studies become more frequent, it is important that models are rigorously developed using published guidelines and take account of regulatory frameworks which depend on model complexity, patient safety, accountability and liability.
Collapse
Affiliation(s)
- Nicole Filipow
- UCL Great Ormond Street Institute of Child Health, University College London, London, UK
| | - Eleanor Main
- UCL Great Ormond Street Institute of Child Health, University College London, London, UK
| | - Neil J Sebire
- Population, Policy and Practice Research and Teaching Department, UCL Great Ormond Street Institute of Child Health, University College London, London, UK.,GOSH NIHR BRC, Great Ormond Street Hospital for Children, London, UK
| | - John Booth
- Population, Policy and Practice Research and Teaching Department, UCL Great Ormond Street Institute of Child Health, University College London, London, UK.,GOSH NIHR BRC, Great Ormond Street Hospital for Children, London, UK
| | - Andrew M Taylor
- GOSH NIHR BRC, Great Ormond Street Hospital for Children, London, UK.,Institute of Cardiovascular Science, University College London, London, UK
| | - Gwyneth Davies
- Population, Policy and Practice Research and Teaching Department, UCL Great Ormond Street Institute of Child Health, University College London, London, UK.,GOSH NIHR BRC, Great Ormond Street Hospital for Children, London, UK
| | - Sanja Stanojevic
- Community Health and Epidemiology, Dalhousie University, Halifax, Nova Scotia, Canada
| |
Collapse
|
58
|
Rodriguez PJ, Veenstra DL, Heagerty PJ, Goss CH, Ramos KJ, Bansal A. A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2022; 25:350-358. [PMID: 35227445 PMCID: PMC9311314 DOI: 10.1016/j.jval.2021.11.1360] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Revised: 10/19/2021] [Accepted: 11/16/2021] [Indexed: 05/06/2023]
Abstract
OBJECTIVES We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study. METHODS We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV1) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects. RESULTS Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV1 and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV1 and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years. CONCLUSIONS Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.
Collapse
Affiliation(s)
- Patricia J Rodriguez
- The Comparative Health Outcomes, Policy & Economics (CHOICE) Institute, University of Washington, Seattle, WA, USA.
| | - David L Veenstra
- The Comparative Health Outcomes, Policy & Economics (CHOICE) Institute, University of Washington, Seattle, WA, USA
| | | | - Christopher H Goss
- Division of Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, University of Washington, Seattle, WA, USA; Division of Pulmonology, Department of Pediatrics, University of Washington, Seattle, WA, USA
| | - Kathleen J Ramos
- Division of Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, University of Washington, Seattle, WA, USA
| | - Aasthaa Bansal
- The Comparative Health Outcomes, Policy & Economics (CHOICE) Institute, University of Washington, Seattle, WA, USA.
| |
Collapse
|
59
|
Clarke SL, Parmesar K, Saleem MA, Ramanan AV. Future of machine learning in paediatrics. Arch Dis Child 2022; 107:223-228. [PMID: 34301619 DOI: 10.1136/archdischild-2020-321023] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Accepted: 07/16/2021] [Indexed: 11/03/2022]
Abstract
Machine learning (ML) is a branch of artificial intelligence (AI) that enables computers to learn without being explicitly programmed, through a combination of statistics and computer science. It encompasses a variety of techniques used to analyse and interpret extremely large amounts of data, which can then be applied to create predictive models. Such applications of this technology are now ubiquitous in our day-to-day lives: predictive text, spam filtering, and recommendation systems in social media, streaming video and e-commerce to name a few examples. It is only more recently that ML has started to be implemented against the vast amount of data generated in healthcare. The emerging role of AI in refining healthcare delivery was recently highlighted in the 'National Health Service Long Term Plan 2019'. In paediatrics, workforce challenges, rising healthcare attendance and increased patient complexity and comorbidity mean that demands on paediatric services are also growing. As healthcare moves into this digital age, this review considers the potential impact ML can have across all aspects of paediatric care from improving workforce efficiency and aiding clinical decision-making to precision medicine and drug development.
Collapse
Affiliation(s)
- Sarah Ln Clarke
- MRC Integrative Epidemiology Unit, University of Bristol, Bristol, UK
- School of Population Health Sciences, University of Bristol, Bristol, UK
- Department of Paediatric Rheumatology, Bristol Royal Hospital for Children, Bristol, UK
| | - Kevon Parmesar
- School of Population Health Sciences, University of Bristol, Bristol, UK
| | - Moin A Saleem
- Bristol Renal, University of Bristol, Bristol, UK
- Children's Renal Unit, Bristol Royal Hospital for Children, Bristol, UK
| | - Athimalaipet V Ramanan
- Department of Paediatric Rheumatology, Bristol Royal Hospital for Children, Bristol, UK
- School of Translational Health Sciences, University of Bristol, Bristol, UK
| |
Collapse
|
60
|
Kim SW, Kim YW, Min YH, Lee KJ, Choi HJ, Kim DW, Jo YH, Lee DK. Development and Validation of Simple Age-Adjusted Objectified Korean Triage and Acuity Scale for Adult Patients Visiting the Emergency Department. Yonsei Med J 2022; 63:272-281. [PMID: 35184430 PMCID: PMC8860940 DOI: 10.3349/ymj.2022.63.3.272] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 11/10/2021] [Accepted: 11/12/2021] [Indexed: 11/27/2022] Open
Abstract
PURPOSE The study aimed to develop an objectified Korean Triage and Acuity Scale (OTAS) that can objectively and quickly classify severity, as well as a simple age-adjusted OTAS (S-OTAS) that reflects age and evaluate its usefulness. MATERIALS AND METHODS A retrospective analysis was performed of all adult patients who had visited the emergency department at three teaching hospitals. Sex, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, O2 saturation, and consciousness level were collected from medical records. The OTAS was developed with objective criterion and minimal OTAS level, and S-OTAS was developed by adding the age variable. For usefulness evaluation, the 30-day mortality, the rates of computed tomography scan and emergency procedures were compared between Korean Triage and Acuity Scale (KTAS) and OTAS. RESULTS A total of 44402 patients were analyzed. For 30-day mortality, S-OTAS showed a higher area under the curve (AUC) compared to KTAS (0.751 vs. 0.812 for KTAS and S-OTAS, respectively, p<0.001). Regarding the rates of emergency procedures, AUC was significantly higher in S-OTAS, compared to KTAS (0.807 vs. 0.830, for KTAS and S-OTAS, respectively, p=0.013). CONCLUSION S-OTAS showed comparative usefulness for adult patients visiting the emergency department as a triage tool compared to KTAS.
Collapse
Affiliation(s)
- Seung Wook Kim
- Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Yong Won Kim
- Department of Emergency Medicine, Dongguk University Ilsan Hospital, Goyang, Korea
| | - Yong Hun Min
- Department of Emergency Medicine, Pohang St. Mary's Hospital, Pohang, Korea
| | - Kui Ja Lee
- Department of Emergency Medical Services, Kyungdong University, Wonju, Korea
| | - Hyo Ju Choi
- Department of Emergency Medical Services, Kyungdong University, Wonju, Korea
| | - Dong Won Kim
- Department of Emergency Medicine, Chuncheon Sacred Heart Hospital, Hallym University College of Medicine, Chuncheon, Korea.
| | - You Hwan Jo
- Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, Korea
| | - Dong Keon Lee
- Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, Korea.
| |
Collapse
|
61
|
Hwang S, Lee B. Machine learning-based prediction of critical illness in children visiting the emergency department. PLoS One 2022; 17:e0264184. [PMID: 35176113 PMCID: PMC8853514 DOI: 10.1371/journal.pone.0264184] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Accepted: 02/04/2022] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVES Triage is an essential emergency department (ED) process designed to provide timely management depending on acuity and severity; however, the process may be inconsistent with clinical and hospitalization outcomes. Therefore, studies have attempted to augment this process with machine learning models, showing advantages in predicting critical conditions and hospitalization outcomes. The aim of this study was to utilize nationwide registry data to develop a machine learning-based classification model to predict the clinical course of pediatric ED visits. METHODS This cross-sectional observational study used data from the National Emergency Department Information System on emergency visits of children under 15 years of age from January 1, 2016, to December 31, 2017. The primary and secondary outcomes were to identify critically ill children and predict hospitalization from triage data, respectively. We developed and tested a random forest model with the under sampled dataset and validated the model using the entire dataset. We compared the model's performance with that of the conventional triage system. RESULTS A total of 2,621,710 children were eligible for the analysis and included 12,951 (0.5%) critical outcomes and 303,808 (11.6%) hospitalizations. After validation, the area under the receiver operating characteristic curve was 0.991 (95% confidence interval [CI] 0.991-0.992) for critical outcomes and 0.943 (95% CI 0.943-0.944) for hospitalization, which were higher than those of the conventional triage system. CONCLUSIONS The machine learning-based model using structured triage data from a nationwide database can effectively predict critical illness and hospitalizations among children visiting the ED.
Collapse
Affiliation(s)
- Soyun Hwang
- Department of Pediatrics, Severance Children’s Hospital, Yonsei University College of Medicine, Seoul, Korea
| | - Bongjin Lee
- Department of Pediatrics, Seoul National University Hospital, Seoul, Korea
| |
Collapse
|
62
|
Stokes K, Castaldo R, Federici C, Pagliara S, Maccaro A, Cappuccio F, Fico G, Salvatore M, Franzese M, Pecchia L. The use of artificial intelligence systems in diagnosis of pneumonia via signs and symptoms: A systematic review. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103325] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
|
63
|
Liu YC, Cheng HY, Chang TH, Ho TW, Liu TC, Yen TY, Chou CC, Chang LY, Lai F. Evaluation of the Need for Intensive Care in Children With Pneumonia: Machine Learning Approach. JMIR Med Inform 2022; 10:e28934. [PMID: 35084358 PMCID: PMC8832265 DOI: 10.2196/28934] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 07/01/2021] [Accepted: 01/02/2022] [Indexed: 01/20/2023] Open
Abstract
Background Timely decision-making regarding intensive care unit (ICU) admission for children with pneumonia is crucial for a better prognosis. Despite attempts to establish a guideline or triage system for evaluating ICU care needs, no clinically applicable paradigm is available. Objective The aim of this study was to develop machine learning (ML) algorithms to predict ICU care needs for pediatric pneumonia patients within 24 hours of admission, evaluate their performance, and identify clinical indices for making decisions for pediatric pneumonia patients. Methods Pneumonia patients admitted to National Taiwan University Hospital from January 2010 to December 2019 aged under 18 years were enrolled. Their underlying diseases, clinical manifestations, and laboratory data at admission were collected. The outcome of interest was ICU transfer within 24 hours of hospitalization. We compared clinically relevant features between early ICU transfer patients and patients without ICU care. ML algorithms were developed to predict ICU admission. The performance of the algorithms was evaluated using sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and average precision. The relative feature importance of the best-performing algorithm was compared with physician-rated feature importance for explainability. Results A total of 8464 pediatric hospitalizations due to pneumonia were recorded, and 1166 (1166/8464, 13.8%) hospitalized patients were transferred to the ICU within 24 hours. Early ICU transfer patients were younger (P<.001), had higher rates of underlying diseases (eg, cardiovascular, neuropsychological, and congenital anomaly/genetic disorders; P<.001), had abnormal laboratory data, had higher pulse rates (P<.001), had higher breath rates (P<.001), had lower oxygen saturation (P<.001), and had lower peak body temperature (P<.001) at admission than patients without ICU transfer. The random forest (RF) algorithm achieved the best performance (sensitivity 0.94, 95% CI 0.92-0.95; specificity 0.94, 95% CI 0.92-0.95; AUC 0.99, 95% CI 0.98-0.99; and average precision 0.93, 95% CI 0.90-0.96). The lowest systolic blood pressure and presence of cardiovascular and neuropsychological diseases ranked in the top 10 in both RF relative feature importance and clinician judgment. Conclusions The ML approach could provide a clinically applicable triage algorithm and identify important clinical indices, such as age, underlying diseases, abnormal vital signs, and laboratory data for evaluating the need for intensive care in children with pneumonia.
Collapse
Affiliation(s)
- Yun-Chung Liu
- Department of Pediatrics, National Taiwan University Hospital, College of Medicine, National Taiwan University, Taipei City, Taiwan.,Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei City, Taiwan
| | - Hao-Yuan Cheng
- Department of Pediatrics, National Taiwan University Hospital, College of Medicine, National Taiwan University, Taipei City, Taiwan.,Taiwan Centers for Disease Control, Taipei City, Taiwan
| | - Tu-Hsuan Chang
- Department of Pediatrics, Chi Mei Medical Center, Tainan City, Taiwan
| | - Te-Wei Ho
- Department of Surgery, College of Medicine, National Taiwan University, Taipei City, Taiwan
| | - Ting-Chi Liu
- Institute of Applied Mechanics, National Taiwan University, Taipei City, Taiwan.,Department of Civil Engineering, National Taiwan University, Taipei City, Taiwan
| | - Ting-Yu Yen
- Department of Pediatrics, National Taiwan University Hospital, College of Medicine, National Taiwan University, Taipei City, Taiwan
| | - Chia-Ching Chou
- Institute of Applied Mechanics, National Taiwan University, Taipei City, Taiwan
| | - Luan-Yin Chang
- Department of Pediatrics, National Taiwan University Hospital, College of Medicine, National Taiwan University, Taipei City, Taiwan
| | - Feipei Lai
- Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei City, Taiwan.,Department of Computer Science and Information Engineering, National Taiwan University, Taipei City, Taiwan.,Department of Electrical Engineering, National Taiwan University, Taipei City, Taiwan
| |
Collapse
|
64
|
Yamanaka S, Goto T, Morikawa K, Watase H, Okamoto H, Hagiwara Y, Hasegawa K. Machine Learning Approaches for Predicting Difficult Airway and First-Pass Success in the Emergency Department: Multicenter Prospective Observational Study. Interact J Med Res 2022; 11:e28366. [PMID: 35076398 PMCID: PMC8826144 DOI: 10.2196/28366] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 06/07/2021] [Accepted: 12/06/2021] [Indexed: 12/13/2022] Open
Abstract
Background There is still room for improvement in the modified LEMON (look, evaluate, Mallampati, obstruction, neck mobility) criteria for difficult airway prediction and no prediction tool for first-pass success in the emergency department (ED). Objective We applied modern machine learning approaches to predict difficult airways and first-pass success. Methods In a multicenter prospective study that enrolled consecutive patients who underwent tracheal intubation in 13 EDs, we developed 7 machine learning models (eg, random forest model) using routinely collected data (eg, demographics, initial airway assessment). The outcomes were difficult airway and first-pass success. Model performance was evaluated using c-statistics, calibration slopes, and association measures (eg, sensitivity) in the test set (randomly selected 20% of the data). Their performance was compared with the modified LEMON criteria for difficult airway success and a logistic regression model for first-pass success. Results Of 10,741 patients who underwent intubation, 543 patients (5.1%) had a difficult airway, and 7690 patients (71.6%) had first-pass success. In predicting a difficult airway, machine learning models—except for k-point nearest neighbor and multilayer perceptron—had higher discrimination ability than the modified LEMON criteria (all, P≤.001). For example, the ensemble method had the highest c-statistic (0.74 vs 0.62 with the modified LEMON criteria; P<.001). Machine learning models—except k-point nearest neighbor and random forest models—had higher discrimination ability for first-pass success. In particular, the ensemble model had the highest c-statistic (0.81 vs 0.76 with the reference regression; P<.001). Conclusions Machine learning models demonstrated greater ability for predicting difficult airway and first-pass success in the ED.
Collapse
Affiliation(s)
- Syunsuke Yamanaka
- Department of Emergency Medicine & General Internal Medicine, The University of Fukui, Fukui, Japan
| | - Tadahiro Goto
- Department of Clinical Epidemiology & Health Economics, School of Public Health, The University of Tokyo, Tokyo, Japan
| | | | - Hiroko Watase
- Department of Surgery, University of Washington, Seattle, WA, United States
| | - Hiroshi Okamoto
- Department of Intensive Care, St. Luke's International Hospital, Tokyo, Japan
| | - Yusuke Hagiwara
- Department of Pediatric Emergency and Critical Care Medicine, Tokyo Metropolitan Children's Medical Center, Tokyo, Japan
| | - Kohei Hasegawa
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, United States
| |
Collapse
|
65
|
Su D, Li Q, Zhang T, Veliz P, Chen Y, He K, Mahajan P, Zhang X. Prediction of acute appendicitis among patients with undifferentiated abdominal pain at emergency department. BMC Med Res Methodol 2022; 22:18. [PMID: 35026994 PMCID: PMC8759254 DOI: 10.1186/s12874-021-01490-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2020] [Accepted: 12/08/2021] [Indexed: 11/12/2022] Open
Abstract
Background Early screening and accurately identifying Acute Appendicitis (AA) among patients with undifferentiated symptoms associated with appendicitis during their emergency visit will improve patient safety and health care quality. The aim of the study was to compare models that predict AA among patients with undifferentiated symptoms at emergency visits using both structured data and free-text data from a national survey. Methods We performed a secondary data analysis on the 2005-2017 United States National Hospital Ambulatory Medical Care Survey (NHAMCS) data to estimate the association between emergency department (ED) patients with the diagnosis of AA, and the demographic and clinical factors present at ED visits during a patient’s ED stay. We used binary logistic regression (LR) and random forest (RF) models incorporating natural language processing (NLP) to predict AA diagnosis among patients with undifferentiated symptoms. Results Among the 40,441 ED patients with assigned International Classification of Diseases (ICD) codes of AA and appendicitis-related symptoms between 2005 and 2017, 655 adults (2.3%) and 256 children (2.2%) had AA. For the LR model identifying AA diagnosis among adult ED patients, the c-statistic was 0.72 (95% CI: 0.69–0.75) for structured variables only, 0.72 (95% CI: 0.69–0.75) for unstructured variables only, and 0.78 (95% CI: 0.76–0.80) when including both structured and unstructured variables. For the LR model identifying AA diagnosis among pediatric ED patients, the c-statistic was 0.84 (95% CI: 0.79–0.89) for including structured variables only, 0.78 (95% CI: 0.72–0.84) for unstructured variables, and 0.87 (95% CI: 0.83–0.91) when including both structured and unstructured variables. The RF method showed similar c-statistic to the corresponding LR model. Conclusions We developed predictive models that can predict the AA diagnosis for adult and pediatric ED patients, and the predictive accuracy was improved with the inclusion of NLP elements and approaches. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-021-01490-9.
Collapse
Affiliation(s)
- Dai Su
- Department of Health Management and Policy, School of Public Health, Capital Medical University, Beijing, China
| | - Qinmengge Li
- Department of Systems, Populations, and Leadership, University of Michigan School of Nursing, Ann Arbor, USA.,Department of Biostatistics, University of Michigan School of Public Health, Ann Arbor, USA
| | - Tao Zhang
- Department of Epidemiology and Biostatistics, West China School of Public Health School, Sichuan University, Chengdu, China
| | - Philip Veliz
- Department of Systems, Populations, and Leadership, University of Michigan School of Nursing, Ann Arbor, USA
| | - Yingchun Chen
- Department of Health Management, School of Medicine and Health Management, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.,Research Center for Rural Health Services, Hubei Province Key Research Institute of Humanities and Social Sciences, Wuhan, China
| | - Kevin He
- Department of Biostatistics, University of Michigan School of Public Health, Ann Arbor, USA
| | - Prashant Mahajan
- Department of Emergency Medicine, University of Michigan School of Medicine, Ann Arbor, USA
| | - Xingyu Zhang
- Thomas E. Starzl Transplantation Institute, University of Pittsburgh Medical Center, Pittsburgh, USA.
| |
Collapse
|
66
|
Pienaar MA, Sempa JB, Luwes N, George EC, Brown SC. Development of artificial neural network models for paediatric critical illness in South Africa. Front Pediatr 2022; 10:1008840. [PMID: 36458145 PMCID: PMC9705750 DOI: 10.3389/fped.2022.1008840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Accepted: 10/17/2022] [Indexed: 11/16/2022] Open
Abstract
OBJECTIVES Failures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation. DESIGN Prospective, analytical cohort study. SETTING A single centre tertiary hospital in South Africa providing acute paediatric services. PATIENTS Children, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations. OUTCOMES Predictive models for a composite outcome of death before discharge from hospital or admission to the PICU. INTERVENTIONS None. MEASUREMENTS AND MAIN RESULTS 765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit. CONCLUSIONS All models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.
Collapse
Affiliation(s)
- Michael A Pienaar
- Paediatric Critical Care Unit, Department of Paediatrics and Child Health, University of the Free State, Bloemfontein, South Africa
| | - Joseph B Sempa
- Department of Biostatistics, Faculty of Health Sciences, University of the Free State, Bloemfontein, South Africa
| | - Nicolaas Luwes
- Department of Electrical, Electronic and Computer Engineering, Faculty of Engineering, Built Environment and Information Technology, Central University of Technology, Bloemfontein, South Africa
| | - Elizabeth C George
- Medical Research Council Clinical Trials Unit, University College London, London, United Kingdom
| | - Stephen C Brown
- Paediatric Cardiology Unit, Department of Paediatrics and Child Health, University of the Free State, Bloemfontein, South Africa
| |
Collapse
|
67
|
Sánchez-Salmerón R, Gómez-Urquiza JL, Albendín-García L, Correa-Rodríguez M, Martos-Cabrera MB, Velando-Soriano A, Suleiman-Martos N. Machine learning methods applied to triage in emergency services: A systematic review. Int Emerg Nurs 2021; 60:101109. [PMID: 34952482 DOI: 10.1016/j.ienj.2021.101109] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2021] [Revised: 08/23/2021] [Accepted: 10/22/2021] [Indexed: 12/23/2022]
Abstract
BACKGROUND In emergency services is important to accurately assess and classify symptoms, which may be improved with the help of technology. One mechanism that could help and improve predictions from health records or patient flow is machine learning (ML). AIM To analyse the effectiveness of ML systems in triage for making predictions at the emergency department in comparison with other triage scales/scores. METHODS Following the PRISMA recommendations, a systematic review was conducted using CINAHL, Cochrane, Cuiden, Medline and Scopus databases with the search equation "Machine learning AND triage AND emergency". RESULTS Eleven studies were identified. The studies show that the use of ML methods consistently predict important outcomes like mortality, critical care outcomes and admission, and the need for hospitalization in comparison with scales like Emergency Severity Index or others. Among the ML models considered, XGBoost and Deep Neural Networks obtained the highest levels of prediction accuracy, while Logistic Regression performed obtained the worst values. CONCLUSIONS Machine learning methods can be a good instrument for helping triage process with the prediction of important emergency variables like mortality or the need for critical care or hospitalization.
Collapse
Affiliation(s)
| | - José L Gómez-Urquiza
- Faculty of Health Sciences, University of Granada, Avenida de la Ilustración N. 60, 18016 Granada, Spain.
| | - Luis Albendín-García
- Faculty of Health Sciences, University of Granada, Avenida de la Ilustración N. 60, 18016 Granada, Spain.
| | - María Correa-Rodríguez
- Faculty of Health Sciences, University of Granada, Avenida de la Ilustración N. 60, 18016 Granada, Spain.
| | - María Begoña Martos-Cabrera
- San Cecilio Clinical University Hospital, Andalusian Health Service, Avenida del Conocimiento s/n, 18016 Granada, Spain.
| | - Almudena Velando-Soriano
- San Cecilio Clinical University Hospital, Andalusian Health Service, Avenida del Conocimiento s/n, 18016 Granada, Spain.
| | - Nora Suleiman-Martos
- Faculty of Health Sciences, Ceuta University Campus, University of Granada, C/Cortadura del Valle SN, 51001 Ceuta, Spain.
| |
Collapse
|
68
|
Greer ML, Davis K, Stack BC. Machine learning can identify patients at risk of hyperparathyroidism without known calcium and intact parathyroid hormone. Head Neck 2021; 44:817-822. [PMID: 34953008 DOI: 10.1002/hed.26970] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Revised: 11/01/2021] [Accepted: 12/16/2021] [Indexed: 01/16/2023] Open
Abstract
BACKGROUND To prove the concept of diagnosing primary hyperparathyroidism (pHPT) without calcium and parathyroid hormone (PTH) values and identifying potential risk factors for pHPT. METHODS Data were extracted from the clinical data warehouse (CDW) at the University of Arkansas for Medical Sciences (UAMS) Epic EHR (2014-2019). RESULTS 1737 patients with over 185 000 rows of clinical data were provided in a relational structure and processed/flattened to facilitate modeling. Phenotype elements were identified for pHPT without advance knowledge of calcium and PTH levels. The area under the curve (AUC) for the prediction of pHPT using our model was 0.86 with sensitivity and specificity of 0.8953 and 0.6686, respectively, using a 0.45 probability threshold. CONCLUSION Primary hyperparathyroidism was predicted from a dataset excluding calcium and PTH data with 86% accuracy. This approach needs to be validated/refined on larger samples of data and plans are in place to do this with other regional/national datasets.
Collapse
Affiliation(s)
- Melody L Greer
- Department of Health Informatics, University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA
| | - Kyle Davis
- Department of Otolaryngology - Head and Neck Surgery, University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA
| | - Brendan C Stack
- Department of Otolaryngology - Head and Neck Surgery, Southern Illinois University School of Medicine, Springfield, Illinois, USA
| |
Collapse
|
69
|
Liu Y, Gao J, Liu J, Walline JH, Liu X, Zhang T, Wu Y, Wu J, Zhu H, Zhu W. Development and validation of a practical machine-learning triage algorithm for the detection of patients in need of critical care in the emergency department. Sci Rep 2021; 11:24044. [PMID: 34911945 PMCID: PMC8674324 DOI: 10.1038/s41598-021-03104-2] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2021] [Accepted: 11/18/2021] [Indexed: 12/23/2022] Open
Abstract
Identifying critically ill patients is a key challenge in emergency department (ED) triage. Mis-triage errors are still widespread in triage systems around the world. Here, we present a machine learning system (MLS) to assist ED triage officers better recognize critically ill patients and provide a text-based explanation of the MLS recommendation. To derive the MLS, an existing dataset of 22,272 patient encounters from 2012 to 2019 from our institution’s electronic emergency triage system (EETS) was used for algorithm training and validation. The area under the receiver operating characteristic curve (AUC) was 0.875 ± 0.006 (CI:95%) in retrospective dataset using fivefold cross validation, higher than that of reference model (0.843 ± 0.005 (CI:95%)). In the prospective cohort study, compared to the traditional triage system’s 1.2% mis-triage rate, the mis-triage rate in the MLS-assisted group was 0.9%. This MLS method with a real-time explanation for triage officers was able to lower the mis-triage rate of critically ill ED patients.
Collapse
Affiliation(s)
- Yecheng Liu
- Emergency Department, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Jiandong Gao
- Department of Electronic Engineering, Tsinghua University, Beijing, China.,Center for Big Data and Clinical Research, Institute for Precision Medicine, Tsinghua University, Beijing, China
| | - Jihai Liu
- Emergency Department, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Joseph Harold Walline
- Accident and Emergency Medicine Academic Unit, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, Hong Kong, China
| | - Xiaoying Liu
- Emergency Department, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Ting Zhang
- Emergency Department, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Yunyang Wu
- Department of Electronic Engineering, Tsinghua University, Beijing, China
| | - Ji Wu
- Department of Electronic Engineering, Tsinghua University, Beijing, China. .,Center for Big Data and Clinical Research, Institute for Precision Medicine, Tsinghua University, Beijing, China.
| | - Huadong Zhu
- Emergency Department, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China.
| | - Weiguo Zhu
- Department of Information Center, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China.
| |
Collapse
|
70
|
Stirling PHC, Strelzow JA, Doornberg JN, White TO, McQueen MM, Duckworth AD. Diagnosis of Suspected Scaphoid Fractures. JBJS Rev 2021; 9:01874474-202112000-00001. [PMID: 34879033 DOI: 10.2106/jbjs.rvw.20.00247] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
» Suspected scaphoid fractures are a diagnostic and therapeutic challenge despite the advances in knowledge regarding these injuries and imaging techniques. The risks and restrictions of routine immobilization as well as the restriction of activities in a young and active population must be weighed against the risks of nonunion that are associated with a missed fracture. » The prevalence of true fractures among suspected fractures is low. This greatly reduces the statistical probability that a positive diagnostic test will correspond with a true fracture, reducing the positive predictive value of an investigation. » There is no consensus reference standard for a true fracture; therefore, alternative statistical methods for calculating sensitivity, specificity, and positive and negative predictive values are required. » Clinical prediction rules that incorporate a set of demographic and clinical factors may allow stratification of secondary imaging, which, in turn, could increase the pretest probability of a scaphoid fracture and improve the diagnostic performance of the sophisticated radiographic investigations that are available. » Machine-learning-derived probability calculators may augment risk stratification and can improve through retraining, although these theoretical benefits need further prospective evaluation. » Convolutional neural networks (CNNs) are a form of artificial intelligence that have demonstrated great promise in the recognition of scaphoid fractures on radiographs. However, in the more challenging diagnostic scenario of a suspected or so-called "clinical" scaphoid fracture, CNNs have not yet proven superior to a diagnosis that has been made by an experienced surgeon.
Collapse
Affiliation(s)
- Paul H C Stirling
- Edinburgh Orthopaedics and University of Edinburgh, Royal Infirmary of Edinburgh, Edinburgh, United Kingdom
| | - Jason A Strelzow
- Orthopaedic Surgery and Rehabilitation Medicine, The University of Chicago Medicine, Chicago, Illinois
| | - Job N Doornberg
- Department of Orthopaedic and Trauma Surgery, University Medical Centre Groningen, UMCG, Groningen, the Netherlands
- Department of Orthopaedic and Trauma Surgery, Flinders Medical Centre, Flinders University, Adelaide, South Australia, Australia
| | - Timothy O White
- Edinburgh Orthopaedics and University of Edinburgh, Royal Infirmary of Edinburgh, Edinburgh, United Kingdom
| | - Margaret M McQueen
- Edinburgh Orthopaedics and University of Edinburgh, Royal Infirmary of Edinburgh, Edinburgh, United Kingdom
| | - Andrew D Duckworth
- Edinburgh Orthopaedics and University of Edinburgh, Royal Infirmary of Edinburgh, Edinburgh, United Kingdom
| |
Collapse
|
71
|
Tarimo CS, Bhuyan SS, Li Q, Mahande MJJ, Wu J, Fu X. Validating machine learning models for the prediction of labour induction intervention using routine data: a registry-based retrospective cohort study at a tertiary hospital in northern Tanzania. BMJ Open 2021; 11:e051925. [PMID: 34857568 PMCID: PMC8647548 DOI: 10.1136/bmjopen-2021-051925] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
OBJECTIVES We aimed at identifying the important variables for labour induction intervention and assessing the predictive performance of machine learning algorithms. SETTING We analysed the birth registry data from a referral hospital in northern Tanzania. Since July 2000, every birth at this facility has been recorded in a specific database. PARTICIPANTS 21 578 deliveries between 2000 and 2015 were included. Deliveries that lacked information regarding the labour induction status were excluded. PRIMARY OUTCOME Deliveries involving labour induction intervention. RESULTS Parity, maternal age, body mass index, gestational age and birth weight were all found to be important predictors of labour induction. Boosting method demonstrated the best discriminative performance (area under curve, AUC=0.75: 95% CI (0.73 to 0.76)) while logistic regression presented the least (AUC=0.71: 95% CI (0.70 to 0.73)). Random forest and boosting algorithms showed the highest net-benefits as per the decision curve analysis. CONCLUSION All of the machine learning algorithms performed well in predicting the likelihood of labour induction intervention. Further optimisation of these classifiers through hyperparameter tuning may result in an improved performance. Extensive research into the performance of other classifier algorithms is warranted.
Collapse
Affiliation(s)
- Clifford Silver Tarimo
- College of Public Health, Zhengzhou University, Zhengzhou, China
- Science and Laboratory Technology, Dar es Salaam Institute of Technology, Dar es Salaam, Tanzania, United Republic of
| | - Soumitra S Bhuyan
- School of Planning and Public Policy, Rutgers University-New Brunswick, New York, New York, USA
| | - Quanman Li
- College of Public Health, Zhengzhou University, Zhengzhou, China
| | - Michael Johnson J Mahande
- Institute of Public Health, Kilimanjaro Christian Medical University College, Moshi, Tanzania, United Republic of
| | - Jian Wu
- College of Public Health, Zhengzhou University, Zhengzhou, China
| | - Xiaoli Fu
- College of Public Health, Zhengzhou University, Zhengzhou, China
| |
Collapse
|
72
|
Choi DH, Hong WP, Song KJ, Kim TH, Shin SD, Hong KJ, Park JH, Jeong J. Modification and Validation of a Complaint-Oriented Emergency Department Triage System: A Multicenter Observational Study. Yonsei Med J 2021; 62:1145-1154. [PMID: 34816645 PMCID: PMC8612858 DOI: 10.3349/ymj.2021.62.12.1145] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/09/2021] [Revised: 09/16/2021] [Accepted: 09/27/2021] [Indexed: 12/23/2022] Open
Abstract
PURPOSE The objective of this study was to modify and validate an emergency department (ED) triage system with improved prediction performance on hospital outcomes by modifying the Korean Triage and Acuity Scale (KTAS). MATERIALS AND METHODS We performed a retrospective observational study at three academic universities in South Korea. The KTAS code, determined by the chief complaint and the selected modifier of a patient, was used to derive the Modified KTAS (MKTAS). We calculated the area under the receiver operating characteristics curve (AUC) and the test characteristics to evaluate the performance of MKTAS to predict hospital mortality, critical outcome, and admission. RESULTS A total of 272402 and 128831 ED visits were used for the derivation and validation of MKTAS, respectively. Compared to KTAS, MKTAS had significantly higher AUC values for the prediction of hospital mortality [MKTAS 0.826 (0.818-0.835) vs. KTAS 0.794 (0.784-0.803)], critical outcome [MKTAS 0.836 (0.830-0.841) vs. 0.798 (0.792-0.804)], and admission [MKTAS 0.725 (0.723-0.728) vs. KTAS 0.685 (0.682-0.688)]. The sensitivity for predicting hospital mortality and critical outcome, as well as the specificity for predicting admission, were significantly improved. CONCLUSION MKTAS was derived by modifying the KTAS, and then validated. Compared with KTAS, MKTAS showed better discriminating ability to predict hospital outcomes. Continuous efforts to evaluate and modify widely used triage systems are required to improve their performance.
Collapse
Affiliation(s)
- Dong Hyun Choi
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Won Pyo Hong
- 119 EMS Division, National Fire Agency, Sejong, Korea.
| | - Kyoung Jun Song
- Department of Emergency Medicine, Seoul National University Boramae Medical Center, Seoul, Korea
| | - Tae Han Kim
- Department of Emergency Medicine, Seoul National University Boramae Medical Center, Seoul, Korea
| | - Sang Do Shin
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Ki Jeong Hong
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Jeong Ho Park
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Joo Jeong
- Department of Emergency Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
| |
Collapse
|
73
|
Cadario R, Longoni C, Morewedge CK. Understanding, explaining, and utilizing medical artificial intelligence. Nat Hum Behav 2021; 5:1636-1642. [PMID: 34183800 DOI: 10.1038/s41562-021-01146-0] [Citation(s) in RCA: 48] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2020] [Accepted: 05/27/2021] [Indexed: 02/06/2023]
Abstract
Medical artificial intelligence is cost-effective and scalable and often outperforms human providers, yet people are reluctant to use it. We show that resistance to the utilization of medical artificial intelligence is driven by both the subjective difficulty of understanding algorithms (the perception that they are a 'black box') and by an illusory subjective understanding of human medical decision-making. In five pre-registered experiments (1-3B: N = 2,699), we find that people exhibit an illusory understanding of human medical decision-making (study 1). This leads people to believe they better understand decisions made by human than algorithmic healthcare providers (studies 2A,B), which makes them more reluctant to utilize algorithmic than human providers (studies 3A,B). Fortunately, brief interventions that increase subjective understanding of algorithmic decision processes increase willingness to utilize algorithmic healthcare providers (studies 3A,B). A sixth study on Google Ads for an algorithmic skin cancer detection app finds that the effectiveness of such interventions generalizes to field settings (study 4: N = 14,013).
Collapse
Affiliation(s)
- Romain Cadario
- Rotterdam School of Management, Erasmus University, Rotterdam, the Netherlands.
| | - Chiara Longoni
- Questrom School of Business, Boston University, Boston, MA, USA
| | | |
Collapse
|
74
|
Miles J, Jacques R, Turner J, Mason S. The Safety INdEx of Prehospital On Scene Triage (SINEPOST) study: the development and validation of a risk prediction model to support ambulance clinical transport decisions on-scene-a protocol. Diagn Progn Res 2021; 5:18. [PMID: 34749832 PMCID: PMC8573562 DOI: 10.1186/s41512-021-00108-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Accepted: 10/25/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Demand for both the ambulance service and the emergency department (ED) is rising every year and when this demand is excessive in both systems, ambulance crews queue at the ED waiting to hand patients over. Some transported ambulance patients are 'low-acuity' and do not require the treatment of the ED. However, paramedics can find it challenging to identify these patients accurately. Decision support tools have been developed using expert opinion to help identify these low acuity patients but have failed to show a benefit beyond regular decision-making. Predictive algorithms may be able to build accurate models, which can be used in the field to support the decision not to take a low-acuity patient to an ED. METHODS AND ANALYSIS All patients in Yorkshire who were transported to the ED by ambulance between July 2019 and February 2020 will be included. Ambulance electronic patient care record (ePCR) clinical data will be used as candidate predictors for the model. These will then be linked to the corresponding ED record, which holds the outcome of a 'non-urgent attendance'. The estimated sample size is 52,958, with 4767 events and an EPP of 7.48. An XGBoost algorithm will be used for model development. Initially, a model will be derived using all the data and the apparent performance will be assessed. Then internal-external validation will use non-random nested cross-validation (CV) with test sets held out for each ED (spatial validation). After all models are created, a random-effects meta-analysis will be undertaken. This will pool performance measures such as goodness of fit, discrimination and calibration. It will also generate a prediction interval and measure heterogeneity between clusters. The performance of the full model will be updated with the pooled results. DISCUSSION Creating a risk prediction model in this area will lead to further development of a clinical decision support tool that ensures every ambulance patient can get to the right place of care, first time. If this study is successful, it could help paramedics evaluate the benefit of transporting a patient to the ED before they leave the scene. It could also reduce congestion in the urgent and emergency care system. TRIAL REGISTRATION This study was retrospectively registered with the ISRCTN: 12121281.
Collapse
Affiliation(s)
- Jamie Miles
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK.
- Yorkshire Ambulance Service, Brindley Way, Wakefield, WF2 0XQ, UK.
| | - Richard Jacques
- School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| | - Janette Turner
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| | - Suzanne Mason
- CURE Group, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
| |
Collapse
|
75
|
Li YX, Shen XP, Yang C, Cao ZZ, Du R, Yu MD, Wang JP, Wang M. Novelelectronic health records applied for prediction of pre-eclampsia: Machine-learning algorithms. Pregnancy Hypertens 2021; 26:102-109. [PMID: 34739939 DOI: 10.1016/j.preghy.2021.10.006] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2021] [Revised: 10/16/2021] [Accepted: 10/22/2021] [Indexed: 10/20/2022]
Abstract
OBJECTIVE To predict risk of pre-eclampsia (PE) in women using machine learning (ML) algorithms, based on electronic health records (EHR) collected at the early second trimester. STUDY DESIGN A total of 3759 cases of pregnancy who received antenatal care at Xinhua hospital Chongming branch Affiliated to Shanghai Jiaotong University were included in this retrospective EHR-based study. Thirty-eight candidate clinical parameters routinely available at the first visit in antenatal care were collected by manual chart review. Logistic regression (LR), random forest (RF), support vector machine (SVM) and extreme gradient boosting (XGBoost) were used to construct the prediction model. Features that contributed to the model predictions were identified using XGBoost. OUTCOME MEASURES The performance of ML models to predict women at risk of PE was quantified in terms of accuracy, precision, recall, false negative score, f1_score, brier score and the area under the receiver operating curve (auROC). RESULTS The XGboost model had the best prediction performance (accuracy = 0.920, precision = 0.447, recall = 0.789, f1_score = 0.571, auROC = 0.955). The most predictive feature of PE development was fasting plasma glucose, followed by mean blood pressure and body mass index. An easy-to-use model that a patient could answer independently still enabled accurate prediction, with auROC of 0.83. CONCLUSION risk of PE development can be predicted with excellent discriminative ability using ML algorithms based on EHR collected at the early second trimester. Future studies are needed to assess the real-world clinical utility of the model.
Collapse
Affiliation(s)
- Yi-Xin Li
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Xiao-Ping Shen
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Chao Yang
- Department of Scientific Research Centre, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Zuo-Zeng Cao
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Rui Du
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Min-da Yu
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Jun-Ping Wang
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Mei Wang
- Department of Obstetrics and Gynecology, Xinhua Hospital Chongming Branch, School of Medicine, Shanghai Jiao Tong University, Shanghai, China.
| |
Collapse
|
76
|
Wu TT, Zheng RF, Lin ZZ, Gong HR, Li H. A machine learning model to predict critical care outcomes in patient with chest pain visiting the emergency department. BMC Emerg Med 2021; 21:112. [PMID: 34620086 PMCID: PMC8496015 DOI: 10.1186/s12873-021-00501-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2021] [Accepted: 09/01/2021] [Indexed: 12/23/2022] Open
Abstract
Background Currently, the risk stratification of critically ill patient with chest pain is a challenge. We aimed to use machine learning approach to predict the critical care outcomes in patients with chest pain, and simultaneously compare its performance with HEART, GRACE, and TIMI scores. Methods This was a retrospective, case-control study in patients with acute non-traumatic chest pain who presented to the emergency department (ED) between January 2017 and December 2019. The outcomes included cardiac arrest, transfer to ICU, and death during treatment in ED. In the randomly sampled training set (70%), a LASSO regression model was developed, and presented with nomogram. The performance was measured in both training set (70% participants) and testing set (30% participants), and findings were compared with the three widely used scores. Results We proposed a LASSO regression model incorporating mode of arrival, reperfusion therapy, Killip class, systolic BP, serum creatinine, creatine kinase-MB, and brain natriuretic peptide as independent predictors of critical care outcomes in patients with chest pain. Our model significantly outperformed the HEART, GRACE, TIMI score with AUC of 0.953 (95%CI: 0.922–0.984), 0.754 (95%CI: 0.675–0.832), 0.747 (95%CI: 0.664–0.829), 0.735 (95%CI: 0.655–0.815), respectively. Consistently, our model demonstrated better outcomes regarding the metrics of accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1 score. Similarly, the decision curve analysis elucidated a greater net benefit of our model over the full ranges of clinical thresholds. Conclusion We present an accurate model for predicting the critical care outcomes in patients with chest pain, and provide substantial support to its application as a decision-making tool in ED.
Collapse
Affiliation(s)
- Ting Ting Wu
- The School of Nursing, Fujian Medical University, Fuzhou, Fujian, China
| | - Ruo Fei Zheng
- Department of Emergency, Fujian Provincial Hospital, Fuzhou, Fujian, China
| | - Zhi Zhong Lin
- Department of Radiotherapy, Fujian Provincial Cancer Hospital, Fuzhou, Fujian, China
| | - Hai Rong Gong
- Department of Nursing, Fujian Health College, Fuzhou, Fujian, China
| | - Hong Li
- The School of Nursing, Fujian Medical University, Fuzhou, Fujian, China. .,Shengli Clinical Medical College of Fujian Medical University, Fuzhou, Fujian, China. .,Department of Nursing, Fujian Provincial Hospital, Fuzhou, Fujian, China.
| |
Collapse
|
77
|
Cardosi JD, Shen H, Groner JI, Armstrong M, Xiang H. Machine learning for outcome predictions of patients with trauma during emergency department care. BMJ Health Care Inform 2021; 28:e100407. [PMID: 34625448 PMCID: PMC8504344 DOI: 10.1136/bmjhci-2021-100407] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Accepted: 09/13/2021] [Indexed: 11/05/2022] Open
Abstract
OBJECTIVES To develop and evaluate a machine learning model for predicting patient with trauma mortality within the US emergency departments. METHODS This was a retrospective prognostic study using deidentified patient visit data from years 2007 to 2014 of the National Trauma Data Bank. The predictive model intelligence building process is designed based on patient demographics, vital signs, comorbid conditions, arrival mode and hospital transfer status. The mortality prediction model was evaluated on its sensitivity, specificity, area under receiver operating curve (AUC), positive and negative predictive value, and Matthews correlation coefficient. RESULTS Our final dataset consisted of 2 007 485 patient visits (36.45% female, mean age of 45), 8198 (0.4%) of which resulted in mortality. Our model achieved AUC and sensitivity-specificity gap of 0.86 (95% CI 0.85 to 0.87), 0.44 for children and 0.85 (95% CI 0.85 to 0.85), 0.44 for adults. The all ages model characteristics indicate it generalised, with an AUC and gap of 0.85 (95% CI 0.85 to 0.85), 0.45. Excluding fall injuries weakened the child model (AUC 0.85, 95% CI 0.84 to 0.86) but strengthened adult (AUC 0.87, 95% CI 0.87 to 0.87) and all ages (AUC 0.86, 95% CI 0.86 to 0.86) models. CONCLUSIONS Our machine learning model demonstrates similar performance to contemporary machine learning models without requiring restrictive criteria or extensive medical expertise. These results suggest that machine learning models for trauma outcome prediction can generalise to patients with trauma across the USA and may be able to provide decision support to medical providers in any healthcare setting.
Collapse
Affiliation(s)
- Joshua David Cardosi
- Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio, USA
| | - Herman Shen
- Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio, USA
| | - Jonathan I Groner
- Center for Pediatric Trauma Research and Center for Injury Research and Policy, Nationwide Children's Hospital, Columbus, Ohio, USA
- Department of Surgery, The Ohio State University, Columbus, Ohio, USA
| | - Megan Armstrong
- Center for Pediatric Trauma Research and Center for Injury Research and Policy, Nationwide Children's Hospital, Columbus, Ohio, USA
| | - Henry Xiang
- Center for Pediatric Trauma Research and Center for Injury Research and Policy, Nationwide Children's Hospital, Columbus, Ohio, USA
- Department of Pediatrics, The Ohio State University, Columbus, Ohio, USA
| |
Collapse
|
78
|
Bhattarai A, Dimitropoulos G, Marriott B, Paget J, Bulloch AGM, Tough SC, Patten SB. Can the adverse childhood experiences (ACEs) checklist be utilized to predict emergency department visits among children and adolescents? BMC Med Res Methodol 2021; 21:195. [PMID: 34563122 PMCID: PMC8465692 DOI: 10.1186/s12874-021-01392-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Accepted: 09/04/2021] [Indexed: 11/29/2022] Open
Abstract
Background Extensive literature has shown an association of Adverse Childhood Experiences (ACEs) with adverse health outcomes; however, its ability to predict events or stratify risks is less known. Individuals with mental illness and ACE exposure have been shown to visit emergency departments (ED) more often than those in the general population. This study thus examined the ability of the ACEs checklist to predict ED visits within the subsequent year among children and adolescents presenting to mental health clinics with pre-existing mental health issues. Methods The study analyzed linked data (n = 6100) from two databases provided by Alberta Health Services (AHS). The Regional Access and Intake System (RAIS 2016–2018) database provided data on the predictors (ACE items, age, sex, residence, mental health program type, and primary diagnosis) regarding children and adolescents (aged 0–17 years) accessing addiction and mental health services within Calgary Zone, and the National Ambulatory Care Reporting System (NACRS 2016–2019) database provided data on ED visits. A 25% random sample of the data was reserved for validation purposes. Two Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression models, each employing a different method to tune the shrinkage parameter lambda (namely cross-validated and adaptive) and performing 10-fold cross-validation for a set of 100 lambdas in each model were examined. Results The adaptive LASSO model had a slightly better fit in the validation dataset than the cross-validated model; however, it still demonstrated poor discrimination (AUC 0.60, sensitivity 37.8%, PPV 49.6%) and poor calibration (over-triaged in low-risk and under-triaged in high-risk subgroups). The model’s poor performance was evident from an out-of-sample deviance ratio of − 0.044. Conclusion The ACEs checklist did not perform well in predicting ED visits among children and adolescents with existing mental health concerns. The diverse causes of ED visits may have hindered accurate predictions, requiring more advanced statistical procedures. Future studies exploring other machine learning approaches and including a more extensive set of childhood adversities and other important predictors may produce better predictions. Furthermore, despite highly significant associations being observed, ACEs may not be deterministic in predicting health-related events at the individual level, such as general ED use. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-021-01392-w.
Collapse
Affiliation(s)
- Asmita Bhattarai
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada. .,Mathison Centre for Research & Education, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.
| | - Gina Dimitropoulos
- Mathison Centre for Research & Education, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Faculty of Social Work, University of Calgary, 2500 University Dr NW, Calgary, AB, T2N 1N4, Canada
| | - Brian Marriott
- Faculty of Social Work, University of Calgary, 2500 University Dr NW, Calgary, AB, T2N 1N4, Canada.,Addiction and Mental Health, Alberta Health Services- Calgary Zone, Calgary, AB, Canada
| | - Jaime Paget
- Addiction and Mental Health, Alberta Health Services- Calgary Zone, Calgary, AB, Canada
| | - Andrew G M Bulloch
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Mathison Centre for Research & Education, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Department of Psychiatry, Cumming School of Medicine, University of Calgary, 2500 University Dr NW, Calgary, AB, T2N 1N4, Canada
| | - Suzanne C Tough
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Department of Pediatrics, Cumming School of Medicine, University of Calgary, 2500 University Dr NW, Calgary, AB, T2N 1N4, Canada
| | - Scott B Patten
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Mathison Centre for Research & Education, University of Calgary, 3280 Hospital Drive NW, Calgary, AB, T2N4Z6, Canada.,Department of Psychiatry, Cumming School of Medicine, University of Calgary, 2500 University Dr NW, Calgary, AB, T2N 1N4, Canada
| |
Collapse
|
79
|
Could a Multi-Marker and Machine Learning Approach Help Stratify Patients with Heart Failure? MEDICINA (KAUNAS, LITHUANIA) 2021; 57:medicina57100996. [PMID: 34684033 PMCID: PMC8538712 DOI: 10.3390/medicina57100996] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/16/2021] [Revised: 09/13/2021] [Accepted: 09/17/2021] [Indexed: 12/23/2022]
Abstract
Half of the patients with heart failure (HF) have preserved ejection fraction (HFpEF). To date, there are no specific markers to distinguish this subgroup. The main objective of this work was to stratify HF patients using current biochemical markers coupled with clinical data. The cohort study included HFpEF (n = 24) and heart failure with reduced ejection fraction (HFrEF) (n = 34) patients as usually considered in clinical practice based on cardiac imaging (EF ≥ 50% for HFpEF; EF < 50% for HFrEF). Routine blood tests consisted of measuring biomarkers of renal and heart functions, inflammation, and iron metabolism. A multi-test approach and analysis of peripheral blood samples aimed to establish a computerized Machine Learning strategy to provide a blood signature to distinguish HFpEF and HFrEF. Based on logistic regression, demographic characteristics and clinical biomarkers showed no statistical significance to differentiate the HFpEF and HFrEF patient subgroups. Hence a multivariate factorial discriminant analysis, performed blindly using the data set, allowed us to stratify the two HF groups. Consequently, a Machine Learning (ML) strategy was developed using the same variables in a genetic algorithm approach. ML provided very encouraging explorative results when considering the small size of the samples applied. The accuracy and the sensitivity were high for both validation and test groups (69% and 100%, 64% and 75%, respectively). Sensitivity was 100% for the validation and 75% for the test group, whereas specificity was 44% and 55% for the validation and test groups because of the small number of samples. Lastly, the precision was acceptable, with 58% in the validation and 60% in the test group. Combining biochemical and clinical markers is an excellent entry to develop a computer classification tool to diagnose HFpEF. This translational approach is a springboard for improving new personalized treatment methods and identifying “high-yield” populations for clinical trials.
Collapse
|
80
|
Konishi T, Goto T, Fujiogi M, Michihata N, Kumazawa R, Matsui H, Fushimi K, Tanabe M, Seto Y, Yasunaga H. New machine learning scoring system for predicting postoperative mortality in gastroduodenal ulcer perforation: A study using a Japanese nationwide inpatient database. Surgery 2021; 171:1036-1042. [PMID: 34538648 DOI: 10.1016/j.surg.2021.08.031] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 08/14/2021] [Accepted: 08/19/2021] [Indexed: 12/12/2022]
Abstract
BACKGROUND Conventional prediction models for estimating risk of postoperative mortality in gastroduodenal ulcer perforation have suboptimal prediction ability. We aimed to develop and validate new machine learning models and an integer-based score for predicting the postoperative mortality. METHODS We retrospectively identified patients with gastroduodenal ulcer perforation who underwent surgical repair, using a nationwide Japanese inpatient database. In a derivation cohort from July 2010 to March 2016, we developed 2 machine learning-based models, Lasso and XGBoost, using 45 candidate predictors, and also developed an integer-based score for clinical use by including important variables in Lasso. In a validation cohort from April 2016 to March 2017, we measured the prediction performances of the models by computing area under the curve and comparing it to the conventional American Society of Anesthesiology risk score. RESULTS Of 25,886 patients, 1,176 (4.5%) died after surgical repair. For the validation cohort, Lasso and XGBoost had significantly higher prediction abilities than the American Society of Anesthesiology score (Lasso area under the curve = 0.84; 95% confidence interval 0.81-0.86; American Society of Anesthesiology score area under the curve = 0.70; 95% confidence interval 0.65-0.74, P < .001). The integer-based risk score, which had 13 factors, had a prediction ability similar to those of Lasso and XGBoost (area under the curve = 0.83; 95% confidence interval 0.81-0.86). According to the integer-based score, the mortalities were 0.1%, 2.3%, 9.3%, and 29.0% for the low (score, 0), moderate (1-2), high (3-4), and very high (≥5) score groups, respectively. CONCLUSION Machine learning models and the integer-based risk score performed well in predicting risk of postoperative mortality in gastroduodenal ulcer perforation. These models will help in decision making.
Collapse
Affiliation(s)
- Takaaki Konishi
- Department of Breast and Endocrine Surgery, Graduate School of Medicine, University of Tokyo, Japan; Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan.
| | - Tadahiro Goto
- Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan; TXP Medical Co. Ltd, Tokyo, Japan
| | - Michimasa Fujiogi
- Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan; Department of Emergency Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - Nobuaki Michihata
- Department of Health Services Research, Graduate School of Medicine, University of Tokyo, Japan
| | - Ryosuke Kumazawa
- Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan
| | - Hiroki Matsui
- Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan
| | - Kiyohide Fushimi
- Department of Health Policy and Informatics, Tokyo Medical and Dental University Graduate School, Japan
| | - Masahiko Tanabe
- Department of Breast and Endocrine Surgery, Graduate School of Medicine, University of Tokyo, Japan
| | - Yasuyuki Seto
- Department of Breast and Endocrine Surgery, Graduate School of Medicine, University of Tokyo, Japan; Department of Gastrointestinal Surgery, Graduate School of Medicine, University of Tokyo, Japan
| | - Hideo Yasunaga
- Department of Clinical Epidemiology and Health Economics, School of Public Health, University of Tokyo, Japan
| |
Collapse
|
81
|
Wang ZB, Ren L, Lu QB, Zhang XA, Miao D, Hu YY, Dai K, Li H, Luo ZX, Fang LQ, Liu EM, Liu W. The Impact of Weather and Air Pollution on Viral Infection and Disease Outcome Among Pediatric Pneumonia Patients in Chongqing, China, from 2009 to 2018: A Prospective Observational Study. Clin Infect Dis 2021; 73:e513-e522. [PMID: 32668459 DOI: 10.1093/cid/ciaa997] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2019] [Indexed: 11/12/2022] Open
Abstract
BACKGROUND For pediatric pneumonia, the meteorological and air pollution indicators have been frequently investigated for their association with viral circulation but not for their impact on disease severity. METHODS We performed a 10-year prospective, observational study in 1 hospital in Chongqing, China, to recruit children with pneumonia. Eight commonly seen respiratory viruses were tested. Autoregressive distributed lag (ADL) and random forest (RF) models were used to fit monthly detection rates of each virus at the population level and to predict the possibility of severe pneumonia at the individual level, respectively. RESULTS Between 2009 and 2018, 6611 pediatric pneumonia patients were included, and 4846 (73.3%) tested positive for at least 1 respiratory virus. The patient median age was 9 months (interquartile range, 4‒20). ADL models demonstrated a decent fitting of detection rates of R2 > 0.7 for respiratory syncytial virus, human rhinovirus, parainfluenza virus, and human metapneumovirus. Based on the RF models, the area under the curve for host-related factors alone was 0.88 (95% confidence interval [CI], .87‒.89) and 0.86 (95% CI, .85‒.88) for meteorological and air pollution indicators alone and 0.62 (95% CI, .60‒.63) for viral infections alone. The final model indicated that 9 weather and air pollution indicators were important determinants of severe pneumonia, with a relative contribution of 62.53%, which is significantly higher than respiratory viral infections (7.36%). CONCLUSIONS Meteorological and air pollution predictors contributed more to severe pneumonia in children than did respiratory viruses. These meteorological data could help predict times when children would be at increased risk for severe pneumonia and when interventions, such as reducing outdoor activities, may be warranted.
Collapse
Affiliation(s)
- Zhi-Bo Wang
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Luo Ren
- Department of Respiratory Medicine, Children's Hospital, Chongqing Medical University, Chongqing, People's Republic of China
| | - Qing-Bin Lu
- Department of Laboratorial Science and Technology, School of Public Health, Peking University, Beijing, People's Republic of China
| | - Xiao-Ai Zhang
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Dong Miao
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Yuan-Yuan Hu
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Ke Dai
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Hao Li
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - Zheng-Xiu Luo
- Department of Respiratory Medicine, Children's Hospital, Chongqing Medical University, Chongqing, People's Republic of China
| | - Li-Qun Fang
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| | - En-Mei Liu
- Department of Respiratory Medicine, Children's Hospital, Chongqing Medical University, Chongqing, People's Republic of China
| | - Wei Liu
- Beijing Institute of Microbiology and Epidemiology, State Key Laboratory of Pathogen and Biosecurity, Beijing, People's Republic of China
| |
Collapse
|
82
|
Xie F, Ong MEH, Liew JNMH, Tan KBK, Ho AFW, Nadarajan GD, Low LL, Kwan YH, Goldstein BA, Matchar DB, Chakraborty B, Liu N. Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions. JAMA Netw Open 2021; 4:e2118467. [PMID: 34448870 PMCID: PMC8397930 DOI: 10.1001/jamanetworkopen.2021.18467] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
IMPORTANCE Triage in the emergency department (ED) is a complex clinical judgment based on the tacit understanding of the patient's likelihood of survival, availability of medical resources, and local practices. Although a scoring tool could be valuable in risk stratification, currently available scores have demonstrated limitations. OBJECTIVES To develop an interpretable machine learning tool based on a parsimonious list of variables available at ED triage; provide a simple, early, and accurate estimate of patients' risk of death; and evaluate the tool's predictive accuracy compared with several established clinical scores. DESIGN, SETTING, AND PARTICIPANTS This single-site, retrospective cohort study assessed all ED patients between January 1, 2009, and December 31, 2016, who were subsequently admitted to a tertiary hospital in Singapore. The Score for Emergency Risk Prediction (SERP) tool was derived using a machine learning framework. To estimate mortality outcomes after emergency admissions, SERP was compared with several triage systems, including Patient Acuity Category Scale, Modified Early Warning Score, National Early Warning Score, Cardiac Arrest Risk Triage, Rapid Acute Physiology Score, and Rapid Emergency Medicine Score. The initial analyses were completed in October 2020, and additional analyses were conducted in May 2021. MAIN OUTCOMES AND MEASURES Three SERP scores, namely SERP-2d, SERP-7d, and SERP-30d, were developed using the primary outcomes of interest of 2-, 7-, and 30-day mortality, respectively. Secondary outcomes included 3-day mortality and inpatient mortality. The SERP's predictive power was measured using the area under the curve in the receiver operating characteristic analysis. RESULTS The study included 224 666 ED episodes in the model training cohort (mean [SD] patient age, 63.60 [16.90] years; 113 426 [50.5%] female), 56 167 episodes in the validation cohort (mean [SD] patient age, 63.58 [16.87] years; 28 427 [50.6%] female), and 42 676 episodes in the testing cohort (mean [SD] patient age, 64.85 [16.80] years; 21 556 [50.5%] female). The mortality rates in the training cohort were 0.8% at 2 days, 2.2% at 7 days, and 5.9% at 30 days. In the testing cohort, the areas under the curve of SERP-30d were 0.821 (95% CI, 0.796-0.847) for 2-day mortality, 0.826 (95% CI, 0.811-0.841) for 7-day mortality, and 0.823 (95% CI, 0.814-0.832) for 30-day mortality and outperformed several benchmark scores. CONCLUSIONS AND RELEVANCE In this retrospective cohort study, SERP had better prediction performance than existing triage scores while maintaining easy implementation and ease of ascertainment in the ED. It has the potential to be widely applied and validated in different circumstances and health care settings.
Collapse
Affiliation(s)
- Feng Xie
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| | | | | | - Andrew Fu Wah Ho
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| | | | - Lian Leng Low
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Family Medicine and Continuing Care, Singapore General Hospital, Singapore
| | - Yu Heng Kwan
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Pharmacy, Faculty of Science, National University of Singapore, Singapore
| | - Benjamin Alan Goldstein
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina
| | - David Bruce Matchar
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Duke University Medical Center, Duke University, Durham, North Carolina
| | - Bibhas Chakraborty
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina
- Department of Statistics and Data Science, National University of Singapore, Singapore
| | - Nan Liu
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Health Service Research Centre, Singapore Health Services, Singapore
- Institute of Data Science, National University of Singapore, Singapore
| |
Collapse
|
83
|
Shah PK, Ginestra JC, Ungar LH, Junker P, Rohrbach JI, Fishman NO, Weissman GE. A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients. Crit Care Med 2021; 49:1312-1321. [PMID: 33711001 PMCID: PMC8282687 DOI: 10.1097/ccm.0000000000004966] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
OBJECTIVES The National Early Warning Score, Modified Early Warning Score, and quick Sepsis-related Organ Failure Assessment can predict clinical deterioration. These scores exhibit only moderate performance and are often evaluated using aggregated measures over time. A simulated prospective validation strategy that assesses multiple predictions per patient-day would provide the best pragmatic evaluation. We developed a deep recurrent neural network deterioration model and conducted a simulated prospective evaluation. DESIGN Retrospective cohort study. SETTING Four hospitals in Pennsylvania. PATIENTS Inpatient adults discharged between July 1, 2017, and June 30, 2019. INTERVENTIONS None. MEASUREMENTS AND MAIN RESULTS We trained a deep recurrent neural network and logistic regression model using data from electronic health records to predict hourly the 24-hour composite outcome of transfer to ICU or death. We analyzed 146,446 hospitalizations with 16.75 million patient-hours. The hourly event rate was 1.6% (12,842 transfers or deaths, corresponding to 260,295 patient-hours within the predictive horizon). On a hold-out dataset, the deep recurrent neural network achieved an area under the precision-recall curve of 0.042 (95% CI, 0.04-0.043), comparable with logistic regression model (0.043; 95% CI 0.041 to 0.045), and outperformed National Early Warning Score (0.034; 95% CI, 0.032-0.035), Modified Early Warning Score (0.028; 95% CI, 0.027- 0.03), and quick Sepsis-related Organ Failure Assessment (0.021; 95% CI, 0.021-0.022). For a fixed sensitivity of 50%, the deep recurrent neural network achieved a positive predictive value of 3.4% (95% CI, 3.4-3.5) and outperformed logistic regression model (3.1%; 95% CI 3.1-3.2), National Early Warning Score (2.0%; 95% CI, 2.0-2.0), Modified Early Warning Score (1.5%; 95% CI, 1.5-1.5), and quick Sepsis-related Organ Failure Assessment (1.5%; 95% CI, 1.5-1.5). CONCLUSIONS Commonly used early warning scores for clinical decompensation, along with a logistic regression model and a deep recurrent neural network model, show very poor performance characteristics when assessed using a simulated prospective validation. None of these models may be suitable for real-time deployment.
Collapse
Affiliation(s)
- Parth K Shah
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
| | - Jennifer C Ginestra
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
| | - Lyle H Ungar
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA
| | - Paul Junker
- Clinical Effectiveness and Quality Improvement, Hospital of the University of Pennsylvania, Philadelphia, PA
| | - Jeff I Rohrbach
- Clinical Effectiveness and Quality Improvement, Hospital of the University of Pennsylvania, Philadelphia, PA
| | - Neil O Fishman
- Department of Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA
| | - Gary E Weissman
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
- Department of Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA
| |
Collapse
|
84
|
Steele RW. Pediatric quality measures: The leap from process to outcomes. Curr Probl Pediatr Adolesc Health Care 2021; 51:101065. [PMID: 34518131 DOI: 10.1016/j.cppeds.2021.101065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
Value-based reimbursement arrangements tie financial incentives to achieving quality measures to ensure savings are not from withholding care. For patients and their families, the delivery of high-quality care is simply the expectation. Defining and measuring pediatric quality, however, is not standardized which has led to a large proliferation of metrics across multiple stakeholders. The majority of these measures are process rather than outcomes metrics often chosen for the ease at which the data can be obtained. In order to drive greater value, outcomes measures should be preferentially selected. However, measuring outcomes in children presents multiple unique challenges. Compared to adults, children are generally healthier, their outcomes may take more time to manifest, and their clinical variability is greater. Another challenge is the amount of healthcare data being generated by providers, provider networks, payors, government agencies, and many others. This should help in understanding pediatric quality outcomes, but the massive volume of data requires new analytic tools. Artificial intelligence techniques such as machine learning offer faster, more precise, and larger scale evaluation of quality outcomes. Its implementation necessitates identifying expertise in the way of data scientists as well as additional infrastructure components to evaluate data governance, security, regulatory compliance, and ethics. Despite these prerequisites, much progress is being made in outcome insights that drive value benefiting children and families.
Collapse
Affiliation(s)
- Robert W Steele
- EVP/Chief Strategy and Innovation Officer, Children's Mercy Kansas City, United States.
| |
Collapse
|
85
|
Song J, Woo K, Shang J, Ojo M, Topaz M. Predictive Risk Models for Wound Infection-Related Hospitalization or ED Visits in Home Health Care Using Machine-Learning Algorithms. Adv Skin Wound Care 2021; 34:1-12. [PMID: 34260423 DOI: 10.1097/01.asw.0000755928.30524.22] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
OBJECTIVE Wound infection is prevalent in home healthcare (HHC) and often leads to hospitalizations. However, none of the previous studies of wounds in HHC have used data from clinical notes. Therefore, the authors created a more accurate description of a patient's condition by extracting risk factors from clinical notes to build predictive models to identify a patient's risk of wound infection in HHC. METHODS The structured data (eg, standardized assessments) and unstructured information (eg, narrative-free text charting) were retrospectively reviewed for HHC patients with wounds who were served by a large HHC agency in 2014. Wound infection risk factors were identified through bivariate analysis and stepwise variable selection. Risk predictive performance of three machine learning models (logistic regression, random forest, and artificial neural network) was compared. RESULTS A total of 754 of 54,316 patients (1.39%) had a hospitalization or ED visit related to wound infection. In the bivariate logistic regression, language describing wound type in the patient's clinical notes was strongly associated with risk (odds ratio, 9.94; P < .05). The areas under the curve were 0.82 in logistic regression, 0.75 in random forest, and 0.78 in artificial neural network. Risk prediction performance of the models improved (by up to 13.2%) after adding risk factors extracted from clinical notes. CONCLUSIONS Logistic regression showed the best risk prediction performance in prediction of wound infection-related hospitalization or ED visits in HHC. The use of data extracted from clinical notes can improve the performance of risk prediction models.
Collapse
Affiliation(s)
- Jiyoun Song
- Jiyoun Song, PhD, RN, AGACNP-BC, is Postdoctoral Fellow, Columbia University School of Nursing, New York, NY. Kyungmi Woo, PhD, RN, is Assistant Professor, The Research Institute of Nursing Science, Seoul National University College of Nursing, Republic of Korea. Jingjing Shang, PhD, RN, is Associate Professor, Columbia University School of Nursing, New York, NY. Marietta Ojo, MPH, is Research Assistant, Columbia University Mailman School of Public Health, New York, NY. Maxim Topaz, PhD, RN, is Associate Professor, Columbia University School of Nursing, New York, NY. Acknowledgments: This study is funded by the Eugenie and Joseph Doyle Research Partnership Fund from Visiting Nurses Service of New York and the Intramural Pilot Grant from Columbia University School of Nursing. At the time of data analysis and manuscript development, Jiyoun Song was supported in part by the Agency for Healthcare Research and Quality (R01HS024915), Nursing Intensity of Patient Care Needs and Rates of Healthcare-Associated Infections, and The Jonas Center for Nursing and Veterans Healthcare. Kyungmi Woo was supported by the Comparative and Cost-Effectiveness Research (T32 NR014205) grant through the National Institute of Nursing Research. The authors have disclosed no other financial relationships related to this article. Submitted August 28, 2020; accepted in revised form December 8, 2020
| | | | | | | | | |
Collapse
|
86
|
Yun H, Choi J, Park JH. XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information. JMIR Med Inform 2021; 9:e30770. [PMID: 34346889 PMCID: PMC8491120 DOI: 10.2196/30770] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2021] [Revised: 06/27/2021] [Accepted: 07/27/2021] [Indexed: 12/23/2022] Open
Abstract
Background The emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge. Objective This study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS). Methods To predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis. Results The AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction. Conclusions Our study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.
Collapse
Affiliation(s)
- Hyoungju Yun
- Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, KR
| | - Jinwook Choi
- Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, KR.,Department of Biomedical Engineering, College of Medicine, Seoul National University, Seoul, KR.,Institute of Medical and Biological Engineering,, Seoul National University Medical Research Center, 103 Daehak-Ro, Jongno-Gu, Seoul, KR
| | - Jeong Ho Park
- Department of Emergency Medicine, College of Medicine, Seoul National University, Seoul, KR.,Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, KR
| |
Collapse
|
87
|
Kim D, Oh J, Im H, Yoon M, Park J, Lee J. Automatic Classification of the Korean Triage Acuity Scale in Simulated Emergency Rooms Using Speech Recognition and Natural Language Processing: a Proof of Concept Study. J Korean Med Sci 2021; 36:e175. [PMID: 34254471 PMCID: PMC8275459 DOI: 10.3346/jkms.2021.36.e175] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/21/2021] [Accepted: 06/07/2021] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Rapid triage reduces the patients' stay time at an emergency department (ED). The Korean Triage Acuity Scale (KTAS) is mandatorily applied at EDs in South Korea. For rapid triage, we studied machine learning-based triage systems composed of a speech recognition model and natural language processing-based classification. METHODS We simulated 762 triage cases that consisted of 18 classes with six types of the main symptom (chest pain, dyspnea, fever, stroke, abdominal pain, and headache) and three levels of KTAS. In addition, we recorded conversations between emergency patients and clinicians during the simulation. We used speech recognition models to transcribe the conversation. Bidirectional Encoder Representation from Transformers (BERT), support vector machine (SVM), random forest (RF), and k-nearest neighbors (KNN) were used for KTAS and symptom classification. Additionally, we evaluated the Shapley Additive exPlanations (SHAP) values of features to interpret the classifiers. RESULTS The character error rate of the speech recognition model was reduced to 25.21% through transfer learning. With auto-transcribed scripts, support vector machine (area under the receiver operating characteristic curve [AUROC], 0.86; 95% confidence interval [CI], 0.81-0.9), KNN (AUROC, 0.89; 95% CI, 0.85-0.93), RF (AUROC, 0.86; 95% CI, 0.82-0.9) and BERT (AUROC, 0.82; 95% CI, 0.75-0.87) achieved excellent classification performance. Based on SHAP, we found "stress", "pain score point", "fever", "breath", "head" and "chest" were the important vocabularies for determining KTAS and symptoms. CONCLUSION We demonstrated the potential of an automatic KTAS classification system using speech recognition models, machine learning and BERT-based classifiers.
Collapse
Affiliation(s)
- Dongkyun Kim
- Department of Electrical and Electronic Engineering, Hanyang University, Ansan, Korea
| | - Jaehoon Oh
- Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Korea
| | - Heeju Im
- Department of Artificial Intelligence, Hanyang University, Seoul, Korea
| | - Myeongseong Yoon
- Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Korea
| | - Jiwoo Park
- Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Korea
| | - Joohyun Lee
- Department of Electrical and Electronic Engineering, Hanyang University, Ansan, Korea.
| |
Collapse
|
88
|
Tahayori B, Chini-Foroush N, Akhlaghi H. Advanced natural language processing technique to predict patient disposition based on emergency triage notes. Emerg Med Australas 2021; 33:480-484. [PMID: 33043570 DOI: 10.1111/1742-6723.13656] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2020] [Revised: 09/23/2020] [Accepted: 09/24/2020] [Indexed: 12/13/2022]
Abstract
OBJECTIVE To demonstrate the potential of machine learning and capability of natural language processing (NLP) to predict disposition of patients based on triage notes in the ED. METHODS A retrospective cohort of ED triage notes from St Vincent's Hospital (Melbourne) was used to develop a deep-learning algorithm that predicts patient disposition. Bidirectional Encoder Representations from Transformers, a recent language representation model developed by Google, was utilised for NLP. Eighty percent of the dataset was used for training the model and 20% was used to test the algorithm performance. Ktrain library, a wrapper for TensorFlow Keras, was employed to develop the model. RESULTS The accuracy of the algorithm was 83% and the area under the curve was 0.88. Sensitivity, specificity, precision and F1-score of the algorithm were 72%, 86%, 56% and 63%, respectively. CONCLUSION Machine learning and NLP can be together applied to the ED triage note to predict patient disposition with a high level of accuracy. The algorithm can potentially assist ED clinicians in early identification of patients requiring admission by mitigating the cognitive load, thus optimises resource allocation in EDs.
Collapse
Affiliation(s)
- Bahman Tahayori
- Department of Biomedical Engineering, The University of Melbourne, Melbourne, Victoria, Australia
- Emergency Department, St Vincent's Hospital, Melbourne, Victoria, Australia
| | | | - Hamed Akhlaghi
- Emergency Department, St Vincent's Hospital, Melbourne, Victoria, Australia
| |
Collapse
|
89
|
Kelz RR, Airoldi EM, Keele L. Strengthsand Limitations of Machine Learning in Surgical Care. J Am Coll Surg 2021; 232:919-920. [PMID: 34030853 PMCID: PMC10906963 DOI: 10.1016/j.jamcollsurg.2021.03.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Accepted: 03/01/2021] [Indexed: 11/26/2022]
|
90
|
Jin L, Zhao C, Li H, Wang R, Wang Q, Wang H. Clinical Profile, Prognostic Factors, and Outcome Prediction in Hospitalized Patients With Bloodstream Infection: Results From a 10-Year Prospective Multicenter Study. Front Med (Lausanne) 2021; 8:629671. [PMID: 34095163 PMCID: PMC8172964 DOI: 10.3389/fmed.2021.629671] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2020] [Accepted: 03/15/2021] [Indexed: 12/23/2022] Open
Abstract
Background: Bloodstream infection (BSI) is one of the most common serious bacterial infections worldwide and also a major contributor to in-hospital mortality. Determining the predictors of mortality is crucial for prevention and improving clinical prognosis in patients with nosocomial BSI. Methods: A nationwide prospective cohort study was conducted from 2007 until 2016 in 16 teaching hospitals across China. Microbiological results, clinical information, and patient outcomes were collected to investigate the pathogenic spectrum and mortality rate in patients with BSI and identify outcome predictors using multivariate regression, prediction model, and Kaplan-Meier analysis. Results: No significant change was observed in the causative pathogen distribution during the 10-year period and the overall in-hospital mortality was 12.83% (480/3,741). An increased trend was found in the mortality of patients infected with Pseudomonas aeruginosa or Acinetobacter baumannii, while a decreased mortality rate was noted in Staphylococcus aureus-related BSI. In multivariable-adjusted models, higher mortality rate was significantly associated with older age, cancer, sepsis diagnosis, ICU admission, and prolonged hospital stay prior to BSI onset, which were also determined using machine learning-based predictive model achieved by random forest algorithm with a satisfactory performance in outcome prediction. Conclusions: Our study described the clinical and microbiological characteristics and mortality predictive factors in patients with BSI. These informative predictors would inform clinical practice to adopt effective therapeutic strategies to improve patient outcomes.
Collapse
Affiliation(s)
- Longyang Jin
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| | - Chunjiang Zhao
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| | - Henan Li
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| | - Ruobing Wang
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| | - Qi Wang
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| | - Hui Wang
- Department of Clinical Laboratory, Peking University People's Hospital, Beijing, China
| |
Collapse
|
91
|
Gillan C, Hodges B, Wiljer D, Dobrow M. Health Care Professional Association Agency in Preparing for Artificial Intelligence: Protocol for a Multi-Case Study. JMIR Res Protoc 2021; 10:e27340. [PMID: 34009136 PMCID: PMC8173392 DOI: 10.2196/27340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2021] [Revised: 04/20/2021] [Accepted: 04/21/2021] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND The emergence of artificial intelligence (AI) in health care has impacted health care systems, including employment, training, education, and professional regulation. It is incumbent on health professional associations to assist their membership in defining and preparing for AI-related change. Health professional associations, or the national groups convened to represent the interests of the members of a profession, play a unique role in establishing the sociocultural, normative, and regulative elements of health care professions. OBJECTIVE The aim of this paper is to present a protocol for a proposed study of how, when faced with AI as a disruptive technology, health professional associations engage in sensemaking and legitimization of change to support their membership in preparing for future practice. METHODS An exploratory multi-case study approach will be used. This study will be informed by the normalization process theory (NPT), which suggests behavioral constructs required for complex change, providing a novel lens through which to consider the agency of macrolevel actors in practice change. A total of 4 health professional associations will be studied, each representing an instrumental case and related fields selected for their early consideration of AI technologies. Data collection will consist of key informant interviews, observation of relevant meetings, and document review. Individual and collective sensemaking activities and action toward change will be identified using stakeholder network mapping. A hybrid inductive and deductive model will be used for a concurrent thematic analysis, mapping emergent themes against the NPT framework to assess fit and identify areas of discordance. RESULTS As of January 2021, we have conducted 17 interviews, with representation across the 4 health professional associations. Of these 17 interviews, 15 (88%) have been transcribed. Document review is underway and complete for one health professional association and nearly complete for another. Observation opportunities have been challenged by competing priorities during COVID-19 and may require revisiting. A linear cross-case analytic approach will be taken to present the data, highlighting both guidance for the implementation of AI and implications for the application of NPT at the macro level. The ability to inform consideration of AI will depend on the degree to which the engaged health professional associations have considered this topic at the time of the study and, hence, what priority it has been assigned within the health professional association and what actions have been taken to consider or prepare for it. The fact that this may differ between health professional associations and practice environments will require consideration throughout the analysis. CONCLUSIONS Ultimately, this protocol outlines a case study approach to understand how, when faced with AI as a disruptive technology, health professional associations engage in sensemaking and legitimization of change to support their membership in preparing for future practice. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/27340.
Collapse
Affiliation(s)
- Caitlin Gillan
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
- Joint Department of Medical Imaging, Sinai Health/University Health Network/Women's College Hospital, Toronto, ON, Canada
- Department of Radiation Oncology, University of Toronto, Toronto, ON, Canada
| | - Brian Hodges
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
- University Health Network, Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - David Wiljer
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
- University Health Network, Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Mark Dobrow
- Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
92
|
Development and Validation of Machine Learning Models to Predict Admission From Emergency Department to Inpatient and Intensive Care Units. Ann Emerg Med 2021; 78:290-302. [PMID: 33972128 DOI: 10.1016/j.annemergmed.2021.02.029] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2020] [Revised: 02/10/2021] [Accepted: 02/25/2021] [Indexed: 12/23/2022]
Abstract
STUDY OBJECTIVE This study aimed to develop and validate 2 machine learning models that use historical and current-visit patient data from electronic health records to predict the probability of patient admission to either an inpatient unit or ICU at each hour (up to 24 hours) of an emergency department (ED) encounter. The secondary goal was to provide a framework for the operational implementation of these machine learning models. METHODS Data were curated from 468,167 adult patient encounters in 3 EDs (1 academic and 2 community-based EDs) of a large academic health system from August 1, 2015, to October 31, 2018. The models were validated using encounter data from January 1, 2019, to December 31, 2019. An operational user dashboard was developed, and the models were run on real-time encounter data. RESULTS For the intermediate admission model, the area under the receiver operating characteristic curve was 0.873 and the area under the precision-recall curve was 0.636. For the ICU admission model, the area under the receiver operating characteristic curve was 0.951 and the area under the precision-recall curve was 0.461. The models had similar performance in both the academic- and community-based settings as well as across the 2019 and real-time encounter data. CONCLUSION Machine learning models were developed to accurately make predictions regarding the probability of inpatient or ICU admission throughout the entire duration of a patient's encounter in ED and not just at the time of triage. These models remained accurate for a patient cohort beyond the time period of the initial training data and were integrated to run on live electronic health record data, with similar performance.
Collapse
|
93
|
Schultz MA, Walden RL, Cato K, Coviak CP, Cruz C, D'Agostino F, Douthit BJ, Forbes T, Gao G, Lee MA, Lekan D, Wieben A, Jeffery AD. Data Science Methods for Nursing-Relevant Patient Outcomes and Clinical Processes: The 2019 Literature Year in Review. Comput Inform Nurs 2021; 39:654-667. [PMID: 34747890 PMCID: PMC8578863 DOI: 10.1097/cin.0000000000000705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
Data science continues to be recognized and used within healthcare due to the increased availability of large data sets and advanced analytics. It can be challenging for nurse leaders to remain apprised of this rapidly changing landscape. In this article, we describe our findings from a scoping literature review of papers published in 2019 that use data science to explore, explain, and/or predict 15 phenomena of interest to nurses. Fourteen of the 15 phenomena were associated with at least one paper published in 2019. We identified the use of many contemporary data science methods (eg, natural language processing, neural networks) for many of the outcomes. We found many studies exploring Readmissions and Pressure Injuries. The topics of Artificial Intelligence/Machine Learning Acceptance, Burnout, Patient Safety, and Unit Culture were poorly represented. We hope that the studies described in this article help readers: (1) understand the breadth and depth of data science's ability to improve clinical processes and patient outcomes that are relevant to nurses and (2) identify gaps in the literature that are in need of exploration.
Collapse
Affiliation(s)
- Mary Anne Schultz
- Author Affiliations: California State University (Dr Schultz); Annette and Irwin Eskind Family Biomedical Library, Vanderbilt University (Ms Walden); Department of Emergency Medicine, Columbia University School of Nursing (Dr Cato); Grand Valley State University (Dr Coviak); Global Health Technology & Informatics, Chevron, San Ramon, CA (Mr Cruz); Saint Camillus International University of Health Sciences, Rome, Italy (Dr D'Agostino); Duke University School of Nursing (Mr Douthit); East Carolina University College of Nursing (Dr Forbes); St Catherine University Department of Nursing (Dr Gao); Texas Woman's University College of Nursing (Dr Lee); Assistant Professor, University of North Carolina at Greensboro School of Nursing (Dr Lekan); University of Wisconsin School of Nursing (Ms Wieben); and Vanderbilt University School of Nursing, and Tennessee Valley Healthcare System, US Department of Veterans Affairs (Dr Jeffery)
| | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
94
|
Heyming TW, Knudsen-Robbins C, Feaster W, Ehwerhemuepha L. Criticality index conducted in pediatric emergency department triage. Am J Emerg Med 2021; 48:209-217. [PMID: 33975133 DOI: 10.1016/j.ajem.2021.05.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2021] [Revised: 04/27/2021] [Accepted: 05/02/2021] [Indexed: 10/21/2022] Open
Abstract
OBJECTIVE To develop and analyze the performance of a machine learning model capable of predicting the disposition of patients presenting to a pediatric emergency department (ED) based on triage assessment and historical information mined from electronic health records. METHODS We retrospectively reviewed data from 585,142 ED visits at a pediatric quaternary care institution between 2013 and 2020. An extreme gradient boosting machine learning model was trained on a randomly selected training data set (50%) to stratify patients into 3 classes: (1) high criticality (patients requiring intensive care unit [ICU] care within 4 h of hospital admission, patients who died within 4 h of admission, and patients who died in the ED); (2) moderate criticality (patients requiring hospitalization without the need for ICU care); and (3) low criticality (patients discharged home). Variables considered during model development included triage vital signs, aspects of triage nursing assessment, demographics, and historical information (diagnoses, medication use, and healthcare utilization). Historical factors were limited to the 6 months preceding the index ED visit. The model was tested on a previously withheld test data set (40%), and its performance analyzed. RESULTS The distribution of criticality among high, moderate, and low was 1.5%, 7.1%, and 91.4%, respectively. The one-versus-all area under the receiver operating characteristic (AUROC) curve for high and moderate criticality was 0.982 (95% CI 0.980, 0.983) and 0.968 (0.967, 0.969). The multi-class macro average AUROC and area under the receiver operating characteristic curve were 0.976 and 0.754. The features most integral to model performance included history of intravenous medications, capillary refill, emergency severity index level, history of hospitalization, use of a supplemental oxygen device, age, and history of admission to the ICU. CONCLUSION Pediatric ED disposition can be accurately predicted using information available at triage, providing an opportunity to improve quality of care and patient outcomes.
Collapse
Affiliation(s)
- Theodore W Heyming
- Children's Hospital of Orange County, Orange, CA, United States; Department of Emergency Medicine, University of California, Irvine, United States.
| | | | - William Feaster
- Children's Hospital of Orange County, Orange, CA, United States
| | | |
Collapse
|
95
|
Some machine’s doin’ that for you* – elektronische Triagesysteme in der Notaufnahme. Notf Rett Med 2021. [DOI: 10.1007/s10049-021-00874-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
Zusammenfassung
Hintergrund
In den letzten 25 Jahren haben sich Triagesysteme zur Dringlichkeitseinschätzung in den Notaufnahmen etabliert. Die bisherigen symptomorientierten Triagesysteme haben allerdings auch Schwächen. Inzwischen ermöglichen die Digitalisierung der Notaufnahmen und die Leistungsfähigkeit der aktuellen Computergeneration bereits zum Triagezeitpunkt einen algorithmenbasierten Datenvergleich und eine Risikostratifizierung für bestimmte klinische Endpunkte über die reine Triagestufe hinaus.
Ziel der Arbeit
Nach selektiver Literaturrecherche erfolgt eine Übersicht über elektronische Triagesysteme (ETS). Das Funktionsprinzip und die aktuellen Möglichkeiten der ETS werden dargestellt. Daneben werden Chancen und Schwierigkeiten einer Etablierung von ETS in deutschen Notaufnahmen betrachtet.
Ergebnisse
Es wurden wesentliche Prädiktorvariablen wie Alter und bestimmte Vitalparameter identifiziert, die bisher nicht standardisiert in die Triagestufen einfließen, aber mithilfe von Modelllernen (ML) in belastbare Vorhersagen für klinische Endpunkte wie stationäre Aufnahme oder Mortalität einfließen können. Die Güte der Ersteinschätzung durch ein ETS ist insgesamt hoch. Ein ETS kann das Triagepersonal evidenzbasiert bei der Disposition der Patienten unterstützen und Über- und Untertriage reduzieren. Es gibt einige Entwicklungen, die günstige Bedingungen für den Einsatz von ETS in deutschen ZNA schaffen. So erleichtern z. B. repräsentative Notaufnahmeregister die Erstellung von Referenzdatensätzen, die zum Aufbau computerbasierter Klassifikationsmodelle benötigt werden. Außerdem müssen individuelle Patientendaten schnell verfügbar sein.
Schlussfolgerung
ETS können zur Erhöhung der Patientensicherheit und zur besseren Ressourcennutzung beitragen. Bislang fehlen allerdings noch objektive Referenzstandards und Leitlinien zum maschinellen Lernen.
Collapse
|
96
|
Sills MR, Ozkaynak M, Jang H. Predicting hospitalization of pediatric asthma patients in emergency departments using machine learning. Int J Med Inform 2021; 151:104468. [PMID: 33940479 DOI: 10.1016/j.ijmedinf.2021.104468] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2021] [Revised: 03/25/2021] [Accepted: 04/15/2021] [Indexed: 11/26/2022]
Abstract
MOTIVATION The timely identification of patients for hospitalization in emergency departments (EDs) can facilitate efficient use of hospital resources. Machine learning can help the early prediction of ED disposition; however, application of machine learning models requires both computer science skills and domain knowledge. This presents a barrier for those who want to apply machine learning to real-world settings. OBJECTIVES The objective of this study was to construct a competitive predictive model with a minimal amount of human effort to facilitate decisions regarding hospitalization of patients. METHODS This study used the electronic health record data from five EDs in a single healthcare system, including an academic urban children's hospital ED, from January 2009 to December 2013. We constructed two machine learning models by using automated machine learning algorithm (autoML) which allows non-experts to use machine learning model: one with data only available at ED triage, the other adding information available one hour into the ED visit. Random forest and logistic regression were employed as bench-marking models. The ratio of the training dataset to the test dataset was 8:2, and the area under the receiver operating characteristic curve (AUC), accuracy, and F1 were calculated to assess the quality of the models. RESULTS Of the 9,069 ED visits analyzed, the study population was made up of males (62.7 %), median (IQR) age was 6 (4, 10) years, and public insurance coverage (66.0 %). The majority had an Emergency Severity Index score of 3 (52.9 %). The prevalence of hospitalization was 22.5 %. The AUCs were 0.914 and 0.942. AUCs were 0.831 and 0.886 for random forests, and 0.795 and 0.823 for logistic regression. Among the predictors, an outcome at a prior visit, ESI level, time to first medication, and time to triage were the most important features for the prediction of the need for hospitalization. CONCLUSIONS In comparison with the conventional approaches, the use of autoML improved the predictive ability for the need for hospitalization. The findings can optimize ED management, hospital-level resource utilization and improve quality. Furthermore, this approach can support the design of a more effective patient ED flow for pediatric asthma care.
Collapse
Affiliation(s)
- Marion R Sills
- School of Medicine, University of Colorado-Anschutz Medical Campus, Aurora, CO, USA
| | - Mustafa Ozkaynak
- College of Nursing, University of Colorado-Anschutz Medical Campus, Aurora, CO, USA
| | - Hoon Jang
- College of Global Business, Korea University, 2511 Sejong-ro, Sejong, Republic of Korea.
| |
Collapse
|
97
|
Leonard F, Gilligan J, Barrett MJ. Predicting Admissions From a Paediatric Emergency Department - Protocol for Developing and Validating a Low-Dimensional Machine Learning Prediction Model. Front Big Data 2021; 4:643558. [PMID: 33937750 PMCID: PMC8085432 DOI: 10.3389/fdata.2021.643558] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Accepted: 03/22/2021] [Indexed: 12/02/2022] Open
Abstract
Introduction: Patients boarding in the Emergency Department can contribute to overcrowding, leading to longer waiting times and patients leaving without being seen or completing their treatment. The early identification of potential admissions could act as an additional decision support tool to alert clinicians that a patient needs to be reviewed for admission and would also be of benefit to bed managers in advance bed planning for the patient. We aim to create a low-dimensional model predicting admissions early from the paediatric Emergency Department. Methods and Analysis: The methodology Cross Industry Standard Process for Data Mining (CRISP-DM) will be followed. The dataset will comprise of 2 years of data, ~76,000 records. Potential predictors were identified from previous research, comprising of demographics, registration details, triage assessment, hospital usage and past medical history. Fifteen models will be developed comprised of 3 machine learning algorithms (Logistic regression, naïve Bayes and gradient boosting machine) and 5 sampling methods, 4 of which are aimed at addressing class imbalance (undersampling, oversampling, and synthetic oversampling techniques). The variables of importance will then be identified from the optimal model (selected based on the highest Area under the curve) and used to develop an additional low-dimensional model for deployment. Discussion: A low-dimensional model comprised of routinely collected data, captured up to post triage assessment would benefit many hospitals without data rich platforms for the development of models with a high number of predictors. Novel to the planned study is the use of data from the Republic of Ireland and the application of sampling techniques aimed at improving model performance impacted by an imbalance between admissions and discharges in the outcome variable.
Collapse
Affiliation(s)
- Fiona Leonard
- Business Intelligence Unit, Children's Health Ireland at Crumlin, Dublin, Ireland
| | - John Gilligan
- School of Computer Science, Technological University Dublin, Dublin, Ireland
| | - Michael J Barrett
- Department of Emergency Medicine, Children's Health Ireland at Crumlin, Dublin, Ireland.,School of Medicine, University College Dublin, Dublin, Ireland
| |
Collapse
|
98
|
Osawa I, Sonoo T, Soeno S, Hara K, Nakamura K, Goto T. Clinical performance of early warning scoring systems for identifying sepsis among anti-hypertensive agent users. Am J Emerg Med 2021; 48:120-127. [PMID: 33878566 DOI: 10.1016/j.ajem.2021.03.091] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Revised: 03/29/2021] [Accepted: 03/30/2021] [Indexed: 12/29/2022] Open
Abstract
BACKGROUND Little is known about the accuracy of the quick Sequential Organ Failure Assessment (qSOFA) and the National Early Warning Score (NEWS) in identifying sepsis patients with a history of hypertension on anti-hypertensive agents, which affect vital signs as components of the scoring systems. We aimed to examine the ability of qSOFA and NEWS to predict sepsis among anti-hypertensive agent users by comparing them with non-users. METHODS We retrospectively identified adult patients (aged ≥18years) with suspected infection who presented to an emergency department (ED) of a large tertiary medical center in Japan between April 2018 and March 2020. Suspected infection was defined based on the chief complaint of fever, high body temperature, or the clinical context on arrival at the ED. We excluded patients who had trauma or cardiac arrest, those who were transported to other hospitals after arrival at the ED, and those whose vital signs data were mostly missing. The predictive performances of qSOFA and NEWS based on initial vital signs were examined separately for sepsis, ICU admission, and in-hospital mortality and compared between anti-hypertensive agent users and non-users. RESULTS Among 2900 patients with suspected infection presenting to the ED, 291 (10%) had sepsis, 1023 (35%) were admitted to the ICU, and 188 (6.5%) died. The prediction performances of qSOFA and NEWS for each outcome among anti-hypertensive agent users were lower than that among non-users (e.g., c-statistics of qSOFA for sepsis, 0.66 vs. 0.71, p = 0.07; and for ICU admission, 0.70 vs. 0.75, p = 0.01). For identifying sepsis, the sensitivity and specificity of qSOFA ≥2 were 0.43 and 0.77 in anti-hypertensive agent users and 0.51 and 0.82 in non-users. Similar associations were observed for identifying ICU admission and in-hospital mortality. Regardless of the use of anti-hypertensive agents, NEWS had better prediction abilities for each outcome than qSOFA. CONCLUSION The clinical performance of qSOFA and NEWS for identifying sepsis among anti-hypertensive agent users was likely lower than that among non-users.
Collapse
Affiliation(s)
- Itsuki Osawa
- Department of Emergency and Critical Care Medicine, The University of Tokyo Hospital, Tokyo, Japan.
| | - Tomohiro Sonoo
- Department of Emergency and Critical Care Medicine, Hitachi General Hospital, Ibaraki, Japan; TXP Medical Co. Ltd., Tokyo, Japan
| | - Shoko Soeno
- Department of Emergency Medicine, Southern Tohoku General Hospital, Fukushima, Japan
| | - Konan Hara
- TXP Medical Co. Ltd., Tokyo, Japan; Department of Public Health, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Kensuke Nakamura
- Department of Emergency and Critical Care Medicine, Hitachi General Hospital, Ibaraki, Japan
| | - Tadahiro Goto
- TXP Medical Co. Ltd., Tokyo, Japan; Department of Clinical Epidemiology and Health Economics, School of Public Health, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
99
|
Extreme gradient boosting machine learning method for predicting medical treatment in patients with acute bronchiolitis. Biocybern Biomed Eng 2021. [DOI: 10.1016/j.bbe.2021.04.015] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
|
100
|
Kim D, Chae J, Oh Y, Lee J, Kim IY. Automated remote decision-making algorithm as a primary triage system using machine learning techniques. Physiol Meas 2021; 42:025006. [PMID: 33567409 DOI: 10.1088/1361-6579/abe524] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE An objective and convenient primary triage procedure is needed for prioritizing patients who need help in mass casualty incident (MCI) situations, where there is a lack of medical staff and available resources. This study aimed to develop an automated remote decision-making algorithm that remotely categorize a patient's emergency level using clinical parameters that can be measured with a wearable device. APPROACH The algorithm was developed according to the following procedures. First, we used the National Trauma Data Bank data set, a large open trauma patient data set assembled by the American College of Surgeons (ACS). In addition, we performed pre-processing to exclude data when the vital sign or consciousness indicator value was missing or physiologically in an abnormal range. Second, we selected the T-RTS method, which classifies emergency levels into four classes (Delayed, Urgent, Immediate and Dead), as the primary outcome. Third, three machine learning methods widely used in the medical field, logistic regression, random forest, and deep neural network (DNN), were applied to build the algorithm. Finally, each method was evaluated using quantitative performance indicators including the macro-averaged f1 score, macro-averaged mean absolute error (MMAE), and the area under the receiver operating characteristic curve (AUC). MAIN RESULTS For total sets, the logistic regression had a macro-averaged f1 score of 0.673, an MMAE of 0.387 and an AUC value of 0.844 (95% CI, 0.843-0.845), while the random forest and DNN had macro-averaged f1 scores of 0.783 and 0.784, MMAEs of 0.297 and 0.298 and AUC values of 0.882 (95% CI, 0.881-0.883) and 0.883(95% CI, 0.881-0.884), respectively. SIGNIFICANCE In a comprehensive analysis of these results, our algorithm demonstrated a viable approach that could be practically adopted in an MCI. In addition, it can be employed to transfer patients and to redistribute available resources according to their priorities.
Collapse
Affiliation(s)
- Dohyun Kim
- Ground Technology Research Institute, Agency for Defense Development, Daejeon, Republic of Korea
| | | | | | | | | |
Collapse
|