1
|
Meerwijk EL, McElfresh DC, Martins S, Tamang SR. Evaluating accuracy and fairness of clinical decision support algorithms when health care resources are limited. J Biomed Inform 2024; 156:104664. [PMID: 38851413 DOI: 10.1016/j.jbi.2024.104664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Revised: 04/02/2024] [Accepted: 06/02/2024] [Indexed: 06/10/2024]
Abstract
OBJECTIVE Guidance on how to evaluate accuracy and algorithmic fairness across subgroups is missing for clinical models that flag patients for an intervention but when health care resources to administer that intervention are limited. We aimed to propose a framework of metrics that would fit this specific use case. METHODS We evaluated the following metrics and applied them to a Veterans Health Administration clinical model that flags patients for intervention who are at risk of overdose or a suicidal event among outpatients who were prescribed opioids (N = 405,817): Receiver - Operating Characteristic and area under the curve, precision - recall curve, calibration - reliability curve, false positive rate, false negative rate, and false omission rate. In addition, we developed a new approach to visualize false positives and false negatives that we named 'per true positive bars.' We demonstrate the utility of these metrics to our use case for three cohorts of patients at the highest risk (top 0.5 %, 1.0 %, and 5.0 %) by evaluating algorithmic fairness across the following age groups: <=30, 31-50, 51-65, and >65 years old. RESULTS Metrics that allowed us to assess group differences more clearly were the false positive rate, false negative rate, false omission rate, and the new 'per true positive bars'. Metrics with limited utility to our use case were the Receiver - Operating Characteristic and area under the curve, the calibration - reliability curve, and the precision - recall curve. CONCLUSION There is no "one size fits all" approach to model performance monitoring and bias analysis. Our work informs future researchers and clinicians who seek to evaluate accuracy and fairness of predictive models that identify patients to intervene on in the context of limited health care resources. In terms of ease of interpretation and utility for our use case, the new 'per true positive bars' may be the most intuitive to a range of stakeholders and facilitates choosing a threshold that allows weighing false positives against false negatives, which is especially important when predicting severe adverse events.
Collapse
Affiliation(s)
- Esther L Meerwijk
- Program Evaluation and Resource Center, Office of Mental Health and Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA; VA Health Systems Research, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA.
| | - Duncan C McElfresh
- Program Evaluation and Resource Center, Office of Mental Health and Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA
| | - Susana Martins
- Program Evaluation and Resource Center, Office of Mental Health and Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA
| | - Suzanne R Tamang
- Program Evaluation and Resource Center, Office of Mental Health and Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA; VA Health Systems Research, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA; Department of Medicine, Stanford University, Stanford, CA, USA
| |
Collapse
|
2
|
Strehlow M, Alvarez A, Blomkalns AL, Caretta-Wyer H, Gharahbaghian L, Imler D, Khan A, Lee M, Lobo V, Newberry JA, Riberia R, Sebok-Syer S, Shen S, Gisondi MA. Precision emergency medicine. Acad Emerg Med 2024. [PMID: 38940478 DOI: 10.1111/acem.14962] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 04/13/2024] [Accepted: 05/23/2024] [Indexed: 06/29/2024]
Abstract
BACKGROUND Precision health is a burgeoning scientific discipline that aims to incorporate individual variability in biological, behavioral, and social factors to develop personalized health solutions. To date, emergency medicine has not deeply engaged in the precision health movement. However, rapid advances in health technology, data science, and medical informatics offer new opportunities for emergency medicine to realize the promises of precision health. METHODS In this article, we conceptualize precision emergency medicine as an emerging paradigm and identify key drivers of its implementation into current and future clinical practice. We acknowledge important obstacles to the specialty-wide adoption of precision emergency medicine and offer solutions that conceive a successful path forward. RESULTS Precision emergency medicine is defined as the use of information and technology to deliver acute care effectively, efficiently, and authentically to individual patients and their communities. Key drivers and opportunities include leveraging human data, capitalizing on technology and digital tools, providing deliberate access to care, advancing population health, and reimagining provider education and roles. Overcoming challenges in equity, privacy, and cost is essential for success. We close with a call to action to proactively incorporate precision health into the clinical practice of emergency medicine, the training of future emergency physicians, and the research agenda of the specialty. CONCLUSIONS Precision emergency medicine leverages new technology and data-driven artificial intelligence to advance diagnostic testing, individualize patient care plans and therapeutics, and strategically refine the convergence of the health system and the community.
Collapse
Affiliation(s)
- Matthew Strehlow
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Al'ai Alvarez
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Andra L Blomkalns
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Holly Caretta-Wyer
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Laleh Gharahbaghian
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Daniel Imler
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Ayesha Khan
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Moon Lee
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Viveta Lobo
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Jennifer A Newberry
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Ryan Riberia
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Stefanie Sebok-Syer
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Sam Shen
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Michael A Gisondi
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
3
|
Thiruganasambandamoorthy V, Probst MA, Poterucha TJ, Sandhu RK, Toarta C, Raj SR, Sheldon R, Rahgozar A, Grant L. Role of Artificial Intelligence in Improving Syncope Management. Can J Cardiol 2024:S0828-282X(24)00429-X. [PMID: 38838932 DOI: 10.1016/j.cjca.2024.05.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 04/25/2024] [Accepted: 05/01/2024] [Indexed: 06/07/2024] Open
Abstract
Syncope is common in the general population and a common presenting symptom in acute care settings. Substantial costs are attributed to the care of patients with syncope. Current challenges include differentiating syncope from its mimickers, identifying serious underlying conditions that caused the syncope, and wide variations in current management. Although validated risk tools exist, especially for short-term prognosis, there is inconsistent application, and the current approach does not meet patient needs and expectations. Artificial intelligence (AI) techniques, such as machine learning methods including natural language processing, can potentially address the current challenges in syncope management. Preliminary evidence from published studies indicates that it is possible to accurately differentiate syncope from its mimickers and predict short-term prognosis and hospitalisation. More recently, AI analysis of electrocardiograms has shown promise in detection of serious structural and functional cardiac abnormalities, which has the potential to improve syncope care. Future AI studies have the potential to address current issues in syncope management. AI can automatically prognosticate risk in real time by accessing traditional and nontraditional data. However, steps to mitigate known problems such as generalisability, patient privacy, data protection, and liability will be needed. In the past AI has had limited impact due to underdeveloped analytical methods, lack of computing power, poor access to powerful computing systems, and availability of reliable high-quality data. All impediments except data have been solved. AI will live up to its promise to transform syncope care if the health care system can satisfy AI requirement of large scale, robust, accurate, and reliable data.
Collapse
Affiliation(s)
- Venkatesh Thiruganasambandamoorthy
- Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada; Ottawa Hospital Research Institute, The Ottawa Hospital, Ottawa, Ontario, Canada; School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada.
| | - Marc A Probst
- Department of Emergency Medicine, Columbia University Irving Medical Center, New York, New York, USA
| | - Timothy J Poterucha
- Seymour, Paul, and Gloria Milstein Division of Cardiology, Department of Medicine, Columbia University Irving Medical Center, New York, New York, USA
| | - Roopinder K Sandhu
- Libin Cardiovascular Institute, Department of Cardiac Sciences, University of Calgary, Calgary, Alberta, Canada
| | - Cristian Toarta
- Department of Emergency Medicine, McGill University, Montréal, Québec, Canada; McGill University Health Centre, Montréal, Québec, Canada
| | - Satish R Raj
- Libin Cardiovascular Institute, Department of Cardiac Sciences, University of Calgary, Calgary, Alberta, Canada
| | - Robert Sheldon
- Libin Cardiovascular Institute, Department of Cardiac Sciences, University of Calgary, Calgary, Alberta, Canada
| | - Arya Rahgozar
- Ottawa Hospital Research Institute, The Ottawa Hospital, Ottawa, Ontario, Canada; School of Engineering Design and Teaching Innovation, University of Ottawa, Ottawa, Ontario, Canada
| | - Lars Grant
- Department of Emergency Medicine, McGill University, Montréal, Québec, Canada; Lady Davis Research Institute, Montréal, Québec, Canada; Jewish General Hospital, Montréal, Québec, Canada
| |
Collapse
|
4
|
Assis de Souza A, Stubbs AP, Hesselink DA, Baan CC, Boer K. Cherry on Top or Real Need? A Review of Explainable Machine Learning in Kidney Transplantation. Transplantation 2024:00007890-990000000-00768. [PMID: 38773859 DOI: 10.1097/tp.0000000000005063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/24/2024]
Abstract
Research on solid organ transplantation has taken advantage of the substantial acquisition of medical data and the use of artificial intelligence (AI) and machine learning (ML) to answer diagnostic, prognostic, and therapeutic questions for many years. Nevertheless, despite the question of whether AI models add value to traditional modeling approaches, such as regression models, their "black box" nature is one of the factors that have hindered the translation from research to clinical practice. Several techniques that make such models understandable to humans were developed with the promise of increasing transparency in the support of medical decision-making. These techniques should help AI to close the gap between theory and practice by yielding trust in the model by doctors and patients, allowing model auditing, and facilitating compliance with emergent AI regulations. But is this also happening in the field of kidney transplantation? This review reports the use and explanation of "black box" models to diagnose and predict kidney allograft rejection, delayed graft function, graft failure, and other related outcomes after kidney transplantation. In particular, we emphasize the discussion on the need (or not) to explain ML models for biological discovery and clinical implementation in kidney transplantation. We also discuss promising future research paths for these computational tools.
Collapse
Affiliation(s)
- Alvaro Assis de Souza
- Department of Internal Medicine, Erasmus MC Transplant Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands
| | - Andrew P Stubbs
- Department of Pathology and Clinical Bioinformatics, Erasmus MC Stubbs Group, University Medical Center Rotterdam, Rotterdam, the Netherlands
| | - Dennis A Hesselink
- Department of Internal Medicine, Erasmus MC Transplant Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands
| | - Carla C Baan
- Department of Internal Medicine, Erasmus MC Transplant Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands
| | - Karin Boer
- Department of Internal Medicine, Erasmus MC Transplant Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands
| |
Collapse
|
5
|
Alrawashdeh A, Alqahtani S, Alkhatib ZI, Kheirallah K, Melhem NY, Alwidyan M, Al-Dekah AM, Alshammari T, Nehme Z. Applications and Performance of Machine Learning Algorithms in Emergency Medical Services: A Scoping Review. Prehosp Disaster Med 2024:1-11. [PMID: 38757150 DOI: 10.1017/s1049023x24000414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/18/2024]
Abstract
OBJECTIVE The aim of this study was to summarize the literature on the applications of machine learning (ML) and their performance in Emergency Medical Services (EMS). METHODS Four relevant electronic databases were searched (from inception through January 2024) for all original studies that employed EMS-guided ML algorithms to enhance the clinical and operational performance of EMS. Two reviewers screened the retrieved studies and extracted relevant data from the included studies. The characteristics of included studies, employed ML algorithms, and their performance were quantitively described across primary domains and subdomains. RESULTS This review included a total of 164 studies published from 2005 through 2024. Of those, 125 were clinical domain focused and 39 were operational. The characteristics of ML algorithms such as sample size, number and type of input features, and performance varied between and within domains and subdomains of applications. Clinical applications of ML algorithms involved triage or diagnosis classification (n = 62), treatment prediction (n = 12), or clinical outcome prediction (n = 50), mainly for out-of-hospital cardiac arrest/OHCA (n = 62), cardiovascular diseases/CVDs (n = 19), and trauma (n = 24). The performance of these ML algorithms varied, with a median area under the receiver operating characteristic curve (AUC) of 85.6%, accuracy of 88.1%, sensitivity of 86.05%, and specificity of 86.5%. Within the operational studies, the operational task of most ML algorithms was ambulance allocation (n = 21), followed by ambulance detection (n = 5), ambulance deployment (n = 5), route optimization (n = 5), and quality assurance (n = 3). The performance of all operational ML algorithms varied and had a median AUC of 96.1%, accuracy of 90.0%, sensitivity of 94.4%, and specificity of 87.7%. Generally, neural network and ensemble algorithms, to some degree, out-performed other ML algorithms. CONCLUSION Triaging and managing different prehospital medical conditions and augmenting ambulance performance can be improved by ML algorithms. Future reports should focus on a specific clinical condition or operational task to improve the precision of the performance metrics of ML models.
Collapse
Affiliation(s)
- Ahmad Alrawashdeh
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | - Saeed Alqahtani
- Department of Emergency Medical Services, Prince Sultan Military College for Health Sciences, Dhahran, Saudi Arabia
| | - Zaid I Alkhatib
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | - Khalid Kheirallah
- Department of Public Health and Family Medicine, Faculty of Medicine, Jordan University of Science and Technology, Irbid, Jordan
| | - Nebras Y Melhem
- Department of Anatomy, Physiology and Biochemistry, Faculty of Medicine, The Hashemite University, Zarqa, Jordan
| | - Mahmoud Alwidyan
- Department of Allied Medical Sciences, Jordan University of Science and Technology, Irbid, Jordan
| | | | - Talal Alshammari
- Department of Emergency Medical Care, College of Applied Medical Sciences, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Ziad Nehme
- Ambulance Victoria, Doncaster, Victoria, Australia
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
6
|
Kolasa K, Admassu B, Hołownia-Voloskova M, Kędzior KJ, Poirrier JE, Perni S. Systematic reviews of machine learning in healthcare: a literature review. Expert Rev Pharmacoecon Outcomes Res 2024; 24:63-115. [PMID: 37955147 DOI: 10.1080/14737167.2023.2279107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 10/31/2023] [Indexed: 11/14/2023]
Abstract
INTRODUCTION The increasing availability of data and computing power has made machine learning (ML) a viable approach to faster, more efficient healthcare delivery. METHODS A systematic literature review (SLR) of published SLRs evaluating ML applications in healthcare settings published between1 January 2010 and 27 March 2023 was conducted. RESULTS In total 220 SLRs covering 10,462 ML algorithms were reviewed. The main application of AI in medicine related to the clinical prediction and disease prognosis in oncology and neurology with the use of imaging data. Accuracy, specificity, and sensitivity were provided in 56%, 28%, and 25% SLRs respectively. Internal and external validation was reported in 53% and less than 1% of the cases respectively. The most common modeling approach was neural networks (2,454 ML algorithms), followed by support vector machine and random forest/decision trees (1,578 and 1,522 ML algorithms, respectively). EXPERT OPINION The review indicated considerable reporting gaps in terms of the ML's performance, both internal and external validation. Greater accessibility to healthcare data for developers can ensure the faster adoption of ML algorithms into clinical practice.
Collapse
Affiliation(s)
- Katarzyna Kolasa
- Division of Health Economics and Healthcare Management, Kozminski University, Warsaw, Poland
| | - Bisrat Admassu
- Division of Health Economics and Healthcare Management, Kozminski University, Warsaw, Poland
| | | | | | | | | |
Collapse
|
7
|
Stewart J, Lu J, Goudie A, Arendts G, Meka SA, Freeman S, Walker K, Sprivulis P, Sanfilippo F, Bennamoun M, Dwivedi G. Applications of natural language processing at emergency department triage: A narrative review. PLoS One 2023; 18:e0279953. [PMID: 38096321 PMCID: PMC10721204 DOI: 10.1371/journal.pone.0279953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Accepted: 11/30/2023] [Indexed: 12/18/2023] Open
Abstract
INTRODUCTION Natural language processing (NLP) uses various computational methods to analyse and understand human language, and has been applied to data acquired at Emergency Department (ED) triage to predict various outcomes. The objective of this scoping review is to evaluate how NLP has been applied to data acquired at ED triage, assess if NLP based models outperform humans or current risk stratification techniques when predicting outcomes, and assess if incorporating free-text improve predictive performance of models when compared to predictive models that use only structured data. METHODS All English language peer-reviewed research that applied an NLP technique to free-text obtained at ED triage was eligible for inclusion. We excluded studies focusing solely on disease surveillance, and studies that used information obtained after triage. We searched the electronic databases MEDLINE, Embase, Cochrane Database of Systematic Reviews, Web of Science, and Scopus for medical subject headings and text keywords related to NLP and triage. Databases were last searched on 01/01/2022. Risk of bias in studies was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Due to the high level of heterogeneity between studies and high risk of bias, a metanalysis was not conducted. Instead, a narrative synthesis is provided. RESULTS In total, 3730 studies were screened, and 20 studies were included. The population size varied greatly between studies ranging from 1.8 million patients to 598 triage notes. The most common outcomes assessed were prediction of triage score, prediction of admission, and prediction of critical illness. NLP models achieved high accuracy in predicting need for admission, triage score, critical illness, and mapping free-text chief complaints to structured fields. Incorporating both structured data and free-text data improved results when compared to models that used only structured data. However, the majority of studies (80%) were assessed to have a high risk of bias, and only one study reported the deployment of an NLP model into clinical practice. CONCLUSION Unstructured free-text triage notes have been used by NLP models to predict clinically relevant outcomes. However, the majority of studies have a high risk of bias, most research is retrospective, and there are few examples of implementation into clinical practice. Future work is needed to prospectively assess if applying NLP to data acquired at ED triage improves ED outcomes when compared to usual clinical practice.
Collapse
Affiliation(s)
- Jonathon Stewart
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Juan Lu
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Adrian Goudie
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Glenn Arendts
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Shiv Akarsh Meka
- HIVE & Data and Digital Innovation, Royal Perth Hospital, Perth, Western Australia, Australia
| | - Sam Freeman
- Department of Emergency Medicine, St Vincent’s Hospital Melbourne, Melbourne, Victoria, Australia
- SensiLab, Monash University, Melbourne, Victoria, Australia
| | - Katie Walker
- School of Clinical Sciences at Monash Health, Monash University, Melbourne, Victoria, Australia
| | - Peter Sprivulis
- Western Australia Department of Health, East Perth, Western Australia, Australia
| | - Frank Sanfilippo
- School of Population and Global Health, University of Western Australia, Crawley, Western Australia, Australia
| | - Mohammed Bennamoun
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Girish Dwivedi
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Cardiology, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| |
Collapse
|
8
|
Leonard F, O’Sullivan D, Gilligan J, O’Shea N, Barrett MJ. Supporting clinical decision making in the emergency department for paediatric patients using machine learning: A scoping review protocol. PLoS One 2023; 18:e0294231. [PMID: 37972029 PMCID: PMC10653406 DOI: 10.1371/journal.pone.0294231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 10/28/2023] [Indexed: 11/19/2023] Open
Abstract
INTRODUCTION Machine learning as a clinical decision support system tool has the potential to assist clinicians who must make complex and accurate medical decisions in fast paced environments such as the emergency department. This paper presents a protocol for a scoping review, with the objective of summarising the existing research on machine learning clinical decision support system tools in the emergency department, focusing on models that can be used for paediatric patients, where a knowledge gap exists. MATERIALS AND METHODS The methodology used will follow the scoping study framework of Arksey and O'Malley, along with other guidelines. Machine learning clinical decision support system tools for any outcome and population (paediatric/adult/mixed) for use in the emergency department will be included. Articles such as grey literature, letters, pre-prints, editorials, scoping/literature/narrative reviews, non-English full text papers, protocols, surveys, abstract or full text not available and models based on synthesised data will be excluded. Articles from the last five years will be included. Four databases will be searched: Medline (EBSCO), CINAHL (EBSCO), EMBASE and Cochrane Central. Independent reviewers will perform the screening in two sequential stages (stage 1: clinician expertise and stage 2: computer science expertise), disagreements will be resolved by discussion. Data relevant to the research question will be collected. Quantitative analysis will be performed to generate the results. DISCUSSION The study results will summarise the existing research on machine learning clinical decision support tools in the emergency department, focusing on models that can be used for paediatric patients. This holds the promise to identify opportunities to both incorporate models in clinical practice and to develop future models by utilising reviewers from diverse backgrounds and relevant expertise.
Collapse
Affiliation(s)
- Fiona Leonard
- School of Computer Science, Technological University Dublin, Dublin, Ireland
- Digital Health Department, Children’s Health Ireland, Crumlin, Dublin, Ireland
| | - Dympna O’Sullivan
- School of Computer Science, Technological University Dublin, Dublin, Ireland
| | - John Gilligan
- School of Computer Science, Technological University Dublin, Dublin, Ireland
| | - Nicola O’Shea
- Library and Information Service, Children’s Health Ireland at Crumlin, Dublin, Ireland
| | - Michael J. Barrett
- Department of Paediatric Emergency Medicine, Children’s Health Ireland at Crumlin, Dublin, Ireland
- Women’s and Children’s Health, School of Medicine, University College Dublin, Dublin, Ireland
| |
Collapse
|
9
|
Zworth M, Kareemi H, Boroumand S, Sikora L, Stiell I, Yadav K. Machine learning for the diagnosis of acute coronary syndrome using a 12-lead ECG: a systematic review. CAN J EMERG MED 2023; 25:818-827. [PMID: 37665551 DOI: 10.1007/s43678-023-00572-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 07/26/2023] [Indexed: 09/05/2023]
Abstract
OBJECTIVES Prompt diagnosis of acute coronary syndrome (ACS) using a 12-lead electrocardiogram (ECG) is a critical task for emergency physicians. While computerized algorithms for ECG interpretation are limited in their accuracy, machine learning (ML) models have shown promise in several areas of clinical medicine. We performed a systematic review to compare the performance of ML-based ECG analysis to clinician or non-ML computerized ECG interpretation in the diagnosis of ACS for emergency department (ED) or prehospital patients. METHODS We searched Medline, Embase, Cochrane Central, and CINAHL databases from inception to May 18, 2022. We included studies that compared ML algorithms to either clinicians or non-ML based software in their ability to diagnose ACS using only a 12-lead ECG, in adult patients experiencing chest pain or symptoms concerning for ACS in the ED or prehospital setting. We used QUADAS-2 for risk of bias assessment. Prospero registration CRD42021264765. RESULTS Our search yielded 1062 abstracts. 10 studies met inclusion criteria. Five model types were tested, including neural networks, random forest, and gradient boosting. In five studies with complete performance data, ML models were more sensitive but less specific (sensitivity range 0.59-0.98, specificity range 0.44-0.95) than clinicians (sensitivity range 0.22-0.93, specificity range 0.63-0.98) in diagnosing ACS. In four studies that reported it, ML models had better discrimination (area under ROC curve range 0.79-0.98) than clinicians (area under ROC curve 0.67-0.78). Heterogeneity in both methodology and reporting methods precluded a meta-analysis. Several studies had high risk of bias due to patient selection, lack of external validation, and unreliable reference standards for ACS diagnosis. CONCLUSIONS ML models have overall higher discrimination and sensitivity but lower specificity than clinicians and non-ML software in ECG interpretation for the diagnosis of ACS. ML-based ECG interpretation could potentially serve a role as a "safety net", alerting emergency care providers to a missed acute MI when it has not been diagnosed. More rigorous primary research is needed to definitively demonstrate the ability of ML to outperform clinicians at ECG interpretation.
Collapse
Affiliation(s)
- Max Zworth
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada.
| | - Hashim Kareemi
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Suzanne Boroumand
- Department of Family Medicine, McMaster University Faculty of Health Sciences, Hamilton, ON, Canada
| | - Lindsey Sikora
- Health Sciences Library, University of Ottawa, Ottawa, ON, Canada
| | - Ian Stiell
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Krishan Yadav
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
10
|
Monahan AC, Feldman SS. The Utility of Predictive Modeling and a Systems Process Approach to Reduce Emergency Department Crowding: A Position Paper. Interact J Med Res 2023; 12:e42016. [PMID: 37428536 PMCID: PMC10366955 DOI: 10.2196/42016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Revised: 01/12/2023] [Accepted: 05/10/2023] [Indexed: 07/11/2023] Open
Abstract
Emergency department (ED) crowding and its main causes, exit block and boarding, continue to threaten the quality and safety of ED care. Most interventions to reduce crowding have not been comprehensive or system solutions, only focusing on part of the care procession and not directly affecting boarding reduction. This position paper proposes that the ED crowding problem can be optimally addressed by applying a systems approach using predictive modeling to identify patients at risk of being admitted to the hospital and uses that information to initiate the time-consuming bed management process earlier in the care continuum, shortening the time during which patients wait in the ED for an inpatient bed assignment, thus removing the exit block that causes boarding and subsequently reducing crowding.
Collapse
Affiliation(s)
| | - Sue S Feldman
- Department of Health Services Administration, University of Alabama at Birmingham, Birmingham, AL, United States
| |
Collapse
|
11
|
Chan SL, Lee JW, Ong MEH, Siddiqui FJ, Graves N, Ho AFW, Liu N. Implementation of Prediction Models in the Emergency Department from an Implementation Science Perspective-Determinants, Outcomes, and Real-World Impact: A Scoping Review. Ann Emerg Med 2023; 82:22-36. [PMID: 36925394 DOI: 10.1016/j.annemergmed.2023.02.001] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 01/26/2023] [Accepted: 02/01/2023] [Indexed: 03/16/2023]
Abstract
STUDY OBJECTIVE Prediction models offer a promising form of clinical decision support in the complex and fast-paced environment of the emergency department (ED). Despite significant advancements in model development and validation, implementation of such models in routine clinical practice remains elusive. This scoping review aims to survey the current state of prediction model implementation in the ED and to provide insights on contributing factors and outcomes from an implementation science perspective. METHODS We searched 4 databases from their inception to May 20, 2022: MEDLINE (through PubMed), Embase, Scopus, and CINAHL. Articles that reported implementation outcomes and/or contextual determinants under the Reach, Effectiveness, Adoption, Implementation Maintenance (RE-AIM)/Practical, Robust, Implementation, and Sustainability Model (PRISM) framework were included. Characteristics of studies, models, and results of the RE-AIM/PRISM domains were summarized narratively. RESULTS Thirty-six reports on 31 implementations were included. The most common prediction models implemented were early warning scores. The most common implementation strategies used were training stakeholders, infrastructural changes, and using evaluative or iterative strategies. Only one report examined ED patients' perspectives, whereas the rest were focused on the experience of health care workers or organizational stakeholders. Key determinants of successful implementation include strong stakeholder engagement, codevelopment of workflows and implementation strategies, education, and usability. CONCLUSION Examining ED prediction models from an implementation science perspective can provide valuable insights and help guide future implementations.
Collapse
Affiliation(s)
- Sze Ling Chan
- Health Services Research Center, Singapore Health Services, Singapore; Program in Health Services and Systems Research, Duke-NUS Medical School, Singapore
| | - Jin Wee Lee
- Center for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Marcus Eng Hock Ong
- Health Services Research Center, Singapore Health Services, Singapore; Program in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore
| | | | - Nicholas Graves
- Program in Health Services and Systems Research, Duke-NUS Medical School, Singapore
| | - Andrew Fu Wah Ho
- Department of Emergency Medicine, Singapore General Hospital, Singapore; Prehospital Emergency Research Center, Duke-NUS Medical School, Singapore
| | - Nan Liu
- Health Services Research Center, Singapore Health Services, Singapore; Program in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Center for Quantitative Medicine, Duke-NUS Medical School, Singapore; SingHealth AI Office, Singapore Health Services, Singapore; Institute of Data Science, National University of Singapore, Singapore.
| |
Collapse
|
12
|
Casano N, Santini SJ, Vittorini P, Sinatti G, Carducci P, Mastroianni CM, Ciardi MR, Pasculli P, Petrucci E, Marinangeli F, Balsano C. Application of machine learning approach in emergency department to support clinical decision making for SARS-CoV-2 infected patients. J Integr Bioinform 2023; 20:jib-2022-0047. [PMID: 36877860 PMCID: PMC10561065 DOI: 10.1515/jib-2022-0047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2022] [Revised: 01/20/2023] [Accepted: 02/08/2023] [Indexed: 03/08/2023] Open
Abstract
To support physicians in clinical decision process on patients affected by Coronavirus Disease 2019 (COVID-19) in areas with a low vaccination rate, we devised and evaluated the performances of several machine learning (ML) classifiers fed with readily available clinical and laboratory data. Our observational retrospective study collected data from a cohort of 779 COVID-19 patients presenting to three hospitals of the Lazio-Abruzzo area (Italy). Based on a different selection of clinical and respiratory (ROX index and PaO2/FiO2 ratio) variables, we devised an AI-driven tool to predict safe discharge from ED, disease severity and mortality during hospitalization. To predict safe discharge our best classifier is an RF integrated with ROX index that reached AUC of 0.96. To predict disease severity the best classifier was an RF integrated with ROX index that reached an AUC of 0.91. For mortality prediction the best classifier was an RF integrated with ROX index, that reached an AUC of 0.91. The results obtained thanks to our algorithms are consistent with the scientific literature an accomplish significant performances to forecast safe discharge from ED and severe clinical course of COVID-19.
Collapse
Affiliation(s)
- Nicolò Casano
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
| | - Silvano Junior Santini
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
- Francesco Balsano Foundation, Via Giovanni Battista Martini 6, 00198, Rome, Italy
| | - Pierpaolo Vittorini
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
| | - Gaia Sinatti
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
- Francesco Balsano Foundation, Via Giovanni Battista Martini 6, 00198, Rome, Italy
| | - Paolo Carducci
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
| | - Claudio Maria Mastroianni
- Department of Public Health and Infectious Diseases, “Sapienza” University of Rome, Policlinico Umberto I Hospital, Rome, Italy
| | - Maria Rosa Ciardi
- Department of Public Health and Infectious Diseases, “Sapienza” University of Rome, Policlinico Umberto I Hospital, Rome, Italy
| | - Patrizia Pasculli
- Department of Public Health and Infectious Diseases, “Sapienza” University of Rome, Policlinico Umberto I Hospital, Rome, Italy
| | - Emiliano Petrucci
- Department of Anesthesiology, Intensive Care and Pain Treatment, University of L’Aquila, L’Aquila, Italy
| | - Franco Marinangeli
- Department of Anesthesiology, Intensive Care and Pain Treatment, University of L’Aquila, L’Aquila, Italy
| | - Clara Balsano
- School of Emergency Medicine, Interdisciplinary BioMedical group on Artificial Intelligence, IBMAI, Department MeSVA, University of L’Aquila, L’Aquila, Italy
- Francesco Balsano Foundation, Via Giovanni Battista Martini 6, 00198, Rome, Italy
| |
Collapse
|
13
|
Dhiman P, Ma J, Andaur Navarro CL, Speich B, Bullock G, Damen JAA, Hooft L, Kirtley S, Riley RD, Van Calster B, Moons KGM, Collins GS. Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review. J Clin Epidemiol 2023; 157:120-133. [PMID: 36935090 DOI: 10.1016/j.jclinepi.2023.03.012] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 03/03/2023] [Accepted: 03/14/2023] [Indexed: 03/19/2023]
Abstract
OBJECTIVES In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction. STUDY DESIGN AND SETTING We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices. RESULTS We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion. CONCLUSION The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.
Collapse
Affiliation(s)
- Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Constanza L Andaur Navarro
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Benjamin Speich
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; Meta-Research Centre, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Garrett Bullock
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Johanna A A Damen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Shona Kirtley
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK, ST5 5BG
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands; EPI-centre, KU Leuven, Leuven, Belgium
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK; NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| |
Collapse
|
14
|
Truchot A, Raynaud M, Kamar N, Naesens M, Legendre C, Delahousse M, Thaunat O, Buchler M, Crespo M, Linhares K, Orandi BJ, Akalin E, Pujol GS, Silva HT, Gupta G, Segev DL, Jouven X, Bentall AJ, Stegall MD, Lefaucheur C, Aubert O, Loupy A. Machine learning does not outperform traditional statistical modelling for kidney allograft failure prediction. Kidney Int 2023; 103:936-948. [PMID: 36572246 DOI: 10.1016/j.kint.2022.12.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Revised: 11/04/2022] [Accepted: 12/15/2022] [Indexed: 12/24/2022]
Abstract
Machine learning (ML) models have recently shown potential for predicting kidney allograft outcomes. However, their ability to outperform traditional approaches remains poorly investigated. Therefore, using large cohorts of kidney transplant recipients from 14 centers worldwide, we developed ML-based prediction models for kidney allograft survival and compared their prediction performances to those achieved by a validated Cox-Based Prognostication System (CBPS). In a French derivation cohort of 4000 patients, candidate determinants of allograft failure including donor, recipient and transplant-related parameters were used as predictors to develop tree-based models (RSF, RSF-ERT, CIF), Support Vector Machine models (LK-SVM, AK-SVM) and a gradient boosting model (XGBoost). Models were externally validated with cohorts of 2214 patients from Europe, 1537 from North America, and 671 from South America. Among these 8422 kidney transplant recipients, 1081 (12.84%) lost their grafts after a median post-transplant follow-up time of 6.25 years (Inter Quartile Range 4.33-8.73). At seven years post-risk evaluation, the ML models achieved a C-index of 0.788 (95% bootstrap percentile confidence interval 0.736-0.833), 0.779 (0.724-0.825), 0.786 (0.735-0.832), 0.527 (0.456-0.602), 0.704 (0.648-0.759) and 0.767 (0.711-0.815) for RSF, RSF-ERT, CIF, LK-SVM, AK-SVM and XGBoost respectively, compared with 0.808 (0.792-0.829) for the CBPS. In validation cohorts, ML models' discrimination performances were in a similar range of those of the CBPS. Calibrations of the ML models were similar or less accurate than those of the CBPS. Thus, when using a transparent methodological pipeline in validated international cohorts, ML models, despite overall good performances, do not outperform a traditional CBPS in predicting kidney allograft failure. Hence, our current study supports the continued use of traditional statistical approaches for kidney graft prognostication.
Collapse
Affiliation(s)
- Agathe Truchot
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France
| | - Marc Raynaud
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France
| | - Nassim Kamar
- Université Paul Sabatier, INSERM, Department of Nephrology and Organ Transplantation, CHU Rangueil and Purpan, Toulouse, France
| | - Maarten Naesens
- Department of Microbiology, Immunology and Transplantation, KU Leuven, Leuven, Belgium
| | - Christophe Legendre
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France; Kidney Transplant Department, Necker Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Michel Delahousse
- Department of Transplantation, Nephrology and Clinical Immunology, Foch Hospital, Suresnes, France
| | - Olivier Thaunat
- Department of Transplantation, Nephrology and Clinical Immunology, Hospices Civils de Lyon, Lyon, France
| | - Matthias Buchler
- Nephrology and Immunology Department, Bretonneau Hospital, Tours, France
| | - Marta Crespo
- Department of Nephrology, Hospital del Mar Barcelona, Barcelona, Spain
| | - Kamilla Linhares
- Hospital do Rim, Escola Paulista de Medicina, Universidade Federal de São Paulo, São Paulo, Brazil
| | - Babak J Orandi
- University of Alabama at Birmingham Heersink School of Medicine, Birmingham, Alabama, USA
| | - Enver Akalin
- Renal Division, Montefiore Medical Centre, Kidney Transplantation Program, Albert Einstein College of Medicine, New York, New York, USA
| | - Gervacio Soler Pujol
- Unidad de Trasplante Renopancreas, Centro de Educacion Medica e Investigaciones Clinicas Buenos Aires, Buenos Aires, Argentina
| | - Helio Tedesco Silva
- Hospital do Rim, Escola Paulista de Medicina, Universidade Federal de São Paulo, São Paulo, Brazil
| | - Gaurav Gupta
- Division of Nephrology, Department of Internal Medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Dorry L Segev
- Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Xavier Jouven
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France; Cardiology Department, European Georges Pompidou Hospital, Paris, France
| | - Andrew J Bentall
- William J von Liebig Centre for Transplantation and Clinical Regeneration, Mayo Clinic, Rochester, Minnesota, USA
| | - Mark D Stegall
- William J von Liebig Centre for Transplantation and Clinical Regeneration, Mayo Clinic, Rochester, Minnesota, USA
| | - Carmen Lefaucheur
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France; Kidney Transplant Department, Saint-Louis Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Olivier Aubert
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France; Kidney Transplant Department, Necker Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Alexandre Loupy
- Université de Paris, INSERM, PARCC, Paris Translational Research Centre for Organ Transplantation, Paris, France; Kidney Transplant Department, Necker Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France.
| |
Collapse
|
15
|
Abstract
BACKGROUND Undertriaged patients have worse outcomes than appropriately triaged patients. Machine learning provides better triage prediction than conventional triage in emergency departments, but no machine learning-based undertriage prediction models have yet been developed for prehospital telephone triage. We developed and validated machine learning models for telephone triage. MATERIALS AND METHODS We conducted a retrospective cohort study with the largest after-hour house-call (AHHC) service dataset in Japan. Participants were ≥16 years and used the AHHC service between 1 November 2018 and 31 January 2021. We developed five prediction models based on support vector machine (SVM), lasso regression (LR), random forest (RF), gradient-boosted decision tree (XGB), and deep neural network (DNN). The primary outcome was undertriage, and predictors were telephone triage level and routinely available telephone-based data, including age, sex, 80 chief complaint categories and 10 comorbidities. We measured the area under the receiver operating characteristic curve (AUROC) for all the models. RESULTS We identified 15,442 eligible patients (age: 38.4 ± 16.6, male: 57.2%), including 298 (1.9%; age: 58.2 ± 23.9, male: 55.0%) undertriaged patients. RF and XGB outperformed the other models, with the AUROC values (95% confidence interval; 95% CI) of the SVM, LR, RF, XGB and DNN for undertriage being 0.62 (0.55-0.69), 0.79 (0.74-0.83), 0.81 (0.76-0.86), 0.80 (0.75-0.84) and 0.77 (0.73-0.82), respectively. CONCLUSIONS We found that RF and XGB outperformed other models. Our findings suggest that machine learning models can facilitate the early detection of undertriage and early intervention to yield substantially improved patient outcomes.KEY MESSAGESUndertriaged patients experience worse outcomes than appropriately triaged patients; thus, we developed machine learning models for predicting undertriage in the prehospital setting. In addition, we identified the predictors of risk factors associated with undertriage.Random forest and gradient-boosted decision tree models demonstrated better prediction performance, and the models identified the risk factors associated with undertriage.Machine learning models aid in the early detection of undertriage, leading to significantly improved patient outcomes and identifying undertriage-associated risk factors, including chief complaint categories, could help prioritize conventional telephone triage protocol revision.
Collapse
Affiliation(s)
- Ryota Inokuchi
- Department of Health Services Research, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan.,Health Services Research and Development Center, University of Tsukuba, Tsukuba, Japan
| | - Masao Iwagami
- Department of Health Services Research, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan.,Health Services Research and Development Center, University of Tsukuba, Tsukuba, Japan
| | - Yu Sun
- Health Services Research and Development Center, University of Tsukuba, Tsukuba, Japan.,Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Japan
| | - Ayaka Sakamoto
- Health Services Research and Development Center, University of Tsukuba, Tsukuba, Japan.,Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Japan
| | - Nanako Tamiya
- Department of Health Services Research, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan.,Health Services Research and Development Center, University of Tsukuba, Tsukuba, Japan
| |
Collapse
|
16
|
Liu Z, Zhang L, Wu J, Zheng Z, Gao J, Lin Y, Liu Y, Xu H, Zhou Y. Machine learning-based classification of circadian rhythm characteristics for mild cognitive impairment in the elderly. Front Public Health 2022; 10:1036886. [PMID: 36388285 PMCID: PMC9650188 DOI: 10.3389/fpubh.2022.1036886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Accepted: 10/10/2022] [Indexed: 01/29/2023] Open
Abstract
Introduction Using wrist-wearable sensors to ecological transient assessment may provide a more valid assessment of physical activity, sedentary time, sleep and circadian rhythm than self-reported questionnaires, but has not been used widely to study the association with mild cognitive impairment and their characteristics. Methods 31 normal cognitive ability participants and 68 MCI participants were monitored with tri-axial accelerometer and nocturnal photo volumetric pulse wave signals for 14 days. Two machine learning algorithms: gradient boosting decision tree and eXtreme gradient boosting were constructed using data on daytime physical activity, sedentary time and nighttime physiological functions, including heart rate, heart rate variability, respiratory rate and oxygen saturation, combined with subjective scale features. The accuracy, precision, recall, F1 value, and AUC of the different models are compared, and the training and model effectiveness are validated by the subject-based leave-one-out method. Results The low physical activity state was higher in the MCI group than in the cognitively normal group between 8:00 and 11:00 (P < 0.05), the daily rhythm trend of the high physical activity state was generally lower in the MCI group than in the cognitively normal group (P < 0.05). The peak rhythms in the sedentary state appeared at 12:00-15:00 and 20:00. The peak rhythms of rMSSD, HRV high frequency output power, and HRV low frequency output power in the 6h HRV parameters at night in the MCI group disappeared at 3:00 a.m., and the amplitude of fluctuations decreased; the amplitude of fluctuations of LHratio nocturnal rhythm increased and the phase was disturbed; the oxygen saturation was between 90 and 95% and less than 90% were increased in all time periods (P < 0.05). The F1 value of the two machine learning algorithms for MCI classification of multi-feature data combined with subjective scales were XGBoost (78.02) and GBDT (84.04). Conclusion By collecting PSQI Scale data combined with circadian rhythm characteristics monitored by wrist-wearable sensors, we are able to construct XGBoost and GBDT machine learning models with good discrimination, thus providing an early warning solution for identifying family and community members with high risk of MCI.
Collapse
Affiliation(s)
- Zhizhen Liu
- National-Local Joint Engineering Research Center of Rehabilitation Medicine Technology, Fujian University of Traditional Chinese Medicine, Fuzhou, China,Zhizhen Liu
| | - Lin Zhang
- College of Rehabilitation Medicine, Fujian University of Traditional Chinese Medicine, Fuzhou, China
| | - Jingsong Wu
- College of Rehabilitation Medicine, Fujian University of Traditional Chinese Medicine, Fuzhou, China
| | - Zhicheng Zheng
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Jiahui Gao
- College of Rehabilitation Medicine, Fujian University of Traditional Chinese Medicine, Fuzhou, China
| | - Yongsheng Lin
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Yinghua Liu
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Haihua Xu
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Yongjin Zhou
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China,Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China,*Correspondence: Yongjin Zhou
| |
Collapse
|
17
|
Martinez-Millana A, Saez-Saez A, Tornero-Costa R, Azzopardi-Muscat N, Traver V, Novillo-Ortiz D. Artificial intelligence and its impact on the domains of universal health coverage, health emergencies and health promotion: An overview of systematic reviews. Int J Med Inform 2022; 166:104855. [PMID: 35998421 PMCID: PMC9551134 DOI: 10.1016/j.ijmedinf.2022.104855] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Revised: 08/01/2022] [Accepted: 08/11/2022] [Indexed: 12/04/2022]
Abstract
An overview of systematic reviews on the application of AI including 129 studies. AI use is prominent in Universal Health Coverage, featuring image analysis in neoplasms. Half of the reviews did not evaluate validation procedures nor reporting guidelines. Risk of bias was only included un a third of the reviews. There is not sufficient evidence to transfer AI to actual healthcare delivery.
Background Artificial intelligence is fueling a new revolution in medicine and in the healthcare sector. Despite the growing evidence on the benefits of artificial intelligence there are several aspects that limit the measure of its impact in people’s health. It is necessary to assess the current status on the application of AI towards the improvement of people’s health in the domains defined by WHO’s Thirteenth General Programme of Work (GPW13) and the European Programme of Work (EPW), to inform about trends, gaps, opportunities, and challenges. Objective To perform a systematic overview of systematic reviews on the application of artificial intelligence in the people’s health domains as defined in the GPW13 and provide a comprehensive and updated map on the application specialties of artificial intelligence in terms of methodologies, algorithms, data sources, outcomes, predictors, performance, and methodological quality. Methods A systematic search in MEDLINE, EMBASE, Cochrane and IEEEXplore was conducted between January 2015 and June 2021 to collect systematic reviews using a combination of keywords related to the domains of universal health coverage, health emergencies protection, and better health and wellbeing as defined by the WHO’s PGW13 and EPW. Eligibility criteria was based on methodological quality and the inclusion of practical implementation of artificial intelligence. Records were classified and labeled using ICD-11 categories into the domains of the GPW13. Descriptors related to the area of implementation, type of modeling, data entities, outcomes and implementation on care delivery were extracted using a structured form and methodological aspects of the included reviews studies was assessed using the AMSTAR checklist. Results The search strategy resulted in the screening of 815 systematic reviews from which 203 were assessed for eligibility and 129 were included in the review. The most predominant domain for artificial intelligence applications was Universal Health Coverage (N = 98) followed by Health Emergencies (N = 16) and Better Health and Wellbeing (N = 15). Neoplasms area on Universal Health Coverage was the disease area featuring most of the applications (21.7 %, N = 28). The reviews featured analytics primarily over both public and private data sources (67.44 %, N = 87). The most used type of data was medical imaging (31.8 %, N = 41) and predictors based on regions of interest and clinical data. The most prominent subdomain of Artificial Intelligence was Machine Learning (43.4 %, N = 56), in which Support Vector Machine method was predominant (20.9 %, N = 27). Regarding the purpose, the application of Artificial Intelligence I is focused on the prediction of the diseases (36.4 %, N = 47). With respect to the validation, more than a half of the reviews (54.3 %, N = 70) did not report a validation procedure and, whenever available, the main performance indicator was the accuracy (28.7 %, N = 37). According to the methodological quality assessment, a third of the reviews (34.9 %, N = 45) implemented methods for analysis the risk of bias and the overall AMSTAR score below was 5 (4.01 ± 1.93) on all the included systematic reviews. Conclusion Artificial intelligence is being used for disease modelling, diagnose, classification and prediction in the three domains of GPW13. However, the evidence is often limited to laboratory and the level of adoption is largely unbalanced between ICD-11 categoriesand diseases. Data availability is a determinant factor on the developmental stage of artificial intelligence applications. Most of the reviewed studies show a poor methodological quality and are at high risk of bias, which limits the reproducibility of the results and the reliability of translating these applications to real clinical scenarios. The analyzed papers show results only in laboratory and testing scenarios and not in clinical trials nor case studies, limiting the supporting evidence to transfer artificial intelligence to actual care delivery.
Collapse
Affiliation(s)
- Antonio Martinez-Millana
- Instituto Universitario de Investigación de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Camino de Vera S/N, Valencia 46022, Spain
| | - Aida Saez-Saez
- Instituto Universitario de Investigación de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Camino de Vera S/N, Valencia 46022, Spain
| | - Roberto Tornero-Costa
- Instituto Universitario de Investigación de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Camino de Vera S/N, Valencia 46022, Spain
| | - Natasha Azzopardi-Muscat
- Division of Country Health Policies and Systems, World Health Organization, Regional Office for Europe, Copenhagen, Denmark
| | - Vicente Traver
- Instituto Universitario de Investigación de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Camino de Vera S/N, Valencia 46022, Spain
| | - David Novillo-Ortiz
- Division of Country Health Policies and Systems, World Health Organization, Regional Office for Europe, Copenhagen, Denmark.
| |
Collapse
|
18
|
Improta G, Majolo M, Raiola E, Russo G, Longo G, Triassi M. A case study to investigate the impact of overcrowding indices in emergency departments. BMC Emerg Med 2022; 22:143. [PMID: 35945503 PMCID: PMC9360659 DOI: 10.1186/s12873-022-00703-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Accepted: 08/01/2022] [Indexed: 11/10/2022] Open
Abstract
Background Emergency department (ED) overcrowding is widespread in hospitals in many countries, causing severe consequences to patient outcomes, staff work and the system, with an overall increase in costs. Therefore, health managers are constantly looking for new preventive and corrective measures to counter this phenomenon. To do this, however, it is necessary to be able to characterize the problem objectively. For this reason, various indices are used in the literature to assess ED crowding. In this work, we explore the use of two of the most widespread crowding indices in an ED of an Italian national hospital, investigate their relationships and discuss their effectiveness. Methods In this study, two of the most widely used indices in the literature, the National Emergency Department Overcrowding Scale (NEDOCS) and the Emergency Department Working Index (EDWIN), were analysed to characterize overcrowding in the ED of A.O.R.N. “A. Cardarelli” of Naples, which included 1678 clinical cases. The measurement was taken every 15 minutes for a period of 7 days. Results The results showed consistency in the use of EDWIN and NEDOCS indices as measures of overcrowding, especially in severe overcrowding conditions. Indeed, in the examined case study, both EDWIN and NEDOCS showed very low rates of occurrence of severe overcrowding (2–3%). In contrast, regarding differences in the estimation of busy to overcrowded ED rates, the EDWIN index proved to be less sensitive in distinguishing these variations in the occupancy of the ED. Furthermore, within the target week considered in the study, the results show that, according to both EDWIN and NEDOCS, higher overcrowding rates occurred during the middle week rather than during the weekend. Finally, a low degree of correlation between the two indices was found. Conclusions The effectiveness of both EDWIN and NEDOCS in measuring ED crowding and overcrowding was investigated, and the main differences and relationships in the use of the indices are highlighted. While both indices are useful ED performance metrics, they are not always interchangeable, and their combined use could provide more details in understanding ED dynamics and possibly predicting future critical conditions, thus enhancing ED management.
Collapse
Affiliation(s)
- Giovanni Improta
- Department of Public Health, University of Naples "Federico II", Via Pansini, No. 5 - ZIP, 80131, Naples, Italy. .,Interdepartmental Center for Research in Healthcare Management and Innovation in Healthcare (CIRMIS), University of Naples "Federico II", Naples, Italy.
| | | | | | | | | | - Maria Triassi
- Department of Public Health, University of Naples "Federico II", Via Pansini, No. 5 - ZIP, 80131, Naples, Italy.,Interdepartmental Center for Research in Healthcare Management and Innovation in Healthcare (CIRMIS), University of Naples "Federico II", Naples, Italy
| |
Collapse
|
19
|
Machine Learning in the Prediction of Trauma Outcomes: A Systematic Review. Ann Emerg Med 2022; 80:440-455. [PMID: 35842343 DOI: 10.1016/j.annemergmed.2022.05.011] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2021] [Revised: 03/20/2022] [Accepted: 05/04/2022] [Indexed: 11/23/2022]
Abstract
STUDY OBJECTIVE Machine learning models carry unique potential as decision-making aids and prediction tools for improving patient care. Traumatically injured patients provide a uniquely heterogeneous population with severe injuries that can be difficult to predict. Given the relative infancy of machine learning applications in medicine, this systematic review aimed to better understand the current state of machine learning development and implementation to help create a basis for future research. METHODS We conducted a systematic review from inception to May 2021, using Embase, MEDLINE through Ovid, Web of Science, Google Scholar, and relevant gray literature, for uses of machine learning in predicting the outcomes of trauma patients. The screening and data extraction were performed by 2 independent reviewers. RESULTS Of the 14,694 identified articles screened, 67 were included for data extraction. Artificial neural networks comprised the most commonly used model, and mortality was the most prevalent outcome of interest. In terms of machine learning model development, there was a lack of studies that employed external validation, feature selection methods, and performed formal calibration testing. Significant heterogeneity in reporting was also observed between the machine learning models employed, patient populations, performance metrics, and features employed. CONCLUSION This review highlights the heterogeneity in the development and reporting of machine learning models for the prediction of trauma outcomes. While these models present an area of opportunity as an ancillary to clinical decision-making, we recommend more standardization and rigorous guidelines for the development of future models.
Collapse
|
20
|
Magnusson C, Hagiwara MA, Norberg-Boysen G, Kauppi W, Herlitz J, Axelsson C, Packendorff N, Larsson G, Wibring K. Suboptimal prehospital decision- making for referral to alternative levels of care - frequency, measurement, acceptance rate and room for improvement. BMC Emerg Med 2022; 22:89. [PMID: 35606694 PMCID: PMC9125920 DOI: 10.1186/s12873-022-00643-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 05/05/2022] [Indexed: 11/15/2022] Open
Abstract
Background The emergency medical services (EMS) have undergone dramatic changes during the past few decades. Increased utilisation, changes in care-seeking behaviour and competence among EMS clinicians have given rise to a shift in EMS strategies in many countries. From transport to the emergency department to at the scene deciding on the most appropriate level of care and mode of transport. Among the non-conveyed patients some may suffer from “time-sensitive conditions” delaying diagnosis and treatment. Thus, four questions arise:How often are time-sensitive cases referred to primary care or self-care advice? How can we measure and define the level of inappropriate clinical decision-making? What is acceptable? How to increase patient safety?
Main text To what extent time-sensitive cases are non-conveyed varies. About 5–25% of referred patients visit the emergency department within 72 hours, 5% are hospitalised, 1–3% are reported to have a time-sensitive condition and seven-day mortality rates range from 0.3 to 6%. The level of inappropriate clinical decision-making can be measured using surrogate measures such as emergency department attendances, hospitalisation and short-term mortality. These measures do not reveal time-sensitive conditions. Defining a scoring system may be one alternative, where misclassifications of time-sensitive cases are rated based on how severely they affected patient outcome. In terms of what is acceptable there is no general agreement. Although a zero-vision approach does not seem to be realistic unless under-triage is split into different levels of severity with zero-vision in the most severe categories. There are several ways to reduce the risk of misclassifications. Implementation of support systems for decision-making using machine learning to improve the initial assessment is one approach. Using a trigger tool to identify adverse events is another. Conclusion A substantial number of patients are non-conveyed, including a small portion with time-sensitive conditions. This poses a threat to patient safety. No general agreement on how to define and measure the extent of such EMS referrals and no agreement of what is acceptable exists, but we conclude an overall zero-vision is not realistic. Developing specific tools supporting decision making regarding EMS referral may be one way to reduce misclassification rates.
Collapse
Affiliation(s)
- Carl Magnusson
- Department of Molecular and Clinical Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, SE-405 30, Gothenburg, Sweden. .,Department of Prehospital Emergency Care , Sahlgrenska University Hospital, SE-411 04, Gothenburg, Sweden.
| | - Magnus Andersson Hagiwara
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Gabriella Norberg-Boysen
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Wivica Kauppi
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Johan Herlitz
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Christer Axelsson
- Department of Prehospital Emergency Care , Sahlgrenska University Hospital, SE-411 04, Gothenburg, Sweden.,Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Niclas Packendorff
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Glenn Larsson
- Prehospen-Centre for Prehospital Research, Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Kristoffer Wibring
- Department of Ambulance and Prehospital Care, Region Halland, SE-302 49, Halmstad, Sweden
| |
Collapse
|
21
|
Ramlakhan S, Saatchi R, Sabir L, Singh Y, Hughes R, Shobayo O, Ventour D. Understanding and interpreting artificial intelligence, machine learning and deep learning in Emergency Medicine. Emerg Med J 2022; 39:380-385. [PMID: 35241440 DOI: 10.1136/emermed-2021-212068] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2021] [Accepted: 01/29/2022] [Indexed: 02/06/2023]
Affiliation(s)
- Shammi Ramlakhan
- Emergency Department, Sheffield Children's Hospital, Sheffield, UK
| | - Reza Saatchi
- Electronics and Computer Engineering Research Institute, Sheffield Hallam University, Sheffield, UK
| | - Lisa Sabir
- Emergency Department, Sheffield Children's Hospital, Sheffield, UK
| | - Yardesh Singh
- Department of Clinical Surgical Sciences, Faculty of Medical Sciences, The University of the West Indies, St Augustine, Trinidad and Tobago
| | - Ruby Hughes
- Simulation and Modelling Unit, Advanced Forming Research Centre, University of Strathclyde, Sheffield, UK
| | - Olamilekan Shobayo
- Electronics and Computer Engineering Research Institute, Sheffield Hallam University, Sheffield, UK
| | - Dale Ventour
- Department of Clinical Surgical Sciences, Faculty of Medical Sciences, The University of the West Indies, St Augustine, Trinidad and Tobago
| |
Collapse
|
22
|
Ramlakhan SL, Saatchi R, Sabir L, Ventour D, Shobayo O, Hughes R, Singh Y. Building artificial intelligence and machine learning models : a primer for emergency physicians. Emerg Med J 2022; 39:e1. [PMID: 35241439 DOI: 10.1136/emermed-2022-212379] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2022] [Accepted: 02/14/2022] [Indexed: 12/23/2022]
Abstract
There has been a rise in the number of studies relating to the role of artificial intelligence (AI) in healthcare. Its potential in Emergency Medicine (EM) has been explored in recent years with operational, predictive, diagnostic and prognostic emergency department (ED) implementations being developed. For EM researchers building models de novo, collaborative working with data scientists is invaluable throughout the process. Synergism and understanding between domain (EM) and data experts increases the likelihood of realising a successful real-world model. Our linked manuscript provided a conceptual framework (including a glossary of AI terms) to support clinicians in interpreting AI research. The aim of this paper is to supplement that framework by exploring the key issues for clinicians and researchers to consider in the process of developing an AI model.
Collapse
Affiliation(s)
- Shammi L Ramlakhan
- Emergency Department, Sheffield Children's NHS Foundation Trust, Sheffield, UK
| | - Reza Saatchi
- Electronics & Computer Engineering Research Institute, Sheffield Hallam University, Sheffield, UK
| | - Lisa Sabir
- Emergency Department, Sheffield Children's NHS Foundation Trust, Sheffield, UK
| | - Dale Ventour
- Faculty of Medical Sciences, The University of the West Indies, St Augustine, Trinidad and Tobago
| | - Olamilekan Shobayo
- Electronics & Computer Engineering Research Institute, Sheffield Hallam University, Sheffield, UK
| | - Ruby Hughes
- Advanced Forming Research Centre, University of Strathclyde, Sheffield, UK
| | - Yardesh Singh
- Faculty of Medical Sciences, The University of the West Indies, St Augustine, Trinidad and Tobago
| |
Collapse
|
23
|
Desai MD, Tootooni MS, Bobay KL. Can Prehospital Data Improve Early Identification of Sepsis in Emergency Department? An Integrative Review of Machine Learning Approaches. Appl Clin Inform 2022; 13:189-202. [PMID: 35108741 PMCID: PMC8810268 DOI: 10.1055/s-0042-1742369] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND Sepsis is associated with high mortality, especially during the novel coronavirus disease 2019 (COVID-19) pandemic. Along with high monetary health care costs for sepsis treatment, there is a lasting impact on lives of sepsis survivors and their caregivers. Early identification is necessary to reduce the negative impact of sepsis and to improve patient outcomes. Prehospital data are among the earliest information collected by health care systems. Using these untapped sources of data in machine learning (ML)-based approaches can identify patients with sepsis earlier in emergency department (ED). OBJECTIVES This integrative literature review aims to discuss the importance of utilizing prehospital data elements in ED, summarize their current use in developing ML-based prediction models, and specifically identify those data elements that can potentially contribute to early identification of sepsis in ED when used in ML-based approaches. METHOD Literature search strategy includes following two separate searches: (1) use of prehospital data in ML models in ED; and (2) ML models that are developed specifically to predict/detect sepsis in ED. In total, 24 articles are used in this review. RESULTS A summary of prehospital data used to identify time-sensitive conditions earlier in ED is provided. Literature related to use of ML models for early identification of sepsis in ED is limited and no studies were found related to ML models using prehospital data in prediction/early identification of sepsis in ED. Among those using ED data, ML models outperform traditional statistical models. In addition, the use of the free-text elements and natural language processing (NLP) methods could result in better prediction of sepsis in ED. CONCLUSION This study reviews the use of prehospital data in early decision-making in ED and suggests that researchers utilize such data elements for prediction/early identification of sepsis in ML-based approaches.
Collapse
Affiliation(s)
- Manushi D. Desai
- Marcella Niehoff School of Nursing, Loyola University Chicago, Maywood, Illinois, United States
| | - Mohammad S. Tootooni
- Department of Health Informatics and Data Science, Parkinson School of Health Sciences and Public Health, Loyola University Chicago, Maywood, Illinois, United States
| | - Kathleen L. Bobay
- Department of Health Informatics and Data Science, Parkinson School of Health Sciences and Public Health, Loyola University Chicago, Maywood, Illinois, United States,Address for correspondence Kathleen L. Bobay, PhD, RN, FAAN Department of Health Informatics and Data Science, Parkinson School of Health Sciences and Public Health, Marcella Niehoff School of Nursing, Loyola University Chicago2160 South First Avenue, Maywood, IL 60153United States
| |
Collapse
|
24
|
Li B, Feridooni T, Cuen-Ojeda C, Kishibe T, de Mestral C, Mamdani M, Al-Omran M. Machine learning in vascular surgery: a systematic review and critical appraisal. NPJ Digit Med 2022; 5:7. [PMID: 35046493 PMCID: PMC8770468 DOI: 10.1038/s41746-021-00552-y] [Citation(s) in RCA: 48] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2021] [Accepted: 12/13/2021] [Indexed: 12/18/2022] Open
Abstract
Machine learning (ML) is a rapidly advancing field with increasing utility in health care. We conducted a systematic review and critical appraisal of ML applications in vascular surgery. MEDLINE, Embase, and Cochrane CENTRAL were searched from inception to March 1, 2021. Study screening, data extraction, and quality assessment were performed by two independent reviewers, with a third author resolving discrepancies. All original studies reporting ML applications in vascular surgery were included. Publication trends, disease conditions, methodologies, and outcomes were summarized. Critical appraisal was conducted using the PROBAST risk-of-bias and TRIPOD reporting adherence tools. We included 212 studies from a pool of 2235 unique articles. ML techniques were used for diagnosis, prognosis, and image segmentation in carotid stenosis, aortic aneurysm/dissection, peripheral artery disease, diabetic foot ulcer, venous disease, and renal artery stenosis. The number of publications on ML in vascular surgery increased from 1 (1991-1996) to 118 (2016-2021). Most studies were retrospective and single center, with no randomized controlled trials. The median area under the receiver operating characteristic curve (AUROC) was 0.88 (range 0.61-1.00), with 79.5% [62/78] studies reporting AUROC ≥ 0.80. Out of 22 studies comparing ML techniques to existing prediction tools, clinicians, or traditional regression models, 20 performed better and 2 performed similarly. Overall, 94.8% (201/212) studies had high risk-of-bias and adherence to reporting standards was poor with a rate of 41.4%. Despite improvements over time, study quality and reporting remain inadequate. Future studies should consider standardized tools such as PROBAST and TRIPOD to improve study quality and clinical applicability.
Collapse
Affiliation(s)
- Ben Li
- Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada
- Division of Vascular Surgery, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada
- Temerty Centre for Artificial Intelligence Research and Education in Medicine (T-CAIREM), University of Toronto, 1 King's College Circle, Toronto, ON, M5S 1A8, Canada
| | - Tiam Feridooni
- Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada
- Division of Vascular Surgery, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada
| | - Cesar Cuen-Ojeda
- Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada
- Division of Vascular Surgery, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada
| | - Teruko Kishibe
- Health Sciences Library, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, ON, M5B 1T8, Canada
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, ON, M5B 1T8, Canada
| | - Charles de Mestral
- Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada
- Division of Vascular Surgery, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, ON, M5B 1T8, Canada
- Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, 155 College St, Toronto, ON, M5T 3M7, Canada
| | - Muhammad Mamdani
- Temerty Centre for Artificial Intelligence Research and Education in Medicine (T-CAIREM), University of Toronto, 1 King's College Circle, Toronto, ON, M5S 1A8, Canada
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, ON, M5B 1T8, Canada
- Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, 155 College St, Toronto, ON, M5T 3M7, Canada
- Leslie Dan Faculty of Pharmacy, University of Toronto, 144 College St, Toronto, ON, M5S 3M2, Canada
| | - Mohammed Al-Omran
- Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada.
- Division of Vascular Surgery, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada.
- Temerty Centre for Artificial Intelligence Research and Education in Medicine (T-CAIREM), University of Toronto, 1 King's College Circle, Toronto, ON, M5S 1A8, Canada.
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, ON, M5B 1T8, Canada.
- Institute of Medical Science, University of Toronto, 1 King's College Circle, Toronto, ON, M5S 1A8, Canada.
- Department of Surgery, King Saud University, ZIP 4545, Riyadh, 11451, Kingdom of Saudi Arabia.
| |
Collapse
|
25
|
Wong XY, Ang YK, Li K, Chin YH, Lam SSW, Tan KBK, Chua MCH, Ong MEH, Liu N, Pourghaderi AR, Ho AFW. Development and validation of the SARICA score to predict survival after return of spontaneous circulation in out of hospital cardiac arrest using an interpretable machine learning framework. Resuscitation 2021; 170:126-133. [PMID: 34843878 DOI: 10.1016/j.resuscitation.2021.11.029] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2021] [Revised: 11/19/2021] [Accepted: 11/22/2021] [Indexed: 01/17/2023]
Abstract
BACKGROUND Accurate and timely prognostication of patients with out-of-hospital cardiac arrest (OHCA) who achieved the return of spontaneous circulation (ROSC) is crucial in clinical decision-making, resource allocation, and communications with next-of-kins. We aimed to develop the Survival After ROSC in Cardiac Arrest (SARICA), a practical clinical decision tool to predict survival in OHCA patients who attained ROSC. METHODS We utilized real-world Singapore data from the population-based Pan-Asian Resuscitation Outcomes Study between 2010-2018. We excluded patients without ROSC. The dataset was segmented into training (60%), validation (20%) and testing (20%) cohorts. The primary endpoint was survival (to 30-days or hospital discharge). AutoScore, an interpretable machine-learning based clinical score generation algorithm, was used to develop SARICA. Candidate factors were chosen based on objective demographic and clinical factors commonly available at the time of admission. Performance of SARICA was evaluated based on receiver-operating curve (ROC) analyses. RESULTS 5970 patients were included, of which 855 (14.3%) survived. A three-variable model was determined to be most parsimonious. Prehospital ROSC, age, and initial heart rhythm were identified for inclusion via random forest selection. Finally, SARICA consisted of these 3 variables and ranged from 0 to 10 points, achieving an area under the ROC (AUC) of 0.87 (95% confidence interval: 0.84-0.90) within the testing cohort. CONCLUSION We developed and internally validated the SARICA score to accurately predict survival of OHCA patients with ROSC at the time of admission. SARICA is clinically practical and developed using an interpretable machine-learning framework. SARICA has unknown generalizability pending external validation studies.
Collapse
Affiliation(s)
- Xiang Yi Wong
- Pre-hospital and Emergency Research Centre, Duke-NUS Medical School, Singapore; Singapore Civil Defence Force, Ministry of Home Affairs, Singapore.
| | - Yu Kai Ang
- Pre-hospital and Emergency Research Centre, Duke-NUS Medical School, Singapore
| | - Keqi Li
- Institute of System Science, National University of Singapore, Singapore
| | - Yip Han Chin
- Yong Loo Lin School of Medicine, National University of Singapore, Singapore
| | | | | | | | - Marcus Eng Hock Ong
- Department of Emergency Medicine, Singapore General Hospital, Singapore; Health Services & Systems Research, Duke-NUS Medical School, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Ahmad Reza Pourghaderi
- Health Services Research Centre, SingHealth, Singapore; Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Andrew Fu Wah Ho
- Pre-hospital and Emergency Research Centre, Duke-NUS Medical School, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore.
| | | |
Collapse
|
26
|
Naemi A, Schmidt T, Mansourvar M, Naghavi-Behzad M, Ebrahimi A, Wiil UK. Machine learning techniques for mortality prediction in emergency departments: a systematic review. BMJ Open 2021; 11:e052663. [PMID: 34728454 PMCID: PMC8565537 DOI: 10.1136/bmjopen-2021-052663] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/24/2021] [Accepted: 09/27/2021] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVES This systematic review aimed to assess the performance and clinical feasibility of machine learning (ML) algorithms in prediction of in-hospital mortality for medical patients using vital signs at emergency departments (EDs). DESIGN A systematic review was performed. SETTING The databases including Medline (PubMed), Scopus and Embase (Ovid) were searched between 2010 and 2021, to extract published articles in English, describing ML-based models utilising vital sign variables to predict in-hospital mortality for patients admitted at EDs. Critical appraisal and data extraction for systematic reviews of prediction modelling studies checklist was used for study planning and data extraction. The risk of bias for included papers was assessed using the prediction risk of bias assessment tool. PARTICIPANTS Admitted patients to the ED. MAIN OUTCOME MEASURE In-hospital mortality. RESULTS Fifteen articles were included in the final review. We found that eight models including logistic regression, decision tree, K-nearest neighbours, support vector machine, gradient boosting, random forest, artificial neural networks and deep neural networks have been applied in this domain. Most studies failed to report essential main analysis steps such as data preprocessing and handling missing values. Fourteen included studies had a high risk of bias in the statistical analysis part, which could lead to poor performance in practice. Although the main aim of all studies was developing a predictive model for mortality, nine articles did not provide a time horizon for the prediction. CONCLUSION This review provided an updated overview of the state-of-the-art and revealed research gaps; based on these, we provide eight recommendations for future studies to make the use of ML more feasible in practice. By following these recommendations, we expect to see more robust ML models applied in the future to help clinicians identify patient deterioration earlier.
Collapse
Affiliation(s)
- Amin Naemi
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Thomas Schmidt
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Marjan Mansourvar
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Mohammad Naghavi-Behzad
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Department of Nuclear Medicine, Odense University Hospital, Odense, Denmark
| | - Ali Ebrahimi
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| | - Uffe Kock Wiil
- Maersk Mc-Kinney Moller Institute, Center for Health Informatics and Technology,University of Southern Denmark, Odense, Denmark
| |
Collapse
|
27
|
Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, Collins GS, Bajpai R, Riley RD, Moons KGM, Hooft L. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ 2021; 375:n2281. [PMID: 34670780 PMCID: PMC8527348 DOI: 10.1136/bmj.n2281] [Citation(s) in RCA: 96] [Impact Index Per Article: 32.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 09/13/2021] [Indexed: 12/23/2022]
Abstract
OBJECTIVE To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties. DESIGN Systematic review. DATA SOURCES PubMed from 1 January 2018 to 31 December 2019. ELIGIBILITY CRITERIA Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes. REVIEW METHODS Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall). RESULTS 152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively. CONCLUSION Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice. SYSTEMATIC REVIEW REGISTRATION PROSPERO CRD42019161764.
Collapse
Affiliation(s)
- Constanza L Andaur Navarro
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Johanna A A Damen
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Toshihiko Takada
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Steven W J Nijman
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Ram Bajpai
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Lotty Hooft
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| |
Collapse
|
28
|
Puig-Campmany M, Blázquez-Andión M, Ris-Romeu J. Triage tools: a cautious (and critical) view towards their use in old patients. Eur Geriatr Med 2021; 13:319-322. [PMID: 34609734 DOI: 10.1007/s41999-021-00572-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Affiliation(s)
- Mireia Puig-Campmany
- Emergency Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Catalonia, Spain. .,Universitat Autònoma de Barcelona, Barcelona, Spain. .,IIB Sant Pau Research, Barcelona, Spain.
| | - Marta Blázquez-Andión
- Emergency Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Catalonia, Spain.,Universitat Autònoma de Barcelona, Barcelona, Spain.,IIB Sant Pau Research, Barcelona, Spain
| | - Josep Ris-Romeu
- Emergency Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Catalonia, Spain.,Universitat Autònoma de Barcelona, Barcelona, Spain.,IIB Sant Pau Research, Barcelona, Spain
| |
Collapse
|
29
|
Monahan AC, Feldman SS. Models Predicting Hospital Admission of Adult Patients Utilizing Prehospital Data: Systematic Review Using PROBAST and CHARMS. JMIR Med Inform 2021; 9:e30022. [PMID: 34528893 PMCID: PMC8485197 DOI: 10.2196/30022] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2021] [Revised: 05/27/2021] [Accepted: 07/28/2021] [Indexed: 12/23/2022] Open
Abstract
Background Emergency department boarding and hospital exit block are primary causes of emergency department crowding and have been conclusively associated with poor patient outcomes and major threats to patient safety. Boarding occurs when a patient is delayed or blocked from transitioning out of the emergency department because of dysfunctional transition or bed assignment processes. Predictive models for estimating the probability of an occurrence of this type could be useful in reducing or preventing emergency department boarding and hospital exit block, to reduce emergency department crowding. Objective The aim of this study was to identify and appraise the predictive performance, predictor utility, model application, and model utility of hospital admission prediction models that utilized prehospital, adult patient data and aimed to address emergency department crowding. Methods We searched multiple databases for studies, from inception to September 30, 2019, that evaluated models predicting adult patients’ imminent hospital admission, with prehospital patient data and regression analysis. We used PROBAST (Prediction Model Risk of Bias Assessment Tool) and CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) to critically assess studies. Results Potential biases were found in most studies, which suggested that each model’s predictive performance required further investigation. We found that select prehospital patient data contribute to the identification of patients requiring hospital admission. Biomarker predictors may add superior value and advantages to models. It is, however, important to note that no models had been integrated with an information system or workflow, operated independently as electronic devices, or operated in real time within the care environment. Several models could be used at the site-of-care in real time without digital devices, which would make them suitable for low-technology or no-electricity environments. Conclusions There is incredible potential for prehospital admission prediction models to improve patient care and hospital operations. Patient data can be utilized to act as predictors and as data-driven, actionable tools to identify patients likely to require imminent hospital admission and reduce patient boarding and crowding in emergency departments. Prediction models can be used to justify earlier patient admission and care, to lower morbidity and mortality, and models that utilize biomarker predictors offer additional advantages.
Collapse
Affiliation(s)
- Ann Corneille Monahan
- Department of Epidemiology & Public Health, School of Public Health, University College Cork, Cork, Ireland
| | - Sue S Feldman
- Department of Health Services Administration, University of Alabama at Birmingham, Birmingham, AL, United States
| |
Collapse
|
30
|
Park JH, Choi J, Lee S, Shin SD, Song KJ. Use of Time-to-Event Analysis to Develop On-Scene Return of Spontaneous Circulation Prediction for Out-of-Hospital Cardiac Arrest Patients. Ann Emerg Med 2021; 79:132-144. [PMID: 34417073 DOI: 10.1016/j.annemergmed.2021.07.121] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Revised: 07/05/2021] [Accepted: 07/14/2021] [Indexed: 12/23/2022]
Abstract
STUDY OBJECTIVE We aimed to train and validate the time to on-scene return of spontaneous circulation prediction models using time-to-event analysis among out-of-hospital cardiac arrest patients. METHODS Using a Korean population-based out-of-hospital cardiac arrest registry, we selected a total of 105,215 adults with presumed cardiac etiologies between 2013 and 2018. Patients from 2013 to 2017 and from 2018 were analyzed for training and test, respectively. We developed 4 time-to-event analyzing models (Cox proportional hazard [Cox], random survival forest, extreme gradient boosting survival, and DeepHit) and 4 classification models (logistic regression, random forest, extreme gradient boosting, and feedforward neural network). Patient characteristics and Utstein elements collected at the scene were used as predictors. Discrimination and calibration were evaluated by Harrell's C-index and integrated Brier score. RESULTS Among the 105,215 patients (mean age 70 years and 64% men), 86,314 and 18,901 patients belonged to the training and test sets, respectively. On-scene return of spontaneous circulation was achieved in 5,240 (6.1%) patients in the former set and 1,709 (9.0%) patients in the latter. The proportion of emergency medical services (EMS) management was higher and scene time interval longer in the latter. Median time from EMS scene arrival to on-scene return of spontaneous circulation was 8 minutes for both datasets. Classification models showed similar discrimination and poor calibration power compared to survival models; Cox showed high discrimination with the best calibration (C-index [95% confidence interval]: 0.873 [0.865 to 0.882]; integrated Brier score at 30 minutes: 0.060). CONCLUSION Incorporating time-to-event analysis could lead to improved performance in prediction models and contribute to personalized field EMS resuscitation decisions.
Collapse
Affiliation(s)
- Jeong Ho Park
- Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea; Department of Emergency Medicine, Seoul National University College of Medicine and Hospital, Seoul, Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, Korea
| | - Jinwook Choi
- Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea; Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Korea.
| | - SangMyeong Lee
- School of Electrical Engineering, Undergraduate School of Korea University, Seoul, Korea
| | - Sang Do Shin
- Department of Emergency Medicine, Seoul National University College of Medicine and Hospital, Seoul, Korea; Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, Korea
| | - Kyoung Jun Song
- Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, Korea; Department of Emergency Medicine, Seoul National University College of Medicine and Seoul National University Boramae Medical Center, Seoul, Korea
| |
Collapse
|