1
|
Preiksaitis C, Ashenburg N, Bunney G, Chu A, Kabeer R, Riley F, Ribeira R, Rose C. The Role of Large Language Models in Transforming Emergency Medicine: Scoping Review. JMIR Med Inform 2024; 12:e53787. [PMID: 38728687 PMCID: PMC11127144 DOI: 10.2196/53787] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Revised: 12/20/2023] [Accepted: 04/05/2024] [Indexed: 05/12/2024] Open
Abstract
BACKGROUND Artificial intelligence (AI), more specifically large language models (LLMs), holds significant potential in revolutionizing emergency care delivery by optimizing clinical workflows and enhancing the quality of decision-making. Although enthusiasm for integrating LLMs into emergency medicine (EM) is growing, the existing literature is characterized by a disparate collection of individual studies, conceptual analyses, and preliminary implementations. Given these complexities and gaps in understanding, a cohesive framework is needed to comprehend the existing body of knowledge on the application of LLMs in EM. OBJECTIVE Given the absence of a comprehensive framework for exploring the roles of LLMs in EM, this scoping review aims to systematically map the existing literature on LLMs' potential applications within EM and identify directions for future research. Addressing this gap will allow for informed advancements in the field. METHODS Using PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) criteria, we searched Ovid MEDLINE, Embase, Web of Science, and Google Scholar for papers published between January 2018 and August 2023 that discussed LLMs' use in EM. We excluded other forms of AI. A total of 1994 unique titles and abstracts were screened, and each full-text paper was independently reviewed by 2 authors. Data were abstracted independently, and 5 authors performed a collaborative quantitative and qualitative synthesis of the data. RESULTS A total of 43 papers were included. Studies were predominantly from 2022 to 2023 and conducted in the United States and China. We uncovered four major themes: (1) clinical decision-making and support was highlighted as a pivotal area, with LLMs playing a substantial role in enhancing patient care, notably through their application in real-time triage, allowing early recognition of patient urgency; (2) efficiency, workflow, and information management demonstrated the capacity of LLMs to significantly boost operational efficiency, particularly through the automation of patient record synthesis, which could reduce administrative burden and enhance patient-centric care; (3) risks, ethics, and transparency were identified as areas of concern, especially regarding the reliability of LLMs' outputs, and specific studies highlighted the challenges of ensuring unbiased decision-making amidst potentially flawed training data sets, stressing the importance of thorough validation and ethical oversight; and (4) education and communication possibilities included LLMs' capacity to enrich medical training, such as through using simulated patient interactions that enhance communication skills. CONCLUSIONS LLMs have the potential to fundamentally transform EM, enhancing clinical decision-making, optimizing workflows, and improving patient outcomes. This review sets the stage for future advancements by identifying key research areas: prospective validation of LLM applications, establishing standards for responsible use, understanding provider and patient perceptions, and improving physicians' AI literacy. Effective integration of LLMs into EM will require collaborative efforts and thorough evaluation to ensure these technologies can be safely and effectively applied.
Collapse
Affiliation(s)
- Carl Preiksaitis
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Nicholas Ashenburg
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Gabrielle Bunney
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Andrew Chu
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Rana Kabeer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Fran Riley
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Ryan Ribeira
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Christian Rose
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| |
Collapse
|
2
|
Lebold KM, Preiksaitis C. Is Artificial Intelligence Ready to Take Over Triage? Ann Emerg Med 2024; 83:500-502. [PMID: 38642978 DOI: 10.1016/j.annemergmed.2024.03.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/22/2024]
Affiliation(s)
- Katie M Lebold
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA
| | - Carl Preiksaitis
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA
| |
Collapse
|
3
|
Lee S, Lee J, Park J, Park J, Kim D, Lee J, Oh J. Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage. Am J Emerg Med 2024; 77:29-38. [PMID: 38096637 DOI: 10.1016/j.ajem.2023.11.063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 11/30/2023] [Accepted: 11/30/2023] [Indexed: 02/16/2024] Open
Abstract
OBJECTIVE The manual recording of electronic health records (EHRs) by clinicians in the emergency department (ED) is time-consuming and challenging. In light of recent advancements in large language models (LLMs) such as GPT and BERT, this study aimed to design and validate LLMs for automatic clinical diagnoses. The models were designed to identify 12 medical symptoms and 2 patient histories from simulated clinician-patient conversations within 6 primary symptom scenarios in emergency triage rooms. MATERIALS AND METHOD We developed classification models by fine-tuning BERT, a transformer-based pre-trained model. We subsequently analyzed these models using eXplainable artificial intelligence (XAI) and the Shapley additive explanation (SHAP) method. A Turing test was conducted to ascertain the reliability of the XAI results by comparing them to the outcomes of tasks performed and explained by medical workers. An emergency medicine specialist assessed the results of both XAI and the medical workers. RESULTS We fine-tuned four pre-trained LLMs and compared their classification performance. The KLUE-RoBERTa-based model demonstrated the highest performance (F1-score: 0.965, AUROC: 0.893) on human-transcribed script data. The XAI results using SHAP showed an average Jaccard similarity of 0.722 when compared with explanations of medical workers for 15 samples. The Turing test results revealed a small 6% gap, with XAI and medical workers receiving the mean scores of 3.327 and 3.52, respectively. CONCLUSION This paper highlights the potential of LLMs for automatic EHR recording in Korean EDs. The KLUE-RoBERTa-based model demonstrated superior classification performance. Furthermore, XAI using SHAP provided reliable explanations for model outputs. The reliability of these explanations was confirmed by a Turing test.
Collapse
Affiliation(s)
- Siryeol Lee
- Department of Applied Artificial Intelligence, Hanyang University ERICA, Ansan, Republic of Korea
| | - Juncheol Lee
- Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Republic of Korea
| | - Juntae Park
- School of Electrical Engineering, Hanyang University ERICA, Ansan, Republic of Korea
| | - Jiwoo Park
- Department of Emergency Medicine, Hanyang University Hospital, Seoul, Republic of Korea
| | - Dohoon Kim
- Department of Translational Medicine, Biomedical Science and Engineering Hanyang University, Seoul, Republic of Korea
| | - Joohyun Lee
- Department of Applied Artificial Intelligence, Hanyang University ERICA, Ansan, Republic of Korea; School of Electrical Engineering, Hanyang University ERICA, Ansan, Republic of Korea.
| | - Jaehoon Oh
- Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Republic of Korea.
| |
Collapse
|
4
|
Stewart J, Lu J, Goudie A, Arendts G, Meka SA, Freeman S, Walker K, Sprivulis P, Sanfilippo F, Bennamoun M, Dwivedi G. Applications of natural language processing at emergency department triage: A narrative review. PLoS One 2023; 18:e0279953. [PMID: 38096321 PMCID: PMC10721204 DOI: 10.1371/journal.pone.0279953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Accepted: 11/30/2023] [Indexed: 12/18/2023] Open
Abstract
INTRODUCTION Natural language processing (NLP) uses various computational methods to analyse and understand human language, and has been applied to data acquired at Emergency Department (ED) triage to predict various outcomes. The objective of this scoping review is to evaluate how NLP has been applied to data acquired at ED triage, assess if NLP based models outperform humans or current risk stratification techniques when predicting outcomes, and assess if incorporating free-text improve predictive performance of models when compared to predictive models that use only structured data. METHODS All English language peer-reviewed research that applied an NLP technique to free-text obtained at ED triage was eligible for inclusion. We excluded studies focusing solely on disease surveillance, and studies that used information obtained after triage. We searched the electronic databases MEDLINE, Embase, Cochrane Database of Systematic Reviews, Web of Science, and Scopus for medical subject headings and text keywords related to NLP and triage. Databases were last searched on 01/01/2022. Risk of bias in studies was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). Due to the high level of heterogeneity between studies and high risk of bias, a metanalysis was not conducted. Instead, a narrative synthesis is provided. RESULTS In total, 3730 studies were screened, and 20 studies were included. The population size varied greatly between studies ranging from 1.8 million patients to 598 triage notes. The most common outcomes assessed were prediction of triage score, prediction of admission, and prediction of critical illness. NLP models achieved high accuracy in predicting need for admission, triage score, critical illness, and mapping free-text chief complaints to structured fields. Incorporating both structured data and free-text data improved results when compared to models that used only structured data. However, the majority of studies (80%) were assessed to have a high risk of bias, and only one study reported the deployment of an NLP model into clinical practice. CONCLUSION Unstructured free-text triage notes have been used by NLP models to predict clinically relevant outcomes. However, the majority of studies have a high risk of bias, most research is retrospective, and there are few examples of implementation into clinical practice. Future work is needed to prospectively assess if applying NLP to data acquired at ED triage improves ED outcomes when compared to usual clinical practice.
Collapse
Affiliation(s)
- Jonathon Stewart
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Juan Lu
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Adrian Goudie
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Glenn Arendts
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Department of Emergency Medicine, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| | - Shiv Akarsh Meka
- HIVE & Data and Digital Innovation, Royal Perth Hospital, Perth, Western Australia, Australia
| | - Sam Freeman
- Department of Emergency Medicine, St Vincent’s Hospital Melbourne, Melbourne, Victoria, Australia
- SensiLab, Monash University, Melbourne, Victoria, Australia
| | - Katie Walker
- School of Clinical Sciences at Monash Health, Monash University, Melbourne, Victoria, Australia
| | - Peter Sprivulis
- Western Australia Department of Health, East Perth, Western Australia, Australia
| | - Frank Sanfilippo
- School of Population and Global Health, University of Western Australia, Crawley, Western Australia, Australia
| | - Mohammed Bennamoun
- Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Western Australia, Australia
| | - Girish Dwivedi
- School of Medicine, The University of Western Australia, Crawley, Western Australia, Australia
- Harry Perkins Institute of Medical Research, Murdoch, Western Australia, Australia
- Department of Cardiology, Fiona Stanley Hospital, Murdoch, Western Australia, Australia
| |
Collapse
|
5
|
Chun SJ, Park JB, Ryu H, Jang BS. Development and benchmarking of a Korean audio speech recognition model for Clinician-Patient conversations in radiation oncology clinics. Int J Med Inform 2023; 176:105112. [PMID: 37276615 DOI: 10.1016/j.ijmedinf.2023.105112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Revised: 05/19/2023] [Accepted: 05/27/2023] [Indexed: 06/07/2023]
Abstract
BACKGROUND The purpose of this study is to develop an audio speech recognition (ASR) deep learning model for transcribing clinician-patient conversations in radiation oncology clinics. METHODS We finetuned the pre-trained English QuartzNet 15x5 model for the Korean language using a publicly available dataset of simulated situations between clinicians and patients. Subsequently, real conversations between a radiation oncologist and 115 patients in actual clinics were then prospectively collected, transcribed, and divided into training (30.26 h) and testing (0.79 h) sets. These datasets were used to develop the ASR model for clinics, which was benchmarked against other ASR models, including the 'Whisper large,' the 'Riva Citrinet-1024 Korean model,' and the 'Riva Conformer Korean model.' RESULTS The pre-trained English ASR model was successfully fine-tuned and converted to recognize the Korean language, resulting in a character error rate (CER) of 0.17. However, we found that this performance was not sustained on the real conversation dataset. To address this, we further fine-tuned the model, resulting in an improved CER of 0.26. Other developed ASR models, including 'Whisper large,' the 'Riva Citrinet-1024 Korean model,' and the 'Riva Conformer Korean model.', showed a CER of 0.31, 0.28, and 0.25, respectively. On the general Korean conversation dataset, 'zeroth-korean,' our model showed a CER of 0.44, while the 'Whisper large,' the 'Riva Citrinet-1024 Korean model,' and the 'Riva Conformer Korean model' resulted in CERs of 0.26, 0.98, and 0.99, respectively. CONCLUSION In conclusion, we developed a Korean ASR model to transcribe real conversations between a radiation oncologist and patients. The performance of the model was deemed acceptable for both specific and general purposes, compared to other models. We anticipate that this model will reduce the time required for clinicians to document the patient's chief complaints or side effects.
Collapse
Affiliation(s)
- Seok-Joo Chun
- Department of Radiation Oncology, Seoul National University Hospital, South Korea; Department of Radiation Oncology, Seoul National University, South Korea
| | - Jung Bin Park
- Department of Radiation Oncology, Seoul National University Hospital, South Korea; Department of Radiation Oncology, Seoul National University, South Korea
| | - Hyejo Ryu
- Department of Radiation Oncology, Seoul National University Hospital, South Korea; Department of Radiation Oncology, Seoul National University, South Korea
| | - Bum-Sup Jang
- Department of Radiation Oncology, Seoul National University Hospital, South Korea; Department of Radiation Oncology, Seoul National University, South Korea.
| |
Collapse
|
6
|
Grabar N, Grouin C. Year 2021: COVID-19, Information Extraction and BERTization among the Hottest Topics in Medical Natural Language Processing. Yearb Med Inform 2022; 31:254-260. [PMID: 36463883 PMCID: PMC9719758 DOI: 10.1055/s-0042-1742547] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/05/2022] Open
Abstract
OBJECTIVES Analyze the content of publications within the medical natural language processing (NLP) domain in 2021. METHODS Automatic and manual preselection of publications to be reviewed, and selection of the best NLP papers of the year. Analysis of the important issues. RESULTS Four best papers have been selected in 2021. We also propose an analysis of the content of the NLP publications in 2021, all topics included. CONCLUSIONS The main issues addressed in 2021 are related to the investigation of COVID-related questions and to the further adaptation and use of transformer models. Besides, the trends from the past years continue, such as information extraction and use of information from social networks.
Collapse
Affiliation(s)
- Natalia Grabar
- STL, CNRS, Université de Lille, Domaine du Pont-de-bois, Villeneuve-d'Ascq cedex, France,Correspondence to: Natalia Grabar STL, CNRS, Université de LilleDomaine du Pont-de-bois, 59653 Villeneuve-d'Ascq cedexFrance
| | - Cyril Grouin
- Université Paris Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, Orsay, France
| |
Collapse
|
7
|
Lee SH, Park J, Yang K, Min J, Choi J. Accuracy of Cloud-Based Speech Recognition Open Application Programming Interface for Medical Terms of Korean. J Korean Med Sci 2022; 37:e144. [PMID: 35535371 PMCID: PMC9091429 DOI: 10.3346/jkms.2022.37.e144] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/20/2022] [Accepted: 04/07/2022] [Indexed: 11/23/2022] Open
Abstract
BACKGROUND There are limited data on the accuracy of cloud-based speech recognition (SR) open application programming interfaces (APIs) for medical terminology. This study aimed to evaluate the medical term recognition accuracy of current available cloud-based SR open APIs in Korean. METHODS We analyzed the SR accuracy of currently available cloud-based SR open APIs using real doctor-patient conversation recordings collected from an outpatient clinic at a large tertiary medical center in Korea. For each original and SR transcription, we analyzed the accuracy rate of each cloud-based SR open API (i.e., the number of medical terms in the SR transcription per number of medical terms in the original transcription). RESULTS A total of 112 doctor-patient conversation recordings were converted with three cloud-based SR open APIs (Naver Clova SR from Naver Corporation; Google Speech-to-Text from Alphabet Inc.; and Amazon Transcribe from Amazon), and each transcription was compared. Naver Clova SR (75.1%) showed the highest accuracy with the recognition of medical terms compared to the other open APIs (Google Speech-to-Text, 50.9%, P < 0.001; Amazon Transcribe, 57.9%, P < 0.001), and Amazon Transcribe demonstrated higher recognition accuracy compared to Google Speech-to-Text (P < 0.001). In the sub-analysis, Naver Clova SR showed the highest accuracy in all areas according to word classes, but the accuracy of words longer than five characters showed no statistical differences (Naver Clova SR, 52.6%; Google Speech-to-Text, 56.3%; Amazon Transcribe, 36.6%). CONCLUSION Among three current cloud-based SR open APIs, Naver Clova SR which manufactured by Korean company showed highest accuracy of medical terms in Korean, compared to Google Speech-to-Text and Amazon Transcribe. Although limitations are existing in the recognition of medical terminology, there is a lot of rooms for improvement of this promising technology by combining strengths of each SR engines.
Collapse
Affiliation(s)
- Seung-Hwa Lee
- Rehabilitation and Prevention Center, Heart Vascular Stroke Institute, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea
- Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea
| | - Jungchan Park
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Korea
| | - Kwangmo Yang
- Center for Health Promotion, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea
| | - Jeongwon Min
- Interdisciplinary Program in Bioengineering, Graduate School, Seoul National University, Seoul, Korea
- Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Korea
| | - Jinwook Choi
- Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea
- Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Korea.
| |
Collapse
|