51
|
Lee JH, Choi E, McDougal R, Lytton WW. GPT-4 Performance for Neurologic Localization. Neurol Clin Pract 2024; 14:e200293. [PMID: 38596779 PMCID: PMC11003355 DOI: 10.1212/cpj.0000000000200293] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2023] [Accepted: 01/23/2024] [Indexed: 04/11/2024]
Abstract
Background and Objectives In health care, large language models such as Generative Pretrained Transformers (GPTs), trained on extensive text datasets, have potential applications in reducing health care disparities across regions and populations. Previous software developed for lesion localization has been limited in scope. This study aims to evaluate the capability of GPT-4 for lesion localization based on clinical presentation. Methods GPT-4 was prompted using history and neurologic physical examination (H&P) from published cases of acute stroke followed by questions for clinical reasoning with answering for "single or multiple lesions," "side," and "brain region" using Zero-Shot Chain-of-Thought and Text Classification prompting. GPT-4 output on 3 separate trials for each of 46 cases was compared with imaging-based localization. Results GPT-4 successfully processed raw text from H&P to generate accurate neuroanatomical localization and detailed clinical reasoning. Performance metrics across trial-based analysis for specificity, sensitivity, precision, and F1-score were 0.87, 0.74, 0.75, and 0.74, respectively, for side; 0.94, 0.85, 0.84, and 0.85, respectively, for brain region. Class labels within the brain region were similarly high for all regions except the cerebellum and were also similar when considering all 3 trials to examine metrics by case. Errors were due to extrinsic causes-inadequate information in the published cases, and intrinsic causes-failures of logic or inadequate knowledge base. Discussion This study reveals capabilities of GPT-4 in the localization of acute stroke lesions, showing a potential future role as a clinical tool in neurology.
Collapse
Affiliation(s)
- Jung-Hyun Lee
- Department of Neurology (J-HL, WWL), State University of New York Downstate Health Sciences University; Department of Neurology (J-HL, WWL), Kings County Hospital; Department of Neurology (J-HL), Maimonides Medical Center, Brooklyn; Department of Internal Medicine (EC), Lincoln Medical Center, Bronx, NY; Department of Biostatistics (RM), Yale School of Public Health; Program in Computational Biology and Bioinformatics (RM); Wu-Tsai Institute (RM); Section of Biomedical Informatics and Data Science (RM), Yale School of Medicine, Yale University, New Haven, CT; and Department of Physiology and Pharmacology (WWL), State University of New York Downstate Health Sciences University, Brooklyn, NY
| | - Eunhee Choi
- Department of Neurology (J-HL, WWL), State University of New York Downstate Health Sciences University; Department of Neurology (J-HL, WWL), Kings County Hospital; Department of Neurology (J-HL), Maimonides Medical Center, Brooklyn; Department of Internal Medicine (EC), Lincoln Medical Center, Bronx, NY; Department of Biostatistics (RM), Yale School of Public Health; Program in Computational Biology and Bioinformatics (RM); Wu-Tsai Institute (RM); Section of Biomedical Informatics and Data Science (RM), Yale School of Medicine, Yale University, New Haven, CT; and Department of Physiology and Pharmacology (WWL), State University of New York Downstate Health Sciences University, Brooklyn, NY
| | - Robert McDougal
- Department of Neurology (J-HL, WWL), State University of New York Downstate Health Sciences University; Department of Neurology (J-HL, WWL), Kings County Hospital; Department of Neurology (J-HL), Maimonides Medical Center, Brooklyn; Department of Internal Medicine (EC), Lincoln Medical Center, Bronx, NY; Department of Biostatistics (RM), Yale School of Public Health; Program in Computational Biology and Bioinformatics (RM); Wu-Tsai Institute (RM); Section of Biomedical Informatics and Data Science (RM), Yale School of Medicine, Yale University, New Haven, CT; and Department of Physiology and Pharmacology (WWL), State University of New York Downstate Health Sciences University, Brooklyn, NY
| | - William W Lytton
- Department of Neurology (J-HL, WWL), State University of New York Downstate Health Sciences University; Department of Neurology (J-HL, WWL), Kings County Hospital; Department of Neurology (J-HL), Maimonides Medical Center, Brooklyn; Department of Internal Medicine (EC), Lincoln Medical Center, Bronx, NY; Department of Biostatistics (RM), Yale School of Public Health; Program in Computational Biology and Bioinformatics (RM); Wu-Tsai Institute (RM); Section of Biomedical Informatics and Data Science (RM), Yale School of Medicine, Yale University, New Haven, CT; and Department of Physiology and Pharmacology (WWL), State University of New York Downstate Health Sciences University, Brooklyn, NY
| |
Collapse
|
52
|
Mat Q, Briganti G, Maniaci A, Lelubre C. Will ChatGPT soon replace otolaryngologists? Eur Arch Otorhinolaryngol 2024; 281:3303-3304. [PMID: 38438614 DOI: 10.1007/s00405-024-08543-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 02/08/2024] [Indexed: 03/06/2024]
Affiliation(s)
- Quentin Mat
- Department of Otorhinolaryngology, C.H.U. Charleroi, Chaussée de Bruxelles 140, 6042, Charleroi, Belgium.
- Faculty of Medicine and Pharmacy, University of Mons (UMons), Mons, Belgium.
| | - Giovanni Briganti
- Faculty of Medicine and Pharmacy, University of Mons (UMons), Mons, Belgium
- Department of Clinical Science, Faculty of Medicine, University of Liège, Quartier Hôpital, Avenue Hippocrate 13, 4000, Liege, Belgium
- Faculty of Medicine, Université Libre de Bruxelles, Route de Lennik 808, 1070, Brussels, Belgium
| | - Antonino Maniaci
- Faculty of Medicine and Surgery, University of Enna "Kore", Enna, Italy
| | - Christophe Lelubre
- Faculty of Medicine and Pharmacy, University of Mons (UMons), Mons, Belgium
- Department of Internal Medicine, C.H.U. Charleroi, Charleroi, Belgium
| |
Collapse
|
53
|
Eltaybani S. The Transformative Role of Large Language Models in Post-Acute and Long-Term Care. J Am Med Dir Assoc 2024; 25:104982. [PMID: 38614135 DOI: 10.1016/j.jamda.2024.03.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2023] [Revised: 02/26/2024] [Accepted: 03/03/2024] [Indexed: 04/15/2024]
Affiliation(s)
- Sameh Eltaybani
- Global Nursing Research Center, The University of Tokyo, Tokyo, Japan.
| |
Collapse
|
54
|
Perez-Lopez R, Ghaffari Laleh N, Mahmood F, Kather JN. A guide to artificial intelligence for cancer researchers. Nat Rev Cancer 2024; 24:427-441. [PMID: 38755439 DOI: 10.1038/s41568-024-00694-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/09/2024] [Indexed: 05/18/2024]
Abstract
Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.
Collapse
Affiliation(s)
- Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology, Vall d'Hebron Barcelona Hospital Campus, Barcelona, Spain
| | - Narmin Ghaffari Laleh
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumour Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
55
|
Hobensack M, von Gerich H, Vyas P, Withall J, Peltonen LM, Block LJ, Davies S, Chan R, Van Bulck L, Cho H, Paquin R, Mitchell J, Topaz M, Song J. A rapid review on current and potential uses of large language models in nursing. Int J Nurs Stud 2024; 154:104753. [PMID: 38560958 DOI: 10.1016/j.ijnurstu.2024.104753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Revised: 03/04/2024] [Accepted: 03/06/2024] [Indexed: 04/04/2024]
Abstract
BACKGROUND The application of large language models across commercial and consumer contexts has grown exponentially in recent years. However, a gap exists in the literature on how large language models can support nursing practice, education, and research. This study aimed to synthesize the existing literature on current and potential uses of large language models across the nursing profession. METHODS A rapid review of the literature, guided by Cochrane rapid review methodology and PRISMA reporting standards, was conducted. An expert health librarian assisted in developing broad inclusion criteria to account for the emerging nature of literature related to large language models. Three electronic databases (i.e., PubMed, CINAHL, and Embase) were searched to identify relevant literature in August 2023. Articles that discussed the development, use, and application of large language models within nursing were included for analysis. RESULTS The literature search identified a total of 2028 articles that met the inclusion criteria. After systematically reviewing abstracts, titles, and full texts, 30 articles were included in the final analysis. Nearly all (93 %; n = 28) of the included articles used ChatGPT as an example, and subsequently discussed the use and value of large language models in nursing education (47 %; n = 14), clinical practice (40 %; n = 12), and research (10 %; n = 3). While the most common assessment of large language models was conducted by human evaluation (26.7 %; n = 8), this analysis also identified common limitations of large language models in nursing, including lack of systematic evaluation, as well as other ethical and legal considerations. DISCUSSION This is the first review to summarize contemporary literature on current and potential uses of large language models in nursing practice, education, and research. Although there are significant opportunities to apply large language models, the use and adoption of these models within nursing have elicited a series of challenges, such as ethical issues related to bias, misuse, and plagiarism. CONCLUSION Given the relative novelty of large language models, ongoing efforts to develop and implement meaningful assessments, evaluations, standards, and guidelines for applying large language models in nursing are recommended to ensure appropriate, accurate, and safe use. Future research along with clinical and educational partnerships is needed to enhance understanding and application of large language models in nursing and healthcare.
Collapse
Affiliation(s)
- Mollie Hobensack
- Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, USA.
| | | | - Pankaj Vyas
- College of Nursing, University of Arizona, Tucson, AZ, USA
| | - Jennifer Withall
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Laura-Maria Peltonen
- Department of Nursing Science, University of Turku, Research Services, Turku University Hospital, Finland
| | - Lorraine J Block
- School of Nursing, University of British Columbia, Vancouver, Canada
| | - Shauna Davies
- Faculty of Nursing, University of Regina, Regina, Canada
| | - Ryan Chan
- Arthur Labatt Family School of Nursing, Western University, London, ON, Canada
| | - Liesbet Van Bulck
- Department of Public Health and Primary Care, KU Leuven - University of Leuven, Leuven, Belgium
| | - Hwayoung Cho
- College of Nursing, University of Florida, Gainesville, FL, USA
| | - Robert Paquin
- Faculty of Nursing, Midwifery, and Palliative Care, King's College London, London, UK
| | - James Mitchell
- Department of Biomedical Informatics, University of Colorado School of Medicine, Denver, CO, USA
| | - Maxim Topaz
- Columbia University School of Nursing, Data Science Institute, Columbia University, VNS Health, New York, NY, USA
| | - Jiyoun Song
- Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, PA, USA
| |
Collapse
|
56
|
Berger-Tal O, Wong BBM, Adams CA, Blumstein DT, Candolin U, Gibson MJ, Greggor AL, Lagisz M, Macura B, Price CJ, Putman BJ, Snijders L, Nakagawa S. Leveraging AI to improve evidence synthesis in conservation. Trends Ecol Evol 2024; 39:548-557. [PMID: 38796352 DOI: 10.1016/j.tree.2024.04.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 04/18/2024] [Accepted: 04/22/2024] [Indexed: 05/28/2024]
Abstract
Systematic evidence syntheses (systematic reviews and maps) summarize knowledge and are used to support decisions and policies in a variety of applied fields, from medicine and public health to biodiversity conservation. However, conducting these exercises in conservation is often expensive and slow, which can impede their use and hamper progress in addressing the current biodiversity crisis. With the explosive growth of large language models (LLMs) and other forms of artificial intelligence (AI), we discuss here the promise and perils associated with their use. We conclude that, when judiciously used, AI has the potential to speed up and hopefully improve the process of evidence synthesis, which can be particularly useful for underfunded applied fields, such as conservation science.
Collapse
Affiliation(s)
- Oded Berger-Tal
- Mitrani Department of Desert Ecology, Jacob Blaustein Institutes for Desert Research, Ben-Gurion University of the Negev, Midreshet Ben-Gurion 8499000, Israel.
| | - Bob B M Wong
- School of Biological Sciences, Monash University, Melbourne, VIC 3800, Australia.
| | - Carrie Ann Adams
- Department of Fish, Wildlife, and Conservation Biology, Colorado State University, 1474 Campus Delivery, Fort Collins, CO 80523-1474, USA
| | - Daniel T Blumstein
- Department of Ecology & Evolutionary Biology, University of California, 621 Young Drive South, Los Angeles, CA 90095-1606, USA
| | - Ulrika Candolin
- Organismal and Evolutionary Biology Research Programme, University of Helsinki, 00014 Helsinki, Finland
| | - Matthew J Gibson
- Evolution & Ecology Research Centre, Centre for Ecosystem Science, and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, NSW 2052, Australia
| | - Alison L Greggor
- Conservation Science and Wildlife Health, San Diego Zoo Wildlife Alliance, 15600 San Pasqual Valley Road, Escondido, CA 92027-7000, USA
| | - Malgorzata Lagisz
- Evolution & Ecology Research Centre, Centre for Ecosystem Science, and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, NSW 2052, Australia
| | - Biljana Macura
- Stockholm Environment Institute (HQ), Box 24218, Stockholm, 10451, Sweden
| | - Catherine J Price
- School of Life and Environmental Sciences, University of Sydney, NSW 2006, Australia
| | - Breanna J Putman
- Department of Biology, California State University, 5500 University Parkway, San Bernardino, CA 92407-2393, USA
| | - Lysanne Snijders
- Behavioural Ecology Group, Wageningen University & Research, De Elst 1, 6708 WD, Wageningen, The Netherlands
| | - Shinichi Nakagawa
- Evolution & Ecology Research Centre, Centre for Ecosystem Science, and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, NSW 2052, Australia.
| |
Collapse
|
57
|
Dennstädt F, Hastings J, Putora PM, Vu E, Fischer G, Süveg K, Glatzer M, Riggenbach E, Hà HL, Cihoric N. In Reply to Daungsupawong and Wiwanitkit. Adv Radiat Oncol 2024; 9:101511. [PMID: 38983697 PMCID: PMC11232351 DOI: 10.1016/j.adro.2024.101511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/11/2024] Open
Affiliation(s)
- Fabio Dennstädt
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| | - Janna Hastings
- School of Medicine, University of St. Gallen, St. Gallen, Switzerland
- Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland
| | - Paul Martin Putora
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| | - Erwin Vu
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Galina Fischer
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Krisztian Süveg
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Markus Glatzer
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Elena Riggenbach
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| | - Hông-Linh Hà
- Department of Radiation Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Nikola Cihoric
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| |
Collapse
|
58
|
Iscoe M, Socrates V, Gilson A, Chi L, Li H, Huang T, Kearns T, Perkins R, Khandjian L, Taylor RA. Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models. Acad Emerg Med 2024; 31:599-610. [PMID: 38567658 DOI: 10.1111/acem.14883] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 01/24/2024] [Accepted: 01/24/2024] [Indexed: 04/04/2024]
Abstract
BACKGROUND Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms. OBJECTIVES This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes. METHODS The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level. RESULTS A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model. CONCLUSIONS The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.
Collapse
Affiliation(s)
- Mark Iscoe
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
| | - Vimig Socrates
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
- Program of Computational Biology and Bioinformatics, Yale University, New Haven, Connecticut, USA
| | - Aidan Gilson
- Yale School of Medicine, New Haven, Connecticut, USA
| | - Ling Chi
- Department of Biostatistics, Yale School of Public Health, New Haven, Connecticut, USA
| | - Huan Li
- Program of Computational Biology and Bioinformatics, Yale University, New Haven, Connecticut, USA
| | - Thomas Huang
- Yale School of Medicine, New Haven, Connecticut, USA
| | - Thomas Kearns
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Rachelle Perkins
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Laura Khandjian
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - R Andrew Taylor
- Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA
- Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA
| |
Collapse
|
59
|
Hajijama S, Juneja D, Nasa P. Large Language Model in Critical Care Medicine: Opportunities and Challenges. Indian J Crit Care Med 2024; 28:523-525. [PMID: 39130386 PMCID: PMC11310681 DOI: 10.5005/jp-journals-10071-24743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/13/2024] Open
Abstract
How to cite this article: Hajijama S, Juneja D, Nasa P. Large Language Model in Critical Care Medicine: Opportunities and Challenges. Indian J Crit Care Med 2024;28(6):523-525.
Collapse
Affiliation(s)
- Sameera Hajijama
- Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
| | - Deven Juneja
- Institute of Critical Care Medicine, Max Super Speciality Hospital, New Delhi, India
| | - Prashant Nasa
- Department of Critical Care Unit, The Royal Wolverhampton Trust, Wolverhampton, United Kingdom
| |
Collapse
|
60
|
Gomez-Cabello CA, Borna S, Pressman SM, Haider SA, Sehgal A, Leibovich BC, Forte AJ. Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery. Healthcare (Basel) 2024; 12:1083. [PMID: 38891158 PMCID: PMC11171524 DOI: 10.3390/healthcare12111083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2024] [Revised: 05/13/2024] [Accepted: 05/23/2024] [Indexed: 06/21/2024] Open
Abstract
Since their release, the medical community has been actively exploring large language models' (LLMs) capabilities, which show promise in providing accurate medical knowledge. One potential application is as a patient resource. This study analyzes and compares the ability of the currently available LLMs, ChatGPT-3.5, GPT-4, and Gemini, to provide postoperative care recommendations to plastic surgery patients. We presented each model with 32 questions addressing common patient concerns after surgical cosmetic procedures and evaluated the medical accuracy, readability, understandability, and actionability of the models' responses. The three LLMs provided equally accurate information, with GPT-3.5 averaging the highest on the Likert scale (LS) (4.18 ± 0.93) (p = 0.849), while Gemini provided significantly more readable (p = 0.001) and understandable responses (p = 0.014; p = 0.001). There was no difference in the actionability of the models' responses (p = 0.830). Although LLMs have shown their potential as adjunctive tools in postoperative patient care, further refinement and research are imperative to enable their evolution into comprehensive standalone resources.
Collapse
Affiliation(s)
| | - Sahar Borna
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | | | - Syed Ali Haider
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Ajai Sehgal
- Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
| | - Bradley C. Leibovich
- Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
- Department of Urology, Mayo Clinic, Rochester, MN 55905, USA
| | - Antonio J. Forte
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
- Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
| |
Collapse
|
61
|
de Zegher I, Norak K, Steiger D, Müller H, Kalra D, Scheenstra B, Cina I, Schulz S, Uma K, Kalendralis P, Lotman EM, Benedikt M, Dumontier M, Celebi R. Artificial intelligence based data curation: enabling a patient-centric European health data space. Front Med (Lausanne) 2024; 11:1365501. [PMID: 38813389 PMCID: PMC11133575 DOI: 10.3389/fmed.2024.1365501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 03/18/2024] [Indexed: 05/31/2024] Open
Abstract
The emerging European Health Data Space (EHDS) Regulation opens new prospects for large-scale sharing and re-use of health data. Yet, the proposed regulation suffers from two important limitations: it is designed to benefit the whole population with limited consideration for individuals, and the generation of secondary datasets from heterogeneous, unlinked patient data will remain burdensome. AIDAVA, a Horizon Europe project that started in September 2022, proposes to address both shortcomings by providing patients with an AI-based virtual assistant that maximises automation in the integration and transformation of their health data into an interoperable, longitudinal health record. This personal record can then be used to inform patient-related decisions at the point of care, whether this is the usual point of care or a possible cross-border point of care. The personal record can also be used to generate population datasets for research and policymaking. The proposed solution will enable a much-needed paradigm shift in health data management, implementing a 'curate once at patient level, use many times' approach, primarily for the benefit of patients and their care providers, but also for more efficient generation of high-quality secondary datasets. After 15 months, the project shows promising preliminary results in achieving automation in the integration and transformation of heterogeneous data of each individual patient, once the content of the data sources managed by the data holders has been formally described. Additionally, the conceptualization phase of the project identified a set of recommendations for the development of a patient-centric EHDS, significantly facilitating the generation of data for secondary use.
Collapse
Affiliation(s)
| | - Kerli Norak
- North Estonia Medical Centre, Tallinn, Estonia
- Department of Health Technologies, Tallinn University of Technology, Tallinn, Estonia
| | | | - Heimo Müller
- Diagnostics and Research Institute of Pathology, Medical University Graz, Graz, Austria
| | - Dipak Kalra
- The European Institute for Innovation Through Health Data, Ghent, Belgium
| | - Bart Scheenstra
- Department of Cardiothoracic Surgery, Cardiovascular Research Institute Maastricht, Maastricht University Medical Centre, Maastricht, Netherlands
| | | | - Stefan Schulz
- Averbis GmbH, Freiburg, Germany
- Institute for Medical Informatics, Statistics and Documentation, Medical University Graz, Graz, Austria
| | - Kanimozhi Uma
- Faculty of Engineering Science, Department of Computer Science (HCI), Leuven, Belgium
| | - Petros Kalendralis
- Department of Radiation Oncology (Maastro), GROW School for Oncology and Reproduction, Maastricht University Medical Centre, Maastricht, Netherlands
| | | | - Martin Benedikt
- Department of Internal Medicine, Division of Cardiology, Medical University of Graz, Graz, Austria
| | - Michel Dumontier
- Department of Advanced Computing Sciences, Institute of Data Science, Maastricht University, Maastricht, Netherlands
| | - Remzi Celebi
- Department of Advanced Computing Sciences, Institute of Data Science, Maastricht University, Maastricht, Netherlands
| |
Collapse
|
62
|
Scott IA, Zuccon G. The new paradigm in machine learning - foundation models, large language models and beyond: a primer for physicians. Intern Med J 2024; 54:705-715. [PMID: 38715436 DOI: 10.1111/imj.16393] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 03/26/2024] [Indexed: 05/18/2024]
Abstract
Foundation machine learning models are deep learning models capable of performing many different tasks using different data modalities such as text, audio, images and video. They represent a major shift from traditional task-specific machine learning prediction models. Large language models (LLM), brought to wide public prominence in the form of ChatGPT, are text-based foundational models that have the potential to transform medicine by enabling automation of a range of tasks, including writing discharge summaries, answering patients questions and assisting in clinical decision-making. However, such models are not without risk and can potentially cause harm if their development, evaluation and use are devoid of proper scrutiny. This narrative review describes the different types of LLM, their emerging applications and potential limitations and bias and likely future translation into clinical practice.
Collapse
Affiliation(s)
- Ian A Scott
- Centre for Health Services Research, University of Queensland, Woolloongabba, Australia
| | - Guido Zuccon
- School of Electrical Engineering and Computer Sciences, The University of Queensland, St Lucia, Queensland, Australia
| |
Collapse
|
63
|
Mondal H, Komarraju S, D S, Muralidharan S. Assessing the Capability of Large Language Models in Naturopathy Consultation. Cureus 2024; 16:e59457. [PMID: 38826991 PMCID: PMC11141616 DOI: 10.7759/cureus.59457] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/30/2024] [Indexed: 06/04/2024] Open
Abstract
Background The rapid advancements in natural language processing have brought about the widespread use of large language models (LLMs) across various medical domains. However, their effectiveness in specialized fields, such as naturopathy, remains relatively unexplored. Objective The study aimed to assess the capability of freely available LLM chatbots in providing naturopathy consultations for various types of diseases and disorders. Methods Five free LLMs (viz., Gemini, Copilot, ChatGPT, Claude, and Perplexity) were used to converse with 20 clinical cases (simulation of real-world scenarios). Each case had the case details and questions pertinent to naturopathy. The responses were presented to three naturopathy doctors with > 5 years of practice. The answers were rated by them on a five-point Likert-like scale for language fluency, coherence, accuracy, and relevancy. The average of these four attributes is termed perfection in his study. Results The overall score of the LLMs were Gemini 3.81±0.23, Copilot 4.34±0.28, ChatGPT 4.43±0.2, Claude 3.8±0.26, and Perplexity 3.91±0.28 (ANOVA F [3.034, 57.64] = 33.47, P <0.0001. Together, they showed overall ~80% perfection in consultation. The average measure intraclass correlation coefficient among the LLMs for the overall score was 0.463 (95% CI = -0.028 to 0.76), P = 0.03. Conclusion Although the LLM chatbots could help in providing naturopathy and yoga treatment consultation with approximately an overall fair level of perfection, their solution to the user varies across different chatbots and there was very low reliability among them.
Collapse
Affiliation(s)
- Himel Mondal
- Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, IND
| | | | - Sathyanath D
- Naturopathy and Yoga, National Institute of Naturopathy, Pune, IND
| | | |
Collapse
|
64
|
Ostrowska M, Kacała P, Onolememen D, Vaughan-Lane K, Sisily Joseph A, Ostrowski A, Pietruszewska W, Banaszewski J, Wróbel MJ. To trust or not to trust: evaluating the reliability and safety of AI responses to laryngeal cancer queries. Eur Arch Otorhinolaryngol 2024:10.1007/s00405-024-08643-8. [PMID: 38652298 DOI: 10.1007/s00405-024-08643-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2024] [Accepted: 03/26/2024] [Indexed: 04/25/2024]
Abstract
PURPOSE As online health information-seeking surges, concerns mount over the quality and safety of accessible content, potentially leading to patient harm through misinformation. On one hand, the emergence of Artificial Intelligence (AI) in healthcare could prevent it; on the other hand, questions raise regarding the quality and safety of the medical information provided. As laryngeal cancer is a prevalent head and neck malignancy, this study aims to evaluate the utility and safety of three large language models (LLMs) as sources of patient information about laryngeal cancer. METHODS A cross-sectional study was conducted using three LLMs (ChatGPT 3.5, ChatGPT 4.0, and Bard). A questionnaire comprising 36 inquiries about laryngeal cancer was categorised into diagnosis (11 questions), treatment (9 questions), novelties and upcoming treatments (4 questions), controversies (8 questions), and sources of information (4 questions). The population of reviewers consisted of 3 groups, including ENT specialists, junior physicians, and non-medicals, who graded the responses. Each physician evaluated each question twice for each model, while non-medicals only once. Everyone was blinded to the model type, and the question order was shuffled. Outcome evaluations were based on a safety score (1-3) and a Global Quality Score (GQS, 1-5). Results were compared between LLMs. The study included iterative assessments and statistical validations. RESULTS Analysis revealed that ChatGPT 3.5 scored highest in both safety (mean: 2.70) and GQS (mean: 3.95). ChatGPT 4.0 and Bard had lower safety scores of 2.56 and 2.42, respectively, with corresponding quality scores of 3.65 and 3.38. Inter-rater reliability was consistent, with less than 3% discrepancy. About 4.2% of responses fell into the lowest safety category (1), particularly in the novelty category. Non-medical reviewers' quality assessments correlated moderately (r = 0.67) with response length. CONCLUSIONS LLMs can be valuable resources for patients seeking information on laryngeal cancer. ChatGPT 3.5 provided the most reliable and safe responses among the models evaluated.
Collapse
Affiliation(s)
- Magdalena Ostrowska
- Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| | - Paulina Kacała
- ENT Scientific Club, Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| | - Deborah Onolememen
- ENT Scientific Club, Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| | - Katie Vaughan-Lane
- ENT Scientific Club, Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland.
| | - Anitta Sisily Joseph
- ENT Scientific Club, Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| | - Adam Ostrowski
- Department of Urology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| | - Wioletta Pietruszewska
- Department of Otolaryngology, Laryngological Oncology, Audiology and Phoniatrics, Medical University of Lodz, ul Żeromskiego 113, 90-549, Lodz, Poland
| | - Jacek Banaszewski
- Department of Otolaryngology, Head and Neck Oncology, Poznan University of Medical Science, ul Przybyszewskiego 49, 60-355, Poznań, Poland
| | - Maciej J Wróbel
- Department of Otolaryngology and Laryngological Oncology, Collegium Medicum, Nicolaus Copernicus University in Torun, ul.Marie Sklodowskiej-Curie 9, 85-094, Bydgoszcz, Poland
| |
Collapse
|
65
|
Gilbert S, Kather JN, Hogan A. Augmented non-hallucinating large language models as medical information curators. NPJ Digit Med 2024; 7:100. [PMID: 38654142 DOI: 10.1038/s41746-024-01081-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Accepted: 03/14/2024] [Indexed: 04/25/2024] Open
Affiliation(s)
- Stephen Gilbert
- Else Kröner Fresenius Center for Digital Health, TUD Dresden University of Technology, Dresden, Germany.
| | - Jakob Nikolas Kather
- Else Kröner Fresenius Center for Digital Health, TUD Dresden University of Technology, Dresden, Germany
| | - Aidan Hogan
- Department of Computer Science, Universidad de Chile, Santiago, Chile
- Millennium Institute for Foundational Research on Data, DCC, Universidad de Chile, Santiago, Chile
| |
Collapse
|
66
|
Lang S, Vitale J, Fekete TF, Haschtmann D, Reitmeir R, Ropelato M, Puhakka J, Galbusera F, Loibl M. Are large language models valid tools for patient information on lumbar disc herniation? The spine surgeons' perspective. BRAIN & SPINE 2024; 4:102804. [PMID: 38706800 PMCID: PMC11067000 DOI: 10.1016/j.bas.2024.102804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/27/2023] [Revised: 02/19/2024] [Accepted: 04/04/2024] [Indexed: 05/07/2024]
Abstract
Introduction Generative AI is revolutionizing patient education in healthcare, particularly through chatbots that offer personalized, clear medical information. Reliability and accuracy are vital in AI-driven patient education. Research question How effective are Large Language Models (LLM), such as ChatGPT and Google Bard, in delivering accurate and understandable patient education on lumbar disc herniation? Material and methods Ten Frequently Asked Questions about lumbar disc herniation were selected from 133 questions and were submitted to three LLMs. Six experienced spine surgeons rated the responses on a scale from "excellent" to "unsatisfactory," and evaluated the answers for exhaustiveness, clarity, empathy, and length. Statistical analysis involved Fleiss Kappa, Chi-square, and Friedman tests. Results Out of the responses, 27.2% were excellent, 43.9% satisfactory with minimal clarification, 18.3% satisfactory with moderate clarification, and 10.6% unsatisfactory. There were no significant differences in overall ratings among the LLMs (p = 0.90); however, inter-rater reliability was not achieved, and large differences among raters were detected in the distribution of answer frequencies. Overall, ratings varied among the 10 answers (p = 0.043). The average ratings for exhaustiveness, clarity, empathy, and length were above 3.5/5. Discussion and conclusion LLMs show potential in patient education for lumbar spine surgery, with generally positive feedback from evaluators. The new EU AI Act, enforcing strict regulation on AI systems, highlights the need for rigorous oversight in medical contexts. In the current study, the variability in evaluations and occasional inaccuracies underline the need for continuous improvement. Future research should involve more advanced models to enhance patient-physician communication.
Collapse
Affiliation(s)
- Siegmund Lang
- Department of Trauma Surgery, University Hospital Regensburg, Regensburg, Germany
| | - Jacopo Vitale
- Spine Center, Schulthess Klinik, Zurich, Switzerland
| | | | | | | | | | - Jani Puhakka
- Spine Center, Schulthess Klinik, Zurich, Switzerland
| | | | - Markus Loibl
- Spine Center, Schulthess Klinik, Zurich, Switzerland
| |
Collapse
|
67
|
Cerasa A, Crowe B. Generative artificial intelligence in neurology: Opportunities and risks. Eur J Neurol 2024; 31:e16232. [PMID: 38328960 PMCID: PMC11235660 DOI: 10.1111/ene.16232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 01/26/2024] [Indexed: 02/09/2024]
Affiliation(s)
- Antonio Cerasa
- Institute for Biomedical Research and InnovationNational Research Council of ItalyMessinaItaly
- S'Anna InstituteCrotoneItaly
- Pharmacotechnology Documentation and Transfer Unit, Preclinical and Translational Pharmacology, Department of Pharmacy, Health Science, and NutritionUniversity of CalabriaRendeItaly
| | - Byron Crowe
- Department of MedicineBeth Israel Deaconess Medical CenterBostonMassachusettsUSA
- Harvard Medical SchoolBostonMassachusettsUSA
| |
Collapse
|
68
|
Baumgart A, Beck G, Ghezel-Ahmadi D. [Artificial intelligence in intensive care medicine]. Med Klin Intensivmed Notfmed 2024; 119:189-198. [PMID: 38546864 DOI: 10.1007/s00063-024-01117-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Revised: 01/29/2024] [Accepted: 02/05/2024] [Indexed: 04/05/2024]
Abstract
The integration of artificial intelligence (AI) into intensive care medicine has made considerable progress in recent studies, particularly in the areas of predictive analytics, early detection of complications, and the development of decision support systems. The main challenges remain availability and quality of data, reduction of bias and the need for explainable results from algorithms and models. Methods to explain these systems are essential to increase trust, understanding, and ethical considerations among healthcare professionals and patients. Proper training of healthcare professionals in AI principles, terminology, ethical considerations, and practical application is crucial for the successful use of AI. Careful assessment of the impact of AI on patient autonomy and data protection is essential for its responsible use in intensive care medicine. A balance between ethical and practical considerations must be maintained to ensure patient-centered care while complying with data protection regulations. Synergistic collaboration between clinicians, AI engineers, and regulators is critical to realizing the full potential of AI in intensive care medicine and maximizing its positive impact on patient care. Future research and development efforts should focus on improving AI models for real-time predictions, increasing the accuracy and utility of AI-based closed-loop systems, and overcoming ethical, technical, and regulatory challenges, especially in generative AI systems.
Collapse
Affiliation(s)
- André Baumgart
- Zentrum für Präventivmedizin und Digitale Gesundheit, Medizinische Fakultät Mannheim der Universität Heidelberg, Theodor-Kutzer-Ufer 1-3, 68167, Mannheim, Deutschland.
| | - Grietje Beck
- Abteilung für Anästhesiologie, Intensivmedizin und Schmerzmedizin, Universitätsmedizin Mannheim gGmbH, Medizinische Fakultät Mannheim der Universität Heidelberg, Mannheim, Deutschland
| | - David Ghezel-Ahmadi
- Abteilung für Anästhesiologie, Intensivmedizin und Schmerzmedizin, Universitätsmedizin Mannheim gGmbH, Medizinische Fakultät Mannheim der Universität Heidelberg, Mannheim, Deutschland
| |
Collapse
|
69
|
Tan RES, Teo WZW, Puhaindran ME. Artificial Intelligence in Hand Surgery - How Generative AI is Transforming the Hand Surgery Landscape. J Hand Surg Asian Pac Vol 2024; 29:81-87. [PMID: 38553849 DOI: 10.1142/s2424835524300019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Artificial intelligence (AI) has witnessed significant advancements, reshaping various industries, including healthcare. The introduction of ChatGPT by OpenAI in November 2022 marked a pivotal moment, showcasing the potential of generative AI in revolutionising patient care, diagnosis and treatment. Generative AI, unlike traditional AI systems, possesses the ability to generate new content by understanding patterns within datasets. This article explores the evolution of AI in healthcare, tracing its roots to the term coined by John McCarthy in 1955 and the contributions of pioneers like John Von Neumann and Alan Turing. Currently, generative AI, particularly Large Language Models, holds promise across three broad categories in healthcare: patient care, education and research. In patient care, it offers solutions in clinical document management, diagnostic support and operative planning. Notable advancements include Microsoft's collaboration with Epic for integrating AI into electronic medical records (EMRs), enhancing clinical data management and patient care. Furthermore, generative AI aids in surgical decision-making, as demonstrated in plastic, orthopaedic and hepatobiliary surgeries. However, challenges such as bias, hallucination and integration with EMR systems necessitate caution and ongoing evaluation. The article also presents insights from the implementation of NUHS Russell-GPT, a generative AI chatbot, in a hand surgery department, showcasing its utility in administrative tasks but highlighting challenges in surgical planning and EMR integration. The survey showed unanimous support for incorporating AI into clinical settings, with all respondents being open to its use. In conclusion, generative AI is poised to enhance patient care and ease physician workloads, starting with automating administrative tasks and evolving to inform diagnoses, tailored treatment plans, as well as aid in surgical planning. As healthcare systems navigate the complexities of integrating AI, the potential benefits for both physicians and patients remain significant, offering a glimpse into a future where AI transforms healthcare delivery. Level of Evidence: Level V (Diagnostic).
Collapse
Affiliation(s)
- Ruth En Si Tan
- Department of Hand and Reconstructive Microsurgery, National University Hospital, Singapore
| | - Wendy Zi Wei Teo
- Department of Hand and Reconstructive Microsurgery, National University Hospital, Singapore
| | - Mark E Puhaindran
- Department of Hand and Reconstructive Microsurgery, National University Hospital, Singapore
| |
Collapse
|
70
|
Caglayan A, Slusarczyk W, Rabbani RD, Ghose A, Papadopoulos V, Boussios S. Large Language Models in Oncology: Revolution or Cause for Concern? Curr Oncol 2024; 31:1817-1830. [PMID: 38668040 PMCID: PMC11049602 DOI: 10.3390/curroncol31040137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 03/13/2024] [Accepted: 03/29/2024] [Indexed: 04/28/2024] Open
Abstract
The technological capability of artificial intelligence (AI) continues to advance with great strength. Recently, the release of large language models has taken the world by storm with concurrent excitement and concern. As a consequence of their impressive ability and versatility, their provide a potential opportunity for implementation in oncology. Areas of possible application include supporting clinical decision making, education, and contributing to cancer research. Despite the promises that these novel systems can offer, several limitations and barriers challenge their implementation. It is imperative that concerns, such as accountability, data inaccuracy, and data protection, are addressed prior to their integration in oncology. As the progression of artificial intelligence systems continues, new ethical and practical dilemmas will also be approached; thus, the evaluation of these limitations and concerns will be dynamic in nature. This review offers a comprehensive overview of the potential application of large language models in oncology, as well as concerns surrounding their implementation in cancer care.
Collapse
Affiliation(s)
- Aydin Caglayan
- Department of Medical Oncology, Medway NHS Foundation Trust, Gillingham ME7 5NY, UK; (A.C.); (R.D.R.); (A.G.)
| | | | - Rukhshana Dina Rabbani
- Department of Medical Oncology, Medway NHS Foundation Trust, Gillingham ME7 5NY, UK; (A.C.); (R.D.R.); (A.G.)
| | - Aruni Ghose
- Department of Medical Oncology, Medway NHS Foundation Trust, Gillingham ME7 5NY, UK; (A.C.); (R.D.R.); (A.G.)
- Department of Medical Oncology, Barts Cancer Centre, St Bartholomew’s Hospital, Barts Heath NHS Trust, London EC1A 7BE, UK
- Department of Medical Oncology, Mount Vernon Cancer Centre, East and North Hertfordshire Trust, London HA6 2RN, UK
- Health Systems and Treatment Optimisation Network, European Cancer Organisation, 1040 Brussels, Belgium
- Oncology Council, Royal Society of Medicine, London W1G 0AE, UK
| | | | - Stergios Boussios
- Department of Medical Oncology, Medway NHS Foundation Trust, Gillingham ME7 5NY, UK; (A.C.); (R.D.R.); (A.G.)
- Kent Medway Medical School, University of Kent, Canterbury CT2 7LX, UK;
- Faculty of Life Sciences & Medicine, School of Cancer & Pharmaceutical Sciences, King’s College London, Strand Campus, London WC2R 2LS, UK
- Faculty of Medicine, Health, and Social Care, Canterbury Christ Church University, Canterbury CT2 7PB, UK
- AELIA Organization, 9th Km Thessaloniki—Thermi, 57001 Thessaloniki, Greece
| |
Collapse
|
71
|
Artsi Y, Sorin V, Konen E, Glicksberg BS, Nadkarni G, Klang E. Large language models for generating medical examinations: systematic review. BMC MEDICAL EDUCATION 2024; 24:354. [PMID: 38553693 PMCID: PMC10981304 DOI: 10.1186/s12909-024-05239-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Accepted: 02/28/2024] [Indexed: 04/01/2024]
Abstract
BACKGROUND Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs. METHODS The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool. RESULTS Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify. CONCLUSIONS LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
Collapse
Affiliation(s)
- Yaara Artsi
- Azrieli Faculty of Medicine, Bar-Ilan University, Ha'Hadas St. 1, Rishon Le Zion, Zefat, 7550598, Israel.
| | - Vera Sorin
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Ramat Gan, Israel
- Tel-Aviv University School of Medicine, Tel Aviv, Israel
- DeepVision Lab, Chaim Sheba Medical Center, Ramat Gan, Israel
| | - Eli Konen
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Ramat Gan, Israel
- Tel-Aviv University School of Medicine, Tel Aviv, Israel
| | - Benjamin S Glicksberg
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Girish Nadkarni
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Eyal Klang
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
72
|
Truhn D, Eckardt JN, Ferber D, Kather JN. Large language models and multimodal foundation models for precision oncology. NPJ Precis Oncol 2024; 8:72. [PMID: 38519519 PMCID: PMC10959931 DOI: 10.1038/s41698-024-00573-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 03/12/2024] [Indexed: 03/25/2024] Open
Abstract
The technological progress in artificial intelligence (AI) has massively accelerated since 2022, with far-reaching implications for oncology and cancer research. Large language models (LLMs) now perform at human-level competency in text processing. Notably, both text and image processing networks are increasingly based on transformer neural networks. This convergence enables the development of multimodal AI models that take diverse types of data as an input simultaneously, marking a qualitative shift from specialized niche models which were prevalent in the 2010s. This editorial summarizes these developments, which are expected to impact precision oncology in the coming years.
Collapse
Affiliation(s)
- Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Jan-Niklas Eckardt
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Dyke Ferber
- National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany
- Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - Jakob Nikolas Kather
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany.
- Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany.
| |
Collapse
|
73
|
Das A, Tariq A, Batalini F, Dhara B, Banerjee I. Exposing Vulnerabilities in Clinical LLMs Through Data Poisoning Attacks: Case Study in Breast Cancer. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.03.20.24304627. [PMID: 38562849 PMCID: PMC10984073 DOI: 10.1101/2024.03.20.24304627] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Training Large Language Models (LLMs) with billions of parameters on a dataset and publishing the model for public access is the standard practice currently. Despite their transformative impact on natural language processing, public LLMs present notable vulnerabilities given the source of training data is often web-based or crowdsourced, and hence can be manipulated by perpetrators. We delve into the vulnerabilities of clinical LLMs, particularly BioGPT which is trained on publicly available biomedical literature and clinical notes from MIMIC-III, in the realm of data poisoning attacks. Exploring susceptibility to data poisoning-based attacks on de-identified breast cancer clinical notes, our approach is the first one to assess the extent of such attacks and our findings reveal successful manipulation of LLM outputs. Through this work, we emphasize on the urgency of comprehending these vulnerabilities in LLMs, and encourage the mindful and responsible usage of LLMs in the clinical domain.
Collapse
Affiliation(s)
- Avisha Das
- Arizona Advanced AI & Innovation (A3I) Hub, Mayo Clinic Arizona
| | - Amara Tariq
- Arizona Advanced AI & Innovation (A3I) Hub, Mayo Clinic Arizona
| | | | | | - Imon Banerjee
- Arizona Advanced AI & Innovation (A3I) Hub, Mayo Clinic Arizona
- Department of Radiology, Mayo Clinic Arizona
- School of Computing and Augmented Intelligence, Arizona State University
| |
Collapse
|
74
|
Reddy S. Generative AI in healthcare: an implementation science informed translational path on application, integration and governance. Implement Sci 2024; 19:27. [PMID: 38491544 PMCID: PMC10941464 DOI: 10.1186/s13012-024-01357-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Accepted: 03/06/2024] [Indexed: 03/18/2024] Open
Abstract
BACKGROUND Artificial intelligence (AI), particularly generative AI, has emerged as a transformative tool in healthcare, with the potential to revolutionize clinical decision-making and improve health outcomes. Generative AI, capable of generating new data such as text and images, holds promise in enhancing patient care, revolutionizing disease diagnosis and expanding treatment options. However, the utility and impact of generative AI in healthcare remain poorly understood, with concerns around ethical and medico-legal implications, integration into healthcare service delivery and workforce utilisation. Also, there is not a clear pathway to implement and integrate generative AI in healthcare delivery. METHODS This article aims to provide a comprehensive overview of the use of generative AI in healthcare, focusing on the utility of the technology in healthcare and its translational application highlighting the need for careful planning, execution and management of expectations in adopting generative AI in clinical medicine. Key considerations include factors such as data privacy, security and the irreplaceable role of clinicians' expertise. Frameworks like the technology acceptance model (TAM) and the Non-Adoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) model are considered to promote responsible integration. These frameworks allow anticipating and proactively addressing barriers to adoption, facilitating stakeholder participation and responsibly transitioning care systems to harness generative AI's potential. RESULTS Generative AI has the potential to transform healthcare through automated systems, enhanced clinical decision-making and democratization of expertise with diagnostic support tools providing timely, personalized suggestions. Generative AI applications across billing, diagnosis, treatment and research can also make healthcare delivery more efficient, equitable and effective. However, integration of generative AI necessitates meticulous change management and risk mitigation strategies. Technological capabilities alone cannot shift complex care ecosystems overnight; rather, structured adoption programs grounded in implementation science are imperative. CONCLUSIONS It is strongly argued in this article that generative AI can usher in tremendous healthcare progress, if introduced responsibly. Strategic adoption based on implementation science, incremental deployment and balanced messaging around opportunities versus limitations helps promote safe, ethical generative AI integration. Extensive real-world piloting and iteration aligned to clinical priorities should drive development. With conscientious governance centred on human wellbeing over technological novelty, generative AI can enhance accessibility, affordability and quality of care. As these models continue advancing rapidly, ongoing reassessment and transparent communication around their strengths and weaknesses remain vital to restoring trust, realizing positive potential and, most importantly, improving patient outcomes.
Collapse
Affiliation(s)
- Sandeep Reddy
- Deakin School of Medicine, Waurn Ponds, Geelong, VIC, 3215, Australia.
| |
Collapse
|
75
|
Xu X, Chen Y, Miao J. Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review. JOURNAL OF EDUCATIONAL EVALUATION FOR HEALTH PROFESSIONS 2024; 21:6. [PMID: 38486402 PMCID: PMC11035906 DOI: 10.3352/jeehp.2024.21.6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/12/2024] [Accepted: 03/05/2024] [Indexed: 03/19/2024]
Abstract
BACKGROUND ChatGPT is a large language model (LLM) based on artificial intelligence (AI) capable of responding in multiple languages and generating nuanced and highly complex responses. While ChatGPT holds promising applications in medical education, its limitations and potential risks cannot be ignored. METHODS A scoping review was conducted for English articles discussing ChatGPT in the context of medical education published after 2022. A literature search was performed using PubMed/MEDLINE, Embase, and Web of Science databases, and information was extracted from the relevant studies that were ultimately included. RESULTS ChatGPT exhibits various potential applications in medical education, such as providing personalized learning plans and materials, creating clinical practice simulation scenarios, and assisting in writing articles. However, challenges associated with academic integrity, data accuracy, and potential harm to learning were also highlighted in the literature. The paper emphasizes certain recommendations for using ChatGPT, including the establishment of guidelines. Based on the review, 3 key research areas were proposed: cultivating the ability of medical students to use ChatGPT correctly, integrating ChatGPT into teaching activities and processes, and proposing standards for the use of AI by medical students. CONCLUSION ChatGPT has the potential to transform medical education, but careful consideration is required for its full integration. To harness the full potential of ChatGPT in medical education, attention should not only be given to the capabilities of AI but also to its impact on students and teachers.
Collapse
Affiliation(s)
- Xiaojun Xu
- Division of Hematology/Oncology, Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Centre for Child Health, Zhejiang, China
| | - Yixiao Chen
- Division of Hematology/Oncology, Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Centre for Child Health, Zhejiang, China
| | - Jing Miao
- Division of Hematology/Oncology, Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Centre for Child Health, Zhejiang, China
| |
Collapse
|
76
|
Cirone K, Akrout M, Abid L, Oakley A. Assessing the Utility of Multimodal Large Language Models (GPT-4 Vision and Large Language and Vision Assistant) in Identifying Melanoma Across Different Skin Tones. JMIR DERMATOLOGY 2024; 7:e55508. [PMID: 38477960 DOI: 10.2196/55508] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 02/16/2024] [Accepted: 03/01/2024] [Indexed: 03/14/2024] Open
Abstract
The large language models GPT-4 Vision and Large Language and Vision Assistant are capable of understanding and accurately differentiating between benign lesions and melanoma, indicating potential incorporation into dermatologic care, medical research, and education.
Collapse
Affiliation(s)
- Katrina Cirone
- Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
- AIPLabs, Budapest, Hungary
| | - Mohamed Akrout
- AIPLabs, Budapest, Hungary
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
| | | | - Amanda Oakley
- Department of Dermatology, Health New Zealand Te Whatu Ora Waikato, Hamilton, New Zealand
- Department of Medicine, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, New Zealand
| |
Collapse
|
77
|
Raghu Subramanian C, Yang DA, Khanna R. Enhancing Health Care Communication With Large Language Models-The Role, Challenges, and Future Directions. JAMA Netw Open 2024; 7:e240347. [PMID: 38466311 DOI: 10.1001/jamanetworkopen.2024.0347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/12/2024] Open
Affiliation(s)
| | | | - Raman Khanna
- Division of Clinical Informatics and Digital Transformation, University of California, San Francisco
| |
Collapse
|
78
|
Coombs A, Ong S. Four change-makers seek impact in medical research. Nature 2024; 627:S8-S10. [PMID: 38480969 DOI: 10.1038/d41586-024-00754-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/17/2024]
|
79
|
Hess BJ, Cupido N, Ross S, Kvern B. Becoming adaptive experts in an era of rapid advances in generative artificial intelligence. MEDICAL TEACHER 2024; 46:300-303. [PMID: 38092006 DOI: 10.1080/0142159x.2023.2289844] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Accepted: 11/28/2023] [Indexed: 02/24/2024]
Affiliation(s)
- Brian J Hess
- College of Family Physicians of Canada, Department of Certification and Assessment, Mississauga, Ontario, Canada
| | - Nathan Cupido
- The Wilson Centre, University Health Network and Temerty Faculty of Medicine, and the Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | - Shelley Ross
- Department of Family Medicine, Faculty of Medicine and Dentistry, College of Health Sciences, University of Alberta, Edmonton, Canada
| | - Brent Kvern
- College of Family Physicians of Canada, Department of Certification and Assessment, Mississauga, Ontario, Canada
| |
Collapse
|
80
|
Denecke K, May R, Rivera-Romero O. Transformer Models in Healthcare: A Survey and Thematic Analysis of Potentials, Shortcomings and Risks. J Med Syst 2024; 48:23. [PMID: 38367119 PMCID: PMC10874304 DOI: 10.1007/s10916-024-02043-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 02/10/2024] [Indexed: 02/19/2024]
Abstract
Large Language Models (LLMs) such as General Pretrained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), which use transformer model architectures, have significantly advanced artificial intelligence and natural language processing. Recognized for their ability to capture associative relationships between words based on shared context, these models are poised to transform healthcare by improving diagnostic accuracy, tailoring treatment plans, and predicting patient outcomes. However, there are multiple risks and potentially unintended consequences associated with their use in healthcare applications. This study, conducted with 28 participants using a qualitative approach, explores the benefits, shortcomings, and risks of using transformer models in healthcare. It analyses responses to seven open-ended questions using a simplified thematic analysis. Our research reveals seven benefits, including improved operational efficiency, optimized processes and refined clinical documentation. Despite these benefits, there are significant concerns about the introduction of bias, auditability issues and privacy risks. Challenges include the need for specialized expertise, the emergence of ethical dilemmas and the potential reduction in the human element of patient care. For the medical profession, risks include the impact on employment, changes in the patient-doctor dynamic, and the need for extensive training in both system operation and data interpretation.
Collapse
Affiliation(s)
- Kerstin Denecke
- Institute Patient-centered Digital Health, Bern University of Applied Sciences, Quellgasse 21, Biel, 2502, Switzerland.
| | - Richard May
- Harz University of Applied Sciences, Friedrichstraße 57-59, 38855, Wernigerode, Germany
| | - Octavio Rivera-Romero
- Instituto de Ingeniería Informática (I3US), Universidad de Sevilla, Sevilla, Spain
- Department of Electronic Technology, Universidad de Sevilla, Avda Reina Mercedes s/n, ETSI Informática, G1.43, Sevilla, 41012, Spain
| |
Collapse
|
81
|
Benítez TM, Xu Y, Boudreau JD, Kow AWC, Bello F, Van Phuoc L, Wang X, Sun X, Leung GKK, Lan Y, Wang Y, Cheng D, Tham YC, Wong TY, Chung KC. Harnessing the potential of large language models in medical education: promise and pitfalls. J Am Med Inform Assoc 2024; 31:776-783. [PMID: 38269644 PMCID: PMC10873781 DOI: 10.1093/jamia/ocad252] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Revised: 12/09/2023] [Accepted: 12/17/2023] [Indexed: 01/26/2024] Open
Abstract
OBJECTIVES To provide balanced consideration of the opportunities and challenges associated with integrating Large Language Models (LLMs) throughout the medical school continuum. PROCESS Narrative review of published literature contextualized by current reports of LLM application in medical education. CONCLUSIONS LLMs like OpenAI's ChatGPT can potentially revolutionize traditional teaching methodologies. LLMs offer several potential advantages to students, including direct access to vast information, facilitation of personalized learning experiences, and enhancement of clinical skills development. For faculty and instructors, LLMs can facilitate innovative approaches to teaching complex medical concepts and fostering student engagement. Notable challenges of LLMs integration include the risk of fostering academic misconduct, inadvertent overreliance on AI, potential dilution of critical thinking skills, concerns regarding the accuracy and reliability of LLM-generated content, and the possible implications on teaching staff.
Collapse
Affiliation(s)
- Trista M Benítez
- Department of Surgery, University of Michigan Medical School, Ann Arbor, MI 48109, United States
| | - Yueyuan Xu
- Tsinghua Medicine, Tsinghua University, Beijing, 100084, China
| | - J Donald Boudreau
- Institute of Health Sciences Education, Faculty of Medicine and Health Sciences, McGill University, Montreal, QC H3A 0G4, Canada
| | - Alfred Wei Chieh Kow
- Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, 117597, Singapore
| | - Fernando Bello
- Technology Enhanced Learning and Innovation Department, Duke-NUS Medical School, National University of Singapore, 169857, Singapore
| | - Le Van Phuoc
- College of Health Sciences, VinUniversity, Hanoi, 100000, Vietnam
| | - Xiaofei Wang
- Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
| | - Xiaodong Sun
- Department of Ophthalmology, Shanghai General Hospital, School of Medicine, Shanghai JiaoTong University, Shanghai, 200240, China
| | - Gilberto Ka-Kit Leung
- Department of Surgery, School of Clinical Medicine, LKS Faculty of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong, 999077, China
| | - Yanyan Lan
- Institute of AI Industrial Research, Tsinghua University, Beijing, 100084, China
| | - Yaxing Wang
- Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Beijing Ophthalmology and Visual Sciences Key Laboratory, Capital University of Medical Science, Beijing, 100730, China
| | - Davy Cheng
- School of Medicine, The Chinese University of Hong Kong (Shenzhen), Shenzhen, 518172, China
| | - Yih-Chung Tham
- Centre for Innovation and Precision Eye Health; and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, 117597, Singapore
- Ophthalmology and Visual Sciences Academic Clinical Program, Duke-NUS Medical School, 169857, Singapore
- Singapore Eye Research Institute, Singapore National Eye Centre, 168751, Singapore
| | - Tien Yin Wong
- Tsinghua Medicine, Tsinghua University, Beijing, 100084, China
- Singapore Eye Research Institute, Singapore National Eye Centre, 168751, Singapore
- School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Beijing, 100084, China
| | - Kevin C Chung
- Department of Surgery, University of Michigan Medical School, Ann Arbor, MI 48109, United States
| |
Collapse
|
82
|
Li C, Mowery DL, Ma X, Yang R, Vurgun U, Hwang S, Donnelly HK, Bandhey H, Akhtar Z, Senathirajah Y, Sadhu EM, Getzen E, Freda PJ, Long Q, Becich MJ. Realizing the Potential of Social Determinants Data: A Scoping Review of Approaches for Screening, Linkage, Extraction, Analysis and Interventions. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.02.04.24302242. [PMID: 38370703 PMCID: PMC10871446 DOI: 10.1101/2024.02.04.24302242] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Background Social determinants of health (SDoH) like socioeconomics and neighborhoods strongly influence outcomes, yet standardized SDoH data is lacking in electronic health records (EHR), limiting research and care quality. Methods We searched PubMed using keywords "SDOH" and "EHR", underwent title/abstract and full-text screening. Included records were analyzed under five domains: 1) SDoH screening and assessment approaches, 2) SDoH data collection and documentation, 3) Use of natural language processing (NLP) for extracting SDoH, 4) SDoH data and health outcomes, and 5) SDoH-driven interventions. Results We identified 685 articles, of which 324 underwent full review. Key findings include tailored screening instruments implemented across settings, census and claims data linkage providing contextual SDoH profiles, rule-based and neural network systems extracting SDoH from notes using NLP, connections found between SDoH data and healthcare utilization/chronic disease control, and integrated care management programs executed. However, considerable variability persists across data sources, tools, and outcomes. Discussion Despite progress identifying patient social needs, further development of standards, predictive models, and coordinated interventions is critical to fulfill the potential of SDoH-EHR integration. Additional database searches could strengthen this scoping review. Ultimately widespread capture, analysis, and translation of multidimensional SDoH data into clinical care is essential for promoting health equity.
Collapse
Affiliation(s)
- Chenyu Li
- University of Pittsburgh School of Medicine Department of Biomedical Informatics
| | - Danielle L. Mowery
- University of Pennsylvania, Institute for Biomedical Informatics
- University of Pennsylvania, Department of Biostatistics, Epidemiology and Informatics
| | - Xiaomeng Ma
- University of Toronto, Institute of Health Policy Management and Evaluations
| | - Rui Yang
- Duke-NUS Medical School, Centre for Quantitative Medicine
| | - Ugurcan Vurgun
- University of Pennsylvania, Institute for Biomedical Informatics
| | - Sy Hwang
- University of Pennsylvania, Institute for Biomedical Informatics
| | | | - Harsh Bandhey
- Cedars-Sinai Medical Center, Department of Computational Biomedicine
| | - Zohaib Akhtar
- Northwestern University, Kellogg School of Management
| | - Yalini Senathirajah
- University of Pittsburgh School of Medicine Department of Biomedical Informatics
| | - Eugene Mathew Sadhu
- University of Pittsburgh School of Medicine Department of Biomedical Informatics
| | - Emily Getzen
- University of Pennsylvania, Department of Biostatistics, Epidemiology and Informatics
| | - Philip J Freda
- Cedars-Sinai Medical Center, Department of Computational Biomedicine
| | - Qi Long
- University of Pennsylvania, Institute for Biomedical Informatics
- University of Pennsylvania, Department of Biostatistics, Epidemiology and Informatics
| | - Michael J. Becich
- University of Pittsburgh School of Medicine Department of Biomedical Informatics
| |
Collapse
|
83
|
Giuffrè M, Shung DL. Scrutinizing ChatGPT Applications in Gastroenterology: A Call for Methodological Rigor to Define Accuracy and Preserve Privacy. Clin Gastroenterol Hepatol 2024:S1542-3565(24)00102-2. [PMID: 38311148 DOI: 10.1016/j.cgh.2024.01.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Revised: 01/04/2024] [Accepted: 01/08/2024] [Indexed: 02/10/2024]
Affiliation(s)
- Mauro Giuffrè
- Department of Medicine (Digestive Diseases), Yale School of Medicine, Yale University, New Haven, Connecticut
| | - Dennis L Shung
- Department of Medicine (Digestive Diseases), Yale School of Medicine, Yale University, New Haven, Connecticut
| |
Collapse
|
84
|
Hagedorn JM, George TK, Aiyer R, Schmidt K, Halamka J, D’Souza RS. Artificial Intelligence and Pain Medicine: An Introduction. J Pain Res 2024; 17:509-518. [PMID: 38328019 PMCID: PMC10848920 DOI: 10.2147/jpr.s429594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2023] [Accepted: 12/28/2023] [Indexed: 02/09/2024] Open
Abstract
Artificial intelligence was introduced 60 years ago and has evolved immensely since that time. While artificial intelligence is found in nearly all aspects of our life, the use of artificial intelligence in the healthcare industry has only recently become apparent and more widely discussed. It is expected that artificial intelligence will allow improved disease recognition, treatment optimization, cost and time savings, product development, decision making, and marketing. For pain medicine specifically, these same benefits will be translatable and we can expect better disease recognition and treatment selection. As adoption occurs with this impressive technology, it will be imperative for the pain medicine community to be informed on proper definitions and expected use cases for artificial intelligence. Our objective was to provide pain medicine physicians an overview of artificial intelligence, including important definitions to aid understanding, and to offer potential clinical applications pertinent to the specialty.
Collapse
Affiliation(s)
- Jonathan M Hagedorn
- Department of Anesthesiology and Perioperative Medicine, Division of Pain Medicine, Mayo Clinic, Rochester, MN, USA
| | - Tony K George
- Department of Physical Medicine and Rehabilitation, University Orthopedic Associates, Somerset, NJ, USA
| | - Rohit Aiyer
- Integrative Sports & Spine, Long Beach, CA, USA
| | - Keith Schmidt
- Comprehensive Pain Management Program, St. Alexius Medical Center, Hoffman Estates, IL, USA
| | - John Halamka
- Mayo Clinic Platform, Mayo Clinic, Rochester, MN, USA
| | - Ryan S D’Souza
- Department of Anesthesiology and Perioperative Medicine, Division of Pain Medicine, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
85
|
Crovella S, Suleman M, Tricarico PM, Al-Khuzaei S, Moltrasio C, El Omri A, Marzano AV. Harnessing artificial intelligence for advancing early diagnosis in hidradenitis suppurativa. Ital J Dermatol Venerol 2024; 159:43-49. [PMID: 38345291 DOI: 10.23736/s2784-8671.23.07829-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/15/2024]
Abstract
This perspective delves into the integration of artificial intelligence (AI) to enhance early diagnosis in hidradenitis suppurativa (HS). Despite significantly impacting Quality of Life, HS presents diagnostic challenges leading to treatment delays. We present a viewpoint on AI-powered clinical decision support system designed for HS, emphasizing the transformative potential of AI in dermatology. HS diagnosis, primarily reliant on clinical evaluation and visual inspection, often results in late-stage identification with substantial tissue damage. The incorporation of AI, utilizing machine learning and deep learning algorithms, addresses this challenge by excelling in image analysis. AI adeptly recognizes subtle patterns in skin lesions, providing objective and standardized analyses to mitigate subjectivity in traditional diagnostic approaches. The AI integration encompasses diverse datasets, including clinical records, images, biochemical and immunological data and OMICs data. AI algorithms enable nuanced comprehension, allowing for precise and customized diagnoses. We underscore AI's potential for continuous learning and adaptation, refining recommendations based on evolving data. Challenges in AI integration, such as data privacy, algorithm bias, and interpretability, are addressed, emphasizing the ethical considerations of responsible AI deployment, including transparency, human oversight, and striking a balance between automation and human intervention. From the dermatologists' standpoint, we illustrate how AI enhances diagnostic accuracy, treatment planning, and long-term follow-up in HS management. Dermatologists leverage AI to analyze clinical records, dermatological images, and various data types, facilitating a proactive and personalized approach. AI's dynamic nature supports continuous learning, refining diagnostic and treatment strategies, ultimately reshaping standards of care in dermatology.
Collapse
Affiliation(s)
- Sergio Crovella
- Laboratory of Animal Research Center (LARC), Qatar University, Doha, Qatar -
| | - Muhammad Suleman
- Laboratory of Animal Research Center (LARC), Qatar University, Doha, Qatar
| | | | - Safaa Al-Khuzaei
- Department of Dermatology and Venereology, Hamad Medical Corporation, Rumailah Hospital, Doha, Qatar
| | - Chiara Moltrasio
- Unit of Dermatology, Foundation IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milan, Italy
| | - Abdelfatteh El Omri
- Surgical Research Section, Department of Surgery, Hamad Medical Corporation, Doha, Qatar
| | - Angelo V Marzano
- Unit of Dermatology, Foundation IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milan, Italy
- Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy
| |
Collapse
|
86
|
Andrew A. Potential applications and implications of large language models in primary care. Fam Med Community Health 2024; 12:e002602. [PMID: 38290759 PMCID: PMC10828839 DOI: 10.1136/fmch-2023-002602] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Accepted: 01/16/2024] [Indexed: 02/01/2024] Open
Abstract
The recent release of highly advanced generative artificial intelligence (AI) chatbots, including ChatGPT and Bard, which are powered by large language models (LLMs), has attracted growing mainstream interest over its diverse applications in clinical practice, including in health and healthcare. The potential applications of LLM-based programmes in the medical field range from assisting medical practitioners in improving their clinical decision-making and streamlining administrative paperwork to empowering patients to take charge of their own health. However, despite the broad range of benefits, the use of such AI tools also comes with several limitations and ethical concerns that warrant further consideration, encompassing issues related to privacy, data bias, and the accuracy and reliability of information generated by AI. The focus of prior research has primarily centred on the broad applications of LLMs in medicine. To the author's knowledge, this is, the first article that consolidates current and pertinent literature on LLMs to examine its potential in primary care. The objectives of this paper are not only to summarise the potential benefits, risks and challenges of using LLMs in primary care, but also to offer insights into considerations that primary care clinicians should take into account when deciding to adopt and integrate such technologies into their clinical practice.
Collapse
Affiliation(s)
- Albert Andrew
- Medical Student, The University of Auckland School of Medicine, Auckland, New Zealand
| |
Collapse
|
87
|
Miao J, Thongprayoon C, Suppadungsuk S, Krisanapan P, Radhakrishnan Y, Cheungpasitporn W. Chain of Thought Utilization in Large Language Models and Application in Nephrology. MEDICINA (KAUNAS, LITHUANIA) 2024; 60:148. [PMID: 38256408 PMCID: PMC10819595 DOI: 10.3390/medicina60010148] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 12/31/2023] [Accepted: 01/11/2024] [Indexed: 01/24/2024]
Abstract
Chain-of-thought prompting enhances the abilities of large language models (LLMs) significantly. It not only makes these models more specific and context-aware but also impacts the wider field of artificial intelligence (AI). This approach broadens the usability of AI, increases its efficiency, and aligns it more closely with human thinking and decision-making processes. As we improve this method, it is set to become a key element in the future of AI, adding more purpose, precision, and ethical consideration to these technologies. In medicine, the chain-of-thought prompting is especially beneficial. Its capacity to handle complex information, its logical and sequential reasoning, and its suitability for ethically and context-sensitive situations make it an invaluable tool for healthcare professionals. Its role in enhancing medical care and research is expected to grow as we further develop and use this technique. Chain-of-thought prompting bridges the gap between AI's traditionally obscure decision-making process and the clear, accountable standards required in healthcare. It does this by emulating a reasoning style familiar to medical professionals, fitting well into their existing practices and ethical codes. While solving AI transparency is a complex challenge, the chain-of-thought approach is a significant step toward making AI more comprehensible and trustworthy in medicine. This review focuses on understanding the workings of LLMs, particularly how chain-of-thought prompting can be adapted for nephrology's unique requirements. It also aims to thoroughly examine the ethical aspects, clarity, and future possibilities, offering an in-depth view of the exciting convergence of these areas.
Collapse
Affiliation(s)
- Jing Miao
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
| | - Charat Thongprayoon
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
| | - Supawadee Suppadungsuk
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
- Chakri Naruebodindra Medical Institute, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Samut Prakan 10540, Thailand
| | - Pajaree Krisanapan
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
- Division of Nephrology, Department of Internal Medicine, Faculty of Medicine, Thammasat University, Pathum Thani 12120, Thailand
- Division of Nephrology, Department of Internal Medicine, Thammasat University Hospital, Pathum Thani 12120, Thailand
| | - Yeshwanter Radhakrishnan
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
| | - Wisit Cheungpasitporn
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (J.M.); (S.S.)
| |
Collapse
|
88
|
Bolat B, Eren OC, Dur-Karasayar AH, Mericoz CA, Gunduz-Demir C, Kulac I. Large Language Models as a Rapid and Objective Tool for Pathology Report Data Extraction. Turk Patoloji Derg 2024; 40:138-141. [PMID: 38530110 PMCID: PMC11129865 DOI: 10.5146/tjpath.2024.13256] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/27/2024] Open
Abstract
Medical institutions continuously create a substantial amount of data that is used for scientific research. One of the departments with a great amount of archived data is the pathology department. Pathology archives hold the potential to create a case series of valuable rare entities or large cohorts of common entities. The major problem in creation of these databases is data extraction which is still commonly done manually and is highly laborious and error prone. For these reasons, we offer using large language models to overcome these challenges. Ten pathology reports of selected resection specimens were retrieved from electronic archives of Koç University Hospital for the initial set. These reports were de-identified and uploaded to ChatGPT and Google Bard. Both algorithms were asked to turn the reports in a synoptic report format that is easy to export to a data editor such as Microsoft Excel or Google Sheets. Both programs created tables with Google Bard facilitating the creation of a spreadsheet from the data automatically. In conclusion, we propose the use of AI-assisted data extraction for academic research purposes, as it may enhance efficiency and precision compared to manual data entry.
Collapse
Affiliation(s)
- Beyza Bolat
- Koc University School of Medicine, Koc University, ISTANBUL, TURKEY
| | - Ozgur Can Eren
- Department of Pathology, School of Medicine, Koc University, ISTANBUL, TURKEY
| | | | - Cisel Aydin Mericoz
- Department of Pathology, School of Medicine, Koc University, ISTANBUL, TURKEY
| | | | - Ibrahim Kulac
- Department of Pathology, School of Medicine, Koc University, ISTANBUL, TURKEY
| |
Collapse
|
89
|
Zhang JS, Yoon C, Williams DKA, Pinkas A. Exploring the Usage of ChatGPT Among Medical Students in the United States. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2024; 11:23821205241264695. [PMID: 39092290 PMCID: PMC11292693 DOI: 10.1177/23821205241264695] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/07/2024] [Accepted: 06/08/2024] [Indexed: 08/04/2024]
Abstract
OBJECTIVES Chat Generative Pretrained Transformer (ChatGPT) is a large language model developed by OpenAI that has gained widespread interest. It has been cited for its potential impact on health care and its beneficial role in medical education. However, there is limited investigation into its use among medical students. In this study, we evaluated the frequency of ChatGPT use, motivations for use, and preference for ChatGPT over existing resources among medical students in the United States. METHODS Data was collected from an original survey consisting of 14 questions assessing the frequency and usage of ChatGPT in various contexts within medical education. The survey was distributed via email lists, group messaging applications, and classroom lectures to medical students across the United States. Responses were collected between August and October 2023. RESULTS One hundred thirty-one participants completed the survey and were included in the analysis. Of the total, 48.9% respondents responded that they have used ChatGPT in medical studies. Among ChatGPT users, 43.7% of respondents report using ChatGPT weekly, several times per week, or daily. ChatGPT is most used for writing, revising, editing, and summarizing purposes. 37.5% and 41.3% of respondents reported using ChatGPT more than 25% of the working time for these tasks respectively. Among respondents who have not used ChatGPT, more than 50% of respondents reported they were extremely unlikely or unlikely to use ChatGPT across all surveyed scenarios. ChatGPT users report they are more likely to use ChatGPT over directly asking professors or attendings (45.3%), textbooks (42.2%), and lectures (31.7%), and least likely to be used over popular flashcard application Anki (11.1%) and medical education videos (9.5%). CONCLUSIONS ChatGPT is an increasingly popular resource among medical students, with many preferring ChatGPT over other traditional resources such as professors, textbooks, and lectures. Its impact on medical education will only continue to grow as its capabilities improve.
Collapse
Affiliation(s)
| | - Christine Yoon
- Albert Einstein College of Medicine, Bronx, New York, USA
| | | | - Adi Pinkas
- Albert Einstein College of Medicine, Bronx, New York, USA
| |
Collapse
|
90
|
Akrout M, Cirone KD, Vender R. Evaluation of Vision LLMs GTP-4V and LLaVA for the Recognition of Features Characteristic of Melanoma. J Cutan Med Surg 2024; 28:98-99. [PMID: 38174854 DOI: 10.1177/12034754231220934] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Affiliation(s)
- Mohamed Akrout
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
- AIP Labs, Budapest, Hungary
| | - Katrina D Cirone
- Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
| | - Ronald Vender
- Department of Medicine, McMaster University, Hamilton, ON, Canada
- Dermatrials Research Inc., Hamilton, ON, Canada
| |
Collapse
|
91
|
Hastings J. Preventing harm from non-conscious bias in medical generative AI. Lancet Digit Health 2024; 6:e2-e3. [PMID: 38123253 DOI: 10.1016/s2589-7500(23)00246-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 11/27/2023] [Indexed: 12/23/2023]
Affiliation(s)
- Janna Hastings
- Institute for Implementation Science in Health Care, Faculty of Medicine, University of Zurich, Zurich, Switzerland; School of Medicine, University of St Gallen, St Gallen, Switzerland.
| |
Collapse
|
92
|
Zhou H, Li M, Xiao Y, Yang H, Zhang R. LLM Instruction-Example Adaptive Prompting (LEAP) Framework for Clinical Relation Extraction. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.12.15.23300059. [PMID: 38168203 PMCID: PMC10760264 DOI: 10.1101/2023.12.15.23300059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Abstract
Objective To investigate the demonstration in Large Language Models (LLMs) for clinical relation extraction. We focus on examining two types of adaptive demonstration: instruction adaptive prompting, and example adaptive prompting to understand their impacts and effectiveness. Materials and Methods The study unfolds in two stages. Initially, we explored a range of demonstration components vital to LLMs' clinical data extraction, such as task descriptions and examples, and tested their combinations. Subsequently, we introduced the Instruction-Example Adaptive Prompting (LEAP) Framework, a system that integrates two types of adaptive prompts: one preceding instruction and another before examples. This framework is designed to systematically explore both adaptive task description and adaptive examples within the demonstration. We evaluated LEAP framework's performance on the DDI and BC5CDR chemical interaction datasets, applying it across LLMs such as Llama2-7b, Llama2-13b, and MedLLaMA_13B. Results The study revealed that Instruction + Options + Examples and its expanded form substantially raised F1-scores over the standard Instruction + Options mode. LEAP framework excelled, especially with example adaptive prompting that outdid traditional instruction tuning across models. Notably, the MedLLAMA-13b model scored an impressive 95.13 F1 on the BC5CDR dataset with this method. Significant improvements were also seen in the DDI 2013 dataset, confirming the method's robustness in sophisticated data extraction. Conclusion The LEAP framework presents a promising avenue for refining LLM training strategies, steering away from extensive finetuning towards more contextually rich and dynamic prompting methodologies.
Collapse
Affiliation(s)
- Huixue Zhou
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Mingchen Li
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, Minnesota, USA
| | - Yongkang Xiao
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Han Yang
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, USA
| | - Rui Zhang
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, Minnesota, USA
| |
Collapse
|
93
|
Ohta K, Ohta S. The Performance of GPT-3.5, GPT-4, and Bard on the Japanese National Dentist Examination: A Comparison Study. Cureus 2023; 15:e50369. [PMID: 38213361 PMCID: PMC10782219 DOI: 10.7759/cureus.50369] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/12/2023] [Indexed: 01/13/2024] Open
Abstract
Purpose This study aims to evaluate the performance of three large language models (LLMs), the Generative Pre-trained Transformer (GPT)-3.5, GPT-4, and Google Bard, on the 2023 Japanese National Dentist Examination (JNDE) and assess their potential clinical applications in Japan. Methods A total of 185 questions from the 2023 JNDE were used. These questions were categorized by question type and category. McNemar's test compared the correct response rates between two LLMs, while Fisher's exact test evaluated the performance of LLMs in each question category. Results The overall correct response rates were 73.5% for GPT-4, 66.5% for Bard, and 51.9% for GPT-3.5. GPT-4 showed a significantly higher correct response rate than Bard and GPT-3.5. In the category of essential questions, Bard achieved a correct response rate of 80.5%, surpassing the passing criterion of 80%. In contrast, both GPT-4 and GPT-3.5 fell short of this benchmark, with GPT-4 attaining 77.6% and GPT-3.5 only 52.5%. The scores of GPT-4 and Bard were significantly higher than that of GPT-3.5 (p<0.01). For general questions, the correct response rates were 71.2% for GPT-4, 58.5% for Bard, and 52.5% for GPT-3.5. GPT-4 outperformed GPT-3.5 and Bard (p<0.01). The correct response rates for professional dental questions were 51.6% for GPT-4, 45.3% for Bard, and 35.9% for GPT-3.5. The differences among the models were not statistically significant. All LLMs demonstrated significantly lower accuracy for dentistry questions compared to other types of questions (p<0.01). Conclusions GPT-4 achieved the highest overall score in the JNDE, followed by Bard and GPT-3.5. However, only Bard surpassed the passing score for essential questions. To further understand the application of LLMs in clinical dentistry worldwide, more research on their performance in dental examinations across different languages is required.
Collapse
Affiliation(s)
| | - Satomi Ohta
- Dentistry, Dentist of Mama and Kodomo, Kobe, JPN
| |
Collapse
|
94
|
Lareyre F. Ethical Concerns Regarding the Use of Large Language Models in Healthcare. EJVES Vasc Forum 2023; 61:1. [PMID: 38025830 PMCID: PMC10679759 DOI: 10.1016/j.ejvsvf.2023.10.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Accepted: 10/26/2023] [Indexed: 12/01/2023] Open
Affiliation(s)
- Fabien Lareyre
- Corresponding author. Department of Vascular Surgery, Hospital of Antibes-Juan-les-Pins, 107 avenue de Nice, 06 600, Antibes, France.
| |
Collapse
|