51
|
Truhn D, Eckardt JN, Ferber D, Kather JN. Large language models and multimodal foundation models for precision oncology. NPJ Precis Oncol 2024; 8:72. [PMID: 38519519 PMCID: PMC10959931 DOI: 10.1038/s41698-024-00573-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 03/12/2024] [Indexed: 03/25/2024] Open
Abstract
The technological progress in artificial intelligence (AI) has massively accelerated since 2022, with far-reaching implications for oncology and cancer research. Large language models (LLMs) now perform at human-level competency in text processing. Notably, both text and image processing networks are increasingly based on transformer neural networks. This convergence enables the development of multimodal AI models that take diverse types of data as an input simultaneously, marking a qualitative shift from specialized niche models which were prevalent in the 2010s. This editorial summarizes these developments, which are expected to impact precision oncology in the coming years.
Collapse
Affiliation(s)
- Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Jan-Niklas Eckardt
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Dyke Ferber
- National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany
- Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - Jakob Nikolas Kather
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany.
- Department of Medical Oncology, Heidelberg University Hospital, Heidelberg, Germany.
| |
Collapse
|
52
|
Ge J, Chen IY, Pletcher MJ, Lai JC. Prompt Engineering for Generative Artificial Intelligence in Gastroenterology and Hepatology. Am J Gastroenterol 2024:00000434-990000000-01003. [PMID: 38294157 DOI: 10.14309/ajg.0000000000002689] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 12/28/2023] [Indexed: 02/01/2024]
Affiliation(s)
- Jin Ge
- Division of Gastroenterology and Hepatology, Department of Medicine, University of California, San Francisco, San Francisco, California, USA
| | - Irene Y Chen
- UCSF and UC Berkeley Joint Program in Computational Precision Health, Berkeley, California, USA
| | - Mark J Pletcher
- Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA
| | - Jennifer C Lai
- Division of Gastroenterology and Hepatology, Department of Medicine, University of California, San Francisco, San Francisco, California, USA
| |
Collapse
|
53
|
Sorin V, Glicksberg BS, Artsi Y, Barash Y, Konen E, Nadkarni GN, Klang E. Utilizing large language models in breast cancer management: systematic review. J Cancer Res Clin Oncol 2024; 150:140. [PMID: 38504034 PMCID: PMC10950983 DOI: 10.1007/s00432-024-05678-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Accepted: 03/01/2024] [Indexed: 03/21/2024]
Abstract
PURPOSE Despite advanced technologies in breast cancer management, challenges remain in efficiently interpreting vast clinical data for patient-specific insights. We reviewed the literature on how large language models (LLMs) such as ChatGPT might offer solutions in this field. METHODS We searched MEDLINE for relevant studies published before December 22, 2023. Keywords included: "large language models", "LLM", "GPT", "ChatGPT", "OpenAI", and "breast". The risk bias was evaluated using the QUADAS-2 tool. RESULTS Six studies evaluating either ChatGPT-3.5 or GPT-4, met our inclusion criteria. They explored clinical notes analysis, guideline-based question-answering, and patient management recommendations. Accuracy varied between studies, ranging from 50 to 98%. Higher accuracy was seen in structured tasks like information retrieval. Half of the studies used real patient data, adding practical clinical value. Challenges included inconsistent accuracy, dependency on the way questions are posed (prompt-dependency), and in some cases, missing critical clinical information. CONCLUSION LLMs hold potential in breast cancer care, especially in textual information extraction and guideline-driven clinical question-answering. Yet, their inconsistent accuracy underscores the need for careful validation of these models, and the importance of ongoing supervision.
Collapse
Affiliation(s)
- Vera Sorin
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Affiliated to the Sackler School of Medicine, Tel-Aviv University, Emek Haela St. 1, 52621, Ramat Gan, Israel.
- DeepVision Lab, Chaim Sheba Medical Center, Tel Hashomer, Israel.
| | - Benjamin S Glicksberg
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Yaara Artsi
- Azrieli Faculty of Medicine, Bar-Ilan University, Zefat, Israel
| | - Yiftach Barash
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Affiliated to the Sackler School of Medicine, Tel-Aviv University, Emek Haela St. 1, 52621, Ramat Gan, Israel
- DeepVision Lab, Chaim Sheba Medical Center, Tel Hashomer, Israel
| | - Eli Konen
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Affiliated to the Sackler School of Medicine, Tel-Aviv University, Emek Haela St. 1, 52621, Ramat Gan, Israel
| | - Girish N Nadkarni
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Eyal Klang
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
54
|
Ge J, Sun S, Owens J, Galvez V, Gologorskaya O, Lai JC, Pletcher MJ, Lai K. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology 2024:01515467-990000000-00791. [PMID: 38451962 DOI: 10.1097/hep.0000000000000834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Accepted: 02/24/2024] [Indexed: 03/09/2024]
Abstract
BACKGROUND AND AIMS Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows the embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. APPROACH AND RESULTS We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. RESULTS We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. CONCLUSIONS In this demonstration, we built disease-specific and protected health information-compliant LLMs using RAG. While LiVersa demonstrated higher accuracy in answering questions related to hepatology, there were some deficiencies due to limitations set by the number of documents used for RAG. LiVersa will likely require further refinement before potential live deployment. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical use cases.
Collapse
Affiliation(s)
- Jin Ge
- Department of Medicine, Division of Gastroenterology and Hepatology, University of California-San Francisco, San Francisco, California, USA
| | - Steve Sun
- UCSF Health Information Technology, University of California-San Francisco, San Francisco, California, USA
| | - Joseph Owens
- UCSF Health Information Technology, University of California-San Francisco, San Francisco, California, USA
| | - Victor Galvez
- UCSF Health Information Technology, University of California-San Francisco, San Francisco, California, USA
| | - Oksana Gologorskaya
- UCSF Health Information Technology, University of California-San Francisco, San Francisco, California, USA
- Bakar Computational Health Sciences Institute, University of California-San Francisco, San Francisco, California, USA
| | - Jennifer C Lai
- Department of Medicine, Division of Gastroenterology and Hepatology, University of California-San Francisco, San Francisco, California, USA
| | - Mark J Pletcher
- Department of Epidemiology and Biostatistics, University of California-San Francisco, San Francisco, California, USA
| | - Ki Lai
- UCSF Health Information Technology, University of California-San Francisco, San Francisco, California, USA
| |
Collapse
|
55
|
Raghu Subramanian C, Yang DA, Khanna R. Enhancing Health Care Communication With Large Language Models-The Role, Challenges, and Future Directions. JAMA Netw Open 2024; 7:e240347. [PMID: 38466311 DOI: 10.1001/jamanetworkopen.2024.0347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/12/2024] Open
Affiliation(s)
| | | | - Raman Khanna
- Division of Clinical Informatics and Digital Transformation, University of California, San Francisco
| |
Collapse
|
56
|
Obradovich N, Johnson T, Paulus MP. Managerial and Organizational Challenges in the Age of AI. JAMA Psychiatry 2024; 81:219-220. [PMID: 38265819 DOI: 10.1001/jamapsychiatry.2023.5247] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/25/2024]
Abstract
This Viewpoint discusses the managerial and organizational challenges that could result from the use of artificial intelligence systems in psychiatric research and care.
Collapse
Affiliation(s)
- Nick Obradovich
- Laureate Institute for Brain Research, University of Tulsa, Tulsa, Oklahoma
| | - Tim Johnson
- Atkinson Graduate School of Management, Willamette University, Salem, Oregon
| | - Martin P Paulus
- Laureate Institute for Brain Research, University of Tulsa, Tulsa, Oklahoma
- Department of Psychiatry, University of California, San Diego
- Deputy Editor, JAMA Psychiatry
| |
Collapse
|
57
|
Valizadeh A, Moassefi M, Nakhostin-Ansari A, Heidari Some'eh S, Hosseini-Asl H, Saghab Torbati M, Aghajani R, Maleki Ghorbani Z, Menbari-Oskouie I, Aghajani F, Mirzamohamadi A, Ghafouri M, Faghani S, Memari AH. Automated diagnosis of autism with artificial intelligence: State of the art. Rev Neurosci 2024; 35:141-163. [PMID: 37678819 DOI: 10.1515/revneuro-2023-0050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Accepted: 07/28/2023] [Indexed: 09/09/2023]
Abstract
Autism spectrum disorder (ASD) represents a panel of conditions that begin during the developmental period and result in impairments of personal, social, academic, or occupational functioning. Early diagnosis is directly related to a better prognosis. Unfortunately, the diagnosis of ASD requires a long and exhausting subjective process. We aimed to review the state of the art for automated autism diagnosis and recognition in this research. In February 2022, we searched multiple databases and sources of gray literature for eligible studies. We used an adapted version of the QUADAS-2 tool to assess the risk of bias in the studies. A brief report of the methods and results of each study is presented. Data were synthesized for each modality separately using the Split Component Synthesis (SCS) method. We assessed heterogeneity using the I 2 statistics and evaluated publication bias using trim and fill tests combined with ln DOR. Confidence in cumulative evidence was assessed using the GRADE approach for diagnostic studies. We included 344 studies from 186,020 participants (51,129 are estimated to be unique) for nine different modalities in this review, from which 232 reported sufficient data for meta-analysis. The area under the curve was in the range of 0.71-0.90 for all the modalities. The studies on EEG data provided the best accuracy, with the area under the curve ranging between 0.85 and 0.93. We found that the literature is rife with bias and methodological/reporting flaws. Recommendations are provided for future research to provide better studies and fill in the current knowledge gaps.
Collapse
Affiliation(s)
- Amir Valizadeh
- Neuroscience Institute, Tehran University of Medical Sciences, PO: 1419733141, Tehran, Iran
| | - Mana Moassefi
- Neuroscience Institute, Tehran University of Medical Sciences, PO: 1419733141, Tehran, Iran
| | - Amin Nakhostin-Ansari
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
| | - Soheil Heidari Some'eh
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Students' Scientific Research Center, Tehran University of Medical Sciences, PO: 1417755331, Tehran, Iran
| | - Hossein Hosseini-Asl
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Students' Scientific Research Center, Tehran University of Medical Sciences, PO: 1417755331, Tehran, Iran
| | | | - Reyhaneh Aghajani
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Students' Scientific Research Center, Tehran University of Medical Sciences, PO: 1417755331, Tehran, Iran
| | - Zahra Maleki Ghorbani
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Students' Scientific Research Center, Tehran University of Medical Sciences, PO: 1417755331, Tehran, Iran
| | - Iman Menbari-Oskouie
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
| | - Faezeh Aghajani
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Research Development Center, Arash Women's Hospital, Tehran University of Medical Sciences, PO: 14695542, Tehran, Iran
| | - Alireza Mirzamohamadi
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
- Students' Scientific Research Center, Tehran University of Medical Sciences, PO: 1417755331, Tehran, Iran
| | - Mohammad Ghafouri
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
| | - Shahriar Faghani
- Shariati Hospital, Department of Radiology, Tehran University of Medical Sciences, PO: 1411713135, Tehran, Iran
- Interdisciplinary Neuroscience Research Program (INRP), Tehran University of Medical Sciences, PO: 1416634793, Tehran, Iran
| | - Amir Hossein Memari
- Sports Medicine Research Center, Neuroscience Institute, Tehran University of Medical Sciences, PO: 14395578, Tehran, Iran
| |
Collapse
|
58
|
Shahab O, El Kurdi B, Shaukat A, Nadkarni G, Soroush A. Large language models: a primer and gastroenterology applications. Therap Adv Gastroenterol 2024; 17:17562848241227031. [PMID: 38390029 PMCID: PMC10883116 DOI: 10.1177/17562848241227031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 01/02/2024] [Indexed: 02/24/2024] Open
Abstract
Over the past year, the emergence of state-of-the-art large language models (LLMs) in tools like ChatGPT has ushered in a rapid acceleration in artificial intelligence (AI) innovation. These powerful AI models can generate tailored and high-quality text responses to instructions and questions without the need for labor-intensive task-specific training data or complex software engineering. As the technology continues to mature, LLMs hold immense potential for transforming clinical workflows, enhancing patient outcomes, improving medical education, and optimizing medical research. In this review, we provide a practical discussion of LLMs, tailored to gastroenterologists. We highlight the technical foundations of LLMs, emphasizing their key strengths and limitations as well as how to interact with them safely and effectively. We discuss some potential LLM use cases for clinical gastroenterology practice, education, and research. Finally, we review critical barriers to implementation and ongoing work to address these issues. This review aims to equip gastroenterologists with a foundational understanding of LLMs to facilitate a more active clinician role in the development and implementation of this rapidly emerging technology.
Collapse
Affiliation(s)
- Omer Shahab
- Division of Gastroenterology, Department of Medicine, VHC Health, Arlington, VA, USA
| | - Bara El Kurdi
- Division of Gastroenterology and Hepatology, Department of Medicine, Virginia Tech Carilion School of Medicine, Roanoke, VA, USA
| | - Aasma Shaukat
- Division of Gastroenterology, Department of Medicine, NYU Grossman School of Medicine, New York, NY, USA VA
- New York Harbor Veterans Affairs Healthcare System New York City, New York, NY, USA
| | - Girish Nadkarni
- Division of Data-Driven and Digital Medicine, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ali Soroush
- Division of Data-Driven and Digital Medicine, Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Place, New York, NY 10029-6574, USA
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Henry D. Janowitz Division of Gastroenterology, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
59
|
Han C, Kim DW, Kim S, Chan You S, Park JY, Bae S, Yoon D. Evaluation of GPT-4 for 10-year cardiovascular risk prediction: Insights from the UK Biobank and KoGES data. iScience 2024; 27:109022. [PMID: 38357664 PMCID: PMC10865411 DOI: 10.1016/j.isci.2024.109022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Revised: 11/28/2023] [Accepted: 01/22/2024] [Indexed: 02/16/2024] Open
Abstract
Cardiovascular disease (CVD) remains a pressing global health concern. While traditional risk prediction methods such as the Framingham and American College of Cardiology/American Heart Association (ACC/AHA) risk scores have been widely used in the practice, artificial intelligence (AI), especially GPT-4, offers new opportunities. Utilizing large scale of multi-center data from 47,468 UK Biobank participants and 5,718 KoGES participants, this study quantitatively evaluated the predictive capabilities of GPT-4 in comparison with traditional models. Our results suggest that the GPT-based score showed commendably comparable performance in CVD prediction when compared to traditional models (AUROC on UKB: 0.725 for GPT-4, 0.733 for ACC/AHA, 0.728 for Framingham; KoGES: 0.664 for GPT-4, 0.674 for ACC/AHA, 0.675 for Framingham). Even with omission of certain variables, GPT-4's performance was robust, demonstrating its adaptability to data-scarce situations. In conclusion, this study emphasizes the promising role of GPT-4 in predicting CVD risks across varied ethnic datasets, pointing toward its expansive future applications in the medical practice.
Collapse
Affiliation(s)
- Changho Han
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Yongin, Republic of Korea
| | - Dong Won Kim
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Yongin, Republic of Korea
| | - Songsoo Kim
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Yongin, Republic of Korea
| | - Seng Chan You
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Yongin, Republic of Korea
- Institute for Innovation in Digital Healthcare, Severance Hospital, Seoul, Republic of Korea
| | - Jin Young Park
- Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System, Yongin, Republic of Korea
- Department of Psychiatry, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin, Republic of Korea
- Institute of Behavioral Science in Medicine, Yonsei University College of Medicine, Yonsei University Health System, Seoul, Republic of Korea
| | - SungA Bae
- Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System, Yongin, Republic of Korea
- Department of Cardiology, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin, Republic of Korea
| | - Dukyong Yoon
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Yongin, Republic of Korea
- Institute for Innovation in Digital Healthcare, Severance Hospital, Seoul, Republic of Korea
- Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System, Yongin, Republic of Korea
| |
Collapse
|
60
|
Bey R, Cohen A, Trebossen V, Dura B, Geoffroy PA, Jean C, Landman B, Petit-Jean T, Chatellier G, Sallah K, Tannier X, Bourmaud A, Delorme R. Natural language processing of multi-hospital electronic health records for public health surveillance of suicidality. NPJ MENTAL HEALTH RESEARCH 2024; 3:6. [PMID: 38609541 PMCID: PMC10955903 DOI: 10.1038/s44184-023-00046-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 12/06/2023] [Indexed: 04/14/2024]
Abstract
There is an urgent need to monitor the mental health of large populations, especially during crises such as the COVID-19 pandemic, to timely identify the most at-risk subgroups and to design targeted prevention campaigns. We therefore developed and validated surveillance indicators related to suicidality: the monthly number of hospitalisations caused by suicide attempts and the prevalence among them of five known risks factors. They were automatically computed analysing the electronic health records of fifteen university hospitals of the Paris area, France, using natural language processing algorithms based on artificial intelligence. We evaluated the relevance of these indicators conducting a retrospective cohort study. Considering 2,911,920 records contained in a common data warehouse, we tested for changes after the pandemic outbreak in the slope of the monthly number of suicide attempts by conducting an interrupted time-series analysis. We segmented the assessment time in two sub-periods: before (August 1, 2017, to February 29, 2020) and during (March 1, 2020, to June 31, 2022) the COVID-19 pandemic. We detected 14,023 hospitalisations caused by suicide attempts. Their monthly number accelerated after the COVID-19 outbreak with an estimated trend variation reaching 3.7 (95%CI 2.1-5.3), mainly driven by an increase among girls aged 8-17 (trend variation 1.8, 95%CI 1.2-2.5). After the pandemic outbreak, acts of domestic, physical and sexual violence were more often reported (prevalence ratios: 1.3, 95%CI 1.16-1.48; 1.3, 95%CI 1.10-1.64 and 1.7, 95%CI 1.48-1.98), fewer patients died (p = 0.007) and stays were shorter (p < 0.001). Our study demonstrates that textual clinical data collected in multiple hospitals can be jointly analysed to compute timely indicators describing mental health conditions of populations. Our findings also highlight the need to better take into account the violence imposed on women, especially at early ages and in the aftermath of the COVID-19 pandemic.
Collapse
Affiliation(s)
- Romain Bey
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Ariel Cohen
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France.
| | - Vincent Trebossen
- Child and Adolescent Psychiatry Department, Robert Debré University Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Basile Dura
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Pierre-Alexis Geoffroy
- Département de psychiatrie et d'addictologie, GHU Paris Nord, DMU neurosciences, Bichat - Claude Bernard Hospital, Assistance Publique-Hôpitaux de Paris, 75018, Paris, France
- GHU Paris - psychiatry & neurosciences, 1, rue Cabanis, 75014, Paris, France
- NeuroDiderot, Inserm, FHU I2-D2, université Paris Cité, 75019, Paris, France
- CNRS UPR 3212, Institute for cellular and integrative neurosciences, 67000, Strasbourg, France
| | - Charline Jean
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France
- Université Paris-Est Créteil, INSERM, IMRB U955, Créteil, France
- Service Santé Publique & URC, Hôpital Henri Mondor, Assistance Publique-Hôpitaux de Paris, Créteil, France
| | - Benjamin Landman
- Child and Adolescent Psychiatry Department, Robert Debré University Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Thomas Petit-Jean
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Gilles Chatellier
- Innovation and Data unit, IT Department, Assistance Publique-Hôpitaux de Paris, Paris, France
- Université Paris Cité, Paris, France
| | - Kankoe Sallah
- URC PNVS, CIC-EC 1425, INSERM, Bichat - Claude Bernard Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
| | - Xavier Tannier
- Sorbonne Université, Inserm, Université Sorbonne Paris Nord, Laboratoire d'Informatique Médicale et d'Ingénierie des Connaissances pour la e-Santé (LIMICS), Paris, France
| | - Aurelie Bourmaud
- Université Paris Cité, Paris, France
- Clinical Epidemiology Unit, Robert Debré University Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
- CIC 1426, Inserm, Paris, France
| | - Richard Delorme
- Child and Adolescent Psychiatry Department, Robert Debré University Hospital, Assistance Publique-Hôpitaux de Paris, Paris, France
- Human Genetics and Cognitive Functions, Institut Pasteur, Paris, France
| |
Collapse
|
61
|
Boonstra MJ, Weissenbacher D, Moore JH, Gonzalez-Hernandez G, Asselbergs FW. Artificial intelligence: revolutionizing cardiology with large language models. Eur Heart J 2024; 45:332-345. [PMID: 38170821 PMCID: PMC10834163 DOI: 10.1093/eurheartj/ehad838] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 12/01/2023] [Accepted: 12/05/2023] [Indexed: 01/05/2024] Open
Abstract
Natural language processing techniques are having an increasing impact on clinical care from patient, clinician, administrator, and research perspective. Among others are automated generation of clinical notes and discharge letters, medical term coding for billing, medical chatbots both for patients and clinicians, data enrichment in the identification of disease symptoms or diagnosis, cohort selection for clinical trial, and auditing purposes. In the review, an overview of the history in natural language processing techniques developed with brief technical background is presented. Subsequently, the review will discuss implementation strategies of natural language processing tools, thereby specifically focusing on large language models, and conclude with future opportunities in the application of such techniques in the field of cardiology.
Collapse
Affiliation(s)
- Machteld J Boonstra
- Department of Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam University Medical Centre, University of Amsterdam, Amsterdam, Netherlands
| | - Davy Weissenbacher
- Department of Computational Biomedicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Jason H Moore
- Department of Computational Biomedicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | | | - Folkert W Asselbergs
- Department of Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam University Medical Centre, University of Amsterdam, Amsterdam, Netherlands
- Institute of Health Informatics, University College London, London, UK
- The National Institute for Health Research University College London Hospitals Biomedical Research Centre, University College London, London, UK
| |
Collapse
|
62
|
Liu X, Wu J, Shao A, Shen W, Ye P, Wang Y, Ye J, Jin K, Yang J. Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study. J Med Internet Res 2024; 26:e51926. [PMID: 38252483 PMCID: PMC10845019 DOI: 10.2196/51926] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 10/07/2023] [Accepted: 11/30/2023] [Indexed: 01/23/2024] Open
Abstract
BACKGROUND Benefiting from rich knowledge and the exceptional ability to understand text, large language models like ChatGPT have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in depth. OBJECTIVE This study aimed to evaluate ChatGPT's diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment. METHODS In this cross-sectional study, we collected 1226 fundus fluorescein angiography reports and corresponding diagnoses written in Chinese and tested ChatGPT with 4 prompting strategies (direct diagnosis or diagnosis with a step-by-step reasoning process and in Chinese or English). RESULTS Compared with ChatGPT using Chinese prompts for direct diagnosis that achieved an F1-score of 70.47%, ChatGPT using English prompts for direct diagnosis achieved the best diagnostic performance (80.05%), which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). As for its inference abilities, although ChatGPT can derive a reasoning process with a low error rate (0.4 per report) for both Chinese and English prompts, ophthalmologists identified that the latter brought more reasoning steps with less incompleteness (44.31%), misinformation (1.96%), and hallucinations (0.59%) (all P<.001). Also, analysis of the robustness of ChatGPT with different language prompts indicated significant differences in the recall (P=.03) and F1-score (P=.04) between Chinese and English prompts. In short, when prompted in English, ChatGPT exhibited enhanced diagnostic and inference capabilities for retinal vascular disease classification based on Chinese fundus fluorescein angiography reports. CONCLUSIONS ChatGPT can serve as a helpful medical assistant to provide diagnosis in non-English clinical environments, but there are still performance gaps, language disparities, and errors compared to professionals, which demonstrate the potential limitations and the need to continually explore more robust large language models in ophthalmology practice.
Collapse
Affiliation(s)
- Xiaocong Liu
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
- School of Public Health, Zhejiang University School of Medicine, Zhejiang, China
| | - Jiageng Wu
- School of Public Health, Zhejiang University School of Medicine, Zhejiang, China
| | - An Shao
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Wenyue Shen
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Panpan Ye
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Yao Wang
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Juan Ye
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Kai Jin
- Eye Center, The Second Affiliated Hospital, Zhejiang University, Zhejiang, China
| | - Jie Yang
- School of Public Health, Zhejiang University School of Medicine, Zhejiang, China
| |
Collapse
|
63
|
Ong JCL, Seng BJJ, Law JZF, Low LL, Kwa ALH, Giacomini KM, Ting DSW. Artificial intelligence, ChatGPT, and other large language models for social determinants of health: Current state and future directions. Cell Rep Med 2024; 5:101356. [PMID: 38232690 PMCID: PMC10829781 DOI: 10.1016/j.xcrm.2023.101356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 10/12/2023] [Accepted: 12/10/2023] [Indexed: 01/19/2024]
Abstract
This perspective highlights the importance of addressing social determinants of health (SDOH) in patient health outcomes and health inequity, a global problem exacerbated by the COVID-19 pandemic. We provide a broad discussion on current developments in digital health and artificial intelligence (AI), including large language models (LLMs), as transformative tools in addressing SDOH factors, offering new capabilities for disease surveillance and patient care. Simultaneously, we bring attention to challenges, such as data standardization, infrastructure limitations, digital literacy, and algorithmic bias, that could hinder equitable access to AI benefits. For LLMs, we highlight potential unique challenges and risks including environmental impact, unfair labor practices, inadvertent disinformation or "hallucinations," proliferation of bias, and infringement of copyrights. We propose the need for a multitiered approach to digital inclusion as an SDOH and the development of ethical and responsible AI practice frameworks globally and provide suggestions on bridging the gap from development to implementation of equitable AI technologies.
Collapse
Affiliation(s)
- Jasmine Chiat Ling Ong
- Division of Pharmacy, Singapore General Hospital, Singapore, Singapore; SingHealth Duke-NUS Medicine Academic Clinical Programme, Singapore, Singapore
| | - Benjamin Jun Jie Seng
- MOHH Holdings (Singapore) Pte., Ltd., Singapore, Singapore; SingHealth Duke-NUS Family Medicine Academic Clinical Programme, Singapore, Singapore
| | | | - Lian Leng Low
- SingHealth Duke-NUS Family Medicine Academic Clinical Programme, Singapore, Singapore; Population Health and Integrated Care Office, Singapore General Hospital, Singapore, Singapore; Centre for Population Health Research and Implementation, SingHealth Regional Health System, Singapore, Singapore; Outram Community Hospital, SingHealth Community Hospitals, Singapore, Singapore
| | - Andrea Lay Hoon Kwa
- Division of Pharmacy, Singapore General Hospital, Singapore, Singapore; SingHealth Duke-NUS Medicine Academic Clinical Programme, Singapore, Singapore; Emerging Infectious Diseases, Duke-NUS Medical School, Singapore, Singapore
| | - Kathleen M Giacomini
- Department of Bioengineering and Therapeutic Sciences, Schools of Pharmacy and Medicine, University of California, San Francisco, San Francisco, CA, USA
| | - Daniel Shu Wei Ting
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research, Singapore, Singapore; Duke-NUS Medical School, National University of Singapore, Singapore, Singapore; Byers Eye Institute, Stanford University, Stanford, CA, USA.
| |
Collapse
|
64
|
Schonfeld E, Mordekai N, Berg A, Johnstone T, Shah A, Shah V, Haider G, Marianayagam NJ, Veeravagu A. Machine Learning in Neurosurgery: Toward Complex Inputs, Actionable Predictions, and Generalizable Translations. Cureus 2024; 16:e51963. [PMID: 38333513 PMCID: PMC10851045 DOI: 10.7759/cureus.51963] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2023] [Accepted: 01/08/2024] [Indexed: 02/10/2024] Open
Abstract
Machine learning can predict neurosurgical diagnosis and outcomes, power imaging analysis, and perform robotic navigation and tumor labeling. State-of-the-art models can reconstruct and generate images, predict surgical events from video, and assist in intraoperative decision-making. In this review, we will detail the neurosurgical applications of machine learning, ranging from simple to advanced models, and their potential to transform patient care. As machine learning techniques, outputs, and methods become increasingly complex, their performance is often more impactful yet increasingly difficult to evaluate. We aim to introduce these advancements to the neurosurgical audience while suggesting major potential roadblocks to their safe and effective translation. Unlike the previous generation of machine learning in neurosurgery, the safe translation of recent advancements will be contingent on neurosurgeons' involvement in model development and validation.
Collapse
Affiliation(s)
- Ethan Schonfeld
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| | | | - Alex Berg
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| | - Thomas Johnstone
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| | - Aaryan Shah
- School of Humanities and Sciences, Stanford University, Stanford, USA
| | - Vaibhavi Shah
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| | - Ghani Haider
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| | | | - Anand Veeravagu
- Neurosurgery, Stanford University School of Medicine, Stanford, USA
| |
Collapse
|
65
|
Chen Q, Zhou F, Xie G, Tang CX, Gao X, Zhang Y, Yin X, Xu H, Zhang LJ. Advances in Artificial Intelligence-Assisted Coronary Computed Tomographic Angiography for Atherosclerotic Plaque Characterization. Rev Cardiovasc Med 2024; 25:27. [PMID: 39077649 PMCID: PMC11262402 DOI: 10.31083/j.rcm2501027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2023] [Revised: 09/01/2023] [Accepted: 09/13/2023] [Indexed: 07/31/2024] Open
Abstract
Coronary artery disease is a leading cause of death worldwide. Major adverse cardiac events are associated not only with coronary luminal stenosis but also with atherosclerotic plaque components. Coronary computed tomography angiography (CCTA) enables non-invasive evaluation of atherosclerotic plaque along the entire coronary tree. However, precise and efficient assessment of plaque features on CCTA is still a challenge for physicians in daily practice. Artificial intelligence (AI) refers to algorithms that can simulate intelligent human behavior to improve clinical work efficiency. Recently, cardiovascular imaging has seen remarkable advancements with the use of AI. AI-assisted CCTA has the potential to facilitate the clinical workflow, offer objective and repeatable quantitative results, accelerate the interpretation of reports, and guide subsequent treatment. Several AI algorithms have been developed to provide a comprehensive assessment of atherosclerotic plaques. This review serves to highlight the cutting-edge applications of AI-assisted CCTA in atherosclerosis plaque characterization, including detecting obstructive plaques, assessing plaque volumes and vulnerability, monitoring plaque progression, and providing risk assessment. Finally, this paper discusses the current problems and future directions for implementing AI in real-world clinical settings.
Collapse
Affiliation(s)
- Qian Chen
- Department of Radiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
- Department of Radiology, Jinling Hospital, Medical School of Nanjing University, 210002 Nanjing, Jiangsu, China
| | - Fan Zhou
- Department of Radiology, Jinling Hospital, Medical School of Nanjing University, 210002 Nanjing, Jiangsu, China
| | - Guanghui Xie
- Department of Radiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
| | - Chun Xiang Tang
- Department of Radiology, Jinling Hospital, Medical School of Nanjing University, 210002 Nanjing, Jiangsu, China
| | - Xiaofei Gao
- Department of Cardiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
| | - Yamei Zhang
- Department of Radiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
| | - Xindao Yin
- Department of Radiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
| | - Hui Xu
- Department of Radiology, Nanjing First Hospital, Nanjing Medical University, 210006 Nanjing, Jiangsu, China
| | - Long Jiang Zhang
- Department of Radiology, Jinling Hospital, Medical School of Nanjing University, 210002 Nanjing, Jiangsu, China
| |
Collapse
|
66
|
Zack T, Lehman E, Suzgun M, Rodriguez JA, Celi LA, Gichoya J, Jurafsky D, Szolovits P, Bates DW, Abdulnour REE, Butte AJ, Alsentzer E. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 2024; 6:e12-e22. [PMID: 38123252 DOI: 10.1016/s2589-7500(23)00225-x] [Citation(s) in RCA: 17] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Revised: 09/30/2023] [Accepted: 10/26/2023] [Indexed: 12/23/2023]
Abstract
BACKGROUND Large language models (LLMs) such as GPT-4 hold great promise as transformative tools in health care, ranging from automating administrative tasks to augmenting clinical decision making. However, these models also pose a danger of perpetuating biases and delivering incorrect medical diagnoses, which can have a direct, harmful impact on medical care. We aimed to assess whether GPT-4 encodes racial and gender biases that impact its use in health care. METHODS Using the Azure OpenAI application interface, this model evaluation study tested whether GPT-4 encodes racial and gender biases and examined the impact of such biases on four potential applications of LLMs in the clinical domain-namely, medical education, diagnostic reasoning, clinical plan generation, and subjective patient assessment. We conducted experiments with prompts designed to resemble typical use of GPT-4 within clinical and medical education applications. We used clinical vignettes from NEJM Healer and from published research on implicit bias in health care. GPT-4 estimates of the demographic distribution of medical conditions were compared with true US prevalence estimates. Differential diagnosis and treatment planning were evaluated across demographic groups using standard statistical tests for significance between groups. FINDINGS We found that GPT-4 did not appropriately model the demographic diversity of medical conditions, consistently producing clinical vignettes that stereotype demographic presentations. The differential diagnoses created by GPT-4 for standardised clinical vignettes were more likely to include diagnoses that stereotype certain races, ethnicities, and genders. Assessment and plans created by the model showed significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception. INTERPRETATION Our findings highlight the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care. We discuss the potential sources of these biases and potential mitigation strategies before clinical implementation. FUNDING Priscilla Chan and Mark Zuckerberg.
Collapse
Affiliation(s)
- Travis Zack
- Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA, USA; Helen Diller Family Comprehensive Cancer Center, University of California San Francisco, San Francisco, CA, USA
| | - Eric Lehman
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Mirac Suzgun
- Department of Computer Science, Stanford University, Stanford, CA, USA; Stanford Law School, Stanford University, Stanford, CA, USA
| | - Jorge A Rodriguez
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, USA
| | - Leo Anthony Celi
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA; Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Judy Gichoya
- Department of Radiology, Emory University, Atlanta, GA, USA
| | - Dan Jurafsky
- Department of Computer Science, Stanford University, Stanford, CA, USA; Department of Linguistics, Stanford University, Stanford, CA, USA
| | - Peter Szolovits
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - David W Bates
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, USA; Department of Health Policy and Management, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Raja-Elie E Abdulnour
- Division of Pulmonary and Critical Care Medicine, Brigham and Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA
| | - Atul J Butte
- Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA, USA; Center for Data-Driven Insights and Innovation, University of California, Office of the President, Oakland, CA, USA
| | - Emily Alsentzer
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
67
|
Wu X, Zhang B. ChatGPT promotes healthcare: current applications and potential challenges. Int J Surg 2024; 110:606-608. [PMID: 37816164 PMCID: PMC10793836 DOI: 10.1097/js9.0000000000000802] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Accepted: 09/17/2023] [Indexed: 10/12/2023]
Affiliation(s)
| | - Bin Zhang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangdong, Guangzhou, People’s Republic of China
| |
Collapse
|
68
|
Wang C, Liu S, Li A, Liu J. Text Dialogue Analysis for Primary Screening of Mild Cognitive Impairment: Development and Validation Study. J Med Internet Res 2023; 25:e51501. [PMID: 38157230 PMCID: PMC10787336 DOI: 10.2196/51501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 09/28/2023] [Accepted: 11/27/2023] [Indexed: 01/03/2024] Open
Abstract
BACKGROUND Artificial intelligence models tailored to diagnose cognitive impairment have shown excellent results. However, it is unclear whether large linguistic models can rival specialized models by text alone. OBJECTIVE In this study, we explored the performance of ChatGPT for primary screening of mild cognitive impairment (MCI) and standardized the design steps and components of the prompts. METHODS We gathered a total of 174 participants from the DementiaBank screening and classified 70% of them into the training set and 30% of them into the test set. Only text dialogues were kept. Sentences were cleaned using a macro code, followed by a manual check. The prompt consisted of 5 main parts, including character setting, scoring system setting, indicator setting, output setting, and explanatory information setting. Three dimensions of variables from published studies were included: vocabulary (ie, word frequency and word ratio, phrase frequency and phrase ratio, and lexical complexity), syntax and grammar (ie, syntactic complexity and grammatical components), and semantics (ie, semantic density and semantic coherence). We used R 4.3.0. for the analysis of variables and diagnostic indicators. RESULTS Three additional indicators related to the severity of MCI were incorporated into the final prompt for the model. These indicators were effective in discriminating between MCI and cognitively normal participants: tip-of-the-tongue phenomenon (P<.001), difficulty with complex ideas (P<.001), and memory issues (P<.001). The final GPT-4 model achieved a sensitivity of 0.8636, a specificity of 0.9487, and an area under the curve of 0.9062 on the training set; on the test set, the sensitivity, specificity, and area under the curve reached 0.7727, 0.8333, and 0.8030, respectively. CONCLUSIONS ChatGPT was effective in the primary screening of participants with possible MCI. Improved standardization of prompts by clinicians would also improve the performance of the model. It is important to note that ChatGPT is not a substitute for a clinician making a diagnosis.
Collapse
Affiliation(s)
- Changyu Wang
- Department of Medical Informatics, West China Medical School, Sichuan University, Chengdu, China
- West China College of Stomatology, Sichuan University, Chengdu, China
| | - Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Aiqing Li
- Department of Neurology, West China Hospital, Sichuan University, Chengdu, China
| | - Jialin Liu
- Department of Medical Informatics, West China Medical School, Sichuan University, Chengdu, China
- Information Center, West China Hospital, Sichuan University, Chengdu, China
- Department of Otolaryngology-Head and Neck Surgery, West China Hospital, Sichuan University, Chengdu, China
| |
Collapse
|
69
|
Koranteng E, Rao A, Flores E, Lev M, Landman A, Dreyer K, Succi M. Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care. JMIR MEDICAL EDUCATION 2023; 9:e51199. [PMID: 38153778 PMCID: PMC10884892 DOI: 10.2196/51199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Revised: 10/01/2023] [Accepted: 10/14/2023] [Indexed: 12/29/2023]
Abstract
The growing presence of large language models (LLMs) in health care applications holds significant promise for innovative advancements in patient care. However, concerns about ethical implications and potential biases have been raised by various stakeholders. Here, we evaluate the ethics of LLMs in medicine along 2 key axes: empathy and equity. We outline the importance of these factors in novel models of care and develop frameworks for addressing these alongside LLM deployment.
Collapse
Affiliation(s)
| | - Arya Rao
- Harvard Medical School, Boston, MA, United States
| | - Efren Flores
- Harvard Medical School, Boston, MA, United States
| | - Michael Lev
- Harvard Medical School, Boston, MA, United States
| | - Adam Landman
- Harvard Medical School, Boston, MA, United States
| | - Keith Dreyer
- Harvard Medical School, Boston, MA, United States
| | - Marc Succi
- Massachusetts General Hospital, Boston, United States
| |
Collapse
|
70
|
Beaulieu-Jones BK, Villamar MF, Scordis P, Bartmann AP, Ali W, Wissel BD, Alsentzer E, de Jong J, Patra A, Kohane I. Predicting seizure recurrence after an initial seizure-like episode from routine clinical notes using large language models: a retrospective cohort study. Lancet Digit Health 2023; 5:e882-e894. [PMID: 38000873 PMCID: PMC10695164 DOI: 10.1016/s2589-7500(23)00179-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 08/08/2023] [Accepted: 08/31/2023] [Indexed: 11/26/2023]
Abstract
BACKGROUND The evaluation and management of first-time seizure-like events in children can be difficult because these episodes are not always directly observed and might be epileptic seizures or other conditions (seizure mimics). We aimed to evaluate whether machine learning models using real-world data could predict seizure recurrence after an initial seizure-like event. METHODS This retrospective cohort study compared models trained and evaluated on two separate datasets between Jan 1, 2010, and Jan 1, 2020: electronic medical records (EMRs) at Boston Children's Hospital and de-identified, patient-level, administrative claims data from the IBM MarketScan research database. The study population comprised patients with an initial diagnosis of either epilepsy or convulsions before the age of 21 years, based on International Classification of Diseases, Clinical Modification (ICD-CM) codes. We compared machine learning-based predictive modelling using structured data (logistic regression and XGBoost) with emerging techniques in natural language processing by use of large language models. FINDINGS The primary cohort comprised 14 021 patients at Boston Children's Hospital matching inclusion criteria with an initial seizure-like event and the comparison cohort comprised 15 062 patients within the IBM MarketScan research database. Seizure recurrence based on a composite expert-derived definition occurred in 57% of patients at Boston Children's Hospital and 63% of patients within IBM MarketScan. Large language models with additional domain-specific and location-specific pre-training on patients excluded from the study (F1-score 0·826 [95% CI 0·817-0·835], AUC 0·897 [95% CI 0·875-0·913]) performed best. All large language models, including the base model without additional pre-training (F1-score 0·739 [95% CI 0·738-0·741], AUROC 0·846 [95% CI 0·826-0·861]) outperformed models trained with structured data. With structured data only, XGBoost outperformed logistic regression and XGBoost models trained with the Boston Children's Hospital EMR (logistic regression: F1-score 0·650 [95% CI 0·643-0·657], AUC 0·694 [95% CI 0·685-0·705], XGBoost: F1-score 0·679 [0·676-0·683], AUC 0·725 [0·717-0·734]) performed similarly to models trained on the IBM MarketScan database (logistic regression: F1-score 0·596 [0·590-0·601], AUC 0·670 [0·664-0·675], XGBoost: F1-score 0·678 [0·668-0·687], AUC 0·710 [0·703-0·714]). INTERPRETATION Physician's clinical notes about an initial seizure-like event include substantial signals for prediction of seizure recurrence, and additional domain-specific and location-specific pre-training can significantly improve the performance of clinical large language models, even for specialised cohorts. FUNDING UCB, National Institute of Neurological Disorders and Stroke (US National Institutes of Health).
Collapse
Affiliation(s)
- Brett K Beaulieu-Jones
- Department of Medicine, University of Chicago, Chicago, IL, USA; Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
| | - Mauricio F Villamar
- Department of Neurology, The Warren Alpert Medical School of Brown University, Providence, RI, USA
| | | | | | | | - Benjamin D Wissel
- Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
| | - Emily Alsentzer
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | | | | | - Isaac Kohane
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
71
|
Overgaard SM, Graham MG, Brereton T, Pencina MJ, Halamka JD, Vidal DE, Economou-Zavlanos NJ. Implementing quality management systems to close the AI translation gap and facilitate safe, ethical, and effective health AI solutions. NPJ Digit Med 2023; 6:218. [PMID: 38007604 PMCID: PMC10676432 DOI: 10.1038/s41746-023-00968-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2023] [Accepted: 11/15/2023] [Indexed: 11/27/2023] Open
|
72
|
Peng C, Yang X, Chen A, Smith KE, PourNejatian N, Costa AB, Martin C, Flores MG, Zhang Y, Magoc T, Lipori G, Mitchell DA, Ospina NS, Ahmed MM, Hogan WR, Shenkman EA, Guo Y, Bian J, Wu Y. A study of generative large language model for medical research and healthcare. NPJ Digit Med 2023; 6:210. [PMID: 37973919 PMCID: PMC10654385 DOI: 10.1038/s41746-023-00958-w] [Citation(s) in RCA: 20] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Accepted: 11/01/2023] [Indexed: 11/19/2023] Open
Abstract
There are enormous enthusiasm and concerns in applying large language models (LLMs) to healthcare. Yet current assumptions are based on general-purpose LLMs such as ChatGPT, which are not developed for medical use. This study develops a generative clinical LLM, GatorTronGPT, using 277 billion words of text including (1) 82 billion words of clinical text from 126 clinical departments and approximately 2 million patients at the University of Florida Health and (2) 195 billion words of diverse general English text. We train GatorTronGPT using a GPT-3 architecture with up to 20 billion parameters and evaluate its utility for biomedical natural language processing (NLP) and healthcare text generation. GatorTronGPT improves biomedical natural language processing. We apply GatorTronGPT to generate 20 billion words of synthetic text. Synthetic NLP models trained using synthetic text generated by GatorTronGPT outperform models trained using real-world clinical text. Physicians' Turing test using 1 (worst) to 9 (best) scale shows that there are no significant differences in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights into the opportunities and challenges of LLMs for medical research and healthcare.
Collapse
Affiliation(s)
- Cheng Peng
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Xi Yang
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA
| | - Aokun Chen
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA
| | | | | | | | | | | | - Ying Zhang
- Research Computing, University of Florida, Gainesville, FL, USA
| | - Tanja Magoc
- Integrated Data Repository Research Services, University of Florida, Gainesville, FL, USA
| | - Gloria Lipori
- Integrated Data Repository Research Services, University of Florida, Gainesville, FL, USA
- Lillian S. Wells Department of Neurosurgery, Clinical and Translational Science Institute, University of Florida, Gainesville, FL, USA
| | - Duane A Mitchell
- Lillian S. Wells Department of Neurosurgery, Clinical and Translational Science Institute, University of Florida, Gainesville, FL, USA
| | - Naykky S Ospina
- Division of Endocrinology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Mustafa M Ahmed
- Division of Cardiovascular Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
| | - William R Hogan
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Elizabeth A Shenkman
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Yi Guo
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA
| | - Jiang Bian
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA
| | - Yonghui Wu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA.
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA.
| |
Collapse
|
73
|
Ge J, Sun S, Owens J, Galvez V, Gologorskaya O, Lai JC, Pletcher MJ, Lai K. Development of a Liver Disease-Specific Large Language Model Chat Interface using Retrieval Augmented Generation. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.11.10.23298364. [PMID: 37986764 PMCID: PMC10659484 DOI: 10.1101/2023.11.10.23298364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2023]
Abstract
Background Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating incorrect or hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. Methods We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information (PHI)-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases (AASLD) guidelines and guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by comparing its responses versus those of trainees from a previously published knowledge assessment study regarding hepatitis B (HBV) treatment and hepatocellular carcinoma (HCC) surveillance. Results LiVersa answered all 10 questions correctly when forced to provide a "yes" or "no" answer. Full detailed responses with justifications and rationales, however, were not completely correct for three of the questions. Discussions In this study, we demonstrated the ability to build disease-specific and PHI-compliant LLMs using RAG. While our LLM, LiVersa, demonstrated more specificity in answering questions related to clinical hepatology - there were some knowledge deficiencies due to limitations set by the number and types of documents used for RAG. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical uses and a potential strategy to realize personalized medicine in the future.
Collapse
Affiliation(s)
- Jin Ge
- Division of Gastroenterology and Hepatology, Department of Medicine, University of California – San Francisco, San Francisco, CA
| | - Steve Sun
- UCSF Health Information Technology, University of California – San Francisco, San Francisco, CA
| | - Joseph Owens
- UCSF Health Information Technology, University of California – San Francisco, San Francisco, CA
| | - Victor Galvez
- UCSF Health Information Technology, University of California – San Francisco, San Francisco, CA
| | - Oksana Gologorskaya
- UCSF Health Information Technology, University of California – San Francisco, San Francisco, CA
- Bakar Computational Health Sciences Institute, University of California – San Francisco, San Francisco, CA
| | - Jennifer C. Lai
- Division of Gastroenterology and Hepatology, Department of Medicine, University of California – San Francisco, San Francisco, CA
| | - Mark J. Pletcher
- Department of Epidemiology and Biostatistics, University of California – San Francisco, San Francisco, CA
| | - Ki Lai
- UCSF Health Information Technology, University of California – San Francisco, San Francisco, CA
| |
Collapse
|
74
|
Decker H, Trang K, Ramirez J, Colley A, Pierce L, Coleman M, Bongiovanni T, Melton GB, Wick E. Large Language Model-Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures. JAMA Netw Open 2023; 6:e2336997. [PMID: 37812419 PMCID: PMC10562939 DOI: 10.1001/jamanetworkopen.2023.36997] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/08/2023] [Accepted: 08/29/2023] [Indexed: 10/10/2023] Open
Abstract
Importance Informed consent is a critical component of patient care before invasive procedures, yet it is frequently inadequate. Electronic consent forms have the potential to facilitate patient comprehension if they provide information that is readable, accurate, and complete; it is not known if large language model (LLM)-based chatbots may improve informed consent documentation by generating accurate and complete information that is easily understood by patients. Objective To compare the readability, accuracy, and completeness of LLM-based chatbot- vs surgeon-generated information on the risks, benefits, and alternatives (RBAs) of common surgical procedures. Design, Setting, and Participants This cross-sectional study compared randomly selected surgeon-generated RBAs used in signed electronic consent forms at an academic referral center in San Francisco with LLM-based chatbot-generated (ChatGPT-3.5, OpenAI) RBAs for 6 surgical procedures (colectomy, coronary artery bypass graft, laparoscopic cholecystectomy, inguinal hernia repair, knee arthroplasty, and spinal fusion). Main Outcomes and Measures Readability was measured using previously validated scales (Flesh-Kincaid grade level, Gunning Fog index, the Simple Measure of Gobbledygook, and the Coleman-Liau index). Scores range from 0 to greater than 20 to indicate the years of education required to understand a text. Accuracy and completeness were assessed using a rubric developed with recommendations from LeapFrog, the Joint Commission, and the American College of Surgeons. Both composite and RBA subgroup scores were compared. Results The total sample consisted of 36 RBAs, with 1 RBA generated by the LLM-based chatbot and 5 RBAs generated by a surgeon for each of the 6 surgical procedures. The mean (SD) readability score for the LLM-based chatbot RBAs was 12.9 (2.0) vs 15.7 (4.0) for surgeon-generated RBAs (P = .10). The mean (SD) composite completeness and accuracy score was lower for surgeons' RBAs at 1.6 (0.5) than for LLM-based chatbot RBAs at 2.2 (0.4) (P < .001). The LLM-based chatbot scores were higher than the surgeon-generated scores for descriptions of the benefits of surgery (2.3 [0.7] vs 1.4 [0.7]; P < .001) and alternatives to surgery (2.7 [0.5] vs 1.4 [0.7]; P < .001). There was no significant difference in chatbot vs surgeon RBA scores for risks of surgery (1.7 [0.5] vs 1.7 [0.4]; P = .38). Conclusions and Relevance The findings of this cross-sectional study suggest that despite not being perfect, LLM-based chatbots have the potential to enhance informed consent documentation. If an LLM were embedded in electronic health records in a manner compliant with the Health Insurance Portability and Accountability Act, it could be used to provide personalized risk information while easing documentation burden for physicians.
Collapse
Affiliation(s)
- Hannah Decker
- Department of Surgery, University of California, San Francisco
| | - Karen Trang
- Department of Surgery, University of California, San Francisco
| | - Joel Ramirez
- Department of Surgery, University of California, San Francisco
| | - Alexis Colley
- Department of Surgery, University of California, San Francisco
| | - Logan Pierce
- Department of Medicine, University of California, San Francisco
| | - Melissa Coleman
- Department of Surgery, University of California, San Francisco
| | | | - Genevieve B. Melton
- Department of Surgery, Institute for Health Informatics, and Center for Learning Health System Sciences, University of Minnesota, Minneapolis
| | - Elizabeth Wick
- Department of Surgery, University of California, San Francisco
| |
Collapse
|
75
|
Jung KH. Uncover This Tech Term: Foundation Model. Korean J Radiol 2023; 24:1038-1041. [PMID: 37793672 PMCID: PMC10550749 DOI: 10.3348/kjr.2023.0790] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Accepted: 08/23/2023] [Indexed: 10/06/2023] Open
Affiliation(s)
- Kyu-Hwan Jung
- Department of Medical Device Management and Research, Samsung Advanced Institute for Health Sciences and Technology, Sungkyunkwan University, Seoul, Republic of Korea
- Dataset Science Research Institute, Research Institute for Future Medicine, Samsung Medical Center, Seoul, Republic of Korea.
| |
Collapse
|
76
|
Brin D, Sorin V, Vaid A, Soroush A, Glicksberg BS, Charney AW, Nadkarni G, Klang E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep 2023; 13:16492. [PMID: 37779171 PMCID: PMC10543445 DOI: 10.1038/s41598-023-43436-9] [Citation(s) in RCA: 42] [Impact Index Per Article: 42.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Accepted: 09/23/2023] [Indexed: 10/03/2023] Open
Abstract
The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was used to assess the models' consistency. The performance of the AI models was compared to that of previous AMBOSS users. GPT-4 outperformed ChatGPT, correctly answering 90% compared to ChatGPT's 62.5%. GPT-4 showed more confidence, not revising any responses, while ChatGPT modified its original answers 82.5% of the time. The performance of GPT-4 was higher than that of AMBOSS's past users. Both AI models, notably GPT-4, showed capacity for empathy, indicating AI's potential to meet the complex interpersonal, ethical, and professional demands intrinsic to the practice of medicine.
Collapse
Affiliation(s)
- Dana Brin
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Ramat Gan, Israel.
- Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel.
| | - Vera Sorin
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Ramat Gan, Israel
- Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| | - Akhil Vaid
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ali Soroush
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Benjamin S Glicksberg
- Hasso Plattner Institute for Digital Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Alexander W Charney
- The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Girish Nadkarni
- Division of Data-Driven and Digital Medicine (D3M), The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Eyal Klang
- Department of Diagnostic Imaging, Chaim Sheba Medical Center, Ramat Gan, Israel
- Faculty of Medicine, Tel-Aviv University, Tel-Aviv, Israel
| |
Collapse
|
77
|
Cheung ATM, Nasir-Moin M, Oermann EK. ChatGPT and the Law of the Horse. THE AMERICAN JOURNAL OF BIOETHICS : AJOB 2023; 23:55-57. [PMID: 37812113 DOI: 10.1080/15265161.2023.2250279] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/10/2023]
|
78
|
Robinson ML, Garibaldi BT, Lindquist MA. When Clinical Prediction Is Steering the Ship, Beware the Drift of Its Wake. Ann Intern Med 2023; 176:1424-1425. [PMID: 37812777 DOI: 10.7326/m23-2345] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/11/2023] Open
Affiliation(s)
- Matthew L Robinson
- Division of Infectious Diseases, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Brian T Garibaldi
- Division of Pulmonary and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | - Martin A Lindquist
- Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland
| |
Collapse
|
79
|
Affiliation(s)
- Atul J Butte
- Bakar Computational Health Sciences Institute, University of California, San Francisco
- Center for Data-Driven Insights and Innovation, University of California, Office of the President, Oakland
| |
Collapse
|