1
|
Bellamkonda N, Farlow JL, Haring CT, Sim MW, Seim NB, Cannon RB, Monroe MM, Agrawal A, Rocco JW, McCrary HC. Evaluating the Accuracy of ChatGPT in Common Patient Questions Regarding HPV+ Oropharyngeal Carcinoma. Ann Otol Rhinol Laryngol 2024; 133:814-819. [PMID: 39075853 DOI: 10.1177/00034894241259137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/31/2024]
Abstract
OBJECTIVES Large language model (LLM)-based chatbots such as ChatGPT have been publicly available and increasingly utilized by the general public since late 2022. This study sought to investigate ChatGPT responses to common patient questions regarding Human Papilloma Virus (HPV) positive oropharyngeal cancer (OPC). METHODS This was a prospective, multi-institutional study, with data collected from high volume institutions that perform >50 transoral robotic surgery cases per year. The 100 most recent discussion threads including the term "HPV" on the American Cancer Society's Cancer Survivors Network's Head and Neck Cancer public discussion board were reviewed. The 11 most common questions were serially queried to ChatGPT 3.5; answers were recorded. A survey was distributed to fellowship trained head and neck oncologic surgeons at 3 institutions to evaluate the responses. RESULTS A total of 8 surgeons participated in the study. For questions regarding HPV contraction and transmission, ChatGPT answers were scored as clinically accurate and aligned with consensus in the head and neck surgical oncology community 84.4% and 90.6% of the time, respectively. For questions involving treatment of HPV+ OPC, ChatGPT was clinically accurate and aligned with consensus 87.5% and 91.7% of the time, respectively. For questions regarding the HPV vaccine, ChatGPT was clinically accurate and aligned with consensus 62.5% and 75% of the time, respectively. When asked about circulating tumor DNA testing, only 12.5% of surgeons thought responses were accurate or consistent with consensus. CONCLUSION ChatGPT 3.5 performed poorly with questions involving evolving therapies and diagnostics-thus, caution should be used when using a platform like ChatGPT 3.5 to assess use of advanced technology. Patients should be counseled on the importance of consulting their surgeons to receive accurate and up to date recommendations, and use LLM's to augment their understanding of these important health-related topics.
Collapse
Affiliation(s)
- Nikhil Bellamkonda
- Department of Otolaryngology-Head and Neck Surgery, University of Utah, Salt Lake City, UT, USA
| | - Janice L Farlow
- Department of Otolaryngology-Head and Neck Surgery, Indiana University, Indianapolis, IN, USA
| | - Catherine T Haring
- Department of Otolaryngology-Head and Neck Surgery, The Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Michael W Sim
- Department of Otolaryngology-Head and Neck Surgery, Indiana University, Indianapolis, IN, USA
| | - Nolan B Seim
- Department of Otolaryngology-Head and Neck Surgery, The Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Richard B Cannon
- Department of Otolaryngology-Head and Neck Surgery, University of Utah, Salt Lake City, UT, USA
| | - Marcus M Monroe
- Department of Otolaryngology-Head and Neck Surgery, University of Utah, Salt Lake City, UT, USA
| | - Amit Agrawal
- Department of Otolaryngology-Head and Neck Surgery, The Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - James W Rocco
- Department of Otolaryngology-Head and Neck Surgery, The Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Hilary C McCrary
- Department of Otolaryngology-Head and Neck Surgery, University of Utah, Salt Lake City, UT, USA
| |
Collapse
|
2
|
Giuffrè M, Kresevic S, Pugliese N, You K, Shung DL. Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes. Liver Int 2024; 44:2114-2124. [PMID: 38819632 DOI: 10.1111/liv.15974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/28/2024] [Revised: 04/26/2024] [Accepted: 05/02/2024] [Indexed: 06/01/2024]
Abstract
Large Language Models (LLMs) are transformer-based neural networks with billions of parameters trained on very large text corpora from diverse sources. LLMs have the potential to improve healthcare due to their capability to parse complex concepts and generate context-based responses. The interest in LLMs has not spared digestive disease academics, who have mainly investigated foundational LLM accuracy, which ranges from 25% to 90% and is influenced by the lack of standardized rules to report methodologies and results for LLM-oriented research. In addition, a critical issue is the absence of a universally accepted definition of accuracy, varying from binary to scalar interpretations, often tied to grader expertise without reference to clinical guidelines. We address strategies and challenges to increase accuracy. In particular, LLMs can be infused with domain knowledge using Retrieval Augmented Generation (RAG) or Supervised Fine-Tuning (SFT) with reinforcement learning from human feedback (RLHF). RAG faces challenges with in-context window limits and accurate information retrieval from the provided context. SFT, a deeper adaptation method, is computationally demanding and requires specialized knowledge. LLMs may increase patient quality of care across the field of digestive diseases, where physicians are often engaged in screening, treatment and surveillance for a broad range of pathologies for which in-context learning or SFT with RLHF could improve clinical decision-making and patient outcomes. However, despite their potential, the safe deployment of LLMs in healthcare still needs to overcome hurdles in accuracy, suggesting a need for strategies that integrate human feedback with advanced model training.
Collapse
Affiliation(s)
- Mauro Giuffrè
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA
- Department of Medical, Surgical, and Health Sciences, University of Trieste, Trieste, Italy
| | - Simone Kresevic
- Department of Engineering and Architecture, University of Trieste, Trieste, Italy
| | - Nicola Pugliese
- Division of Internal Medicine and Hepatology, Department of Gastroenterology, IRCCS Humanitas Research Hospital, Rozzano, Italy
- Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Italy
| | - Kisung You
- Department of Mathematics, Baruch College, City University of New York, New York, New York, USA
| | - Dennis L Shung
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA
| |
Collapse
|
3
|
Tse G, Kuzma N, Khan A. Perspective: Artificial intelligence and patient communication. J Hosp Med 2024; 19:845-849. [PMID: 38340353 DOI: 10.1002/jhm.13301] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Revised: 01/21/2024] [Accepted: 01/28/2024] [Indexed: 02/12/2024]
Affiliation(s)
- Gabriel Tse
- Department of Pediatrics, Division of Pediatric Hospital Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Nicholas Kuzma
- Section of Hospital Medicine, St. Christopher's Hospital for Children, Philadelphia, Pennsylvania, USA
| | - Alisa Khan
- Department of Pediatrics, Divison of General Pediatrics, Boston Children's Hospital, Boston, Massachusetts, USA
| |
Collapse
|
4
|
Mnajjed L, Patel RJ. Assessment of ChatGPT generated educational material for head and neck surgery counseling. Am J Otolaryngol 2024; 45:104410. [PMID: 39059168 DOI: 10.1016/j.amjoto.2024.104410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2024] [Accepted: 07/17/2024] [Indexed: 07/28/2024]
Abstract
BACKGROUND ChatGPT is becoming very popular as an information source for the public. The adequacy of ChatGPT generated patient counseling material has not yet been extensively assessed. METHODS ChatGPT was presented with perioperative counseling and complication questions regarding five different procedure, and accuracy of responses was assessed. The chat was then asked to present an explanation of each procedure, and quality of the responses were compared to online educational material. RESULTS ChatGPT responses were comprehensive when discussing counseling points commonly discussed by a provider prior to a procedure. Responses to questions on surgical complications were less accurate and comprehensive. In comparison to online educational material, ChatGPT scored at or above the median SAM and PEMAT scores for all procedures. CONCLUSIONS ChatGPT did well addressing basic counseling points during the perioperative period, although it did not perform as well when addressing surgical complications. Chat response quality was comparable to currently available online educational material.
Collapse
Affiliation(s)
- Lana Mnajjed
- University of Oklahoma College of Medicine - Tulsa, Tulsa, OK, USA
| | - Rusha J Patel
- University of Oklahoma Department of Otolaryngology - Head and Neck Surgery, Oklahoma City, OK, USA.
| |
Collapse
|
5
|
Ghersin I, Weisshof R, Koifman E, Bar-Yoseph H, Ben Hur D, Maza I, Hasnis E, Nasser R, Ovadia B, Dror Zur D, Waterman M, Gorelik Y. Comparative evaluation of a language model and human specialists in the application of European guidelines for the management of inflammatory bowel diseases and malignancies. Endoscopy 2024; 56:706-709. [PMID: 38499197 DOI: 10.1055/a-2289-5732] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/20/2024]
Abstract
BACKGROUND Society guidelines on colorectal dysplasia screening, surveillance, and endoscopic management in inflammatory bowel disease (IBD) are complex, and physician adherence to them is suboptimal. We aimed to evaluate the use of ChatGPT, a large language model, in generating accurate guideline-based recommendations for colorectal dysplasia screening, surveillance, and endoscopic management in IBD in line with European Crohn's and Colitis Organization (ECCO) guidelines. METHODS 30 clinical scenarios in the form of free text were prepared and presented to three separate sessions of ChatGPT and to eight gastroenterologists (four IBD specialists and four non-IBD gastroenterologists). Two additional IBD specialists subsequently assessed all responses provided by ChatGPT and the eight gastroenterologists, judging their accuracy according to ECCO guidelines. RESULTS ChatGPT had a mean correct response rate of 87.8%. Among the eight gastroenterologists, the mean correct response rates were 85.8% for IBD experts and 89.2% for non-IBD experts. No statistically significant differences in accuracy were observed between ChatGPT and all gastroenterologists (P=0.95), or between ChatGPT and the IBD experts and non-IBD expert gastroenterologists, respectively (P=0.82). CONCLUSIONS This study highlights the potential of language models in enhancing guideline adherence regarding colorectal dysplasia in IBD. Further investigation of additional resources and prospective evaluation in real-world settings are warranted.
Collapse
Affiliation(s)
- Itai Ghersin
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Roni Weisshof
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Eduard Koifman
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Haggai Bar-Yoseph
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
- Rappaport Faculty of Medicine, Technion, Israel Institute of Technology, Haifa, Israel
| | - Dana Ben Hur
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Itay Maza
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Erez Hasnis
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Roni Nasser
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| | - Baruch Ovadia
- Department of Gastroenterology and Hepatology, Hillel Yaffe Medical Center, Hadera, Israel
| | - Dikla Dror Zur
- Department of Gastroenterology, Galilee Medical Center, Nahariya, Israel
| | - Matti Waterman
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
- Rappaport Faculty of Medicine, Technion, Israel Institute of Technology, Haifa, Israel
| | - Yuri Gorelik
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel
| |
Collapse
|
6
|
Oliveira AL, Coelho M, Guedes LC, Cattoni MB, Carvalho H, Duarte-Batista P. Performance of ChatGPT 3.5 and 4 as a tool for patient support before and after DBS surgery for Parkinson's disease. Neurol Sci 2024:10.1007/s10072-024-07732-0. [PMID: 39198356 DOI: 10.1007/s10072-024-07732-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2024] [Accepted: 08/19/2024] [Indexed: 09/01/2024]
Abstract
Deep brain stimulation (DBS) is a neurosurgical procedure that involves implanting electrodes into specific areas of the brain to treat a variety of medical conditions, including Parkinson's disease. Doubts and questions from patients prior to or following surgery should be addressed in line with the most recent scientific and clinical practice. ChatGPT emerges as an example of how artificial intelligence can be used, with its ability to comprehend and answer medical questions in an understandable way, accessible to everyone. However, the risks of these resources still need to be fully understood.ChatGPT models 3.5 and 4 responses to 40 questions in English and Portuguese were independently graded by two experienced specialists in functional neurosurgery and neurological movement disorders and resolved by a third reviewer. ChatGPT 3.5 and 4 demonstrated a good level of accuracy in responding to 80 questions in both English and Portuguese, related to DBS surgery for Parkinson's disease. The proportion of responses graded as correct was 57.5% and 83.8% for GPT 3.5 and GPT 4, respectively. GPT 3.5 provided potentially harmful answers for 6.3% (5/80) of its responses. No responses from GPT 4 were graded as harmful. In general, ChatGPT 3.5 and 4 demonstrated good performance in terms of quality and reliability across two different languages. Nonetheless, harmful responses should not be scorned, and it's crucial to consider this aspect when addressing patients using these resources. Considering the current safety concerns, it's not advisable for patients to use such models for DBS surgery guidance. Performance of ChatGPT 3.5 and 4 as a tool for patient support before and after DBS surgery for Parkinson's disease.
Collapse
Affiliation(s)
- Ana Lúcia Oliveira
- Department of Neurology, Hospital de Vila Franca de Xira, Lisbon, Portugal.
| | - Miguel Coelho
- Department of Neuroscience and Mental Health, Neurology, Centro Hospitalar Lisboa Norte, Lisbon, Portugal
- Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | - Leonor Correia Guedes
- Department of Neuroscience and Mental Health, Neurology, Centro Hospitalar Lisboa Norte, Lisbon, Portugal
- Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | | | - Herculano Carvalho
- Department of Neurosurgery, Centro Hospitalar Lisboa Norte, Lisbon, Portugal
| | | |
Collapse
|
7
|
Hunter N, Allen D, Xiao D, Cox M, Jain K. Patient education resources for oral mucositis: a google search and ChatGPT analysis. Eur Arch Otorhinolaryngol 2024:10.1007/s00405-024-08913-5. [PMID: 39198303 DOI: 10.1007/s00405-024-08913-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2024] [Accepted: 08/11/2024] [Indexed: 09/01/2024]
Abstract
PURPOSE Oral mucositis affects 90% of patients receiving chemotherapy or radiation for head and neck malignancies. Many patients use the internet to learn about their condition and treatments; however, the quality of online resources is not guaranteed. Our objective was to determine the most common Google searches related to "oral mucositis" and assess the quality and readability of available resources compared to ChatGPT-generated responses. METHODS Data related to Google searches for "oral mucositis" were analyzed. People Also Ask (PAA) questions (generated by Google) related to searches for "oral mucositis" were documented. Google resources were rated on quality, understandability, ease of reading, and reading grade level using the Journal of the American Medical Association benchmark criteria, Patient Education Materials Assessment Tool, Flesch Reading Ease Score, and Flesh-Kincaid Grade Level, respectively. ChatGPT-generated responses to the most popular PAA questions were rated using identical metrics. RESULTS Google search popularity for "oral mucositis" has significantly increased since 2004. 78% of the Google resources answered the associated PAA question, and 6% met the criteria for universal readability. 100% of the ChatGPT-generated responses answered the prompt, and 20% met the criteria for universal readability when asked to write for the appropriate audience. CONCLUSION Most resources provided by Google do not meet the criteria for universal readability. When prompted specifically, ChatGPT-generated responses were consistently more readable than Google resources. After verification of accuracy by healthcare professionals, ChatGPT could be a reasonable alternative to generate universally readable patient education resources.
Collapse
Affiliation(s)
- Nathaniel Hunter
- McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - David Allen
- Department of Otorhinolaryngology-Head and Neck Surgery, The University of Texas Health Science Center at Houston, Houston, TX, USA.
| | - Daniel Xiao
- McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Madisyn Cox
- McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Kunal Jain
- Department of Otorhinolaryngology-Head and Neck Surgery, The University of Texas Health Science Center at Houston, Houston, TX, USA
| |
Collapse
|
8
|
Sciberras M, Farrugia Y, Gordon H, Furfaro F, Allocca M, Torres J, Arebi N, Fiorino G, Iacucci M, Verstockt B, Magro F, Katsanos K, Busuttil J, De Giovanni K, Fenech VA, Chetcuti Zammit S, Ellul P. Accuracy of Information given by ChatGPT for Patients with Inflammatory Bowel Disease in Relation to ECCO Guidelines. J Crohns Colitis 2024; 18:1215-1221. [PMID: 38520394 DOI: 10.1093/ecco-jcc/jjae040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/09/2023] [Revised: 01/28/2024] [Accepted: 03/22/2024] [Indexed: 03/25/2024]
Abstract
BACKGROUND As acceptance of artificial intelligence [AI] platforms increases, more patients will consider these tools as sources of information. The ChatGPT architecture utilizes a neural network to process natural language, thus generating responses based on the context of input text. The accuracy and completeness of ChatGPT3.5 in the context of inflammatory bowel disease [IBD] remains unclear. METHODS In this prospective study, 38 questions worded by IBD patients were inputted into ChatGPT3.5. The following topics were covered: [1] Crohn's disease [CD], ulcerative colitis [UC], and malignancy; [2] maternal medicine; [3] infection and vaccination; and [4] complementary medicine. Responses given by ChatGPT were assessed for accuracy [1-completely incorrect to 5-completely correct] and completeness [3-point Likert scale; range 1-incomplete to 3-complete] by 14 expert gastroenterologists, in comparison with relevant ECCO guidelines. RESULTS In terms of accuracy, most replies [84.2%] had a median score of ≥4 (interquartile range [IQR]: 2) and a mean score of 3.87 [SD: ±0.6]. For completeness, 34.2% of the replies had a median score of 3 and 55.3% had a median score of between 2 and <3. Overall, the mean rating was 2.24 [SD: ±0.4, median: 2, IQR: 1]. Though groups 3 and 4 had a higher mean for both accuracy and completeness, there was no significant scoring variation between the four question groups [Kruskal-Wallis test p > 0.05]. However, statistical analysis for the different individual questions revealed a significant difference for both accuracy [p < 0.001] and completeness [p < 0.001]. The questions which rated the highest for both accuracy and completeness were related to smoking, while the lowest rating was related to screening for malignancy and vaccinations especially in the context of immunosuppression and family planning. CONCLUSION This is the first study to demonstrate the capability of an AI-based system to provide accurate and comprehensive answers to real-world patient queries in IBD. AI systems may serve as a useful adjunct for patients, in addition to standard of care in clinics and validated patient information resources. However, responses in specialist areas may deviate from evidence-based guidance and the replies need to give more firm advice.
Collapse
Affiliation(s)
- Martina Sciberras
- Department of Medicine, Division of Gastroenterology, Mater Dei Hospital, Msida, Malta
| | - Yvette Farrugia
- Department of Medicine, Division of Gastroenterology, Mater Dei Hospital, Msida, Malta
| | - Hannah Gordon
- Department of Gastroenterology, Barts Health NHS Trust, London, UK
- Translational Gastroenterology and Liver Unit, John Radcliffe Hospital, University of Oxford, Oxford, UK
| | - Federica Furfaro
- IRCCS OSPEDALE San Raffaele, Gastroenterology and Endoscopy, IBD Center, Milan, Italy
| | - Mariangela Allocca
- IRCCS OSPEDALE San Raffaele, Gastroenterology and Endoscopy, IBD Center, Milan, Italy
| | - Joana Torres
- Division of Gastroenterology, Hospital da Luz, Lisbon, Portugal
- Division of Gastroenterology, Hospital Beatriz Ângelo, Loures, Portugal
- Faculdade de Medicina, Universidade de Lisboa, Lisbon, Portugal
| | - Naila Arebi
- Department of Inflammatory Bowel Disease, St Mark's National Bowel Hospital, London, UK
- Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK
| | - Gionata Fiorino
- IRCCS OSPEDALE San Raffaele, Gastroenterology and Endoscopy, IBD Center, Milan, Italy
- IBD Unit, San Camillo-Forlanini Hospital, Rome, Italy
| | - Marietta Iacucci
- APC Microbiome Ireland, College of Medicine and Health, University College of Cork, Cork, Ireland
| | - Bram Verstockt
- Department of Gastroenterology and Hepatology, University Hospitals Leuven, KU Leuven, Leuven, Belgium
- Department of Chronic Diseases and Metabolism, KU Leuven, Leuven, Belgium
| | - Fernando Magro
- CINTESIS@RISE, Faculty of Medicine of the University of Porto, Porto, Portugal
| | - Kostas Katsanos
- Division of Gastroenterology, Department of Internal Medicine, Faculty of Medicine, University of Ioannina School of Health Sciences, Ioannina, Greece
| | | | | | - Valerie Anne Fenech
- Department of Medicine, Division of Gastroenterology, Mater Dei Hospital, Msida, Malta
| | | | - Pierre Ellul
- Department of Medicine, Division of Gastroenterology, Mater Dei Hospital, Msida, Malta
| |
Collapse
|
9
|
Cavnar Helvaci B, Hepsen S, Candemir B, Boz O, Durantas H, Houssein M, Cakal E. Assessing the accuracy and reliability of ChatGPT's medical responses about thyroid cancer. Int J Med Inform 2024; 191:105593. [PMID: 39151245 DOI: 10.1016/j.ijmedinf.2024.105593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2024] [Revised: 08/04/2024] [Accepted: 08/11/2024] [Indexed: 08/19/2024]
Abstract
PURPOSE ChatGPT has the potential to offer patient-friendly support. Thyroid carcinoma has become increasingly prevalent in recent years. This study aimed to assess ChatGPT's accuracy and adequacy in answering questions about information, management, and emotional support related to thyroid cancer. METHODS We conducted a three-step study. In the first step, ChatGPT responded to 30 questions about thyroid cancer. In the second step, we presented three different cases of thyroid cancer patients and asked ChatGPT about their diagnosis, treatment management, and follow-up. In the third step, we inquired about emotional support for patients and their families. Three expert endocrinologists graded these responses according to ATA guidelines. RESULTS We showed that ChatGPT regurgitated extensive knowledge of thyroid cancer (76.66% correct), but only small proportions (6.66%) were labeled as mixed with correct and incorrect/outdated data. However, it was inadequate in the evaluation of clinical cases of thyroid cancer. It mentioned treatment and follow-up recommendations in a general framework, not patient-specific. Also, it provided practical and multifaceted emotional support advice to patients and caregivers regarding the next steps and adjusting to a new diagnosis. CONCLUSION Our study is the first to evaluate the competence and reliability of ChatGPT in thyroid cancer. Although ChatGPT is moderately competent in obtaining information about thyroid cancer, it has not yet been determined to be sufficiently competent and reliable in case management. It has been found effective in guiding patients and their relatives regarding emotional support.
Collapse
Affiliation(s)
- Burcak Cavnar Helvaci
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey.
| | - Sema Hepsen
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey
| | - Burcu Candemir
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey
| | - Ogulcan Boz
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey
| | - Halil Durantas
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey
| | - Mehdi Houssein
- Department of Endocrinology and Metabolism, Ankara Sincan Training and Research Hospital, Ankara, Turkey
| | - Erman Cakal
- Department of Endocrinology and Metabolism, Ankara Etlik City Hospital, Ankara, Turkey
| |
Collapse
|
10
|
Cherrez-Ojeda I, Gallardo-Bastidas JC, Robles-Velasco K, Osorio MF, Velez Leon EM, Leon Velastegui M, Pauletto P, Aguilar-Díaz FC, Squassi A, González Eras SP, Cordero Carrasco E, Chavez Gonzalez KL, Calderon JC, Bousquet J, Bedbrook A, Faytong-Haro M. Understanding Health Care Students' Perceptions, Beliefs, and Attitudes Toward AI-Powered Language Models: Cross-Sectional Study. JMIR MEDICAL EDUCATION 2024; 10:e51757. [PMID: 39137029 DOI: 10.2196/51757] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 09/26/2023] [Accepted: 04/30/2024] [Indexed: 08/15/2024]
Abstract
BACKGROUND ChatGPT was not intended for use in health care, but it has potential benefits that depend on end-user understanding and acceptability, which is where health care students become crucial. There is still a limited amount of research in this area. OBJECTIVE The primary aim of our study was to assess the frequency of ChatGPT use, the perceived level of knowledge, the perceived risks associated with its use, and the ethical issues, as well as attitudes toward the use of ChatGPT in the context of education in the field of health. In addition, we aimed to examine whether there were differences across groups based on demographic variables. The second part of the study aimed to assess the association between the frequency of use, the level of perceived knowledge, the level of risk perception, and the level of perception of ethics as predictive factors for participants' attitudes toward the use of ChatGPT. METHODS A cross-sectional survey was conducted from May to June 2023 encompassing students of medicine, nursing, dentistry, nutrition, and laboratory science across the Americas. The study used descriptive analysis, chi-square tests, and ANOVA to assess statistical significance across different categories. The study used several ordinal logistic regression models to analyze the impact of predictive factors (frequency of use, perception of knowledge, perception of risk, and ethics perception scores) on attitude as the dependent variable. The models were adjusted for gender, institution type, major, and country. Stata was used to conduct all the analyses. RESULTS Of 2661 health care students, 42.99% (n=1144) were unaware of ChatGPT. The median score of knowledge was "minimal" (median 2.00, IQR 1.00-3.00). Most respondents (median 2.61, IQR 2.11-3.11) regarded ChatGPT as neither ethical nor unethical. Most participants (median 3.89, IQR 3.44-4.34) "somewhat agreed" that ChatGPT (1) benefits health care settings, (2) provides trustworthy data, (3) is a helpful tool for clinical and educational medical information access, and (4) makes the work easier. In total, 70% (7/10) of people used it for homework. As the perceived knowledge of ChatGPT increased, there was a stronger tendency with regard to having a favorable attitude toward ChatGPT. Higher ethical consideration perception ratings increased the likelihood of considering ChatGPT as a source of trustworthy health care information (odds ratio [OR] 1.620, 95% CI 1.498-1.752), beneficial in medical issues (OR 1.495, 95% CI 1.452-1.539), and useful for medical literature (OR 1.494, 95% CI 1.426-1.564; P<.001 for all results). CONCLUSIONS Over 40% of American health care students (1144/2661, 42.99%) were unaware of ChatGPT despite its extensive use in the health field. Our data revealed the positive attitudes toward ChatGPT and the desire to learn more about it. Medical educators must explore how chatbots may be included in undergraduate health care education programs.
Collapse
Affiliation(s)
- Ivan Cherrez-Ojeda
- Universidad Espiritu Santo, Samborondon, Ecuador
- Respiralab Research Group, Guayaquil, Ecuador
| | | | - Karla Robles-Velasco
- Universidad Espiritu Santo, Samborondon, Ecuador
- Respiralab Research Group, Guayaquil, Ecuador
| | - María F Osorio
- Universidad Espiritu Santo, Samborondon, Ecuador
- Respiralab Research Group, Guayaquil, Ecuador
| | | | | | | | - F C Aguilar-Díaz
- Departamento Salud Pública, Escuela Nacional de Estudios Superiores, Universidad Nacional Autónoma de México, Guanajuato, Mexico
| | - Aldo Squassi
- Universidad de Buenos Aires, Facultad de Odontologìa, Cátedra de Odontología Preventiva y Comunitaria, Buenos Aires, Argentina
| | | | - Erita Cordero Carrasco
- Departamento de cirugía y traumatología bucal y maxilofacial, Universidad de Chile, Santiago, Chile
| | | | - Juan C Calderon
- Universidad Espiritu Santo, Samborondon, Ecuador
- Respiralab Research Group, Guayaquil, Ecuador
| | - Jean Bousquet
- Institute of Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Fraunhofer Institute for Translational Medicine and Pharmacology ITMP, Allergology and Immunology, Berlin, Germany
- MASK-air, Montpellier, France
| | | | - Marco Faytong-Haro
- Respiralab Research Group, Guayaquil, Ecuador
- Universidad Estatal de Milagro, Cdla Universitaria "Dr. Rómulo Minchala Murillo", Milagro, Ecuador
- Ecuadorian Development Research Lab, Daule, Ecuador
| |
Collapse
|
11
|
Spina A, Andalib S, Flores D, Vermani R, Halaseh FF, Nelson AM. Evaluation of Generative Language Models in Personalizing Medical Information: Instrument Validation Study. JMIR AI 2024; 3:e54371. [PMID: 39137416 DOI: 10.2196/54371] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 04/28/2024] [Accepted: 06/29/2024] [Indexed: 08/15/2024]
Abstract
BACKGROUND Although uncertainties exist regarding implementation, artificial intelligence-driven generative language models (GLMs) have enormous potential in medicine. Deployment of GLMs could improve patient comprehension of clinical texts and improve low health literacy. OBJECTIVE The goal of this study is to evaluate the potential of ChatGPT-3.5 and GPT-4 to tailor the complexity of medical information to patient-specific input education level, which is crucial if it is to serve as a tool in addressing low health literacy. METHODS Input templates related to 2 prevalent chronic diseases-type II diabetes and hypertension-were designed. Each clinical vignette was adjusted for hypothetical patient education levels to evaluate output personalization. To assess the success of a GLM (GPT-3.5 and GPT-4) in tailoring output writing, the readability of pre- and posttransformation outputs were quantified using the Flesch reading ease score (FKRE) and the Flesch-Kincaid grade level (FKGL). RESULTS Responses (n=80) were generated using GPT-3.5 and GPT-4 across 2 clinical vignettes. For GPT-3.5, FKRE means were 57.75 (SD 4.75), 51.28 (SD 5.14), 32.28 (SD 4.52), and 28.31 (SD 5.22) for 6th grade, 8th grade, high school, and bachelor's, respectively; FKGL mean scores were 9.08 (SD 0.90), 10.27 (SD 1.06), 13.4 (SD 0.80), and 13.74 (SD 1.18). GPT-3.5 only aligned with the prespecified education levels at the bachelor's degree. Conversely, GPT-4's FKRE mean scores were 74.54 (SD 2.6), 71.25 (SD 4.96), 47.61 (SD 6.13), and 13.71 (SD 5.77), with FKGL mean scores of 6.3 (SD 0.73), 6.7 (SD 1.11), 11.09 (SD 1.26), and 17.03 (SD 1.11) for the same respective education levels. GPT-4 met the target readability for all groups except the 6th-grade FKRE average. Both GLMs produced outputs with statistically significant differences (P<.001; 8th grade P<.001; high school P<.001; bachelors P=.003; FKGL: 6th grade P=.001; 8th grade P<.001; high school P<.001; bachelors P<.001) between mean FKRE and FKGL across input education levels. CONCLUSIONS GLMs can change the structure and readability of medical text outputs according to input-specified education. However, GLMs categorize input education designation into 3 broad tiers of output readability: easy (6th and 8th grade), medium (high school), and difficult (bachelor's degree). This is the first result to suggest that there are broader boundaries in the success of GLMs in output text simplification. Future research must establish how GLMs can reliably personalize medical texts to prespecified education levels to enable a broader impact on health care literacy.
Collapse
Affiliation(s)
- Aidin Spina
- School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Saman Andalib
- School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Daniel Flores
- School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Rishi Vermani
- School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Faris F Halaseh
- School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Ariana M Nelson
- School of Medicine, University of California, Irvine, Irvine, CA, United States
- Department of Anesthesiology and Perioperative Care, University of California, Irvine, Irvine, CA, United States
| |
Collapse
|
12
|
Soleimani M, Seyyedi N, Ayyoubzadeh SM, Kalhori SRN, Keshavarz H. Practical Evaluation of ChatGPT Performance for Radiology Report Generation. Acad Radiol 2024:S1076-6332(24)00454-9. [PMID: 39142976 DOI: 10.1016/j.acra.2024.07.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Revised: 07/12/2024] [Accepted: 07/14/2024] [Indexed: 08/16/2024]
Abstract
RATIONALE AND OBJECTIVES The process of generating radiology reports is often time-consuming and labor-intensive, prone to incompleteness, heterogeneity, and errors. By employing natural language processing (NLP)-based techniques, this study explores the potential for enhancing the efficiency of radiology report generation through the remarkable capabilities of ChatGPT (Generative Pre-training Transformer), a prominent large language model (LLM). MATERIALS AND METHODS Using a sample of 1000 records from the Medical Information Mart for Intensive Care (MIMIC) Chest X-ray Database, this investigation employed Claude.ai to extract initial radiological report keywords. ChatGPT then generated radiology reports using a consistent 3-step prompt template outline. Various lexical and sentence similarity techniques were employed to evaluate the correspondence between the AI assistant-generated reports and reference reports authored by medical professionals. RESULTS Results showed varying performance among NLP models, with Bart (Bidirectional and Auto-Regressive Transformers) and XLM (Cross-lingual Language Model) displaying high proficiency (mean similarity scores up to 99.3%), closely mirroring physician reports. Conversely, DeBERTa (Decoding-enhanced BERT with disentangled attention) and sequence-matching models scored lower, indicating less alignment with medical language. In the Impression section, the Word-Embedding model excelled with a mean similarity of 84.4%, while others like the Jaccard index showed lower performance. CONCLUSION Overall, the study highlights significant variations across NLP models in their ability to generate radiology reports consistent with medical professionals' language. Pairwise comparisons and Kruskal-Wallis tests confirmed these differences, emphasizing the need for careful selection and evaluation of NLP models in radiology report generation. This research underscores the potential of ChatGPT to streamline and improve the radiology reporting process, with implications for enhancing efficiency and accuracy in clinical practice.
Collapse
Affiliation(s)
- Mohsen Soleimani
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran
| | - Navisa Seyyedi
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran.
| | - Seyed Mohammad Ayyoubzadeh
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran; Health Information Management Research Centre, Tehran University of Medical Sciences, Tehran, Iran
| | - Sharareh Rostam Niakan Kalhori
- Department of Health Information Management and Medical Informatics, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran; Peter L. Reichertz Institute for Medical Informatics, TU Braunschweig and Hannover Medical School, Braunschweig, Germany
| | - Hamidreza Keshavarz
- Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
| |
Collapse
|
13
|
Daza J, Bezerra LS, Santamaría L, Rueda-Esteban R, Bantel H, Girala M, Ebert M, Van Bömmel F, Geier A, Aldana AG, Yau K, Alvares-da-Silva M, Peck-Radosavljevic M, Ridruejo E, Weinmann A, Teufel A. Evaluation of four chatbots in autoimmune liver disease: A comparative analysis. Ann Hepatol 2024:101537. [PMID: 39147133 DOI: 10.1016/j.aohep.2024.101537] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/26/2024] [Revised: 05/24/2024] [Accepted: 06/13/2024] [Indexed: 08/17/2024]
Abstract
INTRODUCTION AND OBJECTIVES Autoimmune liver diseases (AILDs) are rare and require precise evaluation, which is often challenging for medical providers. Chatbots are innovative solutions to assist healthcare professionals in clinical management. In our study, ten liver specialists systematically evaluated four chatbots to determine their utility as clinical decision support tools in the field of AILDs. MATERIALS AND METHODS We constructed a 56-question questionnaire focusing on AILD evaluation, diagnosis, and management of Autoimmune Hepatitis (AIH), Primary Biliary Cholangitis (PBC), and Primary Sclerosing Cholangitis (PSC). Four chatbots -ChatGPT 3.5, Claude, Microsoft Copilot, and Google Bard- were presented with the questions in their free tiers in December 2023. Responses underwent critical evaluation by ten liver specialists using a standardized 1 to 10 Likert scale. The analysis included mean scores, the number of highest-rated replies, and the identification of common shortcomings in chatbots performance. RESULTS Among the assessed chatbots, specialists rated Claude highest with a mean score of 7.37 (SD = 1.91), followed by ChatGPT (7.17, SD = 1.89), Microsoft Copilot (6.63, SD = 2.10), and Google Bard (6.52, SD = 2.27). Claude also excelled with 27 best-rated replies, outperforming ChatGPT (20), while Microsoft Copilot and Google Bard lagged with only 6 and 9, respectively. Common deficiencies included listing details over specific advice, limited dosing options, inaccuracies for pregnant patients, insufficient recent data, over-reliance on CT and MRI imaging, and inadequate discussion regarding off-label use and fibrates in PBC treatment. Notably, internet access for Microsoft Copilot and Google Bard did not enhance precision compared to pre-trained models. CONCLUSIONS Chatbots hold promise in AILD support, but our study underscores key areas for improvement. Refinement is needed in providing specific advice, accuracy, and focused up-to-date information. Addressing these shortcomings is essential for enhancing the utility of chatbots in AILD management, guiding future development, and ensuring their effectiveness as clinical decision-support tools.
Collapse
Affiliation(s)
- Jimmy Daza
- Department of Medicine II, Division of Hepatology, Division of Clinical Bioinformatics, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Lucas Soares Bezerra
- Department of Medicine II, Division of Hepatology, Division of Clinical Bioinformatics, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | | | | | - Heike Bantel
- Department of Gastroenterology, Hepatology, Infectious Diseases and Endocrinology, Hannover Medical School, Hannover, Germany
| | - Marcos Girala
- Department of Gastroenterology, Hospital de Clínicas, Universidad Nacional de Asunción, Asunción, Paraguay
| | - Matthias Ebert
- Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Florian Van Bömmel
- Department of Medicine II, Clinic of Gastroenterology, Hepatology, Infectious Diseases and Pneumology, Leipzig University Medical Center, Leipzig, Germany
| | - Andreas Geier
- Department of Internal Medicine II, Division of Hepatology, University Hospital Würzburg, Würzburg, Germany
| | - Andres Gomez Aldana
- Texas Liver Institute, University of Texas Health Science Center, San Antonio, United States
| | - Kevin Yau
- Division of Nephrology, Department of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Mario Alvares-da-Silva
- Department of Gastroenterology, Hospital de Clinicas de Porto Alegre, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
| | - Markus Peck-Radosavljevic
- Internal Medicine and Gastroenterology (IMuG), Clinic Klagenfurt am Woerthersee, Klagenfurt, Austria
| | - Ezequiel Ridruejo
- Department of Medicine, Section of Hepatology, Centro de Educación Médica e Investigaciones Clínicas Norberto Quirno "CEMIC", Buenos Aires, Argentina
| | - Arndt Weinmann
- Department of Internal Medicine I, University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany
| | - Andreas Teufel
- Department of Medicine II, Division of Hepatology, Division of Clinical Bioinformatics, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany.
| |
Collapse
|
14
|
Schütz P, Lob S, Chahed H, Dathe L, Löwer M, Reiß H, Weigel A, Albrecht J, Tokgöz P, Dockweiler C. ChatGPT as an Information Source for Patients with Migraines: A Qualitative Case Study. Healthcare (Basel) 2024; 12:1594. [PMID: 39201153 PMCID: PMC11354001 DOI: 10.3390/healthcare12161594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2024] [Revised: 08/07/2024] [Accepted: 08/08/2024] [Indexed: 09/02/2024] Open
Abstract
Migraines are one of the most common and expensive neurological diseases worldwide. Non-pharmacological and digitally delivered treatment options have long been used in the treatment of migraines. For instance, migraine management tools, online migraine diagnosis or digitally networked patients have been used. Recently, applications of ChatGPT are used in fields of healthcare ranging from identifying potential research topics to assisting professionals in clinical diagnosis and helping patients in managing their health. Despite advances in migraine management, only a minority of patients are adequately informed and treated. It is important to provide these patients with information to help them manage the symptoms and their daily activities. The primary aim of this case study was to examine the appropriateness of ChatGPT to handle symptom descriptions responsibly, suggest supplementary assistance from credible sources, provide valuable perspectives on treatment options, and exhibit potential influences on daily life for patients with migraines. Using a deductive, qualitative study, ten interactions with ChatGPT on different migraine types were analyzed through semi-structured interviews. ChatGPT provided relevant information aligned with common scientific patient resources. Responses were generally intelligible and situationally appropriate, providing personalized insights despite occasional discrepancies in interaction. ChatGPT's empathetic tone and linguistic clarity encouraged user engagement. However, source citations were found to be inconsistent and, in some cases, not comprehensible, which affected the overall comprehensibility of the information. ChatGPT might be promising for patients seeking information on migraine conditions. Its user-specific responses demonstrate potential benefits over static web-based sources. However, reproducibility and accuracy issues highlight the need for digital health literacy. The findings underscore the necessity for continuously evaluating AI systems and their broader societal implications in health communication.
Collapse
Affiliation(s)
- Pascal Schütz
- Department Digital Health Sciences and Biomedicine, Professorship of Digital Public Health, School of Life Sciences, University of Siegen, 57076 Siegen, Germany; (S.L.); (H.C.); (L.D.); (M.L.); (H.R.); (A.W.); (J.A.); (P.T.); (C.D.)
| | | | | | | | | | | | | | | | | | | |
Collapse
|
15
|
Alqudah AA, Aleshawi AJ, Baker M, Alnajjar Z, Ayasrah I, Ta’ani Y, Al Salkhadi M, Aljawarneh S. Evaluating accuracy and reproducibility of ChatGPT responses to patient-based questions in Ophthalmology: An observational study. Medicine (Baltimore) 2024; 103:e39120. [PMID: 39121263 PMCID: PMC11315477 DOI: 10.1097/md.0000000000039120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Accepted: 07/08/2024] [Indexed: 08/11/2024] Open
Abstract
Chat Generative Pre-Trained Transformer (ChatGPT) is an online large language model that appears to be a popular source of health information, as it can provide patients with answers in the form of human-like text, although the accuracy and safety of its responses are not evident. This study aims to evaluate the accuracy and reproducibility of ChatGPT responses to patients-based questions in ophthalmology. We collected 150 questions from the "Ask an ophthalmologist" page of the American Academy of Ophthalmology, which were reviewed and refined by two ophthalmologists for their eligibility. Each question was inputted into ChatGPT twice using the "new chat" option. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Totally, 117 questions were inputted into ChatGPT, which provided "comprehensive" responses to 70/117 (59.8%) of questions. Concerning reproducibility, it was defined as no difference in grading categories (1 and 2 vs 3 and 4) between the 2 responses for each question. ChatGPT provided reproducible responses to 91.5% of questions. This study shows moderate accuracy and reproducibility of ChatGPT responses to patients' questions in ophthalmology. ChatGPT may be-after more modifications-a supplementary health information source, which should be used as an adjunct, but not a substitute, to medical advice. The reliability of ChatGPT should undergo more investigations.
Collapse
Affiliation(s)
- Asem A. Alqudah
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| | | | - Mohammed Baker
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| | - Zaina Alnajjar
- Faculty of Medicine, Hashemite University, Zarqa, Jordan
| | - Ibrahim Ayasrah
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| | - Yaqoot Ta’ani
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| | - Mohammad Al Salkhadi
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| | - Shaima’a Aljawarneh
- Faculty of Medicine, Jordan University of Science and Technology (JUST), Irbid, Jordan
| |
Collapse
|
16
|
Fatima A, Shafique MA, Alam K, Fadlalla Ahmed TK, Mustafa MS. ChatGPT in medicine: A cross-disciplinary systematic review of ChatGPT's (artificial intelligence) role in research, clinical practice, education, and patient interaction. Medicine (Baltimore) 2024; 103:e39250. [PMID: 39121303 PMCID: PMC11315549 DOI: 10.1097/md.0000000000039250] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 07/19/2024] [Indexed: 08/11/2024] Open
Abstract
BACKGROUND ChatGPT, a powerful AI language model, has gained increasing prominence in medicine, offering potential applications in healthcare, clinical decision support, patient communication, and medical research. This systematic review aims to comprehensively assess the applications of ChatGPT in healthcare education, research, writing, patient communication, and practice while also delineating potential limitations and areas for improvement. METHOD Our comprehensive database search retrieved relevant papers from PubMed, Medline and Scopus. After the screening process, 83 studies met the inclusion criteria. This review includes original studies comprising case reports, analytical studies, and editorials with original findings. RESULT ChatGPT is useful for scientific research and academic writing, and assists with grammar, clarity, and coherence. This helps non-English speakers and improves accessibility by breaking down linguistic barriers. However, its limitations include probable inaccuracy and ethical issues, such as bias and plagiarism. ChatGPT streamlines workflows and offers diagnostic and educational potential in healthcare but exhibits biases and lacks emotional sensitivity. It is useful in inpatient communication, but requires up-to-date data and faces concerns about the accuracy of information and hallucinatory responses. CONCLUSION Given the potential for ChatGPT to transform healthcare education, research, and practice, it is essential to approach its adoption in these areas with caution due to its inherent limitations.
Collapse
Affiliation(s)
- Afia Fatima
- Department of Medicine, Jinnah Sindh Medical University, Karachi, Pakistan
| | | | - Khadija Alam
- Department of Medicine, Liaquat National Medical College, Karachi, Pakistan
| | | | | |
Collapse
|
17
|
Wang Y, Chen Y, Sheng J. Assessing ChatGPT as a Medical Consultation Assistant for Chronic Hepatitis B: Cross-Language Study of English and Chinese. JMIR Med Inform 2024; 12:e56426. [PMID: 39115930 PMCID: PMC11342014 DOI: 10.2196/56426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Revised: 05/24/2024] [Accepted: 07/21/2024] [Indexed: 08/10/2024] Open
Abstract
BACKGROUND Chronic hepatitis B (CHB) imposes substantial economic and social burdens globally. The management of CHB involves intricate monitoring and adherence challenges, particularly in regions like China, where a high prevalence of CHB intersects with health care resource limitations. This study explores the potential of ChatGPT-3.5, an emerging artificial intelligence (AI) assistant, to address these complexities. With notable capabilities in medical education and practice, ChatGPT-3.5's role is examined in managing CHB, particularly in regions with distinct health care landscapes. OBJECTIVE This study aimed to uncover insights into ChatGPT-3.5's potential and limitations in delivering personalized medical consultation assistance for CHB patients across diverse linguistic contexts. METHODS Questions sourced from published guidelines, online CHB communities, and search engines in English and Chinese were refined, translated, and compiled into 96 inquiries. Subsequently, these questions were presented to both ChatGPT-3.5 and ChatGPT-4.0 in independent dialogues. The responses were then evaluated by senior physicians, focusing on informativeness, emotional management, consistency across repeated inquiries, and cautionary statements regarding medical advice. Additionally, a true-or-false questionnaire was employed to further discern the variance in information accuracy for closed questions between ChatGPT-3.5 and ChatGPT-4.0. RESULTS Over half of the responses (228/370, 61.6%) from ChatGPT-3.5 were considered comprehensive. In contrast, ChatGPT-4.0 exhibited a higher percentage at 74.5% (172/222; P<.001). Notably, superior performance was evident in English, particularly in terms of informativeness and consistency across repeated queries. However, deficiencies were identified in emotional management guidance, with only 3.2% (6/186) in ChatGPT-3.5 and 8.1% (15/154) in ChatGPT-4.0 (P=.04). ChatGPT-3.5 included a disclaimer in 10.8% (24/222) of responses, while ChatGPT-4.0 included a disclaimer in 13.1% (29/222) of responses (P=.46). When responding to true-or-false questions, ChatGPT-4.0 achieved an accuracy rate of 93.3% (168/180), significantly surpassing ChatGPT-3.5's accuracy rate of 65.0% (117/180) (P<.001). CONCLUSIONS In this study, ChatGPT demonstrated basic capabilities as a medical consultation assistant for CHB management. The choice of working language for ChatGPT-3.5 was considered a potential factor influencing its performance, particularly in the use of terminology and colloquial language, and this potentially affects its applicability within specific target populations. However, as an updated model, ChatGPT-4.0 exhibits improved information processing capabilities, overcoming the language impact on information accuracy. This suggests that the implications of model advancement on applications need to be considered when selecting large language models as medical consultation assistants. Given that both models performed inadequately in emotional guidance management, this study highlights the importance of providing specific language training and emotional management strategies when deploying ChatGPT for medical purposes. Furthermore, the tendency of these models to use disclaimers in conversations should be further investigated to understand the impact on patients' experiences in practical applications.
Collapse
Affiliation(s)
- Yijie Wang
- State Key Laboratory for Diagnosis and Treatment of Infectious Diseases, Collaborative Innovation Center for Diagnosis and Treatment of Infectious Disease, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Yining Chen
- Department of Urology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Jifang Sheng
- State Key Laboratory for Diagnosis and Treatment of Infectious Diseases, Collaborative Innovation Center for Diagnosis and Treatment of Infectious Disease, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China
| |
Collapse
|
18
|
Ye F, Zhang H, Luo X, Wu T, Yang Q, Shi Z. Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis. Otolaryngol Head Neck Surg 2024; 171:571-577. [PMID: 38796735 DOI: 10.1002/ohn.832] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 04/30/2024] [Accepted: 05/04/2024] [Indexed: 05/28/2024]
Abstract
OBJECTIVE This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions. STUDY DESIGN This is a cross-sectional study. SETTING Each question was inputted as a separate, independent prompt. METHODS Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT-3.5 and GPT-4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer. RESULTS Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT-4.0's accuracy in responding to AR-related questions significantly exceeded that of GPT-3.5, but distinction not evident in CRS-related questions. Patient-originated questions had a significantly higher accuracy compared to doctor-originated questions when utilizing GPT-4.0 to respond to AR-related questions. This discrepancy was not observed with GPT-3.5 or in the context of CRS-related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT-3.5 and GPT-4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese. CONCLUSION Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.
Collapse
Affiliation(s)
- Fan Ye
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - He Zhang
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Xin Luo
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Tong Wu
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Qintai Yang
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Naso-Orbital-Maxilla and Skull Base Center, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Key Laboratory of Airway Inflammatory Disease Research and Innovative Technology Translation, Guangzhou, China
| | - Zhaohui Shi
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Naso-Orbital-Maxilla and Skull Base Center, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
- Key Laboratory of Airway Inflammatory Disease Research and Innovative Technology Translation, Guangzhou, China
| |
Collapse
|
19
|
Cao JJ, Kwon DH, Ghaziani TT, Kwo P, Tse G, Kesselman A, Kamaya A, Tse JR. Large language models' responses to liver cancer surveillance, diagnosis, and management questions: accuracy, reliability, readability. Abdom Radiol (NY) 2024:10.1007/s00261-024-04501-7. [PMID: 39088019 DOI: 10.1007/s00261-024-04501-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2024] [Revised: 07/10/2024] [Accepted: 07/13/2024] [Indexed: 08/02/2024]
Abstract
PURPOSE To assess the accuracy, reliability, and readability of publicly available large language models in answering fundamental questions on hepatocellular carcinoma diagnosis and management. METHODS Twenty questions on liver cancer diagnosis and management were asked in triplicate to ChatGPT-3.5 (OpenAI), Gemini (Google), and Bing (Microsoft). Responses were assessed by six fellowship-trained physicians from three academic liver transplant centers who actively diagnose and/or treat liver cancer. Responses were categorized as accurate (score 1; all information is true and relevant), inadequate (score 0; all information is true, but does not fully answer the question or provides irrelevant information), or inaccurate (score - 1; any information is false). Means with standard deviations were recorded. Responses were considered as a whole accurate if mean score was > 0 and reliable if mean score was > 0 across all responses for the single question. Responses were also quantified for readability using the Flesch Reading Ease Score and Flesch-Kincaid Grade Level. Readability and accuracy across 60 responses were compared using one-way ANOVAs with Tukey's multiple comparison tests. RESULTS Of the twenty questions, ChatGPT answered nine (45%), Gemini answered 12 (60%), and Bing answered six (30%) questions accurately; however, only six (30%), eight (40%), and three (15%), respectively, were both accurate and reliable. There were no significant differences in accuracy between any chatbot. ChatGPT responses were the least readable (mean Flesch Reading Ease Score 29; college graduate), followed by Gemini (30; college) and Bing (40; college; p < 0.001). CONCLUSION Large language models provide complex responses to basic questions on hepatocellular carcinoma diagnosis and management that are seldomly accurate, reliable, or readable.
Collapse
Affiliation(s)
- Jennie J Cao
- Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Room H-1307, Stanford, CA, 94305, USA
| | - Daniel H Kwon
- Department of Medicine, San Francisco School of Medicine, University of California, 505 Parnassus Ave, MC1286C, San Francisco, CA, 94144, USA
| | - Tara T Ghaziani
- Department of Medicine, Stanford University School of Medicine, 430 Broadway St MC 6341, Redwood City, CA, 94063, USA
| | - Paul Kwo
- Department of Medicine, Stanford University School of Medicine, 430 Broadway St MC 6341, Redwood City, CA, 94063, USA
| | - Gary Tse
- Department of Radiological Sciences, Los Angeles David Geffen School of Medicine, University of California, 757 Westwood Plaza Los Angeles, Los Angeles, CA, 90095, USA
| | - Andrew Kesselman
- Department of Radiology, Stanford University School of Medicine, 875 Blake Wilbur Drive Palo Alto, Stanford, CA, 94304, USA
| | - Aya Kamaya
- Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Room H-1307, Stanford, CA, 94305, USA
| | - Justin R Tse
- Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Room H-1307, Stanford, CA, 94305, USA.
| |
Collapse
|
20
|
Magruder ML, Rodriguez AN, Wong JCJ, Erez O, Piuzzi NS, Scuderi GR, Slover JD, Oh JH, Schwarzkopf R, Chen AF, Iorio R, Goodman SB, Mont MA. Assessing Ability for ChatGPT to Answer Total Knee Arthroplasty-Related Questions. J Arthroplasty 2024; 39:2022-2027. [PMID: 38364879 DOI: 10.1016/j.arth.2024.02.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/09/2023] [Revised: 01/19/2024] [Accepted: 02/08/2024] [Indexed: 02/18/2024] Open
Abstract
BACKGROUND Artificial intelligence in the field of orthopaedics has been a topic of increasing interest and opportunity in recent years. Its applications are widespread both for physicians and patients, including use in clinical decision-making, in the operating room, and in research. In this study, we aimed to assess the quality of ChatGPT answers when asked questions related to total knee arthroplasty. METHODS ChatGPT prompts were created by turning 15 of the American Academy of Orthopaedic Surgeons Clinical Practice Guidelines into questions. An online survey was created, which included screenshots of each prompt and answers to the 15 questions. Surgeons were asked to grade ChatGPT answers from 1 to 5 based on their characteristics: (1) relevance, (2) accuracy, (3) clarity, (4) completeness, (5) evidence-based, and (6) consistency. There were 11 Adult Joint Reconstruction fellowship-trained surgeons who completed the survey. Questions were subclassified based on the subject of the prompt: (1) risk factors, (2) implant/intraoperative, and (3) pain/functional outcomes. The average and standard deviation for all answers, as well as for each subgroup, were calculated. Inter-rater reliability (IRR) was also calculated. RESULTS All answer characteristics were graded as being above average (ie, a score > 3). Relevance demonstrated the highest scores (4.43 ± 0.77) by surgeons surveyed, and consistency demonstrated the lowest scores (3.54 ± 1.10). ChatGPT prompts in the Risk Factors group demonstrated the best responses, while those in the Pain/Functional Outcome group demonstrated the lowest. The overall IRR was found to be 0.33 (poor reliability), with the highest IRR for relevance (0.43) and the lowest for evidence-based (0.28). CONCLUSIONS ChatGPT can answer questions regarding well-established clinical guidelines in total knee arthroplasty with above-average accuracy but demonstrates variable reliability. This investigation is the first step in understanding large language model artificial intelligence like ChatGPT and how well they perform in the field of arthroplasty.
Collapse
Affiliation(s)
- Matthew L Magruder
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, New York
| | - Ariel N Rodriguez
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, New York
| | - Jason C J Wong
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, New York
| | - Orry Erez
- Department of Orthopaedic Surgery, Maimonides Medical Center, Brooklyn, New York
| | - Nicolas S Piuzzi
- Department of Orthopaedic Surgery, Cleveland Clinic, Cleveland, Ohio
| | - Gil R Scuderi
- Department of Orthopaedic Surgery, Lenox Hill Hospital, Northwell Orthopaedic Institute, New York, New York
| | - James D Slover
- Department of Orthopaedic Surgery, Lenox Hill Hospital, Northwell Orthopaedic Institute, New York, New York
| | - Jason H Oh
- Department of Orthopaedic Surgery, Lenox Hill Hospital, Northwell Orthopaedic Institute, New York, New York
| | - Ran Schwarzkopf
- Department of Orthopaedic Surgery, NYU Langone Orthopedics, NYU Langone Health, New York, New York
| | - Antonia F Chen
- Department of Orthopaedic Surgery, Brigham and Women's Hospital, Boston, Massachusetts
| | - Richard Iorio
- Department of Orthopaedic Surgery, Brigham and Women's Hospital, Boston, Massachusetts
| | - Stuart B Goodman
- Department of Orthopaedic Surgery, Stanford University School of Medicine, Redwood City, California
| | - Michael A Mont
- Rubin Institute for Advanced Orthopedics, Sinai Hospital of Baltimore, Baltimore, Maryland
| |
Collapse
|
21
|
Hassona Y, Alqaisi D, Al-Haddad A, Georgakopoulou EA, Malamos D, Alrashdan MS, Sawair F. How good is ChatGPT at answering patients' questions related to early detection of oral (mouth) cancer? Oral Surg Oral Med Oral Pathol Oral Radiol 2024; 138:269-278. [PMID: 38714483 DOI: 10.1016/j.oooo.2024.04.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 03/22/2024] [Accepted: 04/14/2024] [Indexed: 05/10/2024]
Abstract
OBJECTIVES To examine the quality, reliability, readability, and usefulness of ChatGPT in promoting oral cancer early detection. STUDY DESIGN About 108 patient-oriented questions about oral cancer early detection were compiled from expert panels, professional societies, and web-based tools. Questions were categorized into 4 topic domains and ChatGPT 3.5 was asked each question independently. ChatGPT answers were evaluated regarding quality, readability, actionability, and usefulness using. Two experienced reviewers independently assessed each response. RESULTS Questions related to clinical appearance constituted 36.1% (n = 39) of the total questions. ChatGPT provided "very useful" responses to the majority of questions (75%; n = 81). The mean Global Quality Score was 4.24 ± 1.3 of 5. The mean reliability score was 23.17 ± 9.87 of 25. The mean understandability score was 76.6% ± 25.9% of 100, while the mean actionability score was 47.3% ± 18.9% of 100. The mean FKS reading ease score was 38.4% ± 29.9%, while the mean SMOG index readability score was 11.65 ± 8.4. No misleading information was identified among ChatGPT responses. CONCLUSION ChatGPT is an attractive and potentially useful resource for informing patients about early detection of oral cancer. Nevertheless, concerns do exist about readability and actionability of the offered information.
Collapse
Affiliation(s)
- Yazan Hassona
- Faculty of Dentistry, Centre for Oral Diseases Studies (CODS), Al-Ahliyya Amman University, Jordan; School of Dentistry, The University of Jordan, Jordan.
| | - Dua'a Alqaisi
- School of Dentistry, The University of Jordan, Jordan
| | | | - Eleni A Georgakopoulou
- Molecular Carcinogenesis Group, Department of Histology and Embryology, Medical School, National and Kapodistrian University of Athens, Greece
| | - Dimitris Malamos
- Oral Medicine Clinic of the National Organization for the Provision of Health, Athens, Greece
| | - Mohammad S Alrashdan
- Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, Sharjah, United Arab Emirates
| | - Faleh Sawair
- School of Dentistry, The University of Jordan, Jordan
| |
Collapse
|
22
|
Gravina AG, Pellegrino R, Palladino G, Imperio G, Ventura A, Federico A. Charting new AI education in gastroenterology: Cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam. Dig Liver Dis 2024; 56:1304-1311. [PMID: 38503659 DOI: 10.1016/j.dld.2024.02.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Revised: 02/08/2024] [Accepted: 02/28/2024] [Indexed: 03/21/2024]
Abstract
BACKGROUND Conversational chatbots, fueled by large language models, spark debate over their potential in education and medical career exams. There is debate in the literature about the scientific integrity of the outputs produced by these chatbots. AIMS This study evaluates ChatGPT 3.5 and Perplexity AI's cross-sectional performance in responding to questions from the 2023 Italian national residency admission exam (SSM23), comparing results and chatbots' concordance with previous years SSMs. METHODS Gastroenterology-related SSM23 questions were input into ChatGPT 3.5 and Perplexity AI, evaluating their performance in correct responses and total scores. This process was repeated with questions from the three preceding years. Additionally, chatbot concordance was assessed using Cohen's method. RESULTS In SSM23, ChatGPT 3.5 outperforms Perplexity AI with 94.11% correct responses, demonstrating consistency across years. Concordance weakened in 2023 (κ=0.203, P = 0.148), but ChatGPT consistently maintains a high standard compared to Perplexity AI. CONCLUSION ChatGPT 3.5 and Perplexity AI exhibit promise in addressing gastroenterological queries, emphasizing potential educational roles. However, their variable performance mandates cautious use as supplementary tools alongside conventional study methods. Clear guidelines are crucial for educators to balance traditional approaches and innovative systems, enhancing educational standards.
Collapse
Affiliation(s)
- Antonietta Gerarda Gravina
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy
| | - Raffaele Pellegrino
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy.
| | - Giovanna Palladino
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy
| | - Giuseppe Imperio
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy
| | - Andrea Ventura
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy
| | - Alessandro Federico
- Hepatogastroenterology Division, Department of Precision Medicine, University of Campania Luigi Vanvitelli, Via Luigi de Crecchio, 80138, Naples, Italy
| |
Collapse
|
23
|
Amaral JZ, Schultz RJ, Martin BM, Taylor T, Touban B, McGraw-Heinrich J, McKay SD, Rosenfeld SB, Smith BG. Evaluating Chat Generative Pre-trained Transformer Responses to Common Pediatric In-toeing Questions. J Pediatr Orthop 2024; 44:e592-e597. [PMID: 38686934 DOI: 10.1097/bpo.0000000000002695] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/02/2024]
Abstract
OBJECTIVE Chat generative pre-trained transformer (ChatGPT) has garnered attention in health care for its potential to reshape patient interactions. As patients increasingly rely on artificial intelligence platforms, concerns about information accuracy arise. In-toeing, a common lower extremity variation, often leads to pediatric orthopaedic referrals despite observation being the primary treatment. Our study aims to assess ChatGPT's responses to pediatric in-toeing questions, contributing to discussions on health care innovation and technology in patient education. METHODS We compiled a list of 34 common in-toeing questions from the "Frequently Asked Questions" sections of 9 health care-affiliated websites, identifying 25 as the most encountered. On January 17, 2024, we queried ChatGPT 3.5 in separate sessions and recorded the responses. These 25 questions were posed again on January 21, 2024, to assess its reproducibility. Two pediatric orthopaedic surgeons evaluated responses using a scale of "excellent (no clarification)" to "unsatisfactory (substantial clarification)." Average ratings were used when evaluators' grades were within one level of each other. In discordant cases, the senior author provided a decisive rating. RESULTS We found 46% of ChatGPT responses were "excellent" and 44% "satisfactory (minimal clarification)." In addition, 8% of cases were "satisfactory (moderate clarification)" and 2% were "unsatisfactory." Questions had appropriate readability, with an average Flesch-Kincaid Grade Level of 4.9 (±2.1). However, ChatGPT's responses were at a collegiate level, averaging 12.7 (±1.4). No significant differences in ratings were observed between question topics. Furthermore, ChatGPT exhibited moderate consistency after repeated queries, evidenced by a Spearman rho coefficient of 0.55 ( P = 0.005). The chatbot appropriately described in-toeing as normal or spontaneously resolving in 62% of responses and consistently recommended evaluation by a health care provider in 100%. CONCLUSION The chatbot presented a serviceable, though not perfect, representation of the diagnosis and management of pediatric in-toeing while demonstrating a moderate level of reproducibility in its responses. ChatGPT's utility could be enhanced by improving readability and consistency and incorporating evidence-based guidelines. LEVEL OF EVIDENCE Level IV-diagnostic.
Collapse
Affiliation(s)
- Jason Zarahi Amaral
- Department of Orthopaedic Surgery, Texas Children's Hospital and Baylor College of Medicine, Houston, TX
| | | | | | | | | | | | | | | | | |
Collapse
|
24
|
Cohen SA, Brant A, Fisher AC, Pershing S, Do D, Pan C. Dr. Google vs. Dr. ChatGPT: Exploring the Use of Artificial Intelligence in Ophthalmology by Comparing the Accuracy, Safety, and Readability of Responses to Frequently Asked Patient Questions Regarding Cataracts and Cataract Surgery. Semin Ophthalmol 2024; 39:472-479. [PMID: 38516983 DOI: 10.1080/08820538.2024.2326058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 02/25/2024] [Accepted: 02/27/2024] [Indexed: 03/23/2024]
Abstract
PURPOSE Patients are using online search modalities to learn about their eye health. While Google remains the most popular search engine, the use of large language models (LLMs) like ChatGPT has increased. Cataract surgery is the most common surgical procedure in the US, and there is limited data on the quality of online information that populates after searches related to cataract surgery on search engines such as Google and LLM platforms such as ChatGPT. We identified the most common patient frequently asked questions (FAQs) about cataracts and cataract surgery and evaluated the accuracy, safety, and readability of the answers to these questions provided by both Google and ChatGPT. We demonstrated the utility of ChatGPT in writing notes and creating patient education materials. METHODS The top 20 FAQs related to cataracts and cataract surgery were recorded from Google. Responses to the questions provided by Google and ChatGPT were evaluated by a panel of ophthalmologists for accuracy and safety. Evaluators were also asked to distinguish between Google and LLM chatbot answers. Five validated readability indices were used to assess the readability of responses. ChatGPT was instructed to generate operative notes, post-operative instructions, and customizable patient education materials according to specific readability criteria. RESULTS Responses to 20 patient FAQs generated by ChatGPT were significantly longer and written at a higher reading level than responses provided by Google (p < .001), with an average grade level of 14.8 (college level). Expert reviewers were correctly able to distinguish between a human-reviewed and chatbot generated response an average of 31% of the time. Google answers contained incorrect or inappropriate material 27% of the time, compared with 6% of LLM generated answers (p < .001). When expert reviewers were asked to compare the responses directly, chatbot responses were favored (66%). CONCLUSIONS When comparing the responses to patients' cataract FAQs provided by ChatGPT and Google, practicing ophthalmologists overwhelming preferred ChatGPT responses. LLM chatbot responses were less likely to contain inaccurate information. ChatGPT represents a viable information source for eye health for patients with higher health literacy. ChatGPT may also be used by ophthalmologists to create customizable patient education materials for patients with varying health literacy.
Collapse
Affiliation(s)
- Samuel A Cohen
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| | - Arthur Brant
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| | - Ann Caroline Fisher
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| | - Suzann Pershing
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| | - Diana Do
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| | - Carolyn Pan
- Byers Eye Institute, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
25
|
Roldan-Vasquez E, Mitri S, Bhasin S, Bharani T, Capasso K, Haslinger M, Sharma R, James TA. Reliability of artificial intelligence chatbot responses to frequently asked questions in breast surgical oncology. J Surg Oncol 2024; 130:188-203. [PMID: 38837375 DOI: 10.1002/jso.27715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2024] [Accepted: 05/21/2024] [Indexed: 06/07/2024]
Abstract
INTRODUCTION Artificial intelligence (AI)-driven chatbots, capable of simulating human-like conversations, are becoming more prevalent in healthcare. While this technology offers potential benefits in patient engagement and information accessibility, it raises concerns about potential misuse, misinformation, inaccuracies, and ethical challenges. METHODS This study evaluated a publicly available AI chatbot, ChatGPT, in its responses to nine questions related to breast cancer surgery selected from the American Society of Breast Surgeons' frequently asked questions (FAQ) patient education website. Four breast surgical oncologists assessed the responses for accuracy and reliability using a five-point Likert scale and the Patient Education Materials Assessment (PEMAT) Tool. RESULTS The average reliability score for ChatGPT in answering breast cancer surgery questions was 3.98 out of 5.00. Surgeons unanimously found the responses understandable and actionable per the PEMAT criteria. The consensus found ChatGPT's overall performance was appropriate, with minor or no inaccuracies. CONCLUSION ChatGPT demonstrates good reliability in responding to breast cancer surgery queries, with minor, nonharmful inaccuracies. Its answers are accurate, clear, and easy to comprehend. Notably, ChatGPT acknowledged its informational role and did not attempt to replace medical advice or discourage users from seeking input from a healthcare professional.
Collapse
Affiliation(s)
- Estefania Roldan-Vasquez
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Samir Mitri
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Shreya Bhasin
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | - Tina Bharani
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Kathryn Capasso
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Michelle Haslinger
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Ranjna Sharma
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Ted A James
- Department of Surgery, Breast Surgical Oncology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
26
|
Pan ET, Florian-Rodriguez M. Human vs machine: identifying ChatGPT-generated abstracts in Gynecology and Urogynecology. Am J Obstet Gynecol 2024; 231:276.e1-276.e10. [PMID: 38710267 DOI: 10.1016/j.ajog.2024.04.045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Revised: 04/12/2024] [Accepted: 04/30/2024] [Indexed: 05/08/2024]
Abstract
BACKGROUND ChatGPT, a publicly available artificial intelligence large language model, has allowed for sophisticated artificial intelligence technology on demand. Indeed, use of ChatGPT has already begun to make its way into medical research. However, the medical community has yet to understand the capabilities and ethical considerations of artificial intelligence within this context, and unknowns exist regarding ChatGPT's writing abilities, accuracy, and implications for authorship. OBJECTIVE We hypothesize that human reviewers and artificial intelligence detection software differ in their ability to correctly identify original published abstracts and artificial intelligence-written abstracts in the subjects of Gynecology and Urogynecology. We also suspect that concrete differences in writing errors, readability, and perceived writing quality exist between original and artificial intelligence-generated text. STUDY DESIGN Twenty-five articles published in high-impact medical journals and a collection of Gynecology and Urogynecology journals were selected. ChatGPT was prompted to write 25 corresponding artificial intelligence-generated abstracts, providing the abstract title, journal-dictated abstract requirements, and select original results. The original and artificial intelligence-generated abstracts were reviewed by blinded Gynecology and Urogynecology faculty and fellows to identify the writing as original or artificial intelligence-generated. All abstracts were analyzed by publicly available artificial intelligence detection software GPTZero, Originality, and Copyleaks, and were assessed for writing errors and quality by artificial intelligence writing assistant Grammarly. RESULTS A total of 157 reviews of 25 original and 25 artificial intelligence-generated abstracts were conducted by 26 faculty and 4 fellows; 57% of original abstracts and 42.3% of artificial intelligence-generated abstracts were correctly identified, yielding an average accuracy of 49.7% across all abstracts. All 3 artificial intelligence detectors rated the original abstracts as less likely to be artificial intelligence-written than the ChatGPT-generated abstracts (GPTZero, 5.8% vs 73.3%; P<.001; Originality, 10.9% vs 98.1%; P<.001; Copyleaks, 18.6% vs 58.2%; P<.001). The performance of the 3 artificial intelligence detection software differed when analyzing all abstracts (P=.03), original abstracts (P<.001), and artificial intelligence-generated abstracts (P<.001). Grammarly text analysis identified more writing issues and correctness errors in original than in artificial intelligence abstracts, including lower Grammarly score reflective of poorer writing quality (82.3 vs 88.1; P=.006), more total writing issues (19.2 vs 12.8; P<.001), critical issues (5.4 vs 1.3; P<.001), confusing words (0.8 vs 0.1; P=.006), misspelled words (1.7 vs 0.6; P=.02), incorrect determiner use (1.2 vs 0.2; P=.002), and comma misuse (0.3 vs 0.0; P=.005). CONCLUSION Human reviewers are unable to detect the subtle differences between human and ChatGPT-generated scientific writing because of artificial intelligence's ability to generate tremendously realistic text. Artificial intelligence detection software improves the identification of artificial intelligence-generated writing, but still lacks complete accuracy and requires programmatic improvements to achieve optimal detection. Given that reviewers and editors may be unable to reliably detect artificial intelligence-generated texts, clear guidelines for reporting artificial intelligence use by authors and implementing artificial intelligence detection software in the review process will need to be established as artificial intelligence chatbots gain more widespread use.
Collapse
Affiliation(s)
- Evelyn T Pan
- Division of Urogynecology and Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, Dallas, TX.
| | - Maria Florian-Rodriguez
- Division of Urogynecology and Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
27
|
Choi J, Oh AR, Park J, Kang RA, Yoo SY, Lee DJ, Yang K. Evaluation of the quality and quantity of artificial intelligence-generated responses about anesthesia and surgery: using ChatGPT 3.5 and 4.0. Front Med (Lausanne) 2024; 11:1400153. [PMID: 39055693 PMCID: PMC11269144 DOI: 10.3389/fmed.2024.1400153] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Accepted: 07/01/2024] [Indexed: 07/27/2024] Open
Abstract
Introduction The large-scale artificial intelligence (AI) language model chatbot, Chat Generative Pre-Trained Transformer (ChatGPT), is renowned for its ability to provide data quickly and efficiently. This study aimed to assess the medical responses of ChatGPT regarding anesthetic procedures. Methods Two anesthesiologist authors selected 30 questions representing inquiries patients might have about surgery and anesthesia. These questions were inputted into two versions of ChatGPT in English. A total of 31 anesthesiologists then evaluated each response for quality, quantity, and overall assessment, using 5-point Likert scales. Descriptive statistics summarized the scores, and a paired sample t-test compared ChatGPT 3.5 and 4.0. Results Regarding quality, "appropriate" was the most common rating for both ChatGPT 3.5 and 4.0 (40 and 48%, respectively). For quantity, responses were deemed "insufficient" in 59% of cases for 3.5, and "adequate" in 69% for 4.0. In overall assessment, 3 points were most common for 3.5 (36%), while 4 points were predominant for 4.0 (42%). Mean quality scores were 3.40 and 3.73, and mean quantity scores were - 0.31 (between insufficient and adequate) and 0.03 (between adequate and excessive), respectively. The mean overall score was 3.21 for 3.5 and 3.67 for 4.0. Responses from 4.0 showed statistically significant improvement in three areas. Conclusion ChatGPT generated responses mostly ranging from appropriate to slightly insufficient, providing an overall average amount of information. Version 4.0 outperformed 3.5, and further research is warranted to investigate the potential utility of AI chatbots in assisting patients with medical information.
Collapse
Affiliation(s)
- Jisun Choi
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Ah Ran Oh
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Jungchan Park
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Ryung A. Kang
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Seung Yeon Yoo
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Dong Jae Lee
- Department of Anesthesiology and Pain Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Kwangmo Yang
- Center for Health Promotion, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| |
Collapse
|
28
|
Ho RA, Shaari AL, Cowan PT, Yan K. ChatGPT Responses to Frequently Asked Questions on Ménière's Disease: A Comparison to Clinical Practice Guideline Answers. OTO Open 2024; 8:e163. [PMID: 38974175 PMCID: PMC11225079 DOI: 10.1002/oto2.163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2024] [Revised: 06/01/2024] [Accepted: 06/08/2024] [Indexed: 07/09/2024] Open
Abstract
Objective Evaluate the quality of responses from Chat Generative Pre-Trained Transformer (ChatGPT) models compared to the answers for "Frequently Asked Questions" (FAQs) from the American Academy of Otolaryngology-Head and Neck Surgery (AAO-HNS) Clinical Practice Guidelines (CPG) for Ménière's disease (MD). Study Design Comparative analysis. Setting The AAO-HNS CPG for MD includes FAQs that clinicians can give to patients for MD-related questions. The ability of ChatGPT to properly educate patients regarding MD is unknown. Methods ChatGPT-3.5 and 4.0 were each prompted with 16 questions from the MD FAQs. Each response was rated in terms of (1) comprehensiveness, (2) extensiveness, (3) presence of misleading information, and (4) quality of resources. Readability was assessed using Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES). Results ChatGPT-3.5 was comprehensive in 5 responses whereas ChatGPT-4.0 was comprehensive in 9 (31.3% vs 56.3%, P = .2852). ChatGPT-3.5 and 4.0 were extensive in all responses (P = 1.0000). ChatGPT-3.5 was misleading in 5 responses whereas ChatGPT-4.0 was misleading in 3 (31.3% vs 18.75%, P = .6851). ChatGPT-3.5 had quality resources in 10 responses whereas ChatGPT-4.0 had quality resources in 16 (62.5% vs 100%, P = .0177). AAO-HNS CPG FRES (62.4 ± 16.6) demonstrated an appropriate readability score of at least 60, while both ChatGPT-3.5 (39.1 ± 7.3) and 4.0 (42.8 ± 8.5) failed to meet this standard. All platforms had FKGL means that exceeded the recommended level of 6 or lower. Conclusion While ChatGPT-4.0 had significantly better resource reporting, both models have room for improvement in being more comprehensive, more readable, and less misleading for patients.
Collapse
Affiliation(s)
- Rebecca A. Ho
- Department of Otolaryngology–Head and Neck SurgeryRutgers New Jersey Medical SchoolNewarkNew JerseyUSA
| | - Ariana L. Shaari
- Department of Otolaryngology–Head and Neck SurgeryRutgers New Jersey Medical SchoolNewarkNew JerseyUSA
| | - Paul T. Cowan
- Department of Otolaryngology–Head and Neck SurgeryRutgers New Jersey Medical SchoolNewarkNew JerseyUSA
| | - Kenneth Yan
- Department of Otolaryngology–Head and Neck SurgeryRutgers New Jersey Medical SchoolNewarkNew JerseyUSA
| |
Collapse
|
29
|
Giuffrè M, Kresevic S, You K, Dupont J, Huebner J, Grimshaw AA, Shung DL. Systematic review: The use of large language models as medical chatbots in digestive diseases. Aliment Pharmacol Ther 2024; 60:144-166. [PMID: 38798194 DOI: 10.1111/apt.18058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 01/23/2024] [Accepted: 05/11/2024] [Indexed: 05/29/2024]
Abstract
BACKGROUND Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology-related questions is unknown. AIMS To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology. METHODS We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers. RESULTS Among the 1671 reports screened, we identified 33 full-text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question-responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT-3.5 and was between 40% and 91.4% with ChatGPT-4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single-study results. CONCLUSIONS Current general-purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.
Collapse
Affiliation(s)
- Mauro Giuffrè
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA
- Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy
| | - Simone Kresevic
- Department of Engineering and Architecture, University of Trieste, Trieste, Italy
| | - Kisung You
- Department of Mathematics at Baruch College, City University of new York, New York, New York, USA
| | - Johannes Dupont
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA
| | - Jack Huebner
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Alyssa Ann Grimshaw
- Research & Education Librarian (Clinical) at Harvey Cushing/John Hay Whitney Medical Library, Yale University, New Haven, Connecticut, USA
| | - Dennis Legen Shung
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA
| |
Collapse
|
30
|
Musheyev D, Pan A, Gross P, Kamyab D, Kaplinsky P, Spivak M, Bragg MA, Loeb S, Kabarriti AE. Readability and Information Quality in Cancer Information From a Free vs Paid Chatbot. JAMA Netw Open 2024; 7:e2422275. [PMID: 39058491 PMCID: PMC11282443 DOI: 10.1001/jamanetworkopen.2024.22275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 05/15/2024] [Indexed: 07/28/2024] Open
Abstract
Importance The mainstream use of chatbots requires a thorough investigation of their readability and quality of information. Objective To identify readability and quality differences in information between a free and paywalled chatbot cancer-related responses, and to explore if more precise prompting can mitigate any observed differences. Design, Setting, and Participants This cross-sectional study compared readability and information quality of a chatbot's free vs paywalled responses with Google Trends' top 5 search queries associated with breast, lung, prostate, colorectal, and skin cancers from January 1, 2021, to January 1, 2023. Data were extracted from the search tracker, and responses were produced by free and paywalled ChatGPT. Data were analyzed from December 20, 2023, to January 15, 2024. Exposures Free vs paywalled chatbot outputs with and without prompt: "Explain the following at a sixth grade reading level: [nonprompted input]." Main Outcomes and Measures The primary outcome measured the readability of a chatbot's responses using Flesch Reading Ease scores (0 [graduate reading level] to 100 [easy fifth grade reading level]). Secondary outcomes included assessing consumer health information quality with the validated DISCERN instrument (overall score from 1 [low quality] to 5 [high quality]) for each response. Scores were compared between the 2 chatbot models with and without prompting. Results This study evaluated 100 chatbot responses. Nonprompted free chatbot responses had lower readability (median [IQR] Flesh Reading ease scores, 52.60 [44.54-61.46]) than nonprompted paywalled chatbot responses (62.48 [54.83-68.40]) (P < .05). However, prompting the free chatbot to reword responses at a sixth grade reading level was associated with increased reading ease scores than the paywalled chatbot nonprompted responses (median [IQR], 71.55 [68.20-78.99]) (P < .001). Prompting was associated with increases in reading ease in both free (median [IQR], 71.55 [68.20-78.99]; P < .001)and paywalled versions (median [IQR], 75.64 [70.53-81.12]; P < .001). There was no significant difference in overall DISCERN scores between the chatbot models, with and without prompting. Conclusions and Relevance In this cross-sectional study, paying for the chatbot was found to provide easier-to-read responses, but prompting the free version of the chatbot was associated with increased response readability without changing information quality. Educating the public on how to prompt chatbots may help promote equitable access to health information.
Collapse
Affiliation(s)
- David Musheyev
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Alexander Pan
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Preston Gross
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Daniel Kamyab
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Peter Kaplinsky
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Mark Spivak
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| | - Marie A. Bragg
- Department of Urology, New York University and Manhattan Veterans Affairs, New York
- Marketing Department, Stern School of Business, New York University, New York
- Department of Population Health, New York University, New York
| | - Stacy Loeb
- Department of Urology, New York University and Manhattan Veterans Affairs, New York
- Department of Population Health, New York University, New York
| | - Abdo E. Kabarriti
- Department of Urology, State University of New York Downstate Health Sciences University, New York
| |
Collapse
|
31
|
Garg N, Campbell DJ, Yang A, McCann A, Moroco AE, Estephan LE, Palmer WJ, Krein H, Heffelfinger R. Chatbots as Patient Education Resources for Aesthetic Facial Plastic Surgery: Evaluation of ChatGPT and Google Bard Responses. Facial Plast Surg Aesthet Med 2024. [PMID: 38946595 DOI: 10.1089/fpsam.2023.0368] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/02/2024] Open
Abstract
Background: ChatGPT and Google Bard™ are popular artificial intelligence chatbots with utility for patients, including those undergoing aesthetic facial plastic surgery. Objective: To compare the accuracy and readability of chatbot-generated responses to patient education questions regarding aesthetic facial plastic surgery using a response accuracy scale and readability testing. Method: ChatGPT and Google Bard™ were asked 28 identical questions using four prompts: none, patient friendly, eighth-grade level, and references. Accuracy was assessed using Global Quality Scale (range: 1-5). Flesch-Kincaid grade level was calculated, and chatbot-provided references were analyzed for veracity. Results: Although 59.8% of responses were good quality (Global Quality Scale ≥4), ChatGPT generated more accurate responses than Google Bard™ on patient-friendly prompting (p < 0.001). Google Bard™ responses were of a significantly lower grade level than ChatGPT for all prompts (p < 0.05). Despite eighth-grade prompting, response grade level for both chatbots was high: ChatGPT (10.5 ± 1.8) and Google Bard™ (9.6 ± 1.3). Prompting for references yielded 108/108 of chatbot-generated references. Forty-one (38.0%) citations were legitimate. Twenty (18.5%) provided accurately reported information from the reference. Conclusion: Although ChatGPT produced more accurate responses and at a higher education level than Google Bard™, both chatbots provided responses above recommended grade levels for patients and failed to provide accurate references.
Collapse
Affiliation(s)
- Neha Garg
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Daniel J Campbell
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Angela Yang
- Sidney Kimmel Medical College, Philadelphia, Pennsylvania, USA
| | - Adam McCann
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Annie E Moroco
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Leonard E Estephan
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - William J Palmer
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Howard Krein
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| | - Ryan Heffelfinger
- Department of Otolaryngology - Head and Neck Surgery, Thomas Jefferson University Hospitals, Philadelphia, Pennsylvania, USA
| |
Collapse
|
32
|
Aburumman R, Al Annan K, Mrad R, Brunaldi VO, Gala K, Abu Dayyeh BK. Assessing ChatGPT vs. Standard Medical Resources for Endoscopic Sleeve Gastroplasty Education: A Medical Professional Evaluation Study. Obes Surg 2024; 34:2718-2724. [PMID: 38758515 DOI: 10.1007/s11695-024-07283-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2024] [Revised: 05/07/2024] [Accepted: 05/09/2024] [Indexed: 05/18/2024]
Abstract
BACKGROUND AND AIMS The Chat Generative Pre-Trained Transformer (ChatGPT) represents a significant advancement in artificial intelligence (AI) chatbot technology. While ChatGPT offers promising capabilities, concerns remain about its reliability and accuracy. This study aims to evaluate ChatGPT's responses to patients' frequently asked questions about Endoscopic Sleeve Gastroplasty (ESG). METHODS Expert Gastroenterologists and Bariatric Surgeons, with experience in ESG, were invited to evaluate ChatGPT-generated answers to eight ESG-related questions, and answers sourced from hospital websites. The evaluation criteria included ease of understanding, scientific accuracy, and overall answer satisfaction. They were also tasked with discerning whether each response was AI generated or not. RESULTS Twelve medical professionals with expertise in ESG participated, 83.3% of whom had experience performing the procedure independently. The entire cohort possessed substantial knowledge about ESG. ChatGPT's utility among participants, rated on a scale of one to five, averaged 2.75. The raters demonstrated a 54% accuracy rate in distinguishing AI-generated responses, with a sensitivity of 39% and specificity of 60%, resulting in an average of 17.6 correct identifications out of a possible 31. Overall, there were no significant differences between AI-generated and non-AI responses in terms of scientific accuracy, understandability, and satisfaction, with one notable exception. For the question defining ESG, the AI-generated definition scored higher in scientific accuracy (4.33 vs. 3.61, p = 0.007) and satisfaction (4.33 vs. 3.58, p = 0.009) compared to the non-AI versions. CONCLUSIONS This study underscores ChatGPT's efficacy in providing medical information on ESG, demonstrating its comparability to traditional sources in scientific accuracy.
Collapse
Affiliation(s)
- Razan Aburumman
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Karim Al Annan
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Rudy Mrad
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Vitor O Brunaldi
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Khushboo Gala
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Barham K Abu Dayyeh
- Division of Gastroenterology and Hepatology, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA.
| |
Collapse
|
33
|
Knee CJ, Campbell RJ, Graham DJ, Handford C, Symes M, Sivakumar BS. Examining the role of ChatGPT in the management of distal radius fractures: insights into its accuracy and consistency. ANZ J Surg 2024; 94:1391-1396. [PMID: 38967407 DOI: 10.1111/ans.19143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2024] [Revised: 03/19/2024] [Accepted: 06/15/2024] [Indexed: 07/06/2024]
Abstract
BACKGROUND The optimal management of distal radius fractures remains a challenge for orthopaedic surgeons. The emergence of Artificial Intelligence (AI) and Large Language Models (LLMs), especially ChatGPT, affords significant potential in improving healthcare and research. This study aims to assess the accuracy and consistency of ChatGPT's knowledge in managing distal radius fractures, with a focus on its capability to provide information for patients and assist in the decision-making processes of orthopaedic clinicians. METHODS We presented ChatGPT with seven questions on distal radius fracture management over two sessions, resulting in 14 responses. These questions covered a range of topics, including patient inquiries and orthopaedic clinical decision-making. We requested references for each response and involved two orthopaedic registrars and two senior orthopaedic surgeons to evaluate response accuracy and consistency. RESULTS All 14 responses contained a mix of both correct and incorrect information. Among the 47 cited references, 13% were accurate, 28% appeared to be fabricated, 57% were incorrect, and 2% were correct but deemed inappropriate. Consistency was observed in 71% of the responses. CONCLUSION ChatGPT demonstrates significant limitations in accuracy and consistency when providing information on distal radius fractures. In its current format, it offers limited utility for patient education and clinical decision-making.
Collapse
Affiliation(s)
- Christopha J Knee
- Department of Orthopaedics and Trauma Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia
| | - Ryan J Campbell
- Department of Orthopaedics and Trauma Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia
| | - David J Graham
- Australian Research Collaboration on Hands [ARCH], Mudgeeraba, Queensland, Australia
- School of Medicine and Dentistry, Griffith University, Southport, Queensland, Australia
- School of Medicine, University of Queensland, Herston, Queensland, Australia
- Department of Musculoskeletal Services, Gold Coast University Hospital, Southport, Queensland, Australia
- Department of Orthopaedic Surgery, Queensland Children's Hospital, South Brisbane, Queensland, Australia
| | - Cameron Handford
- Department of Orthopaedic Surgery, St George Hospital, Kogarah, New South Wales, Australia
| | - Michael Symes
- Department of Orthopaedics and Trauma Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia
- Department of Orthopaedic Surgery, St George Hospital, Kogarah, New South Wales, Australia
| | - Brahman S Sivakumar
- Australian Research Collaboration on Hands [ARCH], Mudgeeraba, Queensland, Australia
- Department of Hand and Peripheral Nerve Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia
- Faculty of Medicine and Health, Sydney Medical School, The University of Sydney, Camperdown, New South Wales, Australia
- Department of Orthopaedic Surgery, Hornsby Ku-ring-gai Hospital, Hornsby, New South Wales, Australia
- Department of Orthopaedic Surgery, Nepean Hospital, Kingswood, New South Wales, Australia
| |
Collapse
|
34
|
Hassona Y, Alqaisi DA. "My kid has autism": An interesting conversation with ChatGPT. SPECIAL CARE IN DENTISTRY 2024; 44:1296-1299. [PMID: 38415857 DOI: 10.1111/scd.12983] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2024] [Revised: 02/11/2024] [Accepted: 02/16/2024] [Indexed: 02/29/2024]
Affiliation(s)
- Yazan Hassona
- Faculty of Dentistry, Centre for Oral Diseases Studies, Al-Ahliyya Amman University, Amman, Jordan
- School of Dentistry, The University of Jordan, Amman, Jordan
| | - Dua A Alqaisi
- School of Dentistry, The University of Jordan, Amman, Jordan
| |
Collapse
|
35
|
Law S, Oldfield B, Yang W. ChatGPT/GPT-4 (large language models): Opportunities and challenges of perspective in bariatric healthcare professionals. Obes Rev 2024; 25:e13746. [PMID: 38613164 DOI: 10.1111/obr.13746] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 03/14/2024] [Accepted: 03/15/2024] [Indexed: 04/14/2024]
Abstract
ChatGPT/GPT-4 is a conversational large language model (LLM) based on artificial intelligence (AI). The potential application of LLM as a virtual assistant for bariatric healthcare professionals in education and practice may be promising if relevant and valid issues are actively examined and addressed. In general medical terms, it is possible that AI models like ChatGPT/GPT-4 will be deeply integrated into medical scenarios, improving medical efficiency and quality, and allowing doctors more time to communicate with patients and implement personalized health management. Chatbots based on AI have great potential in bariatric healthcare and may play an important role in predicting and intervening in weight loss and obesity-related complications. However, given its potential limitations, we should carefully consider the medical, legal, ethical, data security, privacy, and liability issues arising from medical errors caused by ChatGPT/GPT-4. This concern also extends to ChatGPT/GPT -4's ability to justify wrong decisions, and there is an urgent need for appropriate guidelines and regulations to ensure the safe and responsible use of ChatGPT/GPT-4.
Collapse
Affiliation(s)
- Saikam Law
- Department of Metabolic and Bariatric Surgery, The First Affiliated Hospital of Jinan University, Guangzhou, China
- School of Medicine, Jinan University, Guangzhou, China
| | - Brian Oldfield
- Department of Physiology, Monash Biomedicine Discovery Institute, Monash University, Melbourne, Australia
| | - Wah Yang
- Department of Metabolic and Bariatric Surgery, The First Affiliated Hospital of Jinan University, Guangzhou, China
| |
Collapse
|
36
|
Aghamaliyev U, Karimbayli J, Giessen-Jung C, Matthias I, Unger K, Andrade D, Hofmann FO, Weniger M, Angele MK, Benedikt Westphalen C, Werner J, Renz BW. ChatGPT's Gastrointestinal Tumor Board Tango: A limping dance partner? Eur J Cancer 2024; 205:114100. [PMID: 38729055 DOI: 10.1016/j.ejca.2024.114100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2024] [Accepted: 04/23/2024] [Indexed: 05/12/2024]
Abstract
OBJECTIVES This study aimed to assess the consistency and replicability of treatment recommendations provided by ChatGPT 3.5 compared to gastrointestinal tumor cases presented at multidisciplinary tumor boards (MTBs). It also aimed to distinguish between general and case-specific responses and investigated the precision of ChatGPT's recommendations in replicating exact treatment plans, particularly regarding chemotherapy regimens and follow-up protocols. MATERIAL AND METHODS A retrospective study was carried out on 115 cases of gastrointestinal malignancies, selected from 448 patients reviewed in MTB meetings. A senior resident fed patient data into ChatGPT 3.5 to produce treatment recommendations, which were then evaluated against the tumor board's decisions by senior oncology fellows. RESULTS Among the examined cases, ChatGPT 3.5 provided general information about the malignancy without considering individual patient characteristics in 19% of cases. However, only in 81% of cases, ChatGPT generated responses that were specific to the individual clinical scenarios. In the subset of case-specific responses, 83% of recommendations exhibited overall treatment strategy concordance between ChatGPT and MTB. However, the exact treatment concordance dropped to 65%, notably lower in recommending specific chemotherapy regimens. Cases recommended for surgery showed the highest concordance rates, while those involving chemotherapy recommendations faced challenges in precision. CONCLUSIONS ChatGPT 3.5 demonstrates potential in aligning conceptual approaches to treatment strategies with MTB guidelines. However, it falls short in accurately duplicating specific treatment plans, especially concerning chemotherapy regimens and follow-up procedures. Ethical concerns and challenges in achieving exact replication necessitate prudence when considering ChatGPT 3.5 for direct clinical decision-making in MTBs.
Collapse
Affiliation(s)
- Ughur Aghamaliyev
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany
| | - Javad Karimbayli
- Division of Molecular Oncology, Centro di Riferimento Oncologico di Aviano (CRO), IRCCS, National Cancer Institute, Aviano, Italy
| | - Clemens Giessen-Jung
- Comprehensive Cancer Center Munich & Department of Medicine III, LMU University Hospital, LMU Munich, Germany
| | - Ilmer Matthias
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany; German Cancer Consortium (DKTK), Partner Site Munich, Munich, Germany
| | - Kristian Unger
- German Cancer Consortium (DKTK), Partner Site Munich, Munich, Germany; Department of Radiation Oncology, University Hospital, LMU Munich, 81377; Bavarian Cancer Research Center (BZKF), Munich, Germany
| | - Dorian Andrade
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany
| | - Felix O Hofmann
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany; German Cancer Consortium (DKTK), Partner Site Munich, Munich, Germany
| | - Maximilian Weniger
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany
| | - Martin K Angele
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany
| | - C Benedikt Westphalen
- Comprehensive Cancer Center Munich & Department of Medicine III, LMU University Hospital, LMU Munich, Germany; German Cancer Consortium (DKTK), Partner Site Munich, Munich, Germany
| | - Jens Werner
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany
| | - Bernhard W Renz
- Department of General, Visceral and Transplantation Surgery, LMU University Hospital, LMU Munich, Germany; German Cancer Consortium (DKTK), Partner Site Munich, Munich, Germany.
| |
Collapse
|
37
|
Subramanian B, Rajalakshmi R, Sivaprasad S, Rao C, Raman R. Assessing the appropriateness and completeness of ChatGPT-4's AI-generated responses for queries related to diabetic retinopathy. Indian J Ophthalmol 2024; 72:S684-S687. [PMID: 38953134 PMCID: PMC11338417 DOI: 10.4103/ijo.ijo_2510_23] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Revised: 10/30/2023] [Accepted: 12/17/2023] [Indexed: 07/03/2024] Open
Abstract
OBJECTIVE To evaluate the appropriateness of responses generated by an online chat-based artificial intelligence (AI) model for diabetic retinopathy (DR) related questions. DESIGN Cross-sectional study. METHODS A set of 20 questions framed from the patient's perspective addressing DR-related queries, such as the definition of disease, symptoms, prevention methods, treatment options, diagnostic methods, visual impact, and complications, were formulated for input into ChatGPT-4. Peer-reviewed, literature-based answers were collected from popular search engines for the selected questions and three retinal experts reviewed the responses. An inter-human agreement was analyzed for consensus expert responses and also between experts. The answers generated by the AI model were compared with those provided by the experts. The experts rated the response generated by ChatGPT-4 on a scale of 0-5 for appropriateness and completeness. RESULTS The answers provided by ChatGPT-4 were appropriate and complete for most of the DR-related questions. The response to questions on the adverse effects of laser photocoagulation therapy and compliance to treatment was not perfectly complete. The average rating given by the three retina expert evaluators was 4.84 for appropriateness and 4.38 for completeness of answers provided by the AI model. This corresponds to an overall 96.8% agreement among the experts for appropriateness and 87.6% for completeness regarding AI-generated answers. CONCLUSION ChatGPT-4 exhibits a high level of accuracy in generating appropriate responses for a range of questions in DR. However, there is a need to improvise the model to generate complete answers for certain DR-related topics.
Collapse
Affiliation(s)
- Brughanya Subramanian
- Department of Ophthalmology, Shri Bhagwan Mahavir Vitreoretinal Services, SankaraNethralaya, Chennai, Tamil Nadu, India
| | - Ramachandran Rajalakshmi
- Department of Ophthalmology, Dr. Mohan’s Diabetes Specialities Centre and Madras Diabetes Research Foundation, Chennai, Tamil Nadu, India
| | - Sobha Sivaprasad
- NIHR Biomedical Research Centre, Moorfields Eye Hospital NHS Foundation Trust, London, United Kingdom
| | - Chetan Rao
- Department of Ophthalmology, Shri Bhagwan Mahavir Vitreoretinal Services, SankaraNethralaya, Chennai, Tamil Nadu, India
| | - Rajiv Raman
- Department of Ophthalmology, Shri Bhagwan Mahavir Vitreoretinal Services, SankaraNethralaya, Chennai, Tamil Nadu, India
| |
Collapse
|
38
|
Khromchenko K, Shaikh S, Singh M, Vurture G, Rana RA, Baum JD. ChatGPT-3.5 Versus Google Bard: Which Large Language Model Responds Best to Commonly Asked Pregnancy Questions? Cureus 2024; 16:e65543. [PMID: 39188430 PMCID: PMC11346960 DOI: 10.7759/cureus.65543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/27/2024] [Indexed: 08/28/2024] Open
Abstract
Large language models (LLM) have been widely used to provide information in many fields, including obstetrics and gynecology. Which model performs best in providing answers to commonly asked pregnancy questions is unknown. A qualitative analysis of Chat Generative Pre-Training Transformer Version 3.5 (ChatGPT-3.5) (OpenAI, Inc., San Francisco, California, United States) and Bard, recently renamed Google Gemini (Google LLC, Mountain View, California, United States), was performed in August of 2023. Each LLM was queried on 12 commonly asked pregnancy questions and asked for their references. Review and grading of the responses and references for both LLMs were performed by the co-authors individually and then as a group to formulate a consensus. Query responses were graded as "acceptable" or "not acceptable" based on correctness and completeness in comparison to American College of Obstetricians and Gynecologists (ACOG) publications, PubMed-indexed evidence, and clinical experience. References were classified as "verified," "broken," "irrelevant," "non-existent," and "no references." Grades of "acceptable" were given to 58% of ChatGPT-3.5 responses (seven out of 12) and 83% of Bard responses (10 out of 12). In regard to references, ChatGPT-3.5 had reference issues in 100% of its references, and Bard had discrepancies in 8% of its references (one out of 12). When comparing ChatGPT-3.5 responses between May 2023 and August 2023, a change in "acceptable" responses was noted: 50% versus 58%, respectively. Bard answered more questions correctly than ChatGPT-3.5 when queried on a small sample of commonly asked pregnancy questions. ChatGPT-3.5 performed poorly in terms of reference verification. The overall performance of ChatGPT-3.5 remained stable over time, with approximately one-half of responses being "acceptable" in both May and August of 2023. Both LLMs need further evaluation and vetting before being accepted as accurate and reliable sources of information for pregnant women.
Collapse
Affiliation(s)
- Keren Khromchenko
- Obstetrics and Gynecology, Hackensack Meridian Jersey Shore University Medical Center, Neptune, USA
| | - Sameeha Shaikh
- Obstetrics and Gynecology, Hackensack Meridian School of Medicine, Nutley, USA
| | - Meghana Singh
- Obstetrics and Gynecology, Hackensack Meridian School of Medicine, Nutley, USA
| | - Gregory Vurture
- Obstetrics and Gynecology, Hackensack Meridian Jersey Shore University Medical Center, Neptune, USA
| | - Rima A Rana
- Obstetrics and Gynecology, Hackensack Meridian Jersey Shore University Medical Center, Neptune, USA
| | - Jonathan D Baum
- Obstetrics and Gynecology, Hackensack Meridian Jersey Shore University Medical Center, Neptune, USA
| |
Collapse
|
39
|
Kerci SG, Sahan B. An Analysis of ChatGPT4 to Respond to Glaucoma-Related Questions. J Glaucoma 2024; 33:486-489. [PMID: 38647417 DOI: 10.1097/ijg.0000000000002408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Accepted: 03/11/2024] [Indexed: 04/25/2024]
Abstract
PRCIS In recent years, ChatGPT has been widely used as a source of information. In our study, it was revealed that ChatGPT gives accurate information about glaucoma. PURPOSE We examined the knowledge of ChatGPT about glaucoma. MATERIALS AND METHODS Frequently asked questions about glaucoma found on websites of ophthalmology associations and hospitals, and social media applications were assessed. Evidence-Based Recommendations in the European Glaucoma Society Terminology and Guidelines for Glaucoma, Fifth Edition were evaluated. Using the ChatGPT-4, each question was asked twice on different computers to assess the reproducibility of answers. The answers provided were recorded and 2 specialist ophthalmologists evaluated them independently, assigning scores ranging from 1 to 4. RESULTS The answers to all questions about glaucoma resulted in 88.7% completely correct, 7.5% correct but insufficient, and 3.8% misleading information and correct information. No question was answered completely incorrectly. While 85.8% of the general knowledge questions were answered correctly, 91.7%, 86.6%, and 91.7% of questions about diagnosis, treatment, and prevention were answered correctly, respectively. The number of questions prepared based on the European Glaucoma Society Terminology and Guidelines for Glaucoma was 16. The rate of completely correct answers to these questions was 75.0% (12). While 3 (18.8%) answers were correct but insufficient, 1 response (6.3%) contained false information and correct information. CONCLUSIONS Our study revealed that ChatGPT answered 9 out of 10 questions about general information, diagnosis, treatment, and preventive and follow-up about glaucoma with acceptable and satisfactory accuracy rates. In addition, 3 of 4 answers given by ChatGPT were completely correct according to Terminology and Guidelines for Glaucoma.
Collapse
Affiliation(s)
- Suleyman G Kerci
- Department of Ophthalmology, Medicana International Izmir Hospital, İzmir, Turkey
| | | |
Collapse
|
40
|
Rodrigues Alessi M, Gomes HA, Lopes de Castro M, Terumy Okamoto C. Performance of ChatGPT in Solving Questions From the Progress Test (Brazilian National Medical Exam): A Potential Artificial Intelligence Tool in Medical Practice. Cureus 2024; 16:e64924. [PMID: 39156244 PMCID: PMC11330648 DOI: 10.7759/cureus.64924] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/19/2024] [Indexed: 08/20/2024] Open
Abstract
Background The use of artificial intelligence (AI) is not a recent phenomenon, but the latest advancements in this technology are making a significant impact across various fields of human knowledge. In medicine, this trend is no different, although it has developed at a slower pace. ChatGPT is an example of an AI-based algorithm capable of answering questions, interpreting phrases, and synthesizing complex information, potentially aiding and even replacing humans in various areas of social interest. Some studies have compared its performance in solving medical knowledge exams with medical students and professionals to verify AI accuracy. This study aimed to measure the performance of ChatGPT in answering questions from the Progress Test from 2021 to 2023. Methodology An observational study was conducted in which questions from the 2021 Progress Test and the regional tests (Southern Institutional Pedagogical Support Center II) of 2022 and 2023 were presented to ChatGPT 3.5. The results obtained were compared with the scores of first- to sixth-year medical students from over 120 Brazilian universities. All questions were presented sequentially, without any modification to their structure. After each question was presented, the platform's history was cleared, and the site was restarted. Results The platform achieved an average accuracy rate in 2021, 2022, and 2023 of 69.7%, 68.3%, and 67.2%, respectively, surpassing students from all medical years in the three tests evaluated, reinforcing findings in the current literature. The subject with the best score for the AI was Public Health, with a mean grade of 77.8%. Conclusions ChatGPT demonstrated the ability to answer medical questions with higher accuracy than humans, including students from the last year of medical school.
Collapse
Affiliation(s)
| | - Heitor A Gomes
- School of Medicine, Universidade Positivo, Curitiba, BRA
| | | | | |
Collapse
|
41
|
Small WR, Wiesenfeld B, Brandfield-Harvey B, Jonassen Z, Mandal S, Stevens ER, Major VJ, Lostraglio E, Szerencsy A, Jones S, Aphinyanaphongs Y, Johnson SB, Nov O, Mann D. Large Language Model-Based Responses to Patients' In-Basket Messages. JAMA Netw Open 2024; 7:e2422399. [PMID: 39012633 PMCID: PMC11252893 DOI: 10.1001/jamanetworkopen.2024.22399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Accepted: 05/16/2024] [Indexed: 07/17/2024] Open
Abstract
Importance Virtual patient-physician communications have increased since 2020 and negatively impacted primary care physician (PCP) well-being. Generative artificial intelligence (GenAI) drafts of patient messages could potentially reduce health care professional (HCP) workload and improve communication quality, but only if the drafts are considered useful. Objectives To assess PCPs' perceptions of GenAI drafts and to examine linguistic characteristics associated with equity and perceived empathy. Design, Setting, and Participants This cross-sectional quality improvement study tested the hypothesis that PCPs' ratings of GenAI drafts (created using the electronic health record [EHR] standard prompts) would be equivalent to HCP-generated responses on 3 dimensions. The study was conducted at NYU Langone Health using private patient-HCP communications at 3 internal medicine practices piloting GenAI. Exposures Randomly assigned patient messages coupled with either an HCP message or the draft GenAI response. Main Outcomes and Measures PCPs rated responses' information content quality (eg, relevance), using a Likert scale, communication quality (eg, verbosity), using a Likert scale, and whether they would use the draft or start anew (usable vs unusable). Branching logic further probed for empathy, personalization, and professionalism of responses. Computational linguistics methods assessed content differences in HCP vs GenAI responses, focusing on equity and empathy. Results A total of 16 PCPs (8 [50.0%] female) reviewed 344 messages (175 GenAI drafted; 169 HCP drafted). Both GenAI and HCP responses were rated favorably. GenAI responses were rated higher for communication style than HCP responses (mean [SD], 3.70 [1.15] vs 3.38 [1.20]; P = .01, U = 12 568.5) but were similar to HCPs on information content (mean [SD], 3.53 [1.26] vs 3.41 [1.27]; P = .37; U = 13 981.0) and usable draft proportion (mean [SD], 0.69 [0.48] vs 0.65 [0.47], P = .49, t = -0.6842). Usable GenAI responses were considered more empathetic than usable HCP responses (32 of 86 [37.2%] vs 13 of 79 [16.5%]; difference, 125.5%), possibly attributable to more subjective (mean [SD], 0.54 [0.16] vs 0.31 [0.23]; P < .001; difference, 74.2%) and positive (mean [SD] polarity, 0.21 [0.14] vs 0.13 [0.25]; P = .02; difference, 61.5%) language; they were also numerically longer (mean [SD] word count, 90.5 [32.0] vs 65.4 [62.6]; difference, 38.4%), but the difference was not statistically significant (P = .07) and more linguistically complex (mean [SD] score, 125.2 [47.8] vs 95.4 [58.8]; P = .002; difference, 31.2%). Conclusions In this cross-sectional study of PCP perceptions of an EHR-integrated GenAI chatbot, GenAI was found to communicate information better and with more empathy than HCPs, highlighting its potential to enhance patient-HCP communication. However, GenAI drafts were less readable than HCPs', a significant concern for patients with low health or English literacy.
Collapse
Affiliation(s)
| | | | | | - Zoe Jonassen
- NYU Grossman School of Medicine, New York, New York
| | | | | | | | | | | | - Simon Jones
- NYU Grossman School of Medicine, New York, New York
| | | | | | - Oded Nov
- NYU Tandon School of Engineering, New York, New York
| | - Devin Mann
- NYU Grossman School of Medicine, New York, New York
| |
Collapse
|
42
|
Lai Y, Hu Y, Li Z. Response to Pingping Yang et al. Helicobacter 2024; 29:e13132. [PMID: 39169508 DOI: 10.1111/hel.13132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/04/2024] [Revised: 08/07/2024] [Accepted: 08/08/2024] [Indexed: 08/23/2024]
Affiliation(s)
- Yongkang Lai
- Department of Gastroenterology, Ganzhou People's Hospital, Jiangxi Medical College, Nanchang University, Nanchang, China
- Department of Gastroenterology, Shanghai Changhai Hospital, Naval Medical University, Shanghai, China
| | - Yi Hu
- Department of Gastroenterology, The First Affiliated Hospital, Jiangxi Medical College, Nanchang University, Nanchang, China
| | - Zhaoshen Li
- Department of Gastroenterology, Shanghai Changhai Hospital, Naval Medical University, Shanghai, China
| |
Collapse
|
43
|
Lahat A, Sharif K, Zoabi N, Shneor Patt Y, Sharif Y, Fisher L, Shani U, Arow M, Levin R, Klang E. Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4. J Med Internet Res 2024; 26:e54571. [PMID: 38935937 PMCID: PMC11240076 DOI: 10.2196/54571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 02/02/2024] [Accepted: 04/29/2024] [Indexed: 06/29/2024] Open
Abstract
BACKGROUND Artificial intelligence, particularly chatbot systems, is becoming an instrumental tool in health care, aiding clinical decision-making and patient engagement. OBJECTIVE This study aims to analyze the performance of ChatGPT-3.5 and ChatGPT-4 in addressing complex clinical and ethical dilemmas, and to illustrate their potential role in health care decision-making while comparing seniors' and residents' ratings, and specific question types. METHODS A total of 4 specialized physicians formulated 176 real-world clinical questions. A total of 8 senior physicians and residents assessed responses from GPT-3.5 and GPT-4 on a 1-5 scale across 5 categories: accuracy, relevance, clarity, utility, and comprehensiveness. Evaluations were conducted within internal medicine, emergency medicine, and ethics. Comparisons were made globally, between seniors and residents, and across classifications. RESULTS Both GPT models received high mean scores (4.4, SD 0.8 for GPT-4 and 4.1, SD 1.0 for GPT-3.5). GPT-4 outperformed GPT-3.5 across all rating dimensions, with seniors consistently rating responses higher than residents for both models. Specifically, seniors rated GPT-4 as more beneficial and complete (mean 4.6 vs 4.0 and 4.6 vs 4.1, respectively; P<.001), and GPT-3.5 similarly (mean 4.1 vs 3.7 and 3.9 vs 3.5, respectively; P<.001). Ethical queries received the highest ratings for both models, with mean scores reflecting consistency across accuracy and completeness criteria. Distinctions among question types were significant, particularly for the GPT-4 mean scores in completeness across emergency, internal, and ethical questions (4.2, SD 1.0; 4.3, SD 0.8; and 4.5, SD 0.7, respectively; P<.001), and for GPT-3.5's accuracy, beneficial, and completeness dimensions. CONCLUSIONS ChatGPT's potential to assist physicians with medical issues is promising, with prospects to enhance diagnostics, treatments, and ethics. While integration into clinical workflows may be valuable, it must complement, not replace, human expertise. Continued research is essential to ensure safe and effective implementation in clinical environments.
Collapse
Affiliation(s)
- Adi Lahat
- Department of Gastroenterology, Chaim Sheba Medical Center, Affiliated with Tel Aviv University, Ramat Gan, Israel
- Department of Gastroenterology, Samson Assuta Ashdod Medical Center, Affiliated with Ben Gurion University of the Negev, Be'er Sheva, Israel
| | - Kassem Sharif
- Department of Gastroenterology, Chaim Sheba Medical Center, Affiliated with Tel Aviv University, Ramat Gan, Israel
- Department of Internal Medicine B, Sheba Medical Centre, Tel Aviv, Israel
| | - Narmin Zoabi
- Department of Gastroenterology, Chaim Sheba Medical Center, Affiliated with Tel Aviv University, Ramat Gan, Israel
| | | | - Yousra Sharif
- Department of Internal Medicine C, Hadassah Medical Center, Jerusalem, Israel
| | - Lior Fisher
- Department of Internal Medicine B, Sheba Medical Centre, Tel Aviv, Israel
| | - Uria Shani
- Department of Internal Medicine B, Sheba Medical Centre, Tel Aviv, Israel
| | - Mohamad Arow
- Department of Internal Medicine B, Sheba Medical Centre, Tel Aviv, Israel
| | - Roni Levin
- Department of Internal Medicine B, Sheba Medical Centre, Tel Aviv, Israel
| | - Eyal Klang
- Division of Data-Driven and Digital Medicine (D3M), Icahn School of Medicine at Mount Sinai, New York, NY, United States
| |
Collapse
|
44
|
Reynolds K, Nadelman D, Durgin J, Ansah-Addo S, Cole D, Fayne R, Harrell J, Ratycz M, Runge M, Shepard-Hayes A, Wenzel D, Tejasvi T. Comparing the quality of ChatGPT- and physician-generated responses to patients' dermatology questions in the electronic medical record. Clin Exp Dermatol 2024; 49:715-718. [PMID: 38180108 DOI: 10.1093/ced/llad456] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Revised: 11/28/2023] [Accepted: 12/22/2023] [Indexed: 01/06/2024]
Abstract
BACKGROUND ChatGPT is a free artificial intelligence (AI)-based natural language processing tool that generates complex responses to inputs from users. OBJECTIVES To determine whether ChatGPT is able to generate high-quality responses to patient-submitted questions in the patient portal. METHODS Patient-submitted questions and the corresponding responses from their dermatology physician were extracted from the electronic medical record for analysis. The questions were input into ChatGPT (version 3.5) and the outputs extracted for analysis, with manual removal of verbiage pertaining to ChatGPT's inability to provide medical advice. Ten blinded reviewers (seven physicians and three nonphysicians) rated and selected their preference in terms of 'overall quality', 'readability', 'accuracy', 'thoroughness' and 'level of empathy' of the physician- and ChatGPT-generated responses. RESULTS Thirty-one messages and responses were analysed. Physician-generated responses were vastly preferred over the ChatGPT -responses by the physician and nonphysician reviewers and received significantly higher ratings for 'readability' and 'level of empathy'. CONCLUSIONS The results of this study suggest that physician-generated responses to patients' portal messages are still preferred over ChatGPT, but generative AI tools may be helpful in generating the first drafts of responses and providing information on education resources for patients.
Collapse
Affiliation(s)
- Kelly Reynolds
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Daniel Nadelman
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Joseph Durgin
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | | | - Daniel Cole
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Rachel Fayne
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Jane Harrell
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Madison Ratycz
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Mason Runge
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | | | - Daniel Wenzel
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| | - Trilokraj Tejasvi
- Department of Dermatology, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
45
|
Jo MH, Kim MJ, Oh HK, Choi MJ, Shin HR, Lee TG, Ahn HM, Kim DW, Kang SB. Communicative competence of generative artificial intelligence in responding to patient queries about colorectal cancer surgery. Int J Colorectal Dis 2024; 39:94. [PMID: 38902500 PMCID: PMC11189990 DOI: 10.1007/s00384-024-04670-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/13/2024] [Indexed: 06/22/2024]
Abstract
PURPOSE To examine the ability of generative artificial intelligence (GAI) to answer patients' questions regarding colorectal cancer (CRC). METHODS Ten clinically relevant questions about CRC were selected from top-rated hospitals' websites and patient surveys and presented to three GAI tools (Chatbot Generative Pre-Trained Transformer [GPT-4], Google Bard, and CLOVA X). Their responses were compared with answers from the CRC information book. Response evaluation was performed by two groups, each consisting of five healthcare professionals (HCP) and patients. Each question was scored on a 1-5 Likert scale based on four evaluation criteria (maximum score, 20 points/question). RESULTS In an analysis including only HCPs, the information book scored 11.8 ± 1.2, GPT-4 scored 13.5 ± 1.1, Google Bard scored 11.5 ± 0.7, and CLOVA X scored 12.2 ± 1.4 (P = 0.001). The score of GPT-4 was significantly higher than those of the information book (P = 0.020) and Google Bard (P = 0.001). In an analysis including only patients, the information book scored 14.1 ± 1.4, GPT-4 scored 15.2 ± 1.8, Google Bard scored 15.5 ± 1.8, and CLOVA X scored 14.4 ± 1.8, without significant differences (P = 0.234). When both groups of evaluators were included, the information book scored 13.0 ± 0.9, GPT-4 scored 14.4 ± 1.2, Google Bard scored 13.5 ± 1.0, and CLOVA X scored 13.3 ± 1.5 (P = 0.070). CONCLUSION The three GAIs demonstrated similar or better communicative competence than the information book regarding questions related to CRC surgery in Korean. If high-quality medical information provided by GAI is supervised properly by HCPs and published as an information book, it could be helpful for patients to obtain accurate information and make informed decisions.
Collapse
Affiliation(s)
- Min Hyeong Jo
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
| | - Min-Jun Kim
- Department of Surgery, Seoul National University College of Medicine, Seoul, South Korea
| | - Heung-Kwon Oh
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea.
- Department of Surgery, Seoul National University College of Medicine, Seoul, South Korea.
| | - Mi Jeong Choi
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
| | - Hye-Rim Shin
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
| | - Tae-Gyun Lee
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
| | - Hong-Min Ahn
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
| | - Duck-Woo Kim
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
- Department of Surgery, Seoul National University College of Medicine, Seoul, South Korea
| | - Sung-Bum Kang
- Department of Surgery, Seoul National University Bundang Hospital, 300 Gumi-dong Bundang-gu, Seongnam-si, Gyeonggi-do, 13620, South Korea
- Department of Surgery, Seoul National University College of Medicine, Seoul, South Korea
| |
Collapse
|
46
|
Ríos-Hoyo A, Shan NL, Li A, Pearson AT, Pusztai L, Howard FM. Evaluation of large language models as a diagnostic aid for complex medical cases. Front Med (Lausanne) 2024; 11:1380148. [PMID: 38966538 PMCID: PMC11222590 DOI: 10.3389/fmed.2024.1380148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 06/10/2024] [Indexed: 07/06/2024] Open
Abstract
Background The use of large language models (LLM) has recently gained popularity in diverse areas, including answering questions posted by patients as well as medical professionals. Objective To evaluate the performance and limitations of LLMs in providing the correct diagnosis for a complex clinical case. Design Seventy-five consecutive clinical cases were selected from the Massachusetts General Hospital Case Records, and differential diagnoses were generated by OpenAI's GPT3.5 and 4 models. Results The mean number of diagnoses provided by the Massachusetts General Hospital case discussants was 16.77, by GPT3.5 30 and by GPT4 15.45 (p < 0.0001). GPT4 was more frequently able to list the correct diagnosis as first (22% versus 20% with GPT3.5, p = 0.86), provide the correct diagnosis among the top three generated diagnoses (42% versus 24%, p = 0.075). GPT4 was better at providing the correct diagnosis, when the different diagnoses were classified into groups according to the medical specialty and include the correct diagnosis at any point in the differential list (68% versus 48%, p = 0.0063). GPT4 provided a differential list that was more similar to the list provided by the case discussants than GPT3.5 (Jaccard Similarity Index 0.22 versus 0.12, p = 0.001). Inclusion of the correct diagnosis in the generated differential was correlated with PubMed articles matching the diagnosis (OR 1.40, 95% CI 1.25-1.56 for GPT3.5, OR 1.25, 95% CI 1.13-1.40 for GPT4), but not with disease incidence. Conclusions and relevance The GPT4 model was able to generate a differential diagnosis list with the correct diagnosis in approximately two thirds of cases, but the most likely diagnosis was often incorrect for both models. In its current state, this tool can at most be used as an aid to expand on potential diagnostic considerations for a case, and future LLMs should be trained which account for the discrepancy between disease incidence and availability in the literature.
Collapse
Affiliation(s)
| | - Naing Lin Shan
- Yale Cancer Center, Yale School of Medicine, New Haven, CT, United States
| | - Anran Li
- Department of Medicine, University of Chicago, Chicago, IL, United States
| | | | - Lajos Pusztai
- Yale Cancer Center, Yale School of Medicine, New Haven, CT, United States
| | | |
Collapse
|
47
|
Deeb M, Gangadhar A, Rabindranath M, Rao K, Brudno M, Sidhu A, Wang B, Bhat M. The emerging role of generative artificial intelligence in transplant medicine. Am J Transplant 2024:S1600-6135(24)00382-4. [PMID: 38901561 DOI: 10.1016/j.ajt.2024.06.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Revised: 05/26/2024] [Accepted: 06/11/2024] [Indexed: 06/22/2024]
Abstract
Generative artificial intelligence (AI), a subset of machine learning that creates new content based on training data, has witnessed tremendous advances in recent years. Practical applications have been identified in health care in general, and there is significant opportunity in transplant medicine for generative AI to simplify tasks in research, medical education, and clinical practice. In addition, patients stand to benefit from patient education that is more readily provided by generative AI applications. This review aims to catalyze the development and adoption of generative AI in transplantation by introducing basic AI and generative AI concepts to the transplant clinician and summarizing its current and potential applications within the field. We provide an overview of applications to the clinician, researcher, educator, and patient. We also highlight the challenges involved in bringing these applications to the bedside and need for ongoing refinement of generative AI applications to sustainably augment the transplantation field.
Collapse
Affiliation(s)
- Maya Deeb
- Ajmera Transplant Program, University Health Network Toronto, Ontario, Canada; Division of Gastroenterology and Hepatology, Department of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Anirudh Gangadhar
- Ajmera Transplant Program, University Health Network Toronto, Ontario, Canada
| | | | - Khyathi Rao
- Ajmera Transplant Program, University Health Network Toronto, Ontario, Canada
| | - Michael Brudno
- DATA Team, University Health Network, Toronto, Ontario, Canada
| | - Aman Sidhu
- Ajmera Transplant Program, University Health Network Toronto, Ontario, Canada
| | - Bo Wang
- DATA Team, University Health Network, Toronto, Ontario, Canada
| | - Mamatha Bhat
- Ajmera Transplant Program, University Health Network Toronto, Ontario, Canada; Division of Gastroenterology and Hepatology, Department of Medicine, University of Toronto, Toronto, Ontario, Canada.
| |
Collapse
|
48
|
Özcan F, Örücü Atar M, Köroğlu Ö, Yılmaz B. Assessment of the reliability and usability of ChatGPT in response to spinal cord injury questions. J Spinal Cord Med 2024:1-6. [PMID: 38860862 DOI: 10.1080/10790268.2024.2361551] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/12/2024] Open
Abstract
OBJECTIVE The use of artificial intelligence chatbots to obtain information about patients' diseases is increasing. This study aimed to determine the reliability and usability of ChatGPT for spinal cord injury-related questions. METHODS Three raters simultaneously evaluated a total of 47 questions on a 7-point Likert scale for reliability and usability, based on the three most frequently searched keywords in Google Trends ('general information', 'complications' and 'treatment'). RESULTS Inter-rater Cronbach α scores indicated substantial agreement for both reliability and usability scores (α between 0.558 and 0.839, and α between 0.373 and 0.772, respectively). The highest mean reliability score was for 'complications' (mean 5.38). The lowest average was for the 'general information' section (mean 4.20). The 'treatment' had the highest mean scores for the usability (mean 5.87) and the lowest mean value was recorded in the 'general information' section (mean 4.80). CONCLUSION The answers given by ChatGPT to questions related to spinal cord injury were reliable and useful. Nevertheless, it should be kept in mind that ChatGPT may provide incorrect or incomplete information, especially in the 'general information' section, which may mislead patients and their relatives.
Collapse
Affiliation(s)
- Fatma Özcan
- Department of Physical Medicine and Rehabilitation, University of Health Sciences, Gaziler Physical Medicine and Rehabilitation Training and Research Hospital, Ankara, Turkey
| | - Merve Örücü Atar
- Department of Physical Medicine and Rehabilitation, University of Health Sciences, Gaziler Physical Medicine and Rehabilitation Training and Research Hospital, Ankara, Turkey
| | - Özlem Köroğlu
- Department of Physical Medicine and Rehabilitation, University of Health Sciences, Gaziler Physical Medicine and Rehabilitation Training and Research Hospital, Ankara, Turkey
| | - Bilge Yılmaz
- Department of Physical Medicine and Rehabilitation, University of Health Sciences, Gaziler Physical Medicine and Rehabilitation Training and Research Hospital, Ankara, Turkey
| |
Collapse
|
49
|
Kooraki S, Hosseiny M, Jalili MH, Rahsepar AA, Imanzadeh A, Kim GH, Hassani C, Abtin F, Moriarty JM, Bedayat A. Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures. Acad Radiol 2024:S1076-6332(24)00307-6. [PMID: 38839458 DOI: 10.1016/j.acra.2024.05.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Revised: 05/10/2024] [Accepted: 05/13/2024] [Indexed: 06/07/2024]
Abstract
RATIONALE AND OBJECTIVES This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures. METHODS AND MATERIALS Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets. RESULTS In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001). CONCLUSION ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.
Collapse
Affiliation(s)
- Soheil Kooraki
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA.
| | - Melina Hosseiny
- Department of Radiology, University of California, San Diego (UCSD), San Diego, CA.
| | - Mohamamd H Jalili
- Department of radiology and biomedical imaging, Yale New Haven Health, Bridgeport Hospital, CT.
| | - Amir Ali Rahsepar
- Department of Radiology, Feinberg School of Medicine, Northwestern University, Chicago, IL.
| | - Amir Imanzadeh
- Department of Radiology, University of California, Irvine (UCI), Irvine, CA.
| | - Grace Hyun Kim
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA.
| | - Cameron Hassani
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA.
| | - Fereidoun Abtin
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA.
| | - John M Moriarty
- Department of Radiological Sciences, Division of Interventional Radiology, David Geffen School of Medicine at UCLA, Los Angeles, CA.
| | - Arash Bedayat
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA.
| |
Collapse
|
50
|
Yao Y, Hasan WZW, Jiao W, Dong X, Ramli HR, Norsahperi NMH, Wen D. ChatGPT and BCI-VR: a new integrated diagnostic and therapeutic perspective for the accurate diagnosis and personalized treatment of mild cognitive impairment. Front Hum Neurosci 2024; 18:1426055. [PMID: 38895167 PMCID: PMC11183516 DOI: 10.3389/fnhum.2024.1426055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2024] [Accepted: 05/17/2024] [Indexed: 06/21/2024] Open
Affiliation(s)
- Yiduo Yao
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Selangor, Malaysia
| | - W. Z. W. Hasan
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Selangor, Malaysia
| | - Wenlong Jiao
- Brain Computer Intelligence and Intelligent Health Institute, School of Intelligence Science and Technology, University of Science and Technology Beijing, Chengde, China
| | - Xianling Dong
- Hebei International Research Center of Medical Engineering, Chengde Medical University, Hebei, China
| | - H. R. Ramli
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Selangor, Malaysia
| | - N. M. H. Norsahperi
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Selangor, Malaysia
| | - Dong Wen
- Brain Computer Intelligence and Intelligent Health Institute, School of Intelligence Science and Technology, University of Science and Technology Beijing, Chengde, China
| |
Collapse
|