|
1
|
Mankowski MA, Jaffe IS, Xu J, Bae S, Oermann EK, Aphinyanaphongs Y, McAdams-DeMarco MA, Lonze BE, Orandi BJ, Stewart D, Levan M, Massie A, Gentry S, Segev DL. ChatGPT Solving Complex Kidney Transplant Cases: A Comparative Study With Human Respondents. Clin Transplant 2024; 38:e15466. [PMID: 39329220 PMCID: PMC11441623 DOI: 10.1111/ctr.15466] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2024] [Revised: 08/26/2024] [Accepted: 09/06/2024] [Indexed: 09/28/2024]
Abstract
INTRODUCTION ChatGPT has shown the ability to answer clinical questions in general medicine but may be constrained by the specialized nature of kidney transplantation. Thus, it is important to explore how ChatGPT can be used in kidney transplantation and how its knowledge compares to human respondents. METHODS We prompted ChatGPT versions 3.5, 4, and 4 Visual (4 V) with 12 multiple-choice questions related to six kidney transplant cases from 2013 to 2015 American Society of Nephrology (ASN) fellowship program quizzes. We compared the performance of ChatGPT with US nephrology fellowship program directors, nephrology fellows, and the audience of the ASN's annual Kidney Week meeting. RESULTS Overall, ChatGPT 4 V correctly answered 10 out of 12 questions, showing a performance level comparable to nephrology fellows (group majority correctly answered 9 of 12 questions) and training program directors (11 of 12). This surpassed ChatGPT 4 (7 of 12 correct) and 3.5 (5 of 12). All three ChatGPT versions failed to correctly answer questions where the consensus among human respondents was low. CONCLUSION Each iterative version of ChatGPT performed better than the prior version, with version 4 V achieving performance on par with nephrology fellows and training program directors. While it shows promise in understanding and answering kidney transplantation questions, ChatGPT should be seen as a complementary tool to human expertise rather than a replacement.
Collapse
Affiliation(s)
- Michal A Mankowski
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Ian S Jaffe
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Jingzhi Xu
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Sunjae Bae
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| | - Eric K Oermann
- Department of Neurosurgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Yindalon Aphinyanaphongs
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
- Department of Medicine, NYU Grossman School of Medicine, New York, New York, USA
| | - Mara A McAdams-DeMarco
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| | - Bonnie E Lonze
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Babak J Orandi
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Medicine, NYU Grossman School of Medicine, New York, New York, USA
| | - Darren Stewart
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
| | - Macey Levan
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| | - Allan Massie
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| | - Sommer Gentry
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| | - Dorry L Segev
- Department of Surgery, NYU Grossman School of Medicine, New York, New York, USA
- Department of Population Health, NYU Grossman School of Medicine, New York, New York, USA
| |
Collapse
|