|
1
|
De Nunzio C, Lombardo R. Best of 2023 in Prostate Cancer and Prostatic Diseases. Prostate Cancer Prostatic Dis 2024:10.1038/s41391-024-00790-7. [PMID: 38263282 DOI: 10.1038/s41391-024-00790-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Revised: 01/05/2024] [Accepted: 01/10/2024] [Indexed: 01/25/2024]
Affiliation(s)
- Cosimo De Nunzio
- Department of Urology, Sapienza University of Rome, Rome, Italy.
| | | |
Collapse
|
|
2
|
Lombardo R, Gallo G, Stira J, Turchi B, Santoro G, Riolo S, Romagnoli M, Cicione A, Tema G, Pastore A, Al Salhi Y, Fuschi A, Franco G, Nacchia A, Tubaro A, De Nunzio C. Quality of information and appropriateness of Open AI outputs for prostate cancer. Prostate Cancer Prostatic Dis 2024:10.1038/s41391-024-00789-0. [PMID: 38228809 DOI: 10.1038/s41391-024-00789-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 12/22/2023] [Accepted: 01/05/2024] [Indexed: 01/18/2024]
Abstract
Chat-GPT, a natural language processing (NLP) tool created by Open-AI, can potentially be used as a quick source for obtaining information related to prostate cancer. This study aims to analyze the quality and appropriateness of Chat-GPT's responses to inquiries related to prostate cancer compared to those of the European Urology Association's (EAU) 2023 prostate cancer guidelines. Overall, 195 questions were prepared according to the recommendations gathered in the prostate cancer section of the EAU 2023 Guideline. All questions were systematically presented to Chat-GPT's August 3 Version, and two expert urologists independently assessed and assigned scores ranging from 1 to 4 to each response (1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect). Sub-analysis per chapter and per grade of recommendation were performed. Overall, 195 recommendations were evaluated. Overall, 50/195 (26%) were completely correct, 51/195 (26%) correct but inadequate, 47/195 (24%) a mix of correct and misleading and 47/195 (24%) incorrect. When looking at different chapters Open AI was particularly accurate in answering questions on follow-up and QoL. Worst performance was recorded for the diagnosis and treatment chapters with respectively 19% and 30% of the answers completely incorrect. When looking at the strength of recommendation, no differences in terms of accuracy were recorded when comparing weak and strong recommendations (p > 0,05). Chat-GPT has a poor accuracy when answering questions on the PCa EAU guidelines recommendations. Future studies should assess its performance after adequate training.
Collapse
Affiliation(s)
| | - Giacomo Gallo
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Jordi Stira
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Beatrice Turchi
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Giuseppe Santoro
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Sara Riolo
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Matteo Romagnoli
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Antonio Cicione
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Giorgia Tema
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Antonio Pastore
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Yazan Al Salhi
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Andrea Fuschi
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Giorgio Franco
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Antonio Nacchia
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Andrea Tubaro
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| | - Cosimo De Nunzio
- Department of Urology, 'Sapienza' University of Rome, Rome, Italy
| |
Collapse
|