|
1
|
Wu Y, Zhang Z, Dong X, Hong S, Hu Y, Liang P, Li L, Zou B, Wu X, Wang D, Chen H, Qiu H, Tang H, Kang K, Li Q, Zhai X. Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy. Epilepsy Behav 2024; 151:109645. [PMID: 38244419 DOI: 10.1016/j.yebeh.2024.109645] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 01/02/2024] [Accepted: 01/10/2024] [Indexed: 01/22/2024]
Abstract
OBJECTIVE People with epilepsy desire to acquire accurate information about epilepsy and actively engage in its management throughout the long journey of living with seizures. ChatGPT is a large language model and we aimed to assess the accuracy and consistency of ChatGPT in responding to the common concerns of people with epilepsy and to evaluate its ability to provide emotional support. METHODS Questions were collected from the International League against Epilepsy and the China Association against Epilepsy. The responses were independently assessed by two board-certified epileptologists from the China Association against Epilepsy, and a third reviewer resolved disagreements. The reviewers assessed its ability to provide emotional support subjectively. RESULTS A total of 378 questions related to epilepsy and 5 questions related to emotional support were included. ChatGPT provided "correct and comprehensive" answers to 68.4% of the questions. The model provided reproducible answers for 82.3% questions. The model performed poorly in answering prognostic questions, with only 46.8% of the answers rated as comprehensive. When faced with questions requiring emotional support, the model can generate natural and understandable responses. SIGNIFICANCE ChatGPT provides accurate and reliable answers to patients with epilepsy and is a valuable source of information. It also provides partial emotional support, potentially assisting those experiencing emotional distress. However, ChatGPT may provide incorrect responses, leading users to inadvertently accept incorrect and potentially dangerous advice. Therefore, the direct use of ChatGPT for medical guidance is not recommended and its primary use at present is in patients education.
Collapse
Affiliation(s)
- YuXin Wu
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Zaiyu Zhang
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Xinyu Dong
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Siqi Hong
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Yue Hu
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Ping Liang
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Lusheng Li
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Bin Zou
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Xuanxuan Wu
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Difei Wang
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Hui Chen
- MEG-Center, University of Tübingen, Germany; Abteilung für Neuronale Dynamik und Magnetoenzephalographie, Hertie-Institut für klinische Hirnforschung, Universität Tübingen, Tübingen, Deutschland.
| | - Hanli Qiu
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Haotian Tang
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Kaiyi Kang
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Qinling Li
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| | - Xuan Zhai
- Department of Neurosurgery, Children's Hospital of Chongqing Medical University, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China; Chongqing Key Laboratory of Translational Medical Research in Cognitive Development and Learning and Memory Disorders, Chongqing, China.
| |
Collapse
|