Assessing the performance of ChatGPT's responses to questions related to epilepsy: A cross-sectional study on natural language processing and medical information retrieval

Artificial intelligence: Can it help us better grasp the idea of epilepsy? An exploratory dialogue with ChatGPT and DALL·E 2

BACKGROUND

The conceptual definition of epilepsy has been changing over decades and remains debatable. We assessed how artificial intelligence (AI) conceives epilepsy and its impact on a person's life through verbal and visual material.

METHODS

We asked the Chat Generative Pre-Trained Transformer (ChatGPT, OpenAI) to define epilepsy and its impact. Prompts from ChatGPT were transferred to another AI tool DALL·E 2 (Open AI) to generate visual images based on verbal input.

RESULTS

The ChatGPT definition on epilepsy relied on both its conceptual and practical definitions. It titled epilepsy to be "a neurological disorder characterized by recurring seizures" that has significant impact on patients' lives and is diagnosed after two or more unprovoked seizures or if there is a high risk of future seizures. ChatGPT presented nine issues - seizure-related injuries, limitations on daily activities, emotional and psychological impact, social stigma and isolation, educational and employment challenges, relationship and family dynamics, medication side effects, financial burden, and coexisting conditions - as major consequences of epilepsy. AI-generated images ranged from direct portrayals of these phenomena to abstract imagery but were mostly deprived of symbolic elements and visual metaphors.

CONCLUSION

We showed that AI can identify and visually interpret the burden of epilepsy from medical, societal and economical perspectives. However, the imagery created is not figurative and does not follow allegorical narratives put forward by epilepsy specialists in similar studies. The ability of AI models to lead an in-depth discussion on epilepsy remains questionable and should be explored with more advanced AI tools.

ChatGPT Provides Satisfactory but Occasionally Inaccurate Answers to Common Patient Hip Arthroscopy Questions

PURPOSE

To assess the ability of ChatGPT to answer common patient questions regarding hip arthroscopy, and to analyze the accuracy and appropriateness of its responses.

METHODS

Ten questions were selected from well-known patient education websites, and ChatGPT (version 3.5) responses to these questions were graded by 2 fellowship-trained hip preservation surgeons. Responses were analyzed, compared with the current literature, and graded from A to D (A being the highest, and D being the lowest) in a grading scale on the basis of the accuracy and completeness of the response. If the grading differed between the 2 surgeons, a consensus was reached. Inter-rater agreement was calculated. The readability of responses was also assessed using the Flesch-Kincaid Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL).

RESULTS

Responses received the following consensus grades: A (50%, n = 5), B (30%, n = 3), C (10%, n = 1), D (10%, n = 1). Inter-rater agreement on the basis of initial individual grading was 30%. The mean FRES was 28.2 (± 9.2 standard deviation), corresponding to a college graduate level, ranging from 11.7 to 42.5. The mean FKGL was 14.4 (±1.8 standard deviation), ranging from 12.1 to 18, indicating a college student reading level.

CONCLUSIONS

ChatGPT can answer common patient questions regarding hip arthroscopy with satisfactory accuracy graded by 2 high-volume hip arthroscopists; however, incorrect information was identified in more than one instance. Caution must be observed when using ChatGPT for patient education related to hip arthroscopy.

CLINICAL RELEVANCE

Given the increasing number of hip arthroscopies being performed annually, ChatGPT has the potential to aid physicians in educating their patients about this procedure and addressing any questions they may have.

The promise of AI Large Language Models for Epilepsy care

Artificial intelligence (AI) has been supporting our digital life for decades, but public interest in this has exploded with the recognition of large language models, such as GPT-4. We examine and evaluate the potential uses for generative AI technologies in epilepsy and neurological services. Generative AI could not only improve patient care and safety by refining communication and removing certain barriers to healthcare but may also extend to streamlining a doctor's practice through strategies such as automating paperwork. Challenges with the integration of generative AI in epilepsy services are also explored and include the risk of producing inaccurate and biased information. The impact generative AI could have on the provision of healthcare, both positive and negative, should be understood and considered carefully when deciding on the steps that need to be taken before AI is ready for use in hospitals and epilepsy services.

Potential merits and flaws of large language models in epilepsy care: A critical review

The current pace of development and applications of large language models (LLMs) is unprecedented and will impact future medical care significantly. In this critical review, we provide the background to better understand these novel artificial intelligence (AI) models and how LLMs can be of future use in the daily care of people with epilepsy. Considering the importance of clinical history taking in diagnosing and monitoring epilepsy-combined with the established use of electronic health records-a great potential exists to integrate LLMs in epilepsy care. We present the current available LLM studies in epilepsy. Furthermore, we highlight and compare the most commonly used LLMs and elaborate on how these models can be applied in epilepsy. We further discuss important drawbacks and risks of LLMs, and we provide recommendations for overcoming these limitations.

Evaluating ChatGPT-4.0's data analytic proficiency in epidemiological studies: A comparative analysis with SAS, SPSS, and R

Background

OpenAI's Chat Generative Pre-trained Transformer 4.0 (ChatGPT-4), an emerging artificial intelligence (AI)-based large language model (LLM), has been receiving increasing attention from the medical research community for its innovative 'Data Analyst' feature. We aimed to compare the capabilities of ChatGPT-4 against traditional biostatistical software (i.e. SAS, SPSS, R) in statistically analysing epidemiological research data.

Methods

We used a data set from the China Health and Nutrition Survey, comprising 9317 participants and 29 variables (e.g. gender, age, educational level, marital status, income, occupation, weekly working hours, survival status). Two researchers independently evaluated the data analysis capabilities of GPT-4's 'Data Analyst' feature against SAS, SPSS, and R across three commonly used epidemiological analysis methods: Descriptive statistics, intergroup analysis, and correlation analysis. We used an internally developed evaluation scale to assess and compare the consistency of results, analytical efficiency of coding or operations, user-friendliness, and overall performance between ChatGPT-4, SAS, SPSS, and R.

Results

In descriptive statistics, ChatGPT-4 showed high consistency of results, greater analytical efficiency of code or operations, and more intuitive user-friendliness compared to SAS, SPSS, and R. In intergroup comparisons and correlational analyses, despite minor discrepancies in statistical outcomes for certain analysis tasks with SAS, SPSS, and R, ChatGPT-4 maintained high analytical efficiency and exceptional user-friendliness. Thus, employing ChatGPT-4 can significantly lower the operational threshold for conducting epidemiological data analysis while maintaining consistency with traditional biostatistical software's outcome, requiring only specific, clear analysis instructions without any additional operations or code writing.

Conclusions

We found ChatGPT-4 to be a powerful auxiliary tool for statistical analysis in epidemiological research. However, it showed limitations in result consistency and in applying more advanced statistical methods. Therefore, we advocate for the use of ChatGPT-4 in supporting researchers with intermediate experience in data analysis. With AI technologies like LLMs advancing rapidly, their integration with data analysis platforms promises to lower operational barriers, thereby enabling researchers to dedicate greater focus to the nuanced interpretation of analysis results. This development is likely to significantly advance epidemiological and medical research.

ChatGPT's responses to questions related to epilepsy

This is a correspondence on published article on "ChatGPT's responses to questions related to epilepsy".