ChatGPT: is it good for our glaucoma patients?

Understanding natural language: Potential application of large language models to ophthalmology

Large language models (LLMs), a natural language processing technology based on deep learning, are currently in the spotlight. These models closely mimic natural language comprehension and generation. Their evolution has undergone several waves of innovation similar to convolutional neural networks. The transformer architecture advancement in generative artificial intelligence marks a monumental leap beyond early-stage pattern recognition via supervised learning. With the expansion of parameters and training data (terabytes), LLMs unveil remarkable human interactivity, encompassing capabilities such as memory retention and comprehension. These advances make LLMs particularly well-suited for roles in healthcare communication between medical practitioners and patients. In this comprehensive review, we discuss the trajectory of LLMs and their potential implications for clinicians and patients. For clinicians, LLMs can be used for automated medical documentation, and given better inputs and extensive validation, LLMs may be able to autonomously diagnose and treat in the future. For patient care, LLMs can be used for triage suggestions, summarization of medical documents, explanation of a patient's condition, and customizing patient education materials tailored to their comprehension level. The limitations of LLMs and possible solutions for real-world use are also presented. Given the rapid advancements in this area, this review attempts to briefly cover many roles that LLMs may play in the ophthalmic space, with a focus on improving the quality of healthcare delivery.

Evaluating Chatbot responses to patient questions in the field of glaucoma

Objective

The aim of this study was to evaluate the accuracy, comprehensiveness, and safety of a publicly available large language model (LLM)-ChatGPT in the sub-domain of glaucoma.

Design

Evaluation of diagnostic test or technology.

Subjects participants and/or controls

We seek to evaluate the responses of an artificial intelligence chatbot ChatGPT (version GPT-3.5, OpenAI).

Methods intervention or testing

We curated 24 clinically relevant questions in the domain of glaucoma. The questions spanned four categories: pertaining to diagnosis, treatment, surgeries, and ocular emergencies. Each question was posed to the LLM and the responses obtained were graded by an expert grader panel of three glaucoma specialists with combined experience of more than 30 years in the field. For responses which performed poorly, the LLM was further prompted to self-correct. The subsequent responses were then re-evaluated by the expert panel.

Main outcome measures

Accuracy, comprehensiveness, and safety of the responses of a public domain LLM.

Results

There were a total of 24 questions and three expert graders with a total number of responses of n = 72. The scores were ranked from 1 to 4, where 4 represents the best score with a complete and accurate response. The mean score of the expert panel was 3.29 with a standard deviation of 0.484. Out of the 24 question-response pairs, seven (29.2%) of them had a mean inter-grader score of 3 or less. The mean score of the original seven question-response pairs was 2.96 which rose to 3.58 after an opportunity to self-correct (z-score - 3.27, p = 0.001, Mann-Whitney U). The seven out of 24 question-response pairs which performed poorly were given a chance to self-correct. After self-correction, the proportion of responses obtaining a full score increased from 22/72 (30.6%) to 12/21 (57.1%), (p = 0.026, χ2 test).

Conclusion

LLMs show great promise in the realm of glaucoma with additional capabilities of self-correction. The application of LLMs in glaucoma is still in its infancy, and still requires further research and validation.

An Analysis of ChatGPT4 to Respond to Glaucoma-Related Questions

PRCIS

In recent years, ChatGPT has been widely used as a source of information. In our study, it was revealed that ChatGPT gives accurate information about glaucoma.

PURPOSE

We examined the knowledge of ChatGPT about glaucoma.

MATERIALS AND METHODS

Frequently asked questions about glaucoma found on websites of ophthalmology associations and hospitals, and social media applications were assessed. Evidence-Based Recommendations in the European Glaucoma Society Terminology and Guidelines for Glaucoma, Fifth Edition were evaluated. Using the ChatGPT-4, each question was asked twice on different computers to assess the reproducibility of answers. The answers provided were recorded and 2 specialist ophthalmologists evaluated them independently, assigning scores ranging from 1 to 4.

RESULTS

The answers to all questions about glaucoma resulted in 88.7% completely correct, 7.5% correct but insufficient, and 3.8% misleading information and correct information. No question was answered completely incorrectly. While 85.8% of the general knowledge questions were answered correctly, 91.7%, 86.6%, and 91.7% of questions about diagnosis, treatment, and prevention were answered correctly, respectively. The number of questions prepared based on the European Glaucoma Society Terminology and Guidelines for Glaucoma was 16. The rate of completely correct answers to these questions was 75.0% (12). While 3 (18.8%) answers were correct but insufficient, 1 response (6.3%) contained false information and correct information.

CONCLUSIONS

Our study revealed that ChatGPT answered 9 out of 10 questions about general information, diagnosis, treatment, and preventive and follow-up about glaucoma with acceptable and satisfactory accuracy rates. In addition, 3 of 4 answers given by ChatGPT were completely correct according to Terminology and Guidelines for Glaucoma.

Artificial Versus Human Intelligence in the Diagnostic Approach of Ophthalmic Case Scenarios: A Qualitative Evaluation of Performance and Consistency

PURPOSE

To evaluate the efficiency of three artificial intelligence (AI) chatbots (ChatGPT-3.5 (OpenAI, San Francisco, California, United States), Bing Copilot (Microsoft Corporation, Redmond, Washington, United States), Google Gemini (Google LLC, Mountain View, California, United States)) in assisting the ophthalmologist in the diagnostic approach and management of challenging ophthalmic cases and compare their performance with that of a practicing human ophthalmic specialist. The secondary aim was to assess the short- and medium-term consistency of ChatGPT's responses.

METHODS

Eleven ophthalmic case scenarios of variable complexity were presented to the AI chatbots and to an ophthalmic specialist in a stepwise fashion. Advice regarding the initial differential diagnosis, the final diagnosis, further investigation, and management was asked for. One month later, the same process was repeated twice on the same day for ChatGPT only.

RESULTS

The individual diagnostic performance of all three AI chatbots was inferior to that of the ophthalmic specialist; however, they provided useful complementary input in the diagnostic algorithm. This was especially true for ChatGPT and Bing Copilot. ChatGPT exhibited reasonable short- and medium-term consistency, with the mean Jaccard similarity coefficient of responses varying between 0.58 and 0.76.

CONCLUSION

AI chatbots may act as useful assisting tools in the diagnosis and management of challenging ophthalmic cases; however, their responses should be scrutinized for potential inaccuracies, and by no means can they replace consultation with an ophthalmic specialist.

No-boundary thinking for artificial intelligence in bioinformatics and education

No-boundary thinking enables the scientific community to reflect in a thoughtful manner and discover new opportunities, create innovative solutions, and break through barriers that might have otherwise constrained their progress. This concept encourages thinking without being confined by traditional rules, limitations, or established norms, and a mindset that is not limited by previous work, leading to fresh perspectives and innovative outcomes. So, where do we see the field of artificial intelligence (AI) in bioinformatics going in the next 30 years? That was the theme of a "No-Boundary Thinking" Session as part of the Mid-South Computational Bioinformatics Society's (MCBIOS) 19th annual meeting in Irving, Texas. This session addressed various areas of AI in an open discussion and raised some perspectives on how popular tools like ChatGPT can be integrated into bioinformatics, communicating with scientists in different fields to properly utilize the potential of these algorithms, and how to continue educational outreach to further interest of data science and informatics to the next-generation of scientists.