Evaluating the Accuracy, Comprehensiveness, and Validity of ChatGPT Compared to Evidence-Based Sources Regarding Common Surgical Conditions: Surgeons' Perspectives

Is there any room for ChatGPT AI bot in speech-language pathology?

PURPOSE

This study investigates the potential of the ChatGPT-4.0 artificial intelligence bot to assist speech-language pathologists (SLPs) by assessing its accuracy, comprehensiveness, and relevance in various tasks related to speech, language, and swallowing disorders.

METHOD

In this cross-sectional descriptive study, 15 practicing SLPs evaluated ChatGPT-4.0's responses to task-specific queries across six core areas: report writing, assessment material generation, clinical decision support, therapy stimulus generation, therapy planning, and client/family training material generation. English prompts were created in seven areas: speech sound disorders, motor speech disorders, aphasia, stuttering, childhood language disorders, voice disorders, and swallowing disorders. These prompts were entered into ChatGPT-4.0, and its responses were evaluated. Using a three-point Likert-type scale, participants rated each response for accuracy, relevance, and comprehensiveness based on clinical expectations and their professional judgment.

RESULTS

The study revealed that ChatGPT-4.0 performed with predominantly high accuracy, comprehensiveness, and relevance in tasks related to speech and language disorders. High accuracy, comprehensiveness, and relevance levels were observed in report writing, clinical decision support, and creating education material. However, tasks such as creating therapy stimuli and therapy planning showed more variation with medium and high accuracy levels.

CONCLUSIONS

ChatGPT-4.0 shows promise in assisting SLPs with various professional tasks, particularly report writing, clinical decision support, and education material creation. However, further research is needed to address its limitations in therapy stimulus generation and therapy planning to improve its usability in clinical practice. Integrating AI technologies such as ChatGPT could improve the efficiency and effectiveness of therapeutic processes in speech-language pathology.

Using Large Language Models in the Diagnosis of Acute Cholecystitis: Assessing Accuracy and Guidelines Compliance

BackgroundLarge language models (LLMs) are advanced tools capable of understanding and generating human-like text. This study evaluated the accuracy of several commercial LLMs in addressing clinical questions related to diagnosis and management of acute cholecystitis, as outlined in the Tokyo Guidelines 2018 (TG18). We assessed their congruence with the expert panel discussions presented in the guidelines.MethodsWe evaluated ChatGPT4.0, Gemini Advanced, and GPTo1-preview on ten clinical questions. Eight derived from TG18, and two were formulated by the authors. Two authors independently rated the accuracy of each LLM's responses on a four-point scale: (1) accurate and comprehensive, (2) accurate but not comprehensive, (3) partially accurate, partially inaccurate, and (4) entirely inaccurate. A third author resolved any scoring discrepancies. Then, we comparatively analyzed the performance of ChatGPT4.0 against newer large language models (LLMs), specifically Gemini Advanced and GPTo1-preview, on the same set of questions to delineate their respective strengths and limitations.ResultsChatGPT4.0 provided consistent responses for 90% of the questions. It delivered "accurate and comprehensive" answers for 4/10 (40%) questions and "accurate but not comprehensive" answers for 5/10 (50%). One response (10%) was rated as "partially accurate, partially inaccurate." Gemini Advanced demonstrated higher accuracy on some questions but yielded a similar percentage of "partially accurate, partially inaccurate" responses. Notably, neither model produced "entirely inaccurate" answers.DiscussionLLMs, such as ChatGPT and Gemini Advanced, demonstrate potential in accurately addressing clinical questions regarding acute cholecystitis. With awareness of their limitations, their careful implementation, and ongoing refinement, LLMs could serve as valuable resources for physician education and patient information, potentially improving clinical decision-making in the future.