Evaluating artificial intelligence chatbots for patient education in oral and maxillofacial radiology

Readability, accuracy and appropriateness and quality of AI chatbot responses as a patient information source on root canal retreatment: A comparative assessment

AIM

This study aimed to assess the readability, accuracy, appropriateness, and overall quality of responses generated by large language models (LLMs), including ChatGPT-3.5, Microsoft Copilot, and Gemini (Version 2.0 Flash), to frequently asked questions (FAQs) related to root canal retreatment.

METHODS

Three LLM chatbots-ChatGPT-3.5, Microsoft Copilot, and Gemini (Version 2.0 Flash)-were assessed based on their responses to 10 patient FAQs. Readability was analyzed using seven indices, including Flesch reading ease score (FRES), Flesch-Kincaid grade level (FKGL), Simple Measure of Gobbledygook (SMOG), gunning FOG (GFOG), Linsear Write (LW), Coleman-Liau (CL), and automated readability index (ARI), and compared against the recommended sixth-grade reading level. Response quality was evaluated using the Global Quality Scale (GQS), while accuracy and appropriateness were rated on a five-point Likert scale by two independent reviewers. Statistical analyses were conducted using one-way ANOVA, Tukey or Games-Howell post-hoc tests for continuous variables. Spearman's correlation test was used to assess associations between categorical variables.

RESULTS

All chatbots generated responses exceeding the recommended readability level, making them suitable for readers at or above the 10th-grade level. No significant difference was found between ChatGPT-3.5 and Microsoft Copilot, while Gemini produced significantly more readable responses (p < 0.05). Gemini demonstrated the highest proportion of accurate (80 %) and high-quality responses (80 %) compared to ChatGPT-3.5 and Microsoft Copilot.

CONCLUSIONS

None of the chatbots met the recommended readability standards for patient education materials. While Gemini demonstrated better readability, accuracy, and quality, all three models require further optimization to enhance accessibility and reliability in patient communication.

The impact of the large language model ChatGPT in oral and maxillofacial surgery: a systematic review

This systematic review evaluates the impact of the large language model (LLM) ChatGPT in oral and maxillofacial surgery. Following PRISMA guidelines and registered in PROSPERO (CRD42024625882), the study involved a comprehensive search across PubMed/Medline, Embase, Scopus, and Science Direct. Inclusion criteria focused on ChatGPT's use in clinical decision-making, surgical planning, patient education, and research. Ten studies were reviewed, assessing ChatGPT's performance in diagnostic accuracy, surgical efficiency, and patient satisfaction. GPT-4 achieved the highest accuracy (76.8%) in multiple-choice questions but showed variability, with lower performance in pharmacology and complex clinical scenarios. It excelled in generating informed consent documents, outperforming other AI models and human residents in accuracy, completeness, and readability. It also provided accurate and supportive responses in postoperative follow up. However, limitations were noted in handling complex clinical decisions and providing personalised advice for cases such as oral cancer and orthognathic surgery. While ChatGPT shows potential in improving patient communication, reducing healthcare workloads, and providing up-to-date information, its current limitations in accuracy and personalisation underscore the need for human supervision and integration with clinical databases. AI tools like ChatGPT can complement, but should not replace, human judgment in specialised fields such as oral and maxillofacial surgery.

Comparative evaluation of responses from DeepSeek-R1, ChatGPT-o1, ChatGPT-4, and dental GPT chatbots to patient inquiries about dental and maxillofacial prostheses

BACKGROUND

Artificial intelligence chatbots have the potential to inform and guide patients by providing human-like responses to questions about dental and maxillofacial prostheses. Information regarding the accuracy and qualifications of these responses is limited. This in-silico study aimed to evaluate the accuracy, quality, readability, understandability, and actionability of the responses from DeepSeek-R1, ChatGPT-o1, ChatGPT-4, and Dental GPT chatbots.

METHODS

Four chatbots were queried about 35 of the most frequently asked patient questions about their prostheses. The accuracy, quality, understandability, and actionability of the responses were assessed by two prosthodontists using five-point Likert scale, Global Quality Score, and Patient Education Materials Assessment Tool for Printed Materials scales, respectively. Readability was scored using the Flesch-Kincaid Grade Level and Flesch Reading Ease. The agreement was assessed using the Cohen Kappa test. Differences between chatbots were analyzed using the Kruskal-Wallis test, one-way ANOVA, and post-hoc tests.

RESULTS

Chatbots showed a significant difference in accuracy and readability (p <.05). Dental GPT recorded the highest accuracy score, whereas ChatGPT-4 had the lowest. DeepSeek-R1 performed best, while Dental GPT had the lowest performance in readability. Quality, understandability, actionability, and reader education scores did not show significant differences.

CONCLUSIONS

While accuracy may vary among chatbots, the domain-specific trained AI tool and ChatGPT-o1 demonstrated superior accuracy. Even if accuracy is high, misinformation in health care can have significant consequences. Enhancing the readability of the responses is essential, and chatbots should be chosen accordingly. The accuracy and readability of information from chatbots should be monitored for public health.