Evaluation of Rhinoplasty Information from ChatGPT, Gemini, and Claude for Readability and Accuracy

Are chatbots a reliable source for patient frequently asked questions on neck masses?

PURPOSE

To evaluate the reliability and accuracy of Large Language Models in answering patient Frequently Asked Questions about adult neck masses.

METHODS

Twenty-four questions from the American Academy of Otolaryngology-Head and Neck Surgery were presented to ChatGPT, Claude, and Gemini. Five independent otolaryngologists evaluated responses using six criteria: accuracy, extensiveness, misleading information, resource quality, guideline citations, and overall reliability. Statistical analysis used Fisher's exact tests and Fleiss' Kappa.

RESULTS

All models showed high reliability (91.7-100%). Paid GPT and Gemini achieved highest accuracy (95.8%). Extensiveness varied significantly (p = 0.012), with Gemini scoring lowest (62.5%). Resource quality ranged from 58.3% (Claude) to 100% (Paid GPT). Guideline citations were highest for GPT models (50%) and lowest for Gemini (16.7%). Misleading information was rare (0-16.7%). Inter-rater reliability was near-perfect across five reviewers (κ = 0.95).

CONCLUSION

Large Language Models demonstrate high reliability and accuracy for neck mass patient education, with paid versions showing marginally better performance. While promising as educational tools, variable guideline adherence and occasional misinformation suggest they should complement rather than replace professional medical advice.

Improving Accessibility to Facial Plastic and Reconstructive Surgery Patient Resources Using Artificial Intelligence: A Pilot Study in Patient Education Materials

Background: The applications of artificial intelligence (AI) are evolving, offering new opportunities to enhance patient care. Objective: To determine whether the use of AI platforms for translating patient education materials (PEMs) improves their readability for patients seeking information on facial plastic and reconstructive surgery (FPRS) procedures. Methods: Text from 25 PEMs on topics such as rhytidectomy, rhinoplasty, and blepharoplasty was extracted. ChatGPT 4.o, ChatGPT 3.5, Microsoft Copilot, and Google Gemini were prompted to translate AAFPRS PEMs to the 6th-grade reading level, the accepted readability standard for PEMs. Readability was determined using Flesch Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), and Flesch Kincaid Reading Ease (FKRE). Statistical analysis was performed. Results: A total of 125 PEMs were reviewed. Original PEMs had a mean FKGL, GFI, and FKRE of 10.7, 13.48, and 50.8 respectively, which exceed the recommended level. The translated AI-generated PEMs had a mean FKGL, GFI, and FKRE of 8.41, 10.62, and 64.43 respectively, representing an improvement in readability (p < 0.001). Conclusion: With physician supervision, AI platforms can improve the readability of PEMs for common FPRS procedures. This strategy may increase the accessibility of educational resources for diverse patient populations.

Assessing large language models as assistive tools in medical consultations for Kawasaki disease

Background

Kawasaki disease (KD) presents complex clinical challenges in diagnosis, treatment, and long-term management, requiring a comprehensive understanding by both parents and healthcare providers. With advancements in artificial intelligence (AI), large language models (LLMs) have shown promise in supporting medical practice. This study aims to evaluate and compare the appropriateness and comprehensibility of different LLMs in answering clinically relevant questions about KD and assess the impact of different prompting strategies.

Methods

Twenty-five questions were formulated, incorporating three prompting strategies: No prompting (NO), Parent-friendly (PF), and Doctor-level (DL). These questions were input into three LLMs: ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Responses were evaluated based on appropriateness, educational quality, comprehensibility, cautionary statements, references, and potential misinformation, using Information Quality Grade, Global Quality Scale (GQS), Flesch Reading Ease (FRE) score, and word count.

Results

Significant differences were found among the LLMs in terms of response educational quality, accuracy, and comprehensibility (p < 0.001). Claude 3.5 provided the highest proportion of completely correct responses (51.1%) and achieved the highest median GQS score (5.0), outperforming GPT-4o (4.0) and Gemini 1.5 (3.0) significantly. Gemini 1.5 achieved the highest FRE score (31.5) and provided highest proportion of responses assessed as comprehensible (80.4%). Prompting strategies significantly affected LLM responses. Claude 3.5 Sonnet with DL prompting had the highest completely correct rate (81.3%), while PF prompting yielded the most acceptable responses (97.3%). Gemini 1.5 Pro showed minimal variation across prompts but excelled in comprehensibility (98.7% under PF prompting).

Conclusion

This study indicates that LLMs have great potential in providing information about KD, but their use requires caution due to quality inconsistencies and misinformation risks. Significant discrepancies existed across LLMs and prompting strategies. Claude 3.5 Sonnet offered the best response quality and accuracy, while Gemini 1.5 Pro excelled in comprehensibility. PF prompting with Claude 3.5 Sonnet is most recommended for parents seeking KD information. As AI evolves, expanding research and refining models is crucial to ensure reliable, high-quality information.

Comment on Evaluation of Rhinoplasty Information from ChatGPT, Gemini, and Claude

Level of Evidence V This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .