web search for patient questions: what does ChatGPT do better?

PURPOSE

Chat generative pretrained transformer (ChatGPT) has the potential to significantly impact how patients acquire medical information online. Here, we characterize the readability and appropriateness of ChatGPT responses to a range of patient questions compared to results from traditional web searches.

METHODS

Patient questions related to the published Clinical Practice Guidelines by the American Academy of Otolaryngology-Head and Neck Surgery were sourced from existing online posts. Questions were categorized using a modified Rothwell classification system into (1) fact, (2) policy, and (3) diagnosis and recommendations. These were queried using ChatGPT and traditional web search. All results were evaluated on readability (Flesch Reading Ease and Flesch-Kinkaid Grade Level) and understandability (Patient Education Materials Assessment Tool). Accuracy was assessed by two blinded clinical evaluators using a three-point ordinal scale.

RESULTS

54 questions were organized into fact (37.0%), policy (37.0%), and diagnosis (25.8%). The average readability for ChatGPT responses was lower than traditional web search (FRE: 42.3 ± 13.1 vs. 55.6 ± 10.5, p < 0.001), while the PEMAT understandability was equivalent (93.8% vs. 93.5%, p = 0.17). ChatGPT scored higher than web search for questions the 'Diagnosis' category (p < 0.01); there was no difference in questions categorized as 'Fact' (p = 0.15) or 'Policy' (p = 0.22). Additional prompting improved ChatGPT response readability (FRE 55.6 ± 13.6, p < 0.01).

CONCLUSIONS

ChatGPT outperforms web search in answering patient questions related to symptom-based diagnoses and is equivalent in providing medical facts and established policy. Appropriate prompting can further improve readability while maintaining accuracy. Further patient education is needed to relay the benefits and limitations of this technology as a source of medial information.

Applying Machine Learning to Determine Popular Patient Questions About Mentoplasty on Social Media

PURPOSE

Patient satisfaction in esthetic surgery often necessitates synergy between patient and physician goals. The authors aim to characterize patient questions before and after mentoplasty to reflect the patient perspective and enhance the physician-patient relationship.

METHODS

Mentoplasty reviews were gathered from Realself.com using an automated web crawler. Questions were defined as preoperative or postoperative. Each question was reviewed and characterized by the authors into general categories to best reflect the overall theme of the question. A machine learning approach was utilized to create a list of the most common patient questions, asked both preoperatively and postoperatively.

RESULTS

A total of 2,012 questions were collected. Of these, 1,708 (84.9%) and 304 (15.1%) preoperative and postoperative questions, respectively. The primary category for patients preoperatively was "eligibility for surgery" (86.3%), followed by "surgical techniques and logistics" (5.4%) and "cost" (5.4%). Of the postoperative questions, the most common questions were about "options to revise surgery" (44.1%), "symptoms after surgery" (27.0%), and "appearance" (26.3%). Our machine learning approach generated the 10 most common pre- and postoperative questions about mentoplasty. The majority of preoperative questions dealt with potential surgical indications, while most postoperative questions principally addressed appearance.

CONCLUSIONS

The majority of mentoplasty patient questions were preoperative and asked about eligibility of surgery. Our study also found a significant proportion of postoperative questions inquired about revision, suggesting a small but nontrivial subset of patients highly dissatisfied with their results. Our 10 most common preoperative and postoperative question handout can help better inform physicians about the patient perspective on mentoplasty throughout their surgical course. Level of Evidence V This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .