Readability of online sources of information regarding epilepsy surgery and its impact on decision-making processes

Large language models and bariatric surgery patient education: a comparative readability analysis of GPT-3.5, GPT-4, Bard, and online institutional resources

BACKGROUND

The readability of online bariatric surgery patient education materials (PEMs) often surpasses the recommended 6th grade level. Large language models (LLMs), like ChatGPT and Bard, have the potential to revolutionize PEM delivery. We aimed to evaluate the readability of PEMs produced by U.S. medical institutions compared to LLMs, as well as the ability of LLMs to simplify their responses.

METHODS

Responses to frequently asked questions (FAQs) related to bariatric surgery were gathered from top-ranked health institutions. FAQ responses were also generated from GPT-3.5, GPT-4, and Bard. LLMs were then prompted to improve the readability of their initial responses. The readability of institutional responses, initial LLM responses, and simplified LLM responses were graded using validated readability formulas. Accuracy and comprehensiveness of initial and simplified LLM responses were also compared.

RESULTS

Responses to 66 FAQs were included. All institutional and initial LLM responses had poor readability, with average reading levels ranging from 9th grade to college graduate. Simplified responses from LLMs had significantly improved readability, with reading levels ranging from 6th grade to college freshman. When comparing simplified LLM responses, GPT-4 responses demonstrated the highest readability, with reading levels ranging from 6th to 9th grade. Accuracy was similar between initial and simplified responses from all LLMs. Comprehensiveness was similar between initial and simplified responses from GPT-3.5 and GPT-4. However, 34.8% of Bard's simplified responses were graded as less comprehensive compared to initial.

CONCLUSION

Our study highlights the efficacy of LLMs in enhancing the readability of bariatric surgery PEMs. GPT-4 outperformed other models, generating simplified PEMs from 6th to 9th grade reading levels. Unlike GPT-3.5 and GPT-4, Bard's simplified responses were graded as less comprehensive. We advocate for future studies examining the potential role of LLMs as dynamic and personalized sources of PEMs for diverse patient populations of all literacy levels.

Readability and content gaps in online epilepsy surgery materials as potential health literacy and shared-decision-making barriers

OBJECTIVE

Epilepsy surgery is an effective albeit underused treatment for refractory epilepsy, and online materials are vital to patient understanding of the complex process. Our goal is to analyze the readability and content inclusion of online patient health education materials designed for epilepsy surgery.

METHODS

A private browser setting was used on Google and Bing to identify the top 100 search results for the terms "epilepsy+surgery". Scientific papers, insurance pages, pay-wall access sites, and non-text content were excluded. The website text was reformatted to exclude graphics, contact information, links, and headers. Readability metrics were calculated using an online tool. Text content was analyzed for inclusion of important concepts (pre-surgical evaluation, complications, risks of continued seizures, types of surgery, complimentary diagrams/audiovisual material). Comparison of readability and content inclusion was performed as a function of organization types (epilepsy center, community health organization, pediatric-specific) and location (region, country).

RESULTS

Browser search yielded 82 distinct websites with information regarding epilepsy surgery, with 98.7% of websites exceeding the recommended 6th-grade reading level for health information. Epilepsy centers had significantly worse readability (Flesch-Kincaid Grade Level (FKGL) P < 0.01 and Flesch Reading Ease (FRE) P < 0.05). Content analysis showed that only 37% of websites discuss surgical side effects and only 23% mention the risks of continued seizures. Epilepsy centers were less likely to report information on surgical side effects (P < 0.001). UK-based websites had better readability (FKGL P < 0.01 and FRE P < 0.01) and were more likely to discuss side effects (P = 0.01) compared to US-based websites.

SIGNIFICANCE

The majority of online health content is overly complex and relatively incomplete in multiple key areas important to health literacy and understanding of surgical candidacy. Our findings suggest academic organizations, including level 4 epilepsy centers, need to simplify and broaden online education resources. More comprehensive, publicly accessible, and readable information may lead to better-shared decision-making.