Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material

Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model

OBJECTIVE

To use an artificial intelligence (AI)-powered large language model (LLM) to improve readability of patient handouts.

STUDY DESIGN

Review of online material modified by AI.

SETTING

Academic center.

METHODS

Five handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT-4 after prompting: "Rewrite the following at a 6th-grade reading level." The understandability and actionability of both native and LLM-revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank-sum tests.

RESULTS

The mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to "difficult," with reading categories ranging between high school and university grade levels. Conversely, the LLM-revised handouts had an average seventh-grade reading level. LLM-revised handouts had better readability in nearly all metrics tested: Flesch-Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman-Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM-revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.

CONCLUSION

Patient-facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.

LEVEL OF EVIDENCE

Level VI.

Limited Health Literacy Among Patients With Orthopedic Injuries: A Cross-sectional Survey of Patients Who Underwent Orthopedic Trauma Surgery in a County Hospital Setting

BACKGROUND

Patients with limited health literacy have difficulty understanding their injuries and postoperative treatment, which can negatively affect their outcomes.

MATERIALS AND METHODS

This cross-sectional questionnaire-based study of 103 adult patients sought to quantify patients' health literacy at a single county hospital's orthopedic trauma clinic and to examine their ability to understand injuries and treatment plans. Demographics, Newest Vital Sign (NVS) health literacy assessment, and knowledge scores were used to assess patients' comprehension of their injuries and treatment plan. Patients were grouped by NVS score (NVS <4: limited health literacy). Fisher's exact tests and t tests were used to compare demographic and comprehension scores. Multivariate logistic regression analysis was used to examine the association among low health literacy, sociodemographic variables, and knowledge scores.

RESULTS

Of the 103 patients, 75% were determined to have limited health literacy. Patients younger than 30 years were more likely to have adequate literacy (50% vs 23%, P=.01). Patients who spoke Spanish as their primary language were 8.77 times more likely to have limited health literacy with respect to sociodemographic factors (odds ratio, 8.77; 95% CI, 1.03-76.92; P=.04). Low health literacy was 3.52 and 4.14 times more likely to predict discordance in answers to specific bone fractures and the narcotics prescribed (P=.04 and P=.02, respectively).

CONCLUSION

Spanish-speaking patients have demonstrated limited health literacy and difficulty understanding their injuries and postoperative treatment plans compared with English-speaking patients. Patients with low health literacy are more likely to be unsure regarding which bone they fractured or their prescribed opiates. [Orthopedics. 2024;47(4):249-255.].

Evaluating the Quality, Readability, and Activity of Online Information on Brain Arteriovenous Malformations

Introduction Brain arteriovenous malformations (AVMs) are vascular deformities created by improper connections between arteries and veins, most commonly in the brain and spinal cord. The management is complex and patient-dependent; further understanding of patient education activities is imperative. Internet access has become more ubiquitous, allowing patients to utilize a large database of medical information online. Using Google Trends (GT) (Google LLC, Mountain View, CA, USA), one can see the public interest in a particular topic over time. Further, when presented with numerous search results, patients may not be able to identify the highest-yielding resources, making objective measures of information quality and readability imperative. Methods A GT analysis was conducted for "hereditary hemorrhagic telangiectasia," "cerebral aneurysm," and "arteriovenous malformation". These relative search volumes (RSV) were compared with the 2017 to 2019 annual USA AVM diagnosis quantity for correlation. These RSVs were also compared with the 2017 to 2019 annual USA deaths due to cerebral hemorrhagic conditions. One search was conducted for "brain arteriovenous malformation". Since most users looking for health information online use only the first page of sources, the quality and readability analyses were limited to the first page of results on Google search. Five quality tools and six readability formulas were used. Results Pearson's correlation coefficients showed positive correlations between USA AVM RSVs and annual AVM deaths per capita from 2017 to 2019 (R2=0.932). The AVM annual diagnosis quantity and AVM RSVs showed a strong positive correlation as well (R2=0.998). Hereditary hemorrhagic telangiectasia and cerebral aneurysms had strong positive correlations between their RSVs and their corresponding annual diagnoses in the 2017 to 2019 time period (R2=0.982, R2=0.709). One-way ANOVA, for USA's 2004 to 2021 AVM RSVs and 2004 to 2019 deaths per capita, displayed no month-specific statistically significant repeating pattern (all p>0.483). The DISCERN tool had four websites that qualified as "poor" and five as "good." The average score for the tool was "good." The Journal of the American Medical Association (JAMA) benchmark scores were very low on average, as four websites achieved zero points. There was a wide variance in the currency, relevance, authority, accuracy, and purpose (CRAAP) scores, indicating an inconsistent level of webpage reliability across results. The patient education materials assessment tool (PEMAT) understandability (86.6%) showed much higher scores than the PEMAT actionability (54.6%). No readability score averaged at or below the American Medical Association (AMA)-recommended sixth-grade reading level. Conclusion These GT correlations may be due to patients and families with new diagnoses researching those same conditions online. The seasonality results reflect that no prior research has detected seasonality for AVM diagnosis or presentation. The quality study showed a wide variance in website ethics, treatment information quality, website/author qualifications, and actionable next steps regarding AVMs. Overall, this study showed that patients are routinely attempting to access information regarding these intracranial conditions, but the information available, specifically regarding AVMs, is not routinely reliable and the reading level required to understand them is too high.