Readability of Online Materials Related to Vocal Cord Leukoplakia

Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model

OBJECTIVE

To use an artificial intelligence (AI)-powered large language model (LLM) to improve readability of patient handouts.

STUDY DESIGN

Review of online material modified by AI.

SETTING

Academic center.

METHODS

Five handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT-4 after prompting: "Rewrite the following at a 6th-grade reading level." The understandability and actionability of both native and LLM-revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank-sum tests.

RESULTS

The mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to "difficult," with reading categories ranging between high school and university grade levels. Conversely, the LLM-revised handouts had an average seventh-grade reading level. LLM-revised handouts had better readability in nearly all metrics tested: Flesch-Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman-Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM-revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.

CONCLUSION

Patient-facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.

LEVEL OF EVIDENCE

Level VI.

Readability of Patient Education Materials in Head and Neck Cancer: A Systematic Review

Importance

Patient education materials (PEMs) can promote patient engagement, satisfaction, and treatment adherence. The American Medical Association recommends that PEMs be developed for a sixth-grade or lower reading level. Health literacy (HL) refers to an individual's ability to seek, understand, and use health information to make appropriate decisions regarding their health. Patients with suboptimal HL may not be able to understand or act on health information and are at risk for adverse health outcomes.

Objective

To assess the readability of PEMs on head and neck cancer (HNC) and to evaluate HL among patients with HNC.

Evidence Review

A systematic review of the literature was performed by searching Cochrane, PubMed, and Scopus for peer-reviewed studies published from 1995 to 2024 using the keywords head and neck cancer, readability, health literacy, and related synonyms. Full-text studies in English that evaluated readability and/or HL measures were included. Readability assessments included the Flesch-Kincaid Grade Level (FKGL grade, 0-20, with higher grades indicating greater reading difficulty) and Flesch Reading Ease (FRE score, 1-100, with higher scores indicating easier readability), among others. Reviews, conference materials, opinion letters, and guidelines were excluded. Study quality was assessed using the Appraisal Tool for Cross-Sectional Studies.

Findings

Of the 3235 studies identified, 17 studies assessing the readability of 1124 HNC PEMs produced by professional societies, hospitals, and others were included. The mean FKGL grade ranged from 8.8 to 14.8; none of the studies reported a mean FKGL of grade 6 or lower. Eight studies assessed HL and found inadequate HL prevalence ranging from 11.9% to 47.0%.

Conclusions and Relevance

These findings indicate that more than one-third of patients with HNC demonstrate inadequate HL, yet none of the PEMs assessed were developed for a sixth grade or lower reading level, as recommended by the American Medical Association. This incongruence highlights the need to address the readability of HNC PEMs to improve patient understanding of the disease and to mitigate potential barriers to shared decision-making for patients with HNC. It is crucial to acknowledge the responsibility of health care professionals to produce and promote more effective PEMs to dismantle the potentially preventable literacy barriers.

ChatGPT vs. web search for patient questions: what does ChatGPT do better?

PURPOSE

Chat generative pretrained transformer (ChatGPT) has the potential to significantly impact how patients acquire medical information online. Here, we characterize the readability and appropriateness of ChatGPT responses to a range of patient questions compared to results from traditional web searches.

METHODS

Patient questions related to the published Clinical Practice Guidelines by the American Academy of Otolaryngology-Head and Neck Surgery were sourced from existing online posts. Questions were categorized using a modified Rothwell classification system into (1) fact, (2) policy, and (3) diagnosis and recommendations. These were queried using ChatGPT and traditional web search. All results were evaluated on readability (Flesch Reading Ease and Flesch-Kinkaid Grade Level) and understandability (Patient Education Materials Assessment Tool). Accuracy was assessed by two blinded clinical evaluators using a three-point ordinal scale.

RESULTS

54 questions were organized into fact (37.0%), policy (37.0%), and diagnosis (25.8%). The average readability for ChatGPT responses was lower than traditional web search (FRE: 42.3 ± 13.1 vs. 55.6 ± 10.5, p < 0.001), while the PEMAT understandability was equivalent (93.8% vs. 93.5%, p = 0.17). ChatGPT scored higher than web search for questions the 'Diagnosis' category (p < 0.01); there was no difference in questions categorized as 'Fact' (p = 0.15) or 'Policy' (p = 0.22). Additional prompting improved ChatGPT response readability (FRE 55.6 ± 13.6, p < 0.01).

CONCLUSIONS

ChatGPT outperforms web search in answering patient questions related to symptom-based diagnoses and is equivalent in providing medical facts and established policy. Appropriate prompting can further improve readability while maintaining accuracy. Further patient education is needed to relay the benefits and limitations of this technology as a source of medial information.

Conclusiveness, readability and textual characteristics of plain language summaries from medical and non-medical organizations: a cross-sectional study

This cross-sectional study compared plain language summaries (PLSs) from medical and non-medical organizations regarding conclusiveness, readability and textual characteristics. All Cochrane (medical PLSs, n = 8638) and Campbell Collaboration and International Initiative for Impact Evaluation (non-medical PLSs, n = 163) PLSs of latest versions of systematic reviews published until 10 November 2022 were analysed. PLSs were classified into three conclusiveness categories (conclusive, inconclusive and unclear) using a machine learning tool for medical PLSs and by two experts for non-medical PLSs. A higher proportion of non-medical PLSs were conclusive (17.79% vs 8.40%, P < 0.0001), they had higher readability (median number of years of education needed to read the text with ease 15.23 (interquartile range (IQR) 14.35 to 15.96) vs 15.51 (IQR 14.31 to 16.77), P = 0.010), used more words (median 603 (IQR 539.50 to 658.50) vs 345 (IQR 202 to 476), P < 0.001). Language analysis showed that medical PLSs scored higher for disgust and fear, and non-medical PLSs scored higher for positive emotions. The reason for the observed differences between medical and non-medical fields may be attributed to the differences in publication methodologies or disciplinary differences. This approach to analysing PLSs is crucial for enhancing the overall quality of PLSs and knowledge translation to the general public.

Are Mobile Applications in Laryngology Designed for All Patients?

OBJECTIVES

Mobile applications (apps) are multiplying in laryngology, with little standardization of content, functionality, or accessibility. The purpose of this study is to evaluate the quality, functionality, health literacy, readability, accessibility, and inclusivity of laryngology mobile applications.

METHODS

Of the 3230 apps identified from the Apple and Google Play stores, 28 patient-facing apps met inclusion criteria. Apps were evaluated using validated scales assessing quality and functionality: the Mobile App Rating Scale (MARS) and the Institute for Healthcare Informatics App Functionality Scale. The Clear Communication Index (CDC) Institute of Medicine Strategies for Creating Health Literate Mobile Applications, and Patient Education Materials Assessment Tool (PEMAT) were used to evaluate apps health literacy level. Readability was assessed using established readability formulas. Apps were evaluated for language, accessibility features, and representation of a diverse population.

RESULTS

Twenty-six apps (92%) had adequate quality (MARS score > 3). The mean PEMAT score was 89% for actionability and 86% for understandability. On average, apps utilized 25/33 health literate strategies. Twenty-two apps (79%) did not pass the CDC index threshold of 90% for health literacy. Twenty-four app descriptions (86%) were above an 8th grade reading level. Only 4 apps (14%) showed diverse representation, 3 (11%) had non-English language functions, and 2 (7%) offered subtitles. Inter-rater reliability for MARS was adequate (CA-ICC = 0.715).

CONCLUSION

While most apps scored well in quality and functionality, many laryngology apps did not meet standards for health literacy. Most apps were written at a reading level above the national average, lacked accessibility features, and did not represent diverse populations. Laryngoscope, 133:1540-1549, 2023.

Readability Analysis and Accessibility of Online Materials About Transgender Voice Care

OBJECTIVE

To determine readability, understandability, and actionability of online health information related to transgender voice care.

STUDY DESIGN

Review of online materials.

SETTING

Academic medical center.

METHODS

A Google search of "transgender voice care" was performed with the first 50 websites meeting inclusion criteria included. Readability was assessed using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), and the Simple Measure of Gobbledygook (SMOG). Understandability and actionability were measured by 2 independent reviewers using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). Unpaired t tests were used to compare clinician- and patient-oriented sites, surgical and nonsurgical sites, and sites that discuss nonbinary indications for voice care. Analysis of variance was used to compare sites that discuss voice feminization, masculinization, both, or neither.

RESULTS

Average scores across the cohort for FRES, FKGL, and SMOG were 43.77 ± 13.52, 12.14 ± 2.66, and 11.30 ± 1.93, respectively, indicating materials were above a 12th-grade reading level. PEMAT-P scores for understandability and actionability were 64.95% ± 15.78% and 40.55% ± 23.86%, respectively. Patient-oriented sites were significantly more understandable and actionable than clinician-oriented sites (P < .02). Websites that discussed only voice feminization were significantly more readable according to objective metrics (FKGL, SMOG) than websites that discussed both feminization and masculinization or those that did not differentiate care types (P < .05).

CONCLUSION

Online information written about transgender voice care is written at a level above what is recommended for patient education materials. Providers may improve accessibility of transgender voice care by enhancing readability of online materials.

Quality and readability of online information on plantar fasciitis and calcaneal spur

Plantar fasciitis and calcaneal spur are common causes of heel pain in the community. People use the Internet to obtain medical information about diseases. We reviewed Internet information sources on plantar fasciitis and calcaneal spur for quality and readability. The first 50 websites for each search term ("calcaneal spur", "heel spur", and "plantar fasciitis") were scanned on www.google.com . Six different valid tools were used for information quality and readability assessment. We searched for HONCode (Health On the Net Foundation Code) stamps on included websites. The total mean points for DISCERN were 50.52 ± 14.62, and the total mean points for JAMA (Journal of the American Medical Association) were 2.42 ± 1.26. In total, 25.72% of 97 websites had HONCode stamps. The average scores for the readability indicators were calculated to be Flesch-Kincaid Grade Level (FKGL): 7.27 ± 1.71, Gunning Fog: 8.46 ± 2.17, Simple Measure of Gobbledygook (SMOG): 6.89 ± 1.24, and Coleman Liau Index: 15.56 ± 1.85. In our study, when the website resources were examined, there were profit websites the most and website quality and readability were moderate level. A significant proportion of the websites have a financial bias and provide low-quality information. A mechanism for monitoring the quality and readability of online information must be established and managed systematically.