Readability analysis of patient information on the American Academy of Otolaryngology–Head and Neck Surgery website

The Use of Artificial Intelligence to Improve Readability of Otolaryngology Patient Education Materials

OBJECTIVE

The recommended readability of health education materials is at the sixth-grade level. Artificial intelligence (AI) large language models such as the newly released ChatGPT4 might facilitate the conversion of patient-education materials at scale. We sought to ascertain whether online otolaryngology education materials meet recommended reading levels and whether ChatGPT4 could rewrite these materials to the sixth-grade level. We also wished to ensure that converted materials were accurate and retained sufficient content.

METHODS

Seventy-one articles from patient educational materials published online by the American Academy of Otolaryngology-Head and Neck Surgery were selected. Articles were entered into ChatGPT4 with the prompt "translate this text to a sixth-grade reading level." Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL) were determined for each article before and after AI conversion. Each article and conversion were reviewed for factual inaccuracies, and each conversion was reviewed for content retention.

RESULTS

The 71 articles had an initial average FKGL of 11.03 and FRES of 46.79. After conversion by ChatGPT4, the average FKGL across all articles was 5.80 and FRES was 77.27. Converted materials provided enough detail for patient education with no factual errors.

DISCUSSION

We found that ChatGPT4 improved the reading accessibility of otolaryngology online patient education materials to recommended levels quickly and effectively.

IMPLICATIONS FOR PRACTICE

Physicians can determine whether their patient education materials exceed current recommended reading levels by using widely available measurement tools, and then apply AI dialogue platforms to modify materials to more accessible levels as needed.

LEVEL OF EVIDENCE

Level 5.

Readability of Patient Education Materials in Head and Neck Cancer: A Systematic Review

Importance

Patient education materials (PEMs) can promote patient engagement, satisfaction, and treatment adherence. The American Medical Association recommends that PEMs be developed for a sixth-grade or lower reading level. Health literacy (HL) refers to an individual's ability to seek, understand, and use health information to make appropriate decisions regarding their health. Patients with suboptimal HL may not be able to understand or act on health information and are at risk for adverse health outcomes.

Objective

To assess the readability of PEMs on head and neck cancer (HNC) and to evaluate HL among patients with HNC.

Evidence Review

A systematic review of the literature was performed by searching Cochrane, PubMed, and Scopus for peer-reviewed studies published from 1995 to 2024 using the keywords head and neck cancer, readability, health literacy, and related synonyms. Full-text studies in English that evaluated readability and/or HL measures were included. Readability assessments included the Flesch-Kincaid Grade Level (FKGL grade, 0-20, with higher grades indicating greater reading difficulty) and Flesch Reading Ease (FRE score, 1-100, with higher scores indicating easier readability), among others. Reviews, conference materials, opinion letters, and guidelines were excluded. Study quality was assessed using the Appraisal Tool for Cross-Sectional Studies.

Findings

Of the 3235 studies identified, 17 studies assessing the readability of 1124 HNC PEMs produced by professional societies, hospitals, and others were included. The mean FKGL grade ranged from 8.8 to 14.8; none of the studies reported a mean FKGL of grade 6 or lower. Eight studies assessed HL and found inadequate HL prevalence ranging from 11.9% to 47.0%.

Conclusions and Relevance

These findings indicate that more than one-third of patients with HNC demonstrate inadequate HL, yet none of the PEMs assessed were developed for a sixth grade or lower reading level, as recommended by the American Medical Association. This incongruence highlights the need to address the readability of HNC PEMs to improve patient understanding of the disease and to mitigate potential barriers to shared decision-making for patients with HNC. It is crucial to acknowledge the responsibility of health care professionals to produce and promote more effective PEMs to dismantle the potentially preventable literacy barriers.

What Questions Do Patients Ask About Biologic Therapies for Chronic Sinusitis With Nasal Polyps?

Objective: The objective of this study is to explore the questions commonly asked online about biologic therapies for use in treatment of chronic sinusitis with nasal polyps (CRSwNP) and the quality of the available content. Methods: Most common search terms were identified via Google Trends. People Also Ask (PAA) questions were identified and extracted with their associated website using an online data-scraping program [Search Engine Optimization (SEO) Minion, Keywords Everywhere]. Sources were evaluated using Flesch Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE, higher number = better) score for readability; Journal of American Medical Association (JAMA) Benchmark criteria for quality assessment (0-4, 4 = all criteria met). Results: A total of 143 unique PAA questions and unique websites were identified. Questions were organized into 3 categories: questions about biologic therapies overall (38.46%), about CRSwNP (37.76%), and about treatment options for CRSwNP (23.78%). Websites answering PAA questions were from commercial (60.14%), medical practice (13.99%), academic (13.29%), and government (12.59%) sources. FKGL scores found the average reading level to be at approximately a 12th grade level (SD = 3.297) alongside a low reading ease FRE score of 37.6 (SD = 16.77). Mean JAMA criteria scores were 0.9895 (SD = 0.848), indicating largely low-quality materials. Conclusion: Biologic therapies are a novel treatment option for CRSwNP, and participants are seeking more information about these treatments and disease state. Online resources regarding biologics should be presented at a lower reading level. Sources with evidence-based information are needed. Physicians should be aware of these limitations in online material and counsel accordingly by curating and directing patients to good sources.Level of Evidence: Step 4.

Assessing and Improving the Effectiveness of Online Patient Education Materials on Essential Vocal Tremor: A Comprehensive Evaluation

INTRODUCTION

Health literacy, a strong indicator of health outcomes, is an important aspect of good patient care. With an increasing reliance on the Internet for health information, online patient materials should be easily understood by the average reader. The American Medical Association (AMA) and National Institutes of Health (NIH) recommend that patient education materials be written at a sixth-grade level. Creating effective digital information requires careful consideration of not only word choice, but also many other factors including actionability, comprehensiveness, evidence, and visual organization. To support the creation of valuable online health content, the Office of Disease Prevention and Health Promotion (ODPHP) published Health Literacy Online, a research-based guide that discusses why and how to design digital health information tools.This study aims to assess the effectiveness of online patient education materials regarding vocal tremor, assess the effectiveness of patient education materials published by the American Laryngological Association, and to evaluate the usefulness of the Health Literacy Online guide in creating effective online patient education materials on laryngological diseases.

METHODS

The first 50 unsponsored search results for the terms "vocal tremor" and "essential vocal tremor" were evaluated. Each website was analyzed using the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL) readability tests, the DISCERN instrument, and the Patient Education Materials Assessment Tool (PEMAT). The resources published by the American Laryngological Association were also evaluated in this manner.

RESULTS

Of the 100 websites identified from the initial queries, 14 websites were included in this analysis. The average FRES and FKGL scores were 47.21 ± 10.47 and 10.96 ± 2.46, respectively, indicating that readers need a 11th-grade education to comprehend the materials. The average DISCERN score was 22.50 ± 9.76, indicating "very poor" quality with serious shortcomings and not appropriate sources of information about treatment choices. The average PEMAT understandability score was 68.43% ± 9.80% with an actionability score of 20.00% ± 23.53%, indicating the information was fairly difficult to process and do not help identify next steps. For the materials published by the American Laryngological Association (ALA), the average FRES and FKGL scores were 38.33 ± 12.81 and 12.56 ± 2.15, respectively, indicating a 12th-grade reading level. A DISCERN score of 27 was consistent across each item, indicating "very poor" quality. A PEMAT understandability score was 45% with an actionability score of 0%, indicating they are difficult to process and do not help identify next steps. After writing a revised sample of the information provided by the ALA based on the ODPHP's Health Literacy Online tool, the new FRES and FKGL score was 75.6 and 5.9, respectively. The new DISCERN score was 35. The new PEMAT understandability scores was 79% with actionability scores of 80%.

CONCLUSION

This study found that most publicly available online patient education materials on essential vocal tremor and other laryngological diseases do not use plain language and require reading levels too advanced for the average reader to comprehend. In addition, most websites were of very poor quality readability, and were therefore less likely to benefit individuals in their decision-making. In an age where most people seek information on the Internet, the lack of easily understood online patient resources reduces the usefulness of these resources for many individuals. Professional organizations and societies like the American Laryngological Association may consider the use of the Health Literacy Online tool as a resource to provide both accurate and easily understandable patient education resources.

Assessment of an evidence-based laryngeal cancer fact sheet: A mixed methods study

To evaluate perceptions of a laryngeal cancer fact sheet amongst people with direct experience of the disease and its treatment. A mixed methods study (questionnaire and interview) evaluating the information resource was conducted across two institutions. In total 20 participants responded to the questionnaire. Overall participants reported the information resource was detailed and understandable. Insufficient information was provided on: impact on family in eight participants (40%); impact on work in six (33%); and, second opinions and long-term side effects in five (25%). The majority (67%) wanted a large amount of information with the preferred source being one-on-one meetings with their doctor. The thematic analysis identified three main themes: preferences for information, self-management; and, information sources. People with direct experience of laryngeal cancer and its treatments reported the information resource was comprehensive and clear. There were some gaps in the information provided, particularly related to survivorship issues.

Indian endocrine surgery websites are they comparable?

Introduction

In modern day surgical practice, patients and their relatives garner more knowledge through websites rather than direct interaction with the consultant physician. We aimed to assess whether Indian endocrine surgery websites matched with their counterparts abroad.

Materials & methods

We identified 60 endocrine surgery websites worldwide and 12 endocrine surgery websites maintained by trained endocrine surgeons from India. The website parameters, demographic data of the websites, rank, and other parameters were assessed using a professional website (www.Alexa.com). An endocrine surgeon along with a technical website advisor rated the content, presentation, and likes from a scale of 1–5 (1 minimum score and 5 maximum score).

Results

A total of 72 individual endocrine surgery websites, out of which 60 were from abroad and 12 were from India, were analyzed. A majority of foreign websites were ranked (43/60), whereas 2/12 Indian websites were ranked (P < 0.0001). Foreign websites had a better landscape profile. Except for pancreatic facts, which were significantly different (P = 0.006) between Indian and foreign websites, there was no significant difference in thyroid facts, parathyroid facts, adrenal facts, photographs, videos, postop advice, contact information, publications, and complications.

Conclusion

Most parameters were comparable in both groups. Postoperative advice and complications were present in only a few websites. A well-designed endocrine surgery website can aid both the patient and the treating physician.

•

Websites can help patients garner knowledge.

•

Well designed and unbiased website can be a good tool.

•

Websites can safe unnecessary discussion with the consultant.

•

Website can help take decisions.

Quality, Readability, and Suitability of Hearing Health-Related Materials: A Descriptive Review

Objectives The objective of this descriptive review was to determine the quality, readability, and suitability of ear and hearing health information and materials for patients and their significant others. Method A literature search was conducted between August 2018 and April 2019 in the databases CINAHL Complete, MEDLINE, and PsychInfo. Inclusion and exclusion criteria were used to shortlist studies. Data regarding quality, suitability, and readability were extracted from the included studies. Data were assessed qualitatively. Results There were 34 studies included in this review. Of those, eight examined quality, 33 assessed readability, and four investigated the suitability of materials. The range of materials assessed included diagnostic reports, patient education materials (PEMs), patient-reported outcome measures, and websites. Quality elements were examined in studies focusing on website information. Findings indicated that most websites were of poor quality. Suitability was examined in studies focusing on PEMs such as hearing aid user guides. Findings indicated that most of the existing materials were not suitable for the intended populations. The reading grade level of information across all four categories was found to be higher than the recommended fifth or sixth reading grade level for health-related materials. Revisions of some diagnostic reports and PEMs showed that improvements are possible. Conclusions This review suggests that ear- and hearing-related materials generally have lower quality and suitability with higher readability (more difficult to read). Development of materials that are suitable, of high quality, and at the appropriate readability levels is required to improve accessibility of ear- and hearing-related materials.

Quality of Patient Education Sections on Otitis Media Across Different Website Platforms

OBJECTIVE

To determine the quality, content, and readability of patient education materials pertaining to otitis media across several popular online platforms focused on otolaryngology and pediatric primary care education.

METHODS

Online patient materials related to otitis media and directed toward parents were collected from the American Association for Family Practice (AAFP), ENT-Health section of the American Academy of Otolaryngology-Head and Neck Surgery, Healthychildren.org from the American Academy of Pediatrics, KidsHealth from Nemours, WebMD, and Wikipedia. Materials were analyzed for quality, content, and readability. The DISCERN instrument was used to score quality. A unique content score was generated based on the information provided on each website and on the medical and surgical management of otitis media. Readability scores were calculated using the Flesch-Kincaid Grade Level, Flesch Reading Ease Score, Gunning-Fog Index, Simple Measure of Gobbledygook, Coleman-Liau Index, and Automated Readability Index.

RESULTS

Overall, content was well-balanced. Information from AAFP and Healthychildren.org was focused more on medical management than other sources. The average DISCERN scores showed all sources to be of good quality with minimal shortcomings. The AAFP and KidsHealth websites had some readability scores around the 8th-grade reading level, the National Institute of Health's upper limit recommended for public health information; however, most websites were above this recommended reading level.

CONCLUSION

Patient education materials related to otitis media on academic and certain popular internet sites are good sources to obtain high-quality information on the topic. Patient educational background, prior knowledge and understanding of otitis media, and physician-patient partnership goals should be taken into account when referring patients to online materials.

Innovative Online Educational Portal Improves Disease-Specific Reproductive Knowledge Among Patients With Inflammatory Bowel Disease

BACKGROUND

Inflammatory bowel disease (IBD) is often diagnosed in early adulthood, affecting patients through their reproductive years. Many patients, lacking knowledge about IBD and reproduction, make uninformed decisions. Although patients have turned to the Internet for information, it remains unclear if online resources are effective for improving and retaining IBD-specific reproductive knowledge. We aimed to elucidate if a multimedia vs text-only online educational intervention could improve IBD-specific reproductive knowledge for more than 6 months.

METHODS

We developed a website covering genetics, fertility, surgery, pregnancy, medications, delivery, and postpartum in the context of IBD. Adult IBD patients were randomized into study groups (multimedia or text-only) and provided 60-day access. Participants completed pre-, post-, and 6+ month-postintervention pregnancy knowledge (CCPKnow) questionnaires. Results were compared using nonparametric tests.

RESULTS

Of 111 registered participants, 78 (70.3%) completed pre- and postintervention questionnaires, and 37 (47.4%) subsequently completed the 6+ month questionnaire. Demographics were as follows: median age (interquartile range [IQR]) 29.3 (25.6-32.9) years, Crohn's disease n = 54 (69.2%), females n = 63 (80.3%), of which n = 5 (7.9%) were pregnant and n = 19 (30.2%) had previously been pregnant. The median CCPKnow scores (/17) (IQR) were 8.0 (3.0-10.0) pre-intervention, 16.0 (13.00-17.00) postintervention, and 14.0 (12.0-15.0) 6+ months postintervention. The median within-subject increase in score was 6.5 pre- to postintervention, and 4.0 pre- to 6+ months postintervention (both P < 0.001). The median increase in score (pre- to post-) was 8.0 for the multimedia group and 6.0 for the text-only group (P = 0.216).

CONCLUSIONS

An evidence-based, online educational portal can significantly improve and maintain IBD-specific reproductive patient knowledge for more than 6 months.

Readability Trends of Online Information by the American Academy of Otolaryngology—Head and Neck Surgery Foundation

Objective

Previous studies have shown that patient education materials published by the American Academy of Otolaryngology—Head and Neck Surgery Foundation may be too difficult for the average reader to understand. The purpose of this study was to determine if current educational materials show improvements in readability.

Study Design

Cross-sectional analysis.

Setting

The Patient Health Information section of the American Academy of Otolaryngology—Head and Neck Surgery Foundation website.

Subjects and Methods

All patient education articles were extracted in plain text. Webpage navigation, references, author information, appointment information, acknowledgments, and disclaimers were removed. Follow-up editing was also performed to remove paragraph breaks, colons, semicolons, numbers, percentages, and bullets. Readability grade was calculated with the Flesch-Kincaid Grade Level, Flesch Reading Ease, Gunning-Fog Index, Coleman-Liau Index, Automated Readability Index, and Simple Measure of Gobbledygook. Intra- and interobserver reliability were assessed.

Results

A total of 126 articles from 7 topics were analyzed. Readability levels across all 6 tools showed that the difficulty of patient education materials exceeded the abilities of an average American. As compared with previous studies, current educational materials by the American Academy of Otolaryngology—Head and Neck Surgery Foundation have shown a decrease in difficulty. Intra- and interobserver reliability were both excellent, with intraclass coefficients of 0.99 and 0.96, respectively.

Conclusion

Improvements in readability is an encouraging finding and one that is consistent with recent trends toward improved health literacy. Nevertheless, online patient educational material is still too difficult for the average reader. Revisions may be necessary for current materials to benefit a larger readership.

Are Online Zenker’s Diverticulum Materials Readable and Understandable?

Objective

Patients use a multitude of resources to learn about Zenker’s diverticulum (ZD). The objectives of this study were to assess the readability and understandability of online materials on ZD, evaluate them against the existing criteria, and investigate the relationship between readability and understandability.

Methods

The first 50 webpages from an online search for ZD were analyzed. Twenty-one webpages had materials intended for patients and were included in the study. The patient education materials (PEMs) were analyzed using 6 readability tools. Four individuals used the Patient Education Materials Evaluation Tool (PEMAT) to assess the understandability. Fleiss κ interrater reliability analysis determined consistency among the raters. Finally, Pearson correlation coefficient analyzed the relationship between readability and understandability.

Results

The reading grade level of the materials reviewed ranged from 10th to 16th grade while the understandability ranged from 31% to 74%. Correlation analysis demonstrated a strong negative correlation between readability and understandability ( r = −0.62, P < .05). Fleiss’ κ interrater reliability for the raters demonstrated substantial agreement between the 4 raters (κ = 0.64).

Conclusion

Online PEMs pertaining to ZD are written well above the recommended reading level. Materials written at a lower reading level are more understandable. A wide range of understandability exists among materials with identical reading grade levels.

Implications for Practice

Health care providers need to create new PEMs for ZD that are available online that are both readable and understandable. The PEMAT and readability formulas can provide a framework for authors to create these materials.

Readability and Understandability of Online Vocal Cord Paralysis Materials

Objective

Patients use several online resources to learn about vocal cord paralysis (VCP). The objective of this study was to assess the readability and understandability of online VCP patient education materials (PEMs), with readability assessments and the Patient Education Materials Evaluation Tool (PEMAT), respectively. The relationship between readability and understandability was then analyzed.

Study Design and Setting

Descriptive and correlational design.

Method

Online PEMs were identified by performing a Google search with the term “vocal cord paralysis.” After scientific webpages, news articles, and information for medical professionals were excluded, 29 articles from the first 50 search results were considered. Readability analysis was performed with 6 formulas. Four individuals with different educational backgrounds conducted understandability analysis with the PEMAT. Fleiss’s Kappa interrater reliability analysis determined consistency among raters. Correlation between readability and understandability was determined with Pearson’s correlation test.

Results

The reading level of the reviewed articles ranged from grades 9 to 17. Understandability ranged from 29% to 82%. Correlation analysis demonstrated a strong negative correlation between materials’ readability and understandability ( r = −0.462, P < .05).

Conclusion

Online PEMs pertaining to VCP are written above the recommended reading levels. Overall, materials written at lower grade levels are more understandable. However, articles of identical grade levels had varying levels of understandability. The PEMAT may provide a more critical evaluation of the quality of a PEM when compared with readability formulas. Both readability and understandability should be used to evaluate PEMs.

Readability of Internet Information on Hearing: Systematic Literature Review

PURPOSE

This systematic literature review asks the following question: “ What is the readability of Internet information on hearing that people with hearing impairment and their significant others can access in the context of their hearing care?”

METHOD

Searches were completed in three databases: CINAHL, PubMed, and Scopus. Seventy-eight records were identified and systematically screened for eligibility: 8 records were included that contained data on the readability of Internet information on hearing that people with hear ing impairment and their significant others can access in the context of their hearing care.

RESULTS

Records reported mean readability levels from 9 to over 14. In other words, people with hearing impairment and their significant others need 9 to 14 years of education to read and understand Internet information on hearing that they access in the context of their hearing care.

CONCLUSION

The poor readability of Internet information on hearing has been well documented; it is time to focus on valid and sustainable initiatives that address this problem.

The internet as a source of information for De Quervain's tendinitis

BACKGROUND

Many individuals are turning to the Internet for information about various health conditions, and in our study we evaluated the quality and readability of information on the internet about De Quervain's tendinitis.

METHODS

We chose the search terms "De Quervain's Tendinitis," "De Quervain's Tenosynovitis," and "De Quervain's Syndrome," and entered them into the search engines "Google," "Bing," and "Yahoo" and compiled the search results. The websites were then evaluated and assigned a quality score, a Flesch-Kincaid (FK) reading level, and a subjective quality score. They were then grouped according to the search term used, search engine used, and the order (priority) returned by the search engines. ANOVA analysis and pairwise comparisons of quality and readability among groups, as well as correlation analysis were performed.

RESULTS

The FK readability average was 10.3, above the recommended level. The search term De Quervain's tenosynovitis returned the highest objective quality results. There was no statistical difference found between the different search engines. The first 10 results from the searches were of higher quality than results 11-20, and there was a positive correlation between objective and subjective quality scores but no correlation between readability and objective quality.

CONCLUSIONS

We concluded that quality information about De Quervain's tendinitis is available on the internet and is most likely to be found using the search term De Quervain's tenosynovitis and in the first 10 results of an internet search. However, most information is written above the recommended 6th grade reading level.

Readability Assessment of Internet-Based Patient Education Materials Related to Parathyroid Surgery

Objectives:

Patient education is critical in obtaining informed consent and reducing preoperative anxiety. Written patient education material (PEM) can supplement verbal communication to improve understanding and satisfaction. Published guidelines recommend that health information be presented at or below a sixth-grade reading level to facilitate comprehension. We investigate the grade level of online PEMs regarding parathyroid surgery.

Methods:

A popular internet search engine was used to identify PEM discussing parathyroid surgery. Four formulas were used to calculate readability scores: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Frequency of Gobbledygook (GFOG), and Simple Measure of Gobbledygook (SMOG).

Results:

Thirty web-based articles discussing parathyroid surgery were identified. The average FRE score was 42.8 (± 1 standard deviation [SD] 16.3; 95% confidence interval [CI], 36.6-48.8; range, 6.1-71.3). The average FKGL score was 11.7 (± 1 SD 3.3; 95% CI, 10.5-12.9; range, 6.1-19.0). The SMOG scores averaged 14.2 (± 1 SD 2.6; 95% CI, 13.2-15.2; range, 10.7-21.9), and the GFOG scores averaged 15.0 (± 1 SD 3.5; 95% CI, 13.7-16.3; range, 10.6-24.8).

Conclusion:

Online PEM on parathyroid surgery is written above the recommended sixth-grade reading level. Improving readability of PEM may promote better health education and compliance.

Readability assessment of online tracheostomy care resources

OBJECTIVE

To assess the readability of online tracheostomy care resources.

STUDY DESIGN

Cross-sectional study.

SETTING

Academic center.

SUBJECTS AND METHODS

A Google search was performed for "tracheostomy care" in January 2014. The top 50 results were categorized into major versus minor websites and patient-oriented versus professional-oriented resources. These websites were evaluated with the following readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Gunning Frequency of Gobbledygook (GFOG).

RESULTS

Readability scores for the websites were FRES 57.21 ± 16.71 (possible range = 0-100), FKGL 8.33 ± 2.84 (possible range = 3-12), SMOG 11.25 ± 2.49 (possible range = 3-19), and GFOG 11.43 ± 4.07 (possible range = 3-19). There was no significant difference in all 4 readability scores between major (n = 41) and minor (n = 9) websites. Professional-oriented websites (n = 19) had the following readability scores: FRES 40.77 ± 11.69, FKGL 10.93 ± 2.48, SMOG 13.29 ± 2.32, and GFOG 14.91 ± 3.98. Patient-oriented websites (n = 31) had the following readability scores: FRES 67.29 ± 9.91, FKGL 6.73 ± 1.61, SMOG 10.01 ± 1.64, and GFOG 9.30 ± 2.27. Professional-oriented websites had more difficult readability scores than patient-oriented websites for FRES (P < .00), FKGL (P < .00), SMOG (P < .00), and GFOG (P < .00).

CONCLUSION

Online tracheostomy care resources were written at a level more difficult than the recommended fourth- to sixth-grade level for written health information. There was no significant difference in readability between major and minor websites. Professional-oriented websites were more difficult to read than patient-oriented websites.

Are we effectively informing patients? A quantitative analysis of on-line patient education resources from the American Society of Neuroradiology

BACKGROUND AND PURPOSE

The ubiquitous use of the Internet by the public in an attempt to better understand their health care requires the on-line resources written at an appropriate level to maximize comprehension for the average user. The National Institutes of Health and the American Medical Association recommend on-line patient education resources written at a third-to-seventh grade level. We evaluated the readability of the patient education resources provided on the Web site of the American Society of Neuroradiology (http://www.asnr.org/patientinfo/).

MATERIALS AND METHODS

All patient education material from the ASNR Web site and the Society of Neurointerventional Surgery Web site were downloaded and evaluated with the computer software, Readability Studio Professional Edition, by using 10 quantitative readability scales: the Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Gunning Fog Index, New Dale-Chall, FORCAST Formula, Fry Graph, Raygor Reading Estimate, and New Fog Count. An unpaired t test was used to compare the readability level of resources available on the American Society of Neuroradiology and the Society of Neurointerventional Surgery Web sites.

RESULTS

The 20 individual patient education articles were written at a 13.9 ± 1.4 grade level with only 5% written at <11th grade level. There was no statistical difference between the level of readability of the resources on the American Society of Neuroradiology and Society of Neurointerventional Surgery Web sites.

CONCLUSIONS

The patient education resources on these Web sites fail to meet the guidelines of the National Institutes of Health and American Medical Association. Members of the public may fail to fully understand these resources and would benefit from revisions that result in more comprehensible information cast in simpler language.

Internet search term affects the quality and accuracy of online information about developmental hip dysplasia

BACKGROUND

The recent emphasis on shared decision-making has increased the role of the Internet as a readily accessible medical reference source for patients and families. However, the lack of professional review creates concern over the quality, accuracy, and readability of medical information available to patients on the Internet.

METHODS

Three Internet search engines (Google, Yahoo, and Bing) were evaluated prospectively using 3 difference search terms of varying sophistication ("congenital hip dislocation," "developmental dysplasia of the hip," and "hip dysplasia in children"). Sixty-three unique Web sites were evaluated by each of 3 surgeons (2 fellowship-trained pediatric orthopaedic attendings and 1 orthopaedic chief resident) for quality and accuracy using a set of scoring criteria based on the AAOS/POSNA patient education Web site. The readability (literacy grade level) of each Web site was assessed using the Fleisch-Kincaid score.

RESULTS

There were significant differences noted in quality, accuracy, and readability of information depending on the search term used. The search term "developmental dysplasia of the hip" provided higher quality and accuracy compared with the search term "congenital hip dislocation." Of the 63 total Web sites, 1 (1.6%) was below the sixth grade reading level recommended by the NIH for health education materials and 8 (12.7%) Web sites were below the average American reading level (eighth grade).

CONCLUSIONS

The quality and accuracy of information available on the Internet regarding developmental hip dysplasia significantly varied with the search term used. Patients seeking information about DDH on the Internet may not understand the materials found because nearly all of the Web sites are written at a level above that recommended for publically distributed health information.

CLINICAL RELEVANCE

Physicians should advise their patients to search for information using the term "developmental dysplasia of the hip" or, better yet, should refer patients to Web sites that they have personally reviewed for content and clarity. Orthopaedic surgeons, professional societies, and search engines should undertake efforts to ensure that patients have access to information about DDH that is both accurate and easily understandable.

Readability of Questionnaires Assessing Listening Difficulties Associated With (Central) Auditory Processing Disorders

Purpose

Eight English-language, student- or parent proxy-administered questionnaires for (central) auditory processing disorders, or (C)APD, were analyzed for readability. For student questionnaires, readability levels were checked against the approximate reading grade levels by intended administration age per the questionnaires' developers. For proxy questionnaires, results were compared to the reading grade level for the average U.S. adult and the minimum 5th- to 6th-grade reading level for health materials as recommended by adult health literacy experts.

Method

This was a descriptive study that was based on the results of a commercially available readability calculations software package. Text-based files of the 8 questionnaires were analyzed using 3 readability formulas: FORCAST (Caylor & Sticht, 1973), Flesch Reading Ease (FRE; Flesch, 1948), and Gunning’s Fog index (FOG; Gunning, 1952).

Results

The FORCAST formula, the most appropriate for nonnarrative-type materials, indicated that all 8 questionnaires were written at reading levels between the 8th and 10th grades. The FRE and FOG formulas, designed for narrative-type materials, were generally in good agreement with one another but varied widely between the upper 4th- and 12th-grade levels.

Conclusion

In an effort to decrease respondent burden, developers should consider readability as another testable psychometric construct. Clinicians should take into account the functional health literacy skills of adult proxy respondents when giving self-administered questionnaires.

Does the quality, accuracy, and readability of information about lateral epicondylitis on the internet vary with the search term used?

BACKGROUND

Concern exists over the quality, accuracy, and accessibility of online information about health care conditions. The goal of this study is to evaluate the quality, accuracy, and readability of information available on the internet about lateral epicondylitis.

METHODS

We used three different search terms ("tennis elbow," "lateral epicondylitis," and "elbow pain") in three search engines (Google, Bing, and Yahoo) to generate a list of 75 unique websites. Three orthopedic surgeons reviewed the content of each website and assessed the quality and accuracy of information. We assessed each website's readability using the Flesch-Kincaid method. Statistical comparisons were made using ANOVA with post hoc pairwise comparisons.

RESULTS

The mean reading grade level was 11.1. None of the sites were under the recommended sixth grade reading level for the general public. Higher quality information was found when using the terms "tennis elbow" and "lateral epicondylitis" compared to "elbow pain" (p < 0.001). Specialty society websites had higher quality than all other websites (p < 0.001). The information was more accurate if the website was authored by a health care provider when compared to non-health care providers (p = 0.003). Websites seeking commercial gain and those found after the first five search results had lower quality information.

CONCLUSIONS

Reliable information about lateral epicondylitis is available online, especially from specialty societies. However, the quality and accuracy of information vary significantly with the search term, website author, and order of search results. This leaves less educated patients at a disadvantage, particularly because the information we encountered is above the reading level recommended for the general public.

Readability of CAMHS clinical letters

BACKGROUND

  Sending clinical letters to patients is common practice in CAMHS. Ease of reading and understanding is important for patients. Readability formulae are feasible and quick ways to assess this.

METHOD

  Clinical letters sent to patients were analysed for readability and whether they reached a preset threshold. Factors influencing readability were investigated.

RESULTS

  Most letters were less easily readable than is appropriate. Letters written directly to patients were more readable than copied letters.

CONCLUSIONS

  Patients sometimes receive letters that are unlikely to be easily readable. Word processing packages contain readability formulae that could be used to check and improve readability of letters.

The effect of search term on the quality and accuracy of online information regarding distal radius fractures

PURPOSE

Recent emphasis on shared decision making and patient-centered research has increased the importance of patient education and health literacy. The internet is rapidly growing as a source of self-education for patients. However, concern exists over the quality, accuracy, and readability of the information. Our objective was to determine whether the quality, accuracy, and readability of information online about distal radius fractures vary with the search term.

METHODS

This was a prospective evaluation of 3 search engines using 3 different search terms of varying sophistication ("distal radius fracture," "wrist fracture," and "broken wrist"). We evaluated 70 unique Web sites for quality, accuracy, and readability. We used comparative statistics to determine whether the search term affected the quality, accuracy, and readability of the Web sites found. Three orthopedic surgeons independently gauged quality and accuracy of information using a set of predetermined scoring criteria. We evaluated the readability of the Web site using the Fleisch-Kincaid score for reading grade level.

RESULTS

There were significant differences in the quality, accuracy, and readability of information found, depending on the search term. We found higher quality and accuracy resulted from the search term "distal radius fracture," particularly compared with Web sites resulting from the term "broken wrist." The reading level was higher than recommended in 65 of the 70 Web sites and was significantly higher when searching with "distal radius fracture" than "wrist fracture" or "broken wrist." There was no correlation between Web site reading level and quality or accuracy.

CONCLUSIONS

The readability of information about distal radius fractures in most Web sites was higher than the recommended reading level for the general public. The quality and accuracy of the information found significantly varied with the sophistication of the search term used.

CLINICAL RELEVANCE

Physicians, professional societies, and search engines should consider efforts to improve internet access to high-quality information at an understandable level.

Readability assessment of patient education materials on major otolaryngology association websites

OBJECTIVE

Various otolaryngology associations provide Internet-based patient education material (IPEM) to the general public. However, this information may be written above the fourth- to sixth-grade reading level recommended by the American Medical Association (AMA) and National Institutes of Health (NIH). The purpose of this study was to assess the readability of otolaryngology-related IPEMs on various otolaryngology association websites and to determine whether they are above the recommended reading level for patient education materials.

STUDY DESIGN AND SETTING

Analysis of patient education materials from 9 major otolaryngology association websites.

METHODS

The readability of 262 otolaryngology-related IPEMs was assessed with 8 numerical and 2 graphical readability tools. Averages were evaluated against national recommendations and between each source using analysis of variance (ANOVA) with post hoc Tukey's honestly significant difference (HSD) analysis. Mean readability scores for each otolaryngology association website were compared.

RESULTS

Mean website readability scores using Flesch Reading Ease test, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG grading, Gunning Fog Index, New Dale-Chall Readability Formula, FORCAST Formula, New Fog Count Test, Raygor Readability Estimate, and the Fry Readability Graph ranged from 20.0 to 57.8, 9.7 to 17.1, 10.7 to 15.9, 11.6 to 18.2, 10.9 to 15.0, 8.6 to 16.0, 10.4 to 12.1, 8.5 to 11.8, 10.5 to 17.0, and 10.0 to 17.0, respectively. ANOVA results indicate a significant difference (P < .05) between the websites for each individual assessment.

CONCLUSION

The IPEMs found on all otolaryngology association websites exceed the recommended fourth- to sixth-grade reading level.

The digital divide in Internet-based patient education materials

The ubiquity of the Internet has led to the widespread availability of health-related information to the public, and the subsequent empowerment of patients has fundamentally altered the patient-physician relationship. Among several concerns of physicians is the possibility that patients may be misinformed by information obtained from the Internet. One opportunity for health care providers to address this problem exists within Internet-based patient education materials (IPEMs). According to recent research in Otolaryngology-Head and Neck Surgery, IPEMs found within professional otolaryngology websites are written at the 8th- to 18th-grade reading comprehension level, essentially unchanged over the past 3 years. This greatly exceeds the fourth- to sixth-grade reading level recommended by the National Institutes of Health. Benefits, strategies, and challenges to improving the readability of IPEMs are discussed.

Quality and readability of English-language internet information for adults with hearing impairment and their significant others

OBJECTIVE

This study evaluated the quality and readability of English-language internet information for adults with hearing impairment and their significant others.

DESIGN

Two keyword pairs (hearing loss and hearing aids) were entered into five country-specific versions of the most commonly used internet search engine in May 2011.

SAMPLE

For each of the 10 searches, the first 10 relevant websites were included. After removing duplicates, a total of 66 websites were assessed. Their origin (commercial, non-profit organization, or government), date of last update, quality (Health On the Net (HON) certification and DISCERN scores), and readability (Flesch Reading Ease Score, Flesch-Kincaid Grade Level Formula, and Simple Measure Of Gobbledygook) were assessed.

RESULTS

Most websites were of commercial origin and had been updated within the last 18 months. Their quality and readability was highly variable. Only 14% of the websites had HON certification. Websites that were of non-profit organization origin had significantly higher DISCERN scores. Readability measures show that on average, only people with at least 11-12 years of education could read and understand the internet information presented.

CONCLUSIONS

Based on these results, this article provides a list of recommendations for website developers and clinicians wishing to incorporate internet information into their practice.

Readability of online patient education materials on adult reconstruction Web sites

Recommended readability of patient education materials is sixth-grade level or lower. Readability of 212 patient education materials pertaining to adult reconstruction topics available from the American Academy of Orthopaedic Surgeons, American Association of Hip and Knee Surgeons, and 3 other specialty and private practitioner Web sites was assessed using the Flesch-Kincaid grade formula. The mean Flesch-Kincaid score was 11.1 (range, 3-26.5). Only 5 (2%) articles had a readability level of sixth grade or lower. Readability of most of the articles for patient education on adult reconstruction Web sites evaluated may be too advanced for a substantial portion of patients. Further studies are needed to assess the optimal readability level of health information on the Internet.

Are informed consent forms for organ transplantation and donation too difficult to read?

Informed consent for organ transplantation and donation is an ethical obligation, legally required, and considered as part of the Patient's Rights Condition of Medicare Participation for hospitals. National policy-makers recommend that informed consent forms and patient education materials be written at a low reading level (5th-8th grade level) to facilitate patient comprehension. We assessed reading levels of informed consent forms (CFs) for adult organ transplant recipients and living organ donors across US transplant centers. CFs were analyzed using three measures of reading level: Lexile Measure, Flesch-Kincaid Grade Level, and the Gunning Fog Index. Of active transplant centers contacted (N=209), 75 (36%) sent a total of 332 CFs. CFs were written, on average, at the college level, which is a considerably higher reading level than the standards set by policy-makers. CF reading levels were negatively correlated with transplant center volume (r=-0.119; p<0.03). CFs for intestine transplantation and for evaluation/listing were the easiest to read, while consent forms for liver transplantation/donation and pre-transplant agreements were the most difficult to read. Reducing CFs' reading level may help to increase patient comprehension for adequate informed consent.